test(job-array-single-wal-record): chaotic-failover array retry scenario
Extends the HA soak harness with --array-chaos-retry: submits an array, kills the primary essentially immediately (before commit is known), retries the identical submission against the new primary, and hard-asserts exactly one committed instance results -- no silent loss, no double-submit, no partial duplication across the array's full index range. Confirms (no production code changes needed) that the existing single-job cross-primary dedup mechanism (bug-1419: client-side jobid minted once and never re-sent on retry; server-side jobid_range_exists()/inflight_range_try_add(), both genuinely range-shaped over [base,base+array_size); xce_failover_classify()'s idempotent-replay rule) generalizes correctly to arrays, treating the whole range as one dedup unit rather than per-element. No mutation-test RED phase against production code was performed (correctly out of this task's boundary); reviewer compensated by extracting the harness's own assertion logic into a standalone mutation-harness with stubbed collaborators, confirming real fail()-gated teeth on all four outcomes (clean/silent-loss/ double-submit/partial-duplication). Live-verified across 4 independent fresh cluster deployments (2 by implementer, 2 by reviewer), array sizes 6/10/15, 17 total chaotic rounds, zero failures, genuine election/term diversity each time. Task 6.2 of job-array-single-wal-record (Requirements 8.2, 8.3).
This commit is contained in:
parent
0d8986a6a9
commit
c6138496bd
4 changed files with 274 additions and 2 deletions
|
|
@ -202,6 +202,70 @@ run before this verdict line is ever reached. The line's job is to report the
|
|||
actual measurements (and the bound they were checked against) for visibility,
|
||||
not to gate pass/fail itself.
|
||||
|
||||
### `--array-chaos-retry` mode: chaotic kill-during-submit array retry (job-array-single-wal-record task 6.2)
|
||||
|
||||
Extends `--array-soak`'s correctness claims to the case task 6.1 deliberately
|
||||
left out of scope: submits a job array via `jobsub -a N` and kills the
|
||||
primary **immediately** — before it is known whether the array's
|
||||
`WAL_JOB_ARRAY_SUBMIT` record reached majority commit — instead of waiting
|
||||
for a clean, known-committed state first. The array submission runs in the
|
||||
**background**; the client's own transport-level failover retry
|
||||
(`XCECLI_FAILOVER_CALL`, `client_failover.c`) is what actually resends the
|
||||
identical submission (same base jobid — `idgen()` mints it exactly once,
|
||||
client-side, in `cmd_jobsub()` before the retriable call, so every retry
|
||||
attempt inside the one `jobsub` command resends the same struct, never a
|
||||
fresh id) against the newly-elected primary. This is the array-scoped analog
|
||||
of bug-1419's single-job cross-primary dedup (`jobid_range_exists()` +
|
||||
`inflight_range_try_add()`, `job.c`), which requirements.md's Req 8.3 calls
|
||||
out by name — see `run_array_chaos_retry_round()`'s header comment
|
||||
(`run-ha-soak.sh`) for the full mechanism trace.
|
||||
|
||||
Two legitimate outcomes exist, both of which the round asserts converge to
|
||||
**exactly one** committed record:
|
||||
|
||||
- the original attempt already reached majority commit before the
|
||||
connection died — the retry hits `jobid_range_exists()`==true on the new
|
||||
primary, is rejected `-EEXIST`, and the client's own idempotent-replay rule
|
||||
(`client_failover.c`, `is_write && attempt>0`) reclassifies that as
|
||||
success; no second record is ever appended; or
|
||||
- the original attempt never reached majority commit — the retry is a
|
||||
genuinely fresh first commit on the new primary.
|
||||
|
||||
Per round, asserts (all `fail()`-gated, not just displayed):
|
||||
- the WAL-record delta for that specific submission is **exactly 1** (never
|
||||
0 — a silent loss; never ≥2 — a double-submit), counted via `waldump`'s
|
||||
exact `JOB_ARRAY_SUBMIT` type name (task 6.1's own addition), the same
|
||||
offline-read technique `--array-soak` uses; and
|
||||
- **every element of the array's full index range** `[0, N)` is individually
|
||||
addressable afterward (not a sampled subset) — a stronger check than
|
||||
`--array-soak`'s low/mid/high sampling, since this scenario's specific
|
||||
concern is a *partial* duplication across the range, not scale.
|
||||
|
||||
Repeats `XCE_HA_ARRAYRETRY_ROUNDS` times (default 5) since a kill's exact
|
||||
timing relative to the commit is inherently racy — a single lucky/unlucky
|
||||
interleaving must not decide the verdict. Like `--array-soak`, this is a
|
||||
correctness invariant, not a chaotic/racy outcome class, so every round
|
||||
hard-fails immediately on a violation rather than counting and tolerating.
|
||||
|
||||
```bash
|
||||
SKILL_DIR=.claude/skills/xce-test-ha
|
||||
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --array-chaos-retry ~/tmp/xce-ha-arrayretry
|
||||
# tune array size / round count
|
||||
XCE_HA_ARRAYRETRY_SIZE=15 XCE_HA_ARRAYRETRY_ROUNDS=4 \
|
||||
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --array-chaos-retry ~/tmp/xce-ha-arrayretry
|
||||
```
|
||||
|
||||
Verdict line (`soak-verdict.sh`'s `verdict_emit_arrayretry`):
|
||||
|
||||
```
|
||||
[xce-test-ha] VERDICT mode=array-chaos-retry rounds=5 array_size=6 double_submit=0 silent_loss=0 partial_duplication=0 result=PASS
|
||||
```
|
||||
|
||||
Like `--array-soak`, `result` is unconditionally `PASS` here — every hard
|
||||
invariant is asserted with an immediate `fail()` inside
|
||||
`run_array_chaos_retry_round()`, so a violation aborts the run before this
|
||||
line is ever reached.
|
||||
|
||||
### Verdict self-test (no cluster)
|
||||
|
||||
The verdict parser has a deterministic, pure-bash self-test that feeds fixture
|
||||
|
|
@ -298,7 +362,7 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|
|||
|---|---|---|
|
||||
| `XCE_HA_TOP` | deploy directory | `/home/claude/tmp/xce-ha-deploy` |
|
||||
| `XCE_HA_DATADIR` | config/certs source (`cluster.bin`, `xce.crt`, `xce.key`) | `<repo>/ha/data` |
|
||||
| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, `factstream`, or `array-soak` (its `--flag` wins) | `sequential` |
|
||||
| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, `factstream`, `array-soak`, or `array-chaos-retry` (its `--flag` wins) | `sequential` |
|
||||
| `XCE_HA_BIN` | `xce_controller` binary (point at real-fs release for the gate) | `<repo>/src/xce.build/controller/xce_controller` |
|
||||
| `XCE_HA_XT` | `xce_test` binary | `<repo>/src/xce.build/controller/xce_test` |
|
||||
| `XCE_HA_ROUNDS` | kill/restart rounds (5–50) | `6` |
|
||||
|
|
@ -315,6 +379,9 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|
|||
| `XCE_HA_ARRAYSOAK_REPLAY_RATIO_MAX` | `--array-soak` max tolerated large/small replay-time ratio before the floor/margin (Req 4.3 bound) | `20` |
|
||||
| `XCE_HA_ARRAYSOAK_REPLAY_FLOOR_MS` | `--array-soak` minimum bound floor in ms before margin is added | `2000` |
|
||||
| `XCE_HA_ARRAYSOAK_REPLAY_MARGIN_MS` | `--array-soak` ms added on top of the scaled/floor bound to absorb election-timing jitter | `3000` |
|
||||
| `XCE_HA_ARRAYRETRY_ROUNDS` | `--array-chaos-retry` kill-during-submit rounds | `5` |
|
||||
| `XCE_HA_ARRAYRETRY_SIZE` | `--array-chaos-retry` element count of the array each round submits | `6` |
|
||||
| `XCE_HA_ARRAYRETRY_KILL_DELAY` | `--array-chaos-retry` seconds slept after backgrounding the submit before killing the primary | `0` |
|
||||
|
||||
## What the driver does (maps to the requested steps)
|
||||
|
||||
|
|
|
|||
|
|
@ -56,6 +56,27 @@
|
|||
# --force-snapshot, hard-fails (not count-and-tolerate) on
|
||||
# any violated invariant, since these are the feature's core
|
||||
# correctness claims, not a chaotic/racy outcome class.
|
||||
# --array-chaos-retry job-array-single-wal-record task 6.2 (Req 8.2, 8.3):
|
||||
# submits a job ARRAY and kills the primary IMMEDIATELY
|
||||
# (before it is known whether the array's WAL_JOB_ARRAY_SUBMIT
|
||||
# record reached majority commit — chaotic, unlike
|
||||
# --array-soak's clean kill-strictly-after-commit pattern),
|
||||
# relying on the CLIENT's own transport-level failover retry
|
||||
# (XCECLI_FAILOVER_CALL) to resend the IDENTICAL submission
|
||||
# (same base jobid — idgen() mints it once, client-side,
|
||||
# before the retriable call) against the newly-elected
|
||||
# primary. Asserts the array-scoped analog of bug-1419's
|
||||
# single-job cross-primary dedup: exactly ONE committed
|
||||
# WAL_JOB_ARRAY_SUBMIT record results (never zero — a
|
||||
# silently-lost write; never two-or-more — a double-submit),
|
||||
# and every element of the array's FULL index range is
|
||||
# addressable exactly once afterward (no partial
|
||||
# duplication). Repeats XCE_HA_ARRAYRETRY_ROUNDS times
|
||||
# (default 5) since the kill's exact timing relative to the
|
||||
# commit is inherently racy; each round is individually
|
||||
# hard-failed (fail(), not count-and-tolerate) since "exactly
|
||||
# one record, full range addressable" is a correctness
|
||||
# invariant, not a chaotic/racy outcome class.
|
||||
# --selftest run the verdict-parser self-test (soak-verdict-selftest.sh)
|
||||
# and exit — no cluster, no binaries needed.
|
||||
#
|
||||
|
|
@ -117,6 +138,23 @@
|
|||
# XCE_HA_ARRAYSOAK_REPLAY_MARGIN_MS ms added on top of the scaled/floor
|
||||
# bound to absorb ordinary election-timing jitter
|
||||
# shared by both sub-rounds (default 3000)
|
||||
# XCE_HA_ARRAYRETRY_ROUNDS --array-chaos-retry kill-during-submit rounds
|
||||
# (default 5 — independent of the general
|
||||
# --chaotic ROUNDS knob; a kill's exact timing
|
||||
# relative to the commit is racy, so several
|
||||
# rounds build confidence no single lucky/
|
||||
# unlucky interleaving decided the verdict)
|
||||
# XCE_HA_ARRAYRETRY_SIZE element count of the array each
|
||||
# --array-chaos-retry round submits (default 6 —
|
||||
# small enough that the FULL index range can be
|
||||
# addressability-checked cheaply, per Req 8.3's
|
||||
# "extended to the array's full index range as
|
||||
# one unit")
|
||||
# XCE_HA_ARRAYRETRY_KILL_DELAY seconds slept after backgrounding the
|
||||
# array submit before killing the primary
|
||||
# (default 0 — immediate, to maximize "chaotic"
|
||||
# uncertainty about whether the WAL append/
|
||||
# commit happened before the kill)
|
||||
set -u
|
||||
|
||||
# ── Mode / flag parsing (flags may precede the positional [ROUNDS] [DEPLOY_DIR]) ─
|
||||
|
|
@ -136,6 +174,7 @@ while [ $# -gt 0 ]; do
|
|||
--force-snapshot) MODE="force-snapshot" ;;
|
||||
--factstream) MODE="factstream" ;;
|
||||
--array-soak) MODE="array-soak" ;;
|
||||
--array-chaos-retry) MODE="array-chaos-retry" ;;
|
||||
--selftest) exec bash "$SELF_DIR/soak-verdict-selftest.sh" ;;
|
||||
*[!0-9]*|'') ARGS+=("$1") ;; # non-integer ⇒ deploy dir
|
||||
*) ROUNDS_CLI="$1" ;; # bare integer ⇒ rounds
|
||||
|
|
@ -1471,6 +1510,128 @@ run_array_soak_round() {
|
|||
ARRAYSOAK_BOUND_VERDICT="$bound_verdict"
|
||||
}
|
||||
|
||||
# ── Chaotic array-retry primitives (job-array-single-wal-record task 6.2) ───
|
||||
#
|
||||
# Proves the SAME cross-primary dedup mechanism single-job submission already
|
||||
# relies on (bug-1419: jobid_range_exists() + the process-local in-flight
|
||||
# registry inflight_range_try_add()/inflight_range_remove(), job.c)
|
||||
# generalizes correctly to a job ARRAY treated as one unit.
|
||||
#
|
||||
# Mechanism, traced through the actual source (job.c, client.c,
|
||||
# client_failover.c, test.c) rather than assumed:
|
||||
# - idgen() (src/lib/idgen.c) is called EXACTLY ONCE per `jobsub` command,
|
||||
# client-side, inside cmd_jobsub() (testsuite/test.c:1879) — BEFORE
|
||||
# xcecli_jobsubmit() is ever invoked. The resulting base jobid is baked
|
||||
# into the one xce_job_submit_t struct passed to xcecli_jobsubmit().
|
||||
# - xcecli_jobsubmit() (client.c:326) wraps the actual wire call in
|
||||
# XCECLI_FAILOVER_CALL, whose internal retry loop (client_failover.c,
|
||||
# max_retries=20 / max_retry_ms=5000, XCE_RETRY_DEFAULTS) reconnects and
|
||||
# RESENDS THE SAME submit struct — including the SAME jobid — on a
|
||||
# transport failure (a connection dying because the primary was just
|
||||
# SIGKILLed lands XCE_FO_RETRY_NOW/RETRY_BACKOFF, never a fresh idgen()
|
||||
# call). So every retry attempt inside ONE `jobsub` command targets the
|
||||
# exact same base jobid, by construction, without this harness doing
|
||||
# anything special to force it.
|
||||
# - Server-side, handle_jobsubmit() (job.c:1326) checks jobid_range_exists()
|
||||
# then inflight_range_try_add() BEFORE branching into the array append
|
||||
# path (job.c:1388) — both checks are RANGE-shaped (the full
|
||||
# [base_jobid, base_jobid+array_size) interval, task 2.2's own doc
|
||||
# comment), so the array's full index range is dedup-checked as one
|
||||
# unit, not per-element.
|
||||
# Two legitimate outcomes, both of which must resolve to exactly one
|
||||
# committed record:
|
||||
# (a) the original attempt's WAL_JOB_ARRAY_SUBMIT already reached majority
|
||||
# commit before the connection died — once the new primary applies it
|
||||
# (restore_job_array()'s idempotent-per-index skip, Req 3.3), the
|
||||
# client's automatic retry hits jobid_range_exists()==true on the new
|
||||
# primary, is rejected -EEXIST, and xce_failover_classify()'s
|
||||
# "Idempotent-replay rule" (client_failover.c:47, is_write && attempt>0)
|
||||
# reclassifies that as SUCCESS — no second record is ever appended.
|
||||
# (b) the original attempt never reached majority commit before the
|
||||
# connection died — the retry is a genuinely fresh, first commit for
|
||||
# that exact jobid range on the new primary — also exactly one record.
|
||||
# Unlike task 6.1's clean single-kill-STRICTLY-AFTER-commit pattern, this
|
||||
# scenario kills the primary the moment the array submission is
|
||||
# backgrounded, deliberately BEFORE it is known which of (a)/(b) will occur —
|
||||
# that uncertainty is the whole point ("chaotic"). Hard-fails (fail(), not
|
||||
# count-and-tolerate) on any violation, matching --array-soak's convention:
|
||||
# "exactly one committed record, full index range addressable" is a
|
||||
# correctness invariant here, not a racy/sometimes-expected outcome class.
|
||||
|
||||
# Background-safe wrapper around submit_array() (task 6.1, reused unmodified
|
||||
# above): writes "OK <base>" or "FAIL" as its only stdout line so the caller
|
||||
# can parse the outcome after `wait`-ing on the backgrounded job.
|
||||
submit_array_bg() {
|
||||
local n="$1" base
|
||||
if base="$(submit_array "$n")"; then
|
||||
echo "OK $base"
|
||||
else
|
||||
echo "FAIL"
|
||||
fi
|
||||
}
|
||||
|
||||
# One chaotic kill-during-submit-then-retry round. $1 = round label (logging/
|
||||
# filenames only), $2 = array size. Sets ARRAYRETRY_LAST_BASE for the caller.
|
||||
run_array_chaos_retry_round() {
|
||||
local round="$1" n="$2"
|
||||
local outfile="$TOP/logs/arrayretry-round$round.out"
|
||||
local rec_before rec_after delta prev_term newp result base
|
||||
|
||||
PRIMARY="$(discover_primary)"; [ -n "$PRIMARY" ] || fail "array-chaos-retry round $round: no serving primary before submit"
|
||||
prev_term="$(latest_primary_term)"; prev_term="${prev_term:-0}"
|
||||
rec_before="$(count_array_wal_records "$PRIMARY")"
|
||||
|
||||
log "array-chaos-retry round $round: backgrounding a $n-element array submit against $PRIMARY (term=$prev_term), then killing it — chaotic: commit-before-kill is NOT guaranteed either way"
|
||||
: >"$outfile"
|
||||
submit_array_bg "$n" >"$outfile" 2>&1 &
|
||||
local submit_pid=$!
|
||||
|
||||
local kill_delay="${XCE_HA_ARRAYRETRY_KILL_DELAY:-0}"
|
||||
case "$kill_delay" in
|
||||
0|0.0|"") : ;;
|
||||
*) sleep "$kill_delay" ;;
|
||||
esac
|
||||
kill_node "$PRIMARY" || fail "array-chaos-retry round $round: could not kill $PRIMARY"
|
||||
DOWN="$PRIMARY"
|
||||
|
||||
newp="$(wait_for_new_primary "$prev_term")" || fail "array-chaos-retry round $round: no new primary elected after the kill-during-submit"
|
||||
log "array-chaos-retry round $round: new primary $newp (term=$(latest_primary_term)) — waiting for the client's own cross-primary failover retry to resolve"
|
||||
record_primary "$newp"
|
||||
|
||||
wait "$submit_pid" 2>/dev/null
|
||||
result="$(cat "$outfile" 2>/dev/null)"
|
||||
case "$result" in
|
||||
"OK "*)
|
||||
base="${result#OK }"
|
||||
log "array-chaos-retry round $round: client-side retry resolved OK (base=$base)"
|
||||
;;
|
||||
*)
|
||||
fail "array-chaos-retry round $round: the array submission never resolved to a clean ack even after the client's own cross-primary failover retry (output: '$result') — see $outfile"
|
||||
;;
|
||||
esac
|
||||
|
||||
start_node "$DOWN"
|
||||
wait_for_full_convergence || fail "array-chaos-retry round $round: cluster did not reconverge to 3-of-3 after the kill-during-submit"
|
||||
DOWN=""
|
||||
|
||||
rec_after="$(count_array_wal_records "$PRIMARY")"
|
||||
delta=$((rec_after - rec_before))
|
||||
log "array-chaos-retry round $round: WAL_JOB_ARRAY_SUBMIT record delta = $delta (before=$rec_before after=$rec_after, node=$PRIMARY, base=$base)"
|
||||
if [ "$delta" -eq 0 ]; then
|
||||
fail "array-chaos-retry round $round: ZERO WAL_JOB_ARRAY_SUBMIT records committed for base=$base despite the client reporting a clean ack — the write was silently lost across the failover"
|
||||
elif [ "$delta" -gt 1 ]; then
|
||||
fail "array-chaos-retry round $round: $delta WAL_JOB_ARRAY_SUBMIT records committed for ONE logical submission (base=$base) — DOUBLE-SUBMIT: the cross-primary dedup mechanism did not treat the retry as the array's full index range as one unit"
|
||||
fi
|
||||
log "array-chaos-retry round $round: CONFIRMED exactly ONE committed WAL_JOB_ARRAY_SUBMIT record for base=$base (delta=1) — retry deduped correctly"
|
||||
|
||||
verify_array_elements "$base" $(seq 0 $((n - 1))) \
|
||||
|| fail "array-chaos-retry round $round: not every element of base=$base's FULL index range [0,$n) is addressable after the chaotic kill-and-retry — partial duplication/loss"
|
||||
log "array-chaos-retry round $round: all $n elements of base=$base's FULL index range individually addressable — no partial duplication"
|
||||
|
||||
walcheck_count "0"
|
||||
ARRAYRETRY_LAST_BASE="$base"
|
||||
}
|
||||
|
||||
# Drive a SPECIFIC node to primary — used to exercise leadership on the
|
||||
# lowest-priority node (ctrl-3), which the natural single-failover loop NEVER
|
||||
# elects: the election timeout is priority-staggered (my_priority * step_ms, p0
|
||||
|
|
@ -1679,6 +1840,32 @@ if [ "$MODE" = "array-soak" ]; then
|
|||
exit 0
|
||||
fi
|
||||
|
||||
# ── Array chaos-retry mode: job-array-single-wal-record task 6.2 ────────────
|
||||
# Opt-in (--array-chaos-retry / XCE_HA_MODE=array-chaos-retry). See
|
||||
# run_array_chaos_retry_round's own header comment for the full mechanism
|
||||
# trace and rationale. Repeats XCE_HA_ARRAYRETRY_ROUNDS times (default 5 —
|
||||
# independent of the general --chaotic ROUNDS knob) since a kill-during-
|
||||
# submit's exact timing relative to the commit is inherently racy and a
|
||||
# single lucky/unlucky interleaving should not decide the verdict; each round
|
||||
# is nonetheless individually hard-failed (fail()), matching --array-soak's
|
||||
# convention, since "exactly one committed record, full range addressable"
|
||||
# is a correctness invariant, not a chaotic/racy outcome class. Does NOT run
|
||||
# the sequential loop or leadership-coverage maneuver.
|
||||
if [ "$MODE" = "array-chaos-retry" ]; then
|
||||
AR_ROUNDS="${XCE_HA_ARRAYRETRY_ROUNDS:-5}"
|
||||
AR_SIZE="${XCE_HA_ARRAYRETRY_SIZE:-6}"
|
||||
log "MODE=array-chaos-retry — chaotic kill-during-array-submit retry soak ($AR_ROUNDS rounds, size=$AR_SIZE)"
|
||||
for ar_round in $(seq 1 "$AR_ROUNDS"); do
|
||||
log "=== array-chaos-retry round $ar_round/$AR_ROUNDS ==="
|
||||
run_array_chaos_retry_round "$ar_round" "$AR_SIZE"
|
||||
done
|
||||
verdict_emit_arrayretry "$AR_ROUNDS" "$AR_SIZE"
|
||||
log "PASS (array-chaos-retry): $AR_ROUNDS/$AR_ROUNDS rounds each resolved to exactly one committed WAL_JOB_ARRAY_SUBMIT record (no double-submit, no silent loss) with the full $AR_SIZE-element index range addressable, across a chaotic kill-during-submit failover"
|
||||
cleanup
|
||||
trap - EXIT INT TERM
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Chaotic mode: overlapping-kill rounds → isolated verdict ──────────────────
|
||||
# Opt-in (--chaotic / XCE_HA_MODE=chaotic). Kills land inside election/catch-up
|
||||
# windows; outcomes are COUNTED (committed divergence + committed job loss) with
|
||||
|
|
|
|||
|
|
@ -186,3 +186,20 @@ verdict_emit_arraysoak() {
|
|||
"replay_bound_verdict=$bound_verdict result=PASS"
|
||||
return 0
|
||||
}
|
||||
|
||||
# Emit the machine-parseable verdict line for the array-chaos-retry
|
||||
# (job-array-single-wal-record task 6.2) scenario. Like verdict_emit_
|
||||
# arraysoak, this is NOT a count-and-classify verdict: every hard invariant
|
||||
# (exactly one committed WAL_JOB_ARRAY_SUBMIT record per round, never zero,
|
||||
# never two-or-more; the array's full index range addressable exactly once)
|
||||
# is asserted with an immediate fail() inside run_array_chaos_retry_round()
|
||||
# (run-ha-soak.sh) — a violation aborts the run before this line is ever
|
||||
# reached, so result is unconditionally PASS here. This function's job is to
|
||||
# report how many rounds ran and at what array size, for visibility, not to
|
||||
# gate pass/fail itself. Args: rounds size
|
||||
verdict_emit_arrayretry() {
|
||||
local rounds="$1" size="$2"
|
||||
echo "[xce-test-ha] VERDICT mode=array-chaos-retry rounds=$rounds array_size=$size" \
|
||||
"double_submit=0 silent_loss=0 partial_duplication=0 result=PASS"
|
||||
return 0
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue