test(job-array-single-wal-record): chaotic-failover array retry scenario

Extends the HA soak harness with --array-chaos-retry: submits an array, kills the primary essentially immediately (before commit is known), retries the identical submission against the new primary, and hard-asserts exactly one committed instance results -- no silent loss, no double-submit, no partial duplication across the array's full index range. Confirms (no production code changes needed) that the existing single-job cross-primary dedup mechanism (bug-1419: client-side jobid minted once and never re-sent on retry; server-side jobid_range_exists()/inflight_range_try_add(), both genuinely range-shaped over [base,base+array_size); xce_failover_classify()'s idempotent-replay rule) generalizes correctly to arrays, treating the whole range as one dedup unit rather than per-element. No mutation-test RED phase against production code was performed (correctly out of this task's boundary); reviewer compensated by extracting the harness's own assertion logic into a standalone mutation-harness with stubbed collaborators, confirming real fail()-gated teeth on all four outcomes (clean/silent-loss/ double-submit/partial-duplication). Live-verified across 4 independent fresh cluster deployments (2 by implementer, 2 by reviewer), array sizes 6/10/15, 17 total chaotic rounds, zero failures, genuine election/term diversity each time. Task 6.2 of job-array-single-wal-record (Requirements 8.2, 8.3).
2026-07-16 05:27:24 +00:00 · 2026-07-16 05:27:24 +00:00 · c6138496bd
commit c6138496bd
parent 0d8986a6a9
4 changed files with 274 additions and 2 deletions
--- a/.claude/skills/xce-test-ha/SKILL.md
+++ b/.claude/skills/xce-test-ha/SKILL.md
@ -202,6 +202,70 @@ run before this verdict line is ever reached. The line's job is to report the
 actual measurements (and the bound they were checked against) for visibility,
 not to gate pass/fail itself.

+### `--array-chaos-retry` mode: chaotic kill-during-submit array retry (job-array-single-wal-record task 6.2)
+
+Extends `--array-soak`'s correctness claims to the case task 6.1 deliberately
+left out of scope: submits a job array via `jobsub -a N` and kills the
+primary **immediately** — before it is known whether the array's
+`WAL_JOB_ARRAY_SUBMIT` record reached majority commit — instead of waiting
+for a clean, known-committed state first. The array submission runs in the
+**background**; the client's own transport-level failover retry
+(`XCECLI_FAILOVER_CALL`, `client_failover.c`) is what actually resends the
+identical submission (same base jobid — `idgen()` mints it exactly once,
+client-side, in `cmd_jobsub()` before the retriable call, so every retry
+attempt inside the one `jobsub` command resends the same struct, never a
+fresh id) against the newly-elected primary. This is the array-scoped analog
+of bug-1419's single-job cross-primary dedup (`jobid_range_exists()` +
+`inflight_range_try_add()`, `job.c`), which requirements.md's Req 8.3 calls
+out by name — see `run_array_chaos_retry_round()`'s header comment
+(`run-ha-soak.sh`) for the full mechanism trace.
+
+Two legitimate outcomes exist, both of which the round asserts converge to
+**exactly one** committed record:
+
+- the original attempt already reached majority commit before the
+  connection died — the retry hits `jobid_range_exists()`==true on the new
+  primary, is rejected `-EEXIST`, and the client's own idempotent-replay rule
+  (`client_failover.c`, `is_write && attempt>0`) reclassifies that as
+  success; no second record is ever appended; or
+- the original attempt never reached majority commit — the retry is a
+  genuinely fresh first commit on the new primary.
+
+Per round, asserts (all `fail()`-gated, not just displayed):
+- the WAL-record delta for that specific submission is **exactly 1** (never
+  0 — a silent loss; never ≥2 — a double-submit), counted via `waldump`'s
+  exact `JOB_ARRAY_SUBMIT` type name (task 6.1's own addition), the same
+  offline-read technique `--array-soak` uses; and
+- **every element of the array's full index range** `[0, N)` is individually
+  addressable afterward (not a sampled subset) — a stronger check than
+  `--array-soak`'s low/mid/high sampling, since this scenario's specific
+  concern is a *partial* duplication across the range, not scale.
+
+Repeats `XCE_HA_ARRAYRETRY_ROUNDS` times (default 5) since a kill's exact
+timing relative to the commit is inherently racy — a single lucky/unlucky
+interleaving must not decide the verdict. Like `--array-soak`, this is a
+correctness invariant, not a chaotic/racy outcome class, so every round
+hard-fails immediately on a violation rather than counting and tolerating.
+
+```bash
+SKILL_DIR=.claude/skills/xce-test-ha
+nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --array-chaos-retry ~/tmp/xce-ha-arrayretry
+# tune array size / round count
+XCE_HA_ARRAYRETRY_SIZE=15 XCE_HA_ARRAYRETRY_ROUNDS=4 \
+  nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --array-chaos-retry ~/tmp/xce-ha-arrayretry
+```
+
+Verdict line (`soak-verdict.sh`'s `verdict_emit_arrayretry`):
+
+```
+[xce-test-ha] VERDICT mode=array-chaos-retry rounds=5 array_size=6 double_submit=0 silent_loss=0 partial_duplication=0 result=PASS
+```
+
+Like `--array-soak`, `result` is unconditionally `PASS` here — every hard
+invariant is asserted with an immediate `fail()` inside
+`run_array_chaos_retry_round()`, so a violation aborts the run before this
+line is ever reached.
+
 ### Verdict self-test (no cluster)

 The verdict parser has a deterministic, pure-bash self-test that feeds fixture
@ -298,7 +362,7 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
 |---|---|---|
 | `XCE_HA_TOP` | deploy directory | `/home/claude/tmp/xce-ha-deploy` |
 | `XCE_HA_DATADIR` | config/certs source (`cluster.bin`, `xce.crt`, `xce.key`) | `<repo>/ha/data` |
-| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, `factstream`, or `array-soak` (its `--flag` wins) | `sequential` |
+| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, `factstream`, `array-soak`, or `array-chaos-retry` (its `--flag` wins) | `sequential` |
 | `XCE_HA_BIN` | `xce_controller` binary (point at real-fs release for the gate) | `<repo>/src/xce.build/controller/xce_controller` |
 | `XCE_HA_XT` | `xce_test` binary | `<repo>/src/xce.build/controller/xce_test` |
 | `XCE_HA_ROUNDS` | kill/restart rounds (5–50) | `6` |
@ -315,6 +379,9 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
 | `XCE_HA_ARRAYSOAK_REPLAY_RATIO_MAX` | `--array-soak` max tolerated large/small replay-time ratio before the floor/margin (Req 4.3 bound) | `20` |
 | `XCE_HA_ARRAYSOAK_REPLAY_FLOOR_MS` | `--array-soak` minimum bound floor in ms before margin is added | `2000` |
 | `XCE_HA_ARRAYSOAK_REPLAY_MARGIN_MS` | `--array-soak` ms added on top of the scaled/floor bound to absorb election-timing jitter | `3000` |
+| `XCE_HA_ARRAYRETRY_ROUNDS` | `--array-chaos-retry` kill-during-submit rounds | `5` |
+| `XCE_HA_ARRAYRETRY_SIZE` | `--array-chaos-retry` element count of the array each round submits | `6` |
+| `XCE_HA_ARRAYRETRY_KILL_DELAY` | `--array-chaos-retry` seconds slept after backgrounding the submit before killing the primary | `0` |

 ## What the driver does (maps to the requested steps)

--- a/.claude/skills/xce-test-ha/run-ha-soak.sh
+++ b/.claude/skills/xce-test-ha/run-ha-soak.sh
@ -56,6 +56,27 @@
 #                  --force-snapshot, hard-fails (not count-and-tolerate) on
 #                  any violated invariant, since these are the feature's core
 #                  correctness claims, not a chaotic/racy outcome class.
+#     --array-chaos-retry job-array-single-wal-record task 6.2 (Req 8.2, 8.3):
+#                  submits a job ARRAY and kills the primary IMMEDIATELY
+#                  (before it is known whether the array's WAL_JOB_ARRAY_SUBMIT
+#                  record reached majority commit — chaotic, unlike
+#                  --array-soak's clean kill-strictly-after-commit pattern),
+#                  relying on the CLIENT's own transport-level failover retry
+#                  (XCECLI_FAILOVER_CALL) to resend the IDENTICAL submission
+#                  (same base jobid — idgen() mints it once, client-side,
+#                  before the retriable call) against the newly-elected
+#                  primary. Asserts the array-scoped analog of bug-1419's
+#                  single-job cross-primary dedup: exactly ONE committed
+#                  WAL_JOB_ARRAY_SUBMIT record results (never zero — a
+#                  silently-lost write; never two-or-more — a double-submit),
+#                  and every element of the array's FULL index range is
+#                  addressable exactly once afterward (no partial
+#                  duplication). Repeats XCE_HA_ARRAYRETRY_ROUNDS times
+#                  (default 5) since the kill's exact timing relative to the
+#                  commit is inherently racy; each round is individually
+#                  hard-failed (fail(), not count-and-tolerate) since "exactly
+#                  one record, full range addressable" is a correctness
+#                  invariant, not a chaotic/racy outcome class.
 #     --selftest   run the verdict-parser self-test (soak-verdict-selftest.sh)
 #                  and exit — no cluster, no binaries needed.
 #
@ -117,6 +138,23 @@
 #   XCE_HA_ARRAYSOAK_REPLAY_MARGIN_MS   ms added on top of the scaled/floor
 #                            bound to absorb ordinary election-timing jitter
 #                            shared by both sub-rounds (default 3000)
+#   XCE_HA_ARRAYRETRY_ROUNDS   --array-chaos-retry kill-during-submit rounds
+#                              (default 5 — independent of the general
+#                              --chaotic ROUNDS knob; a kill's exact timing
+#                              relative to the commit is racy, so several
+#                              rounds build confidence no single lucky/
+#                              unlucky interleaving decided the verdict)
+#   XCE_HA_ARRAYRETRY_SIZE     element count of the array each
+#                              --array-chaos-retry round submits (default 6 —
+#                              small enough that the FULL index range can be
+#                              addressability-checked cheaply, per Req 8.3's
+#                              "extended to the array's full index range as
+#                              one unit")
+#   XCE_HA_ARRAYRETRY_KILL_DELAY   seconds slept after backgrounding the
+#                              array submit before killing the primary
+#                              (default 0 — immediate, to maximize "chaotic"
+#                              uncertainty about whether the WAL append/
+#                              commit happened before the kill)
 set -u

 # ── Mode / flag parsing (flags may precede the positional [ROUNDS] [DEPLOY_DIR]) ─
@ -136,6 +174,7 @@ while [ $# -gt 0 ]; do
        --force-snapshot) MODE="force-snapshot" ;;
        --factstream)     MODE="factstream" ;;
        --array-soak)     MODE="array-soak" ;;
+        --array-chaos-retry) MODE="array-chaos-retry" ;;
        --selftest)   exec bash "$SELF_DIR/soak-verdict-selftest.sh" ;;
        *[!0-9]*|'')  ARGS+=("$1") ;;            # non-integer ⇒ deploy dir
        *)            ROUNDS_CLI="$1" ;;         # bare integer ⇒ rounds
@ -1471,6 +1510,128 @@ run_array_soak_round() {
    ARRAYSOAK_BOUND_VERDICT="$bound_verdict"
 }

+# ── Chaotic array-retry primitives (job-array-single-wal-record task 6.2) ───
+#
+# Proves the SAME cross-primary dedup mechanism single-job submission already
+# relies on (bug-1419: jobid_range_exists() + the process-local in-flight
+# registry inflight_range_try_add()/inflight_range_remove(), job.c)
+# generalizes correctly to a job ARRAY treated as one unit.
+#
+# Mechanism, traced through the actual source (job.c, client.c,
+# client_failover.c, test.c) rather than assumed:
+#   - idgen() (src/lib/idgen.c) is called EXACTLY ONCE per `jobsub` command,
+#     client-side, inside cmd_jobsub() (testsuite/test.c:1879) — BEFORE
+#     xcecli_jobsubmit() is ever invoked. The resulting base jobid is baked
+#     into the one xce_job_submit_t struct passed to xcecli_jobsubmit().
+#   - xcecli_jobsubmit() (client.c:326) wraps the actual wire call in
+#     XCECLI_FAILOVER_CALL, whose internal retry loop (client_failover.c,
+#     max_retries=20 / max_retry_ms=5000, XCE_RETRY_DEFAULTS) reconnects and
+#     RESENDS THE SAME submit struct — including the SAME jobid — on a
+#     transport failure (a connection dying because the primary was just
+#     SIGKILLed lands XCE_FO_RETRY_NOW/RETRY_BACKOFF, never a fresh idgen()
+#     call). So every retry attempt inside ONE `jobsub` command targets the
+#     exact same base jobid, by construction, without this harness doing
+#     anything special to force it.
+#   - Server-side, handle_jobsubmit() (job.c:1326) checks jobid_range_exists()
+#     then inflight_range_try_add() BEFORE branching into the array append
+#     path (job.c:1388) — both checks are RANGE-shaped (the full
+#     [base_jobid, base_jobid+array_size) interval, task 2.2's own doc
+#     comment), so the array's full index range is dedup-checked as one
+#     unit, not per-element.
+# Two legitimate outcomes, both of which must resolve to exactly one
+# committed record:
+#   (a) the original attempt's WAL_JOB_ARRAY_SUBMIT already reached majority
+#       commit before the connection died — once the new primary applies it
+#       (restore_job_array()'s idempotent-per-index skip, Req 3.3), the
+#       client's automatic retry hits jobid_range_exists()==true on the new
+#       primary, is rejected -EEXIST, and xce_failover_classify()'s
+#       "Idempotent-replay rule" (client_failover.c:47, is_write && attempt>0)
+#       reclassifies that as SUCCESS — no second record is ever appended.
+#   (b) the original attempt never reached majority commit before the
+#       connection died — the retry is a genuinely fresh, first commit for
+#       that exact jobid range on the new primary — also exactly one record.
+# Unlike task 6.1's clean single-kill-STRICTLY-AFTER-commit pattern, this
+# scenario kills the primary the moment the array submission is
+# backgrounded, deliberately BEFORE it is known which of (a)/(b) will occur —
+# that uncertainty is the whole point ("chaotic"). Hard-fails (fail(), not
+# count-and-tolerate) on any violation, matching --array-soak's convention:
+# "exactly one committed record, full index range addressable" is a
+# correctness invariant here, not a racy/sometimes-expected outcome class.
+
+# Background-safe wrapper around submit_array() (task 6.1, reused unmodified
+# above): writes "OK <base>" or "FAIL" as its only stdout line so the caller
+# can parse the outcome after `wait`-ing on the backgrounded job.
+submit_array_bg() {
+    local n="$1" base
+    if base="$(submit_array "$n")"; then
+        echo "OK $base"
+    else
+        echo "FAIL"
+    fi
+}
+
+# One chaotic kill-during-submit-then-retry round. $1 = round label (logging/
+# filenames only), $2 = array size. Sets ARRAYRETRY_LAST_BASE for the caller.
+run_array_chaos_retry_round() {
+    local round="$1" n="$2"
+    local outfile="$TOP/logs/arrayretry-round$round.out"
+    local rec_before rec_after delta prev_term newp result base
+
+    PRIMARY="$(discover_primary)"; [ -n "$PRIMARY" ] || fail "array-chaos-retry round $round: no serving primary before submit"
+    prev_term="$(latest_primary_term)"; prev_term="${prev_term:-0}"
+    rec_before="$(count_array_wal_records "$PRIMARY")"
+
+    log "array-chaos-retry round $round: backgrounding a $n-element array submit against $PRIMARY (term=$prev_term), then killing it — chaotic: commit-before-kill is NOT guaranteed either way"
+    : >"$outfile"
+    submit_array_bg "$n" >"$outfile" 2>&1 &
+    local submit_pid=$!
+
+    local kill_delay="${XCE_HA_ARRAYRETRY_KILL_DELAY:-0}"
+    case "$kill_delay" in
+        0|0.0|"") : ;;
+        *) sleep "$kill_delay" ;;
+    esac
+    kill_node "$PRIMARY" || fail "array-chaos-retry round $round: could not kill $PRIMARY"
+    DOWN="$PRIMARY"
+
+    newp="$(wait_for_new_primary "$prev_term")" || fail "array-chaos-retry round $round: no new primary elected after the kill-during-submit"
+    log "array-chaos-retry round $round: new primary $newp (term=$(latest_primary_term)) — waiting for the client's own cross-primary failover retry to resolve"
+    record_primary "$newp"
+
+    wait "$submit_pid" 2>/dev/null
+    result="$(cat "$outfile" 2>/dev/null)"
+    case "$result" in
+        "OK "*)
+            base="${result#OK }"
+            log "array-chaos-retry round $round: client-side retry resolved OK (base=$base)"
+            ;;
+        *)
+            fail "array-chaos-retry round $round: the array submission never resolved to a clean ack even after the client's own cross-primary failover retry (output: '$result') — see $outfile"
+            ;;
+    esac
+
+    start_node "$DOWN"
+    wait_for_full_convergence || fail "array-chaos-retry round $round: cluster did not reconverge to 3-of-3 after the kill-during-submit"
+    DOWN=""
+
+    rec_after="$(count_array_wal_records "$PRIMARY")"
+    delta=$((rec_after - rec_before))
+    log "array-chaos-retry round $round: WAL_JOB_ARRAY_SUBMIT record delta = $delta (before=$rec_before after=$rec_after, node=$PRIMARY, base=$base)"
+    if [ "$delta" -eq 0 ]; then
+        fail "array-chaos-retry round $round: ZERO WAL_JOB_ARRAY_SUBMIT records committed for base=$base despite the client reporting a clean ack — the write was silently lost across the failover"
+    elif [ "$delta" -gt 1 ]; then
+        fail "array-chaos-retry round $round: $delta WAL_JOB_ARRAY_SUBMIT records committed for ONE logical submission (base=$base) — DOUBLE-SUBMIT: the cross-primary dedup mechanism did not treat the retry as the array's full index range as one unit"
+    fi
+    log "array-chaos-retry round $round: CONFIRMED exactly ONE committed WAL_JOB_ARRAY_SUBMIT record for base=$base (delta=1) — retry deduped correctly"
+
+    verify_array_elements "$base" $(seq 0 $((n - 1))) \
+        || fail "array-chaos-retry round $round: not every element of base=$base's FULL index range [0,$n) is addressable after the chaotic kill-and-retry — partial duplication/loss"
+    log "array-chaos-retry round $round: all $n elements of base=$base's FULL index range individually addressable — no partial duplication"
+
+    walcheck_count "0"
+    ARRAYRETRY_LAST_BASE="$base"
+}
+
 # Drive a SPECIFIC node to primary — used to exercise leadership on the
 # lowest-priority node (ctrl-3), which the natural single-failover loop NEVER
 # elects: the election timeout is priority-staggered (my_priority * step_ms, p0
@ -1679,6 +1840,32 @@ if [ "$MODE" = "array-soak" ]; then
    exit 0
 fi

+# ── Array chaos-retry mode: job-array-single-wal-record task 6.2 ────────────
+# Opt-in (--array-chaos-retry / XCE_HA_MODE=array-chaos-retry). See
+# run_array_chaos_retry_round's own header comment for the full mechanism
+# trace and rationale. Repeats XCE_HA_ARRAYRETRY_ROUNDS times (default 5 —
+# independent of the general --chaotic ROUNDS knob) since a kill-during-
+# submit's exact timing relative to the commit is inherently racy and a
+# single lucky/unlucky interleaving should not decide the verdict; each round
+# is nonetheless individually hard-failed (fail()), matching --array-soak's
+# convention, since "exactly one committed record, full range addressable"
+# is a correctness invariant, not a chaotic/racy outcome class. Does NOT run
+# the sequential loop or leadership-coverage maneuver.
+if [ "$MODE" = "array-chaos-retry" ]; then
+    AR_ROUNDS="${XCE_HA_ARRAYRETRY_ROUNDS:-5}"
+    AR_SIZE="${XCE_HA_ARRAYRETRY_SIZE:-6}"
+    log "MODE=array-chaos-retry — chaotic kill-during-array-submit retry soak ($AR_ROUNDS rounds, size=$AR_SIZE)"
+    for ar_round in $(seq 1 "$AR_ROUNDS"); do
+        log "=== array-chaos-retry round $ar_round/$AR_ROUNDS ==="
+        run_array_chaos_retry_round "$ar_round" "$AR_SIZE"
+    done
+    verdict_emit_arrayretry "$AR_ROUNDS" "$AR_SIZE"
+    log "PASS (array-chaos-retry): $AR_ROUNDS/$AR_ROUNDS rounds each resolved to exactly one committed WAL_JOB_ARRAY_SUBMIT record (no double-submit, no silent loss) with the full $AR_SIZE-element index range addressable, across a chaotic kill-during-submit failover"
+    cleanup
+    trap - EXIT INT TERM
+    exit 0
+fi
+
 # ── Chaotic mode: overlapping-kill rounds → isolated verdict ──────────────────
 # Opt-in (--chaotic / XCE_HA_MODE=chaotic). Kills land inside election/catch-up
 # windows; outcomes are COUNTED (committed divergence + committed job loss) with
--- a/.claude/skills/xce-test-ha/soak-verdict.sh
+++ b/.claude/skills/xce-test-ha/soak-verdict.sh
@ -186,3 +186,20 @@ verdict_emit_arraysoak() {
        "replay_bound_verdict=$bound_verdict result=PASS"
    return 0
 }
+
+# Emit the machine-parseable verdict line for the array-chaos-retry
+# (job-array-single-wal-record task 6.2) scenario. Like verdict_emit_
+# arraysoak, this is NOT a count-and-classify verdict: every hard invariant
+# (exactly one committed WAL_JOB_ARRAY_SUBMIT record per round, never zero,
+# never two-or-more; the array's full index range addressable exactly once)
+# is asserted with an immediate fail() inside run_array_chaos_retry_round()
+# (run-ha-soak.sh) — a violation aborts the run before this line is ever
+# reached, so result is unconditionally PASS here. This function's job is to
+# report how many rounds ran and at what array size, for visibility, not to
+# gate pass/fail itself. Args: rounds size
+verdict_emit_arrayretry() {
+    local rounds="$1" size="$2"
+    echo "[xce-test-ha] VERDICT mode=array-chaos-retry rounds=$rounds array_size=$size" \
+        "double_submit=0 silent_loss=0 partial_duplication=0 result=PASS"
+    return 0
+}