test(job-array-single-wal-record): chaotic-failover array retry scenario

Extends the HA soak harness with --array-chaos-retry: submits an
array, kills the primary essentially immediately (before commit is
known), retries the identical submission against the new primary, and
hard-asserts exactly one committed instance results -- no silent loss,
no double-submit, no partial duplication across the array's full
index range.

Confirms (no production code changes needed) that the existing
single-job cross-primary dedup mechanism (bug-1419: client-side jobid
minted once and never re-sent on retry; server-side
jobid_range_exists()/inflight_range_try_add(), both genuinely
range-shaped over [base,base+array_size); xce_failover_classify()'s
idempotent-replay rule) generalizes correctly to arrays, treating the
whole range as one dedup unit rather than per-element.

No mutation-test RED phase against production code was performed
(correctly out of this task's boundary); reviewer compensated by
extracting the harness's own assertion logic into a standalone
mutation-harness with stubbed collaborators, confirming real
fail()-gated teeth on all four outcomes (clean/silent-loss/
double-submit/partial-duplication).

Live-verified across 4 independent fresh cluster deployments (2 by
implementer, 2 by reviewer), array sizes 6/10/15, 17 total chaotic
rounds, zero failures, genuine election/term diversity each time.

Task 6.2 of job-array-single-wal-record (Requirements 8.2, 8.3).

.claude/skills/xce-test-ha/SKILL.md

@@ -202,6 +202,70 @@ run before this verdict line is ever reached. The line's job is to report the
 actual measurements (and the bound they were checked against) for visibility,
 not to gate pass/fail itself.
 
+### `--array-chaos-retry` mode: chaotic kill-during-submit array retry (job-array-single-wal-record task 6.2)
+
+Extends `--array-soak`'s correctness claims to the case task 6.1 deliberately
+left out of scope: submits a job array via `jobsub -a N` and kills the
+primary **immediately** — before it is known whether the array's
+`WAL_JOB_ARRAY_SUBMIT` record reached majority commit — instead of waiting
+for a clean, known-committed state first. The array submission runs in the
+**background**; the client's own transport-level failover retry
+(`XCECLI_FAILOVER_CALL`, `client_failover.c`) is what actually resends the
+identical submission (same base jobid — `idgen()` mints it exactly once,
+client-side, in `cmd_jobsub()` before the retriable call, so every retry
+attempt inside the one `jobsub` command resends the same struct, never a
+fresh id) against the newly-elected primary. This is the array-scoped analog
+of bug-1419's single-job cross-primary dedup (`jobid_range_exists()` +
+`inflight_range_try_add()`, `job.c`), which requirements.md's Req 8.3 calls
+out by name — see `run_array_chaos_retry_round()`'s header comment
+(`run-ha-soak.sh`) for the full mechanism trace.
+
+Two legitimate outcomes exist, both of which the round asserts converge to
+**exactly one** committed record:
+
+- the original attempt already reached majority commit before the
+  connection died — the retry hits `jobid_range_exists()`==true on the new
+  primary, is rejected `-EEXIST`, and the client's own idempotent-replay rule
+  (`client_failover.c`, `is_write && attempt>0`) reclassifies that as
+  success; no second record is ever appended; or
+- the original attempt never reached majority commit — the retry is a
+  genuinely fresh first commit on the new primary.
+
+Per round, asserts (all `fail()`-gated, not just displayed):
+- the WAL-record delta for that specific submission is **exactly 1** (never
+  0 — a silent loss; never ≥2 — a double-submit), counted via `waldump`'s
+  exact `JOB_ARRAY_SUBMIT` type name (task 6.1's own addition), the same
+  offline-read technique `--array-soak` uses; and
+- **every element of the array's full index range** `[0, N)` is individually
+  addressable afterward (not a sampled subset) — a stronger check than
+  `--array-soak`'s low/mid/high sampling, since this scenario's specific
+  concern is a *partial* duplication across the range, not scale.
+
+Repeats `XCE_HA_ARRAYRETRY_ROUNDS` times (default 5) since a kill's exact
+timing relative to the commit is inherently racy — a single lucky/unlucky
+interleaving must not decide the verdict. Like `--array-soak`, this is a
+correctness invariant, not a chaotic/racy outcome class, so every round
+hard-fails immediately on a violation rather than counting and tolerating.
+
+```bash
+SKILL_DIR=.claude/skills/xce-test-ha
+nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --array-chaos-retry ~/tmp/xce-ha-arrayretry
+# tune array size / round count
+XCE_HA_ARRAYRETRY_SIZE=15 XCE_HA_ARRAYRETRY_ROUNDS=4 \
+  nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --array-chaos-retry ~/tmp/xce-ha-arrayretry
+```
+
+Verdict line (`soak-verdict.sh`'s `verdict_emit_arrayretry`):
+
+```
+[xce-test-ha] VERDICT mode=array-chaos-retry rounds=5 array_size=6 double_submit=0 silent_loss=0 partial_duplication=0 result=PASS
+```
+
+Like `--array-soak`, `result` is unconditionally `PASS` here — every hard
+invariant is asserted with an immediate `fail()` inside
+`run_array_chaos_retry_round()`, so a violation aborts the run before this
+line is ever reached.
+
 ### Verdict self-test (no cluster)
 
 The verdict parser has a deterministic, pure-bash self-test that feeds fixture
@@ -298,7 +362,7 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
 |---|---|---|
 | `XCE_HA_TOP` | deploy directory | `/home/claude/tmp/xce-ha-deploy` |
 | `XCE_HA_DATADIR` | config/certs source (`cluster.bin`, `xce.crt`, `xce.key`) | `<repo>/ha/data` |
-| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, `factstream`, or `array-soak` (its `--flag` wins) | `sequential` |
+| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, `factstream`, `array-soak`, or `array-chaos-retry` (its `--flag` wins) | `sequential` |
 | `XCE_HA_BIN` | `xce_controller` binary (point at real-fs release for the gate) | `<repo>/src/xce.build/controller/xce_controller` |
 | `XCE_HA_XT` | `xce_test` binary | `<repo>/src/xce.build/controller/xce_test` |
 | `XCE_HA_ROUNDS` | kill/restart rounds (5–50) | `6` |
@@ -315,6 +379,9 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
 | `XCE_HA_ARRAYSOAK_REPLAY_RATIO_MAX` | `--array-soak` max tolerated large/small replay-time ratio before the floor/margin (Req 4.3 bound) | `20` |
 | `XCE_HA_ARRAYSOAK_REPLAY_FLOOR_MS` | `--array-soak` minimum bound floor in ms before margin is added | `2000` |
 | `XCE_HA_ARRAYSOAK_REPLAY_MARGIN_MS` | `--array-soak` ms added on top of the scaled/floor bound to absorb election-timing jitter | `3000` |
+| `XCE_HA_ARRAYRETRY_ROUNDS` | `--array-chaos-retry` kill-during-submit rounds | `5` |
+| `XCE_HA_ARRAYRETRY_SIZE` | `--array-chaos-retry` element count of the array each round submits | `6` |
+| `XCE_HA_ARRAYRETRY_KILL_DELAY` | `--array-chaos-retry` seconds slept after backgrounding the submit before killing the primary | `0` |
 
 ## What the driver does (maps to the requested steps)
 

.claude/skills/xce-test-ha/run-ha-soak.sh

@@ -56,6 +56,27 @@
 #                  --force-snapshot, hard-fails (not count-and-tolerate) on
 #                  any violated invariant, since these are the feature's core
 #                  correctness claims, not a chaotic/racy outcome class.
+#     --array-chaos-retry job-array-single-wal-record task 6.2 (Req 8.2, 8.3):
+#                  submits a job ARRAY and kills the primary IMMEDIATELY
+#                  (before it is known whether the array's WAL_JOB_ARRAY_SUBMIT
+#                  record reached majority commit — chaotic, unlike
+#                  --array-soak's clean kill-strictly-after-commit pattern),
+#                  relying on the CLIENT's own transport-level failover retry
+#                  (XCECLI_FAILOVER_CALL) to resend the IDENTICAL submission
+#                  (same base jobid — idgen() mints it once, client-side,
+#                  before the retriable call) against the newly-elected
+#                  primary. Asserts the array-scoped analog of bug-1419's
+#                  single-job cross-primary dedup: exactly ONE committed
+#                  WAL_JOB_ARRAY_SUBMIT record results (never zero — a
+#                  silently-lost write; never two-or-more — a double-submit),
+#                  and every element of the array's FULL index range is
+#                  addressable exactly once afterward (no partial
+#                  duplication). Repeats XCE_HA_ARRAYRETRY_ROUNDS times
+#                  (default 5) since the kill's exact timing relative to the
+#                  commit is inherently racy; each round is individually
+#                  hard-failed (fail(), not count-and-tolerate) since "exactly
+#                  one record, full range addressable" is a correctness
+#                  invariant, not a chaotic/racy outcome class.
 #     --selftest   run the verdict-parser self-test (soak-verdict-selftest.sh)
 #                  and exit — no cluster, no binaries needed.
 #
@@ -117,6 +138,23 @@
 #   XCE_HA_ARRAYSOAK_REPLAY_MARGIN_MS   ms added on top of the scaled/floor
 #                            bound to absorb ordinary election-timing jitter
 #                            shared by both sub-rounds (default 3000)
+#   XCE_HA_ARRAYRETRY_ROUNDS   --array-chaos-retry kill-during-submit rounds
+#                              (default 5 — independent of the general
+#                              --chaotic ROUNDS knob; a kill's exact timing
+#                              relative to the commit is racy, so several
+#                              rounds build confidence no single lucky/
+#                              unlucky interleaving decided the verdict)
+#   XCE_HA_ARRAYRETRY_SIZE     element count of the array each
+#                              --array-chaos-retry round submits (default 6 —
+#                              small enough that the FULL index range can be
+#                              addressability-checked cheaply, per Req 8.3's
+#                              "extended to the array's full index range as
+#                              one unit")
+#   XCE_HA_ARRAYRETRY_KILL_DELAY   seconds slept after backgrounding the
+#                              array submit before killing the primary
+#                              (default 0 — immediate, to maximize "chaotic"
+#                              uncertainty about whether the WAL append/
+#                              commit happened before the kill)
 set -u
 
 # ── Mode / flag parsing (flags may precede the positional [ROUNDS] [DEPLOY_DIR]) ─
@@ -136,6 +174,7 @@ while [ $# -gt 0 ]; do
         --force-snapshot) MODE="force-snapshot" ;;
         --factstream)     MODE="factstream" ;;
         --array-soak)     MODE="array-soak" ;;
+        --array-chaos-retry) MODE="array-chaos-retry" ;;
         --selftest)   exec bash "$SELF_DIR/soak-verdict-selftest.sh" ;;
         *[!0-9]*|'')  ARGS+=("$1") ;;            # non-integer ⇒ deploy dir
         *)            ROUNDS_CLI="$1" ;;         # bare integer ⇒ rounds
@@ -1471,6 +1510,128 @@ run_array_soak_round() {
     ARRAYSOAK_BOUND_VERDICT="$bound_verdict"
 }
 
+# ── Chaotic array-retry primitives (job-array-single-wal-record task 6.2) ───
+#
+# Proves the SAME cross-primary dedup mechanism single-job submission already
+# relies on (bug-1419: jobid_range_exists() + the process-local in-flight
+# registry inflight_range_try_add()/inflight_range_remove(), job.c)
+# generalizes correctly to a job ARRAY treated as one unit.
+#
+# Mechanism, traced through the actual source (job.c, client.c,
+# client_failover.c, test.c) rather than assumed:
+#   - idgen() (src/lib/idgen.c) is called EXACTLY ONCE per `jobsub` command,
+#     client-side, inside cmd_jobsub() (testsuite/test.c:1879) — BEFORE
+#     xcecli_jobsubmit() is ever invoked. The resulting base jobid is baked
+#     into the one xce_job_submit_t struct passed to xcecli_jobsubmit().
+#   - xcecli_jobsubmit() (client.c:326) wraps the actual wire call in
+#     XCECLI_FAILOVER_CALL, whose internal retry loop (client_failover.c,
+#     max_retries=20 / max_retry_ms=5000, XCE_RETRY_DEFAULTS) reconnects and
+#     RESENDS THE SAME submit struct — including the SAME jobid — on a
+#     transport failure (a connection dying because the primary was just
+#     SIGKILLed lands XCE_FO_RETRY_NOW/RETRY_BACKOFF, never a fresh idgen()
+#     call). So every retry attempt inside ONE `jobsub` command targets the
+#     exact same base jobid, by construction, without this harness doing
+#     anything special to force it.
+#   - Server-side, handle_jobsubmit() (job.c:1326) checks jobid_range_exists()
+#     then inflight_range_try_add() BEFORE branching into the array append
+#     path (job.c:1388) — both checks are RANGE-shaped (the full
+#     [base_jobid, base_jobid+array_size) interval, task 2.2's own doc
+#     comment), so the array's full index range is dedup-checked as one
+#     unit, not per-element.
+# Two legitimate outcomes, both of which must resolve to exactly one
+# committed record:
+#   (a) the original attempt's WAL_JOB_ARRAY_SUBMIT already reached majority
+#       commit before the connection died — once the new primary applies it
+#       (restore_job_array()'s idempotent-per-index skip, Req 3.3), the
+#       client's automatic retry hits jobid_range_exists()==true on the new
+#       primary, is rejected -EEXIST, and xce_failover_classify()'s
+#       "Idempotent-replay rule" (client_failover.c:47, is_write && attempt>0)
+#       reclassifies that as SUCCESS — no second record is ever appended.
+#   (b) the original attempt never reached majority commit before the
+#       connection died — the retry is a genuinely fresh, first commit for
+#       that exact jobid range on the new primary — also exactly one record.
+# Unlike task 6.1's clean single-kill-STRICTLY-AFTER-commit pattern, this
+# scenario kills the primary the moment the array submission is
+# backgrounded, deliberately BEFORE it is known which of (a)/(b) will occur —
+# that uncertainty is the whole point ("chaotic"). Hard-fails (fail(), not
+# count-and-tolerate) on any violation, matching --array-soak's convention:
+# "exactly one committed record, full index range addressable" is a
+# correctness invariant here, not a racy/sometimes-expected outcome class.
+
+# Background-safe wrapper around submit_array() (task 6.1, reused unmodified
+# above): writes "OK <base>" or "FAIL" as its only stdout line so the caller
+# can parse the outcome after `wait`-ing on the backgrounded job.
+submit_array_bg() {
+    local n="$1" base
+    if base="$(submit_array "$n")"; then
+        echo "OK $base"
+    else
+        echo "FAIL"
+    fi
+}
+
+# One chaotic kill-during-submit-then-retry round. $1 = round label (logging/
+# filenames only), $2 = array size. Sets ARRAYRETRY_LAST_BASE for the caller.
+run_array_chaos_retry_round() {
+    local round="$1" n="$2"
+    local outfile="$TOP/logs/arrayretry-round$round.out"
+    local rec_before rec_after delta prev_term newp result base
+
+    PRIMARY="$(discover_primary)"; [ -n "$PRIMARY" ] || fail "array-chaos-retry round $round: no serving primary before submit"
+    prev_term="$(latest_primary_term)"; prev_term="${prev_term:-0}"
+    rec_before="$(count_array_wal_records "$PRIMARY")"
+
+    log "array-chaos-retry round $round: backgrounding a $n-element array submit against $PRIMARY (term=$prev_term), then killing it — chaotic: commit-before-kill is NOT guaranteed either way"
+    : >"$outfile"
+    submit_array_bg "$n" >"$outfile" 2>&1 &
+    local submit_pid=$!
+
+    local kill_delay="${XCE_HA_ARRAYRETRY_KILL_DELAY:-0}"
+    case "$kill_delay" in
+        0|0.0|"") : ;;
+        *) sleep "$kill_delay" ;;
+    esac
+    kill_node "$PRIMARY" || fail "array-chaos-retry round $round: could not kill $PRIMARY"
+    DOWN="$PRIMARY"
+
+    newp="$(wait_for_new_primary "$prev_term")" || fail "array-chaos-retry round $round: no new primary elected after the kill-during-submit"
+    log "array-chaos-retry round $round: new primary $newp (term=$(latest_primary_term)) — waiting for the client's own cross-primary failover retry to resolve"
+    record_primary "$newp"
+
+    wait "$submit_pid" 2>/dev/null
+    result="$(cat "$outfile" 2>/dev/null)"
+    case "$result" in
+        "OK "*)
+            base="${result#OK }"
+            log "array-chaos-retry round $round: client-side retry resolved OK (base=$base)"
+            ;;
+        *)
+            fail "array-chaos-retry round $round: the array submission never resolved to a clean ack even after the client's own cross-primary failover retry (output: '$result') — see $outfile"
+            ;;
+    esac
+
+    start_node "$DOWN"
+    wait_for_full_convergence || fail "array-chaos-retry round $round: cluster did not reconverge to 3-of-3 after the kill-during-submit"
+    DOWN=""
+
+    rec_after="$(count_array_wal_records "$PRIMARY")"
+    delta=$((rec_after - rec_before))
+    log "array-chaos-retry round $round: WAL_JOB_ARRAY_SUBMIT record delta = $delta (before=$rec_before after=$rec_after, node=$PRIMARY, base=$base)"
+    if [ "$delta" -eq 0 ]; then
+        fail "array-chaos-retry round $round: ZERO WAL_JOB_ARRAY_SUBMIT records committed for base=$base despite the client reporting a clean ack — the write was silently lost across the failover"
+    elif [ "$delta" -gt 1 ]; then
+        fail "array-chaos-retry round $round: $delta WAL_JOB_ARRAY_SUBMIT records committed for ONE logical submission (base=$base) — DOUBLE-SUBMIT: the cross-primary dedup mechanism did not treat the retry as the array's full index range as one unit"
+    fi
+    log "array-chaos-retry round $round: CONFIRMED exactly ONE committed WAL_JOB_ARRAY_SUBMIT record for base=$base (delta=1) — retry deduped correctly"
+
+    verify_array_elements "$base" $(seq 0 $((n - 1))) \
+        || fail "array-chaos-retry round $round: not every element of base=$base's FULL index range [0,$n) is addressable after the chaotic kill-and-retry — partial duplication/loss"
+    log "array-chaos-retry round $round: all $n elements of base=$base's FULL index range individually addressable — no partial duplication"
+
+    walcheck_count "0"
+    ARRAYRETRY_LAST_BASE="$base"
+}
+
 # Drive a SPECIFIC node to primary — used to exercise leadership on the
 # lowest-priority node (ctrl-3), which the natural single-failover loop NEVER
 # elects: the election timeout is priority-staggered (my_priority * step_ms, p0
@@ -1679,6 +1840,32 @@ if [ "$MODE" = "array-soak" ]; then
     exit 0
 fi
 
+# ── Array chaos-retry mode: job-array-single-wal-record task 6.2 ────────────
+# Opt-in (--array-chaos-retry / XCE_HA_MODE=array-chaos-retry). See
+# run_array_chaos_retry_round's own header comment for the full mechanism
+# trace and rationale. Repeats XCE_HA_ARRAYRETRY_ROUNDS times (default 5 —
+# independent of the general --chaotic ROUNDS knob) since a kill-during-
+# submit's exact timing relative to the commit is inherently racy and a
+# single lucky/unlucky interleaving should not decide the verdict; each round
+# is nonetheless individually hard-failed (fail()), matching --array-soak's
+# convention, since "exactly one committed record, full range addressable"
+# is a correctness invariant, not a chaotic/racy outcome class. Does NOT run
+# the sequential loop or leadership-coverage maneuver.
+if [ "$MODE" = "array-chaos-retry" ]; then
+    AR_ROUNDS="${XCE_HA_ARRAYRETRY_ROUNDS:-5}"
+    AR_SIZE="${XCE_HA_ARRAYRETRY_SIZE:-6}"
+    log "MODE=array-chaos-retry — chaotic kill-during-array-submit retry soak ($AR_ROUNDS rounds, size=$AR_SIZE)"
+    for ar_round in $(seq 1 "$AR_ROUNDS"); do
+        log "=== array-chaos-retry round $ar_round/$AR_ROUNDS ==="
+        run_array_chaos_retry_round "$ar_round" "$AR_SIZE"
+    done
+    verdict_emit_arrayretry "$AR_ROUNDS" "$AR_SIZE"
+    log "PASS (array-chaos-retry): $AR_ROUNDS/$AR_ROUNDS rounds each resolved to exactly one committed WAL_JOB_ARRAY_SUBMIT record (no double-submit, no silent loss) with the full $AR_SIZE-element index range addressable, across a chaotic kill-during-submit failover"
+    cleanup
+    trap - EXIT INT TERM
+    exit 0
+fi
+
 # ── Chaotic mode: overlapping-kill rounds → isolated verdict ──────────────────
 # Opt-in (--chaotic / XCE_HA_MODE=chaotic). Kills land inside election/catch-up
 # windows; outcomes are COUNTED (committed divergence + committed job loss) with

.claude/skills/xce-test-ha/soak-verdict.sh

@@ -186,3 +186,20 @@ verdict_emit_arraysoak() {
         "replay_bound_verdict=$bound_verdict result=PASS"
     return 0
 }
+
+# Emit the machine-parseable verdict line for the array-chaos-retry
+# (job-array-single-wal-record task 6.2) scenario. Like verdict_emit_
+# arraysoak, this is NOT a count-and-classify verdict: every hard invariant
+# (exactly one committed WAL_JOB_ARRAY_SUBMIT record per round, never zero,
+# never two-or-more; the array's full index range addressable exactly once)
+# is asserted with an immediate fail() inside run_array_chaos_retry_round()
+# (run-ha-soak.sh) — a violation aborts the run before this line is ever
+# reached, so result is unconditionally PASS here. This function's job is to
+# report how many rounds ran and at what array size, for visibility, not to
+# gate pass/fail itself. Args: rounds size
+verdict_emit_arrayretry() {
+    local rounds="$1" size="$2"
+    echo "[xce-test-ha] VERDICT mode=array-chaos-retry rounds=$rounds array_size=$size" \
+        "double_submit=0 silent_loss=0 partial_duplication=0 result=PASS"
+    return 0
+}

.kiro/specs/job-array-single-wal-record/tasks.md

@@ -173,7 +173,7 @@
   - _Boundary: Non-Regression & Testing_
   - _Depends: 6.0a_
 
-- [ ] 6.2 Chaotic-failover array retry scenario
+- [x] 6.2 Chaotic-failover array retry scenario
   - Extend the soak harness with a scenario submitting an array immediately before killing the primary, then retrying the identical submission against the new primary
   - Assert the retry follows the same cross-primary dedup mechanism as single-job submission, extended to the array's full index range as one unit, and does not double-submit or partially duplicate
   - Observable: a soak run showing the retried array resolves to exactly one committed instance of the array, with no partial duplication
@@ -206,3 +206,4 @@
 - **Task 5.5**: Cross-daemon integration test, test-only (zero net production diff, confirmed on `request.c` and every task 5.1-5.4 file). Drives real `xce_encode()`/`xce_decode()` wire bytes through the REAL production dispatch chain on both daemons — `extract_data()` → `scheduler_req_handler()` → `sched_handle_dataset()` → `sched_on_job_array()` (C, via `#include "../src/handler.c"` in the test) and `go_push_handler()` → `handleDataset()` → `cJobArrayToModels()`/`dispatchJobArray()` (Go/CGO) — skipping only `vtask_entry()`'s ~10 lines of `static` glue, which the test mirrors verbatim (independently verified line-by-line accurate on the one behavior that mattered: `if (callnext)` treating any nonzero `extract_data()` return, including a negative errno, as "proceed"). Redelivery is tested via genuinely independent fresh decodes of separately-encoded bytes, not a cheaper decode-once-call-twice shortcut. Bug-catching capability independently proven twice (implementer, then reviewer): temporarily reverting task 5.3b's fix makes both the C and Go tests fail meaningfully; restoring returns both to green. Go/cgo toolchain constraint discovered: `import "C"` is disallowed in any `_test.go` file, forcing a two-file split (`wire_dispatch_support.go` for cgo-preamble helpers, `wire_dispatch_test.go` for the actual tests) — logged as bug-1440/1441/1442. One low-severity, non-blocking nit (not fixed): the C test file's own header comment misdescribes which assertion fails first under the RED-phase revert (claims the `callnext != 0` check; actually a downstream `>= 0` check) — cosmetic only, the test's real bug-catching property is independently confirmed accurate.
 - **Task 6.0a**: Fixed `replay_wal()`'s job-count tally (persist.c ~3902) to add a `WAL_JOB_ARRAY_SUBMIT` branch that tallies `apply_rc` (the genuine per-record element count `apply_committed_entry()`'s array case already returns) when `apply_rc > 0`. Independently traced the FULL return-value space of `restore_job_array()`/`apply_committed_entry()`'s array case (6 distinct cases: 3 negative pre-mutation failures, `-EEXIST`→`apply_rc=0` idempotent no-op, the task-3.1 bulk-insert partial-failure case (stays negative, correctly still NOT tallied — a pre-existing, out-of-boundary undercount that still surfaces as a detectable anomaly rather than masking one), and positive `created` on genuine success) and confirmed every positive `apply_rc` this fix tallies is PROVABLY equal to real jobmgr growth for that record — the fix cannot overcount, and therefore cannot weaken `persist_verify_apply_correctness()`'s ability to catch a genuine divergence. Verified end-to-end via a from-scratch live 3-node HA soak re-run (`--array-soak`, both by the implementer and independently by the reviewer): zero `APPLY_DIVERGENT` across all three of the original bug's trigger paths (restart-mid-small-array, standby election-path recovery fallback, restart-mid-N=10000-array), one-WAL-record/one-commit-barrier confirmed for both N=2 and N=10000, per-element post-failover status/cancel/completion confirmed. This soak run also directly satisfies task 6.1's own Observable — its results should be reused rather than re-run from scratch. **New non-blocking finding (pre-existing, NOT introduced by this task)**: `controller_unit_test`'s single outer `UNITY_BEGIN()` is silently reset by ~19 of its called test-group functions' OWN nested `UNITY_BEGIN()/UNITY_END()` calls, making the process exit code / meson `Ok:/Fail:` summary line unreliable for detecting a regression in any group running before the last such nested call — both the implementer and reviewer worked around this by grepping raw `testlog.txt` for `:FAIL:` lines directly rather than trusting the summary, and this should be the standard practice going forward for this binary until a dedicated follow-up fixes the harness itself.
 - **Task 6.1**: Extended `.claude/skills/xce-test-ha/run-ha-soak.sh`/`soak-verdict.sh` with a new `--array-soak` mode (small N=2 + large N=10000 sub-rounds, one clean kill/failover per sub-round — deliberately single-kill, not chaotic, matching this task's correctness-not-chaos scope; chaos-under-array-submission is task 6.2's separate job). `waldump.c` gained a `WAL_JOB_ARRAY_SUBMIT` case in `wal_type_name()` so the scenario can grep an exact type name for its live before/after WAL-record-count delta check. **Round 1 REJECTED**: the replay-time measurement used whole-second (`date +%s`) resolution — both sub-rounds' timings collapsed to `0s`/`n/a` on a real run — and, more fundamentally, nothing ever asserted on the comparison (`verdict_emit_arraysoak()` always returned PASS regardless of measured values), leaving Requirement 4.3 ("replay time governed by record size, not element count") with zero actual evidence and unable to catch a genuine O(N) regression. **Round 2 fixed**: switched to `$EPOCHREALTIME` (millisecond resolution; discovered and worked around a genuine environment trap — this sandbox's BusyBox `date` precedes GNU coreutils on `$PATH` and silently passes `%N`/`%3N` through as a literal string suffix rather than expanding or erroring, which would have poisoned any `date +%s%3N`-based approach silently); added `verdict_check_arraysoak_replay_bound()` with a real `bound = max(t_small*ratio_max, floor_ms) + margin_ms` assertion that `run_array_soak_round()` hard-`fail()`s on when exceeded — independently verified to have real teeth (both the implementer's and reviewer's own hand-picked regression cases correctly trip `BOUND_EXCEEDED`). Live-reproduced 3 times total (once round 1, twice round 2) with genuinely discriminating millisecond numbers each time (final: small=210ms, large=163ms — large array replay was NOT slower, live evidence of the feature's O(1) claim). Also fixed opportunistically: status/cancel/completion checks spread across low/mid/high indices (previously clustered at 1,2,3) matching the already-good basic-addressability spread; selftest extended 17→28 cases covering the new bound-check logic; `SKILL.md` documented. "One commit barrier" remains an inference from the WAL-record-count delta (not independently observed live) — defensible given task 2.1/2.3's unit-tested 1:1 append↔continuation coupling, but flagged as an inference, not a direct check, should a future task want to close that gap.
+- **Task 6.2**: Extended the same harness with `--array-chaos-retry` (`submit_array_bg()`, `run_array_chaos_retry_round()`, `verdict_emit_arrayretry()`). Traced the full cross-primary dedup chain end-to-end (bug-1419's mechanism): client-side `idgen()` mints the base jobid once, `xcecli_jobsubmit()`'s `XCECLI_FAILOVER_CALL` retry loop resends the IDENTICAL submit (same jobid, never re-minted); server-side `handle_jobsubmit()` checks `jobid_range_exists()`/`inflight_range_try_add()` — both genuinely RANGE-shaped over `[base,base+array_size)`, run BEFORE the array-append branch, so the whole array is dedup-checked as one unit, not per-element; `xce_failover_classify()`'s "idempotent-replay rule" reclassifies a resulting `-EEXIST` as success on retry. Reviewer went further and confirmed a subtle apply-lag race (freshly-promoted primary's process-local in-flight registry starting empty) is closed by a PRE-EXISTING, unrelated mechanism (`raft_cb_become_leader`'s synchronous apply-pump drain before the promotion callback returns, added 2026-07-10 for an unrelated bug) — not something this task needed to build, just confirmed already-sound. No production code touched or needed — the existing single-job dedup mechanism generalizes correctly to arrays with zero changes. No mutation-test RED phase was performed against production code (correctly declined per this task's explicit constraint); reviewer compensated by extracting the harness's own assertion logic into a standalone mutation-harness with stubbed collaborators and confirming all four scenarios (clean/silent-loss/double-submit/partial-duplication) correctly trigger or don't trigger `fail()` as expected. Live-verified across 4 independent fresh cluster deployments total (2 by implementer, 2 by reviewer) spanning array sizes 6, 10, and 15, 17 total chaotic kill-during-submit rounds, zero double-submits, zero silent losses, zero partial duplications, genuine election/term diversity confirmed each time. `walcheck_count`'s divergence check is informational-only here, matching task 6.1's own established convention exactly (not a new decision) — judged acceptable since `wait_for_full_convergence` already requires matching `committedLsn` across all 3 nodes before this scenario's checks run.