From c6138496bd8fb7093a05cd87d22555cf014f0768 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Thu, 16 Jul 2026 05:27:24 +0000 Subject: [PATCH] test(job-array-single-wal-record): chaotic-failover array retry scenario Extends the HA soak harness with --array-chaos-retry: submits an array, kills the primary essentially immediately (before commit is known), retries the identical submission against the new primary, and hard-asserts exactly one committed instance results -- no silent loss, no double-submit, no partial duplication across the array's full index range. Confirms (no production code changes needed) that the existing single-job cross-primary dedup mechanism (bug-1419: client-side jobid minted once and never re-sent on retry; server-side jobid_range_exists()/inflight_range_try_add(), both genuinely range-shaped over [base,base+array_size); xce_failover_classify()'s idempotent-replay rule) generalizes correctly to arrays, treating the whole range as one dedup unit rather than per-element. No mutation-test RED phase against production code was performed (correctly out of this task's boundary); reviewer compensated by extracting the harness's own assertion logic into a standalone mutation-harness with stubbed collaborators, confirming real fail()-gated teeth on all four outcomes (clean/silent-loss/ double-submit/partial-duplication). Live-verified across 4 independent fresh cluster deployments (2 by implementer, 2 by reviewer), array sizes 6/10/15, 17 total chaotic rounds, zero failures, genuine election/term diversity each time. Task 6.2 of job-array-single-wal-record (Requirements 8.2, 8.3). --- .claude/skills/xce-test-ha/SKILL.md | 69 ++++++- .claude/skills/xce-test-ha/run-ha-soak.sh | 187 ++++++++++++++++++ .claude/skills/xce-test-ha/soak-verdict.sh | 17 ++ .../job-array-single-wal-record/tasks.md | 3 +- 4 files changed, 274 insertions(+), 2 deletions(-) diff --git a/.claude/skills/xce-test-ha/SKILL.md b/.claude/skills/xce-test-ha/SKILL.md index 2bc2a50..ec8f543 100644 --- a/.claude/skills/xce-test-ha/SKILL.md +++ b/.claude/skills/xce-test-ha/SKILL.md @@ -202,6 +202,70 @@ run before this verdict line is ever reached. The line's job is to report the actual measurements (and the bound they were checked against) for visibility, not to gate pass/fail itself. +### `--array-chaos-retry` mode: chaotic kill-during-submit array retry (job-array-single-wal-record task 6.2) + +Extends `--array-soak`'s correctness claims to the case task 6.1 deliberately +left out of scope: submits a job array via `jobsub -a N` and kills the +primary **immediately** — before it is known whether the array's +`WAL_JOB_ARRAY_SUBMIT` record reached majority commit — instead of waiting +for a clean, known-committed state first. The array submission runs in the +**background**; the client's own transport-level failover retry +(`XCECLI_FAILOVER_CALL`, `client_failover.c`) is what actually resends the +identical submission (same base jobid — `idgen()` mints it exactly once, +client-side, in `cmd_jobsub()` before the retriable call, so every retry +attempt inside the one `jobsub` command resends the same struct, never a +fresh id) against the newly-elected primary. This is the array-scoped analog +of bug-1419's single-job cross-primary dedup (`jobid_range_exists()` + +`inflight_range_try_add()`, `job.c`), which requirements.md's Req 8.3 calls +out by name — see `run_array_chaos_retry_round()`'s header comment +(`run-ha-soak.sh`) for the full mechanism trace. + +Two legitimate outcomes exist, both of which the round asserts converge to +**exactly one** committed record: + +- the original attempt already reached majority commit before the + connection died — the retry hits `jobid_range_exists()`==true on the new + primary, is rejected `-EEXIST`, and the client's own idempotent-replay rule + (`client_failover.c`, `is_write && attempt>0`) reclassifies that as + success; no second record is ever appended; or +- the original attempt never reached majority commit — the retry is a + genuinely fresh first commit on the new primary. + +Per round, asserts (all `fail()`-gated, not just displayed): +- the WAL-record delta for that specific submission is **exactly 1** (never + 0 — a silent loss; never ≥2 — a double-submit), counted via `waldump`'s + exact `JOB_ARRAY_SUBMIT` type name (task 6.1's own addition), the same + offline-read technique `--array-soak` uses; and +- **every element of the array's full index range** `[0, N)` is individually + addressable afterward (not a sampled subset) — a stronger check than + `--array-soak`'s low/mid/high sampling, since this scenario's specific + concern is a *partial* duplication across the range, not scale. + +Repeats `XCE_HA_ARRAYRETRY_ROUNDS` times (default 5) since a kill's exact +timing relative to the commit is inherently racy — a single lucky/unlucky +interleaving must not decide the verdict. Like `--array-soak`, this is a +correctness invariant, not a chaotic/racy outcome class, so every round +hard-fails immediately on a violation rather than counting and tolerating. + +```bash +SKILL_DIR=.claude/skills/xce-test-ha +nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --array-chaos-retry ~/tmp/xce-ha-arrayretry +# tune array size / round count +XCE_HA_ARRAYRETRY_SIZE=15 XCE_HA_ARRAYRETRY_ROUNDS=4 \ + nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --array-chaos-retry ~/tmp/xce-ha-arrayretry +``` + +Verdict line (`soak-verdict.sh`'s `verdict_emit_arrayretry`): + +``` +[xce-test-ha] VERDICT mode=array-chaos-retry rounds=5 array_size=6 double_submit=0 silent_loss=0 partial_duplication=0 result=PASS +``` + +Like `--array-soak`, `result` is unconditionally `PASS` here — every hard +invariant is asserted with an immediate `fail()` inside +`run_array_chaos_retry_round()`, so a violation aborts the run before this +line is ever reached. + ### Verdict self-test (no cluster) The verdict parser has a deterministic, pure-bash self-test that feeds fixture @@ -298,7 +362,7 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK |---|---|---| | `XCE_HA_TOP` | deploy directory | `/home/claude/tmp/xce-ha-deploy` | | `XCE_HA_DATADIR` | config/certs source (`cluster.bin`, `xce.crt`, `xce.key`) | `/ha/data` | -| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, `factstream`, or `array-soak` (its `--flag` wins) | `sequential` | +| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, `factstream`, `array-soak`, or `array-chaos-retry` (its `--flag` wins) | `sequential` | | `XCE_HA_BIN` | `xce_controller` binary (point at real-fs release for the gate) | `/src/xce.build/controller/xce_controller` | | `XCE_HA_XT` | `xce_test` binary | `/src/xce.build/controller/xce_test` | | `XCE_HA_ROUNDS` | kill/restart rounds (5–50) | `6` | @@ -315,6 +379,9 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK | `XCE_HA_ARRAYSOAK_REPLAY_RATIO_MAX` | `--array-soak` max tolerated large/small replay-time ratio before the floor/margin (Req 4.3 bound) | `20` | | `XCE_HA_ARRAYSOAK_REPLAY_FLOOR_MS` | `--array-soak` minimum bound floor in ms before margin is added | `2000` | | `XCE_HA_ARRAYSOAK_REPLAY_MARGIN_MS` | `--array-soak` ms added on top of the scaled/floor bound to absorb election-timing jitter | `3000` | +| `XCE_HA_ARRAYRETRY_ROUNDS` | `--array-chaos-retry` kill-during-submit rounds | `5` | +| `XCE_HA_ARRAYRETRY_SIZE` | `--array-chaos-retry` element count of the array each round submits | `6` | +| `XCE_HA_ARRAYRETRY_KILL_DELAY` | `--array-chaos-retry` seconds slept after backgrounding the submit before killing the primary | `0` | ## What the driver does (maps to the requested steps) diff --git a/.claude/skills/xce-test-ha/run-ha-soak.sh b/.claude/skills/xce-test-ha/run-ha-soak.sh index 3995b47..b249290 100755 --- a/.claude/skills/xce-test-ha/run-ha-soak.sh +++ b/.claude/skills/xce-test-ha/run-ha-soak.sh @@ -56,6 +56,27 @@ # --force-snapshot, hard-fails (not count-and-tolerate) on # any violated invariant, since these are the feature's core # correctness claims, not a chaotic/racy outcome class. +# --array-chaos-retry job-array-single-wal-record task 6.2 (Req 8.2, 8.3): +# submits a job ARRAY and kills the primary IMMEDIATELY +# (before it is known whether the array's WAL_JOB_ARRAY_SUBMIT +# record reached majority commit — chaotic, unlike +# --array-soak's clean kill-strictly-after-commit pattern), +# relying on the CLIENT's own transport-level failover retry +# (XCECLI_FAILOVER_CALL) to resend the IDENTICAL submission +# (same base jobid — idgen() mints it once, client-side, +# before the retriable call) against the newly-elected +# primary. Asserts the array-scoped analog of bug-1419's +# single-job cross-primary dedup: exactly ONE committed +# WAL_JOB_ARRAY_SUBMIT record results (never zero — a +# silently-lost write; never two-or-more — a double-submit), +# and every element of the array's FULL index range is +# addressable exactly once afterward (no partial +# duplication). Repeats XCE_HA_ARRAYRETRY_ROUNDS times +# (default 5) since the kill's exact timing relative to the +# commit is inherently racy; each round is individually +# hard-failed (fail(), not count-and-tolerate) since "exactly +# one record, full range addressable" is a correctness +# invariant, not a chaotic/racy outcome class. # --selftest run the verdict-parser self-test (soak-verdict-selftest.sh) # and exit — no cluster, no binaries needed. # @@ -117,6 +138,23 @@ # XCE_HA_ARRAYSOAK_REPLAY_MARGIN_MS ms added on top of the scaled/floor # bound to absorb ordinary election-timing jitter # shared by both sub-rounds (default 3000) +# XCE_HA_ARRAYRETRY_ROUNDS --array-chaos-retry kill-during-submit rounds +# (default 5 — independent of the general +# --chaotic ROUNDS knob; a kill's exact timing +# relative to the commit is racy, so several +# rounds build confidence no single lucky/ +# unlucky interleaving decided the verdict) +# XCE_HA_ARRAYRETRY_SIZE element count of the array each +# --array-chaos-retry round submits (default 6 — +# small enough that the FULL index range can be +# addressability-checked cheaply, per Req 8.3's +# "extended to the array's full index range as +# one unit") +# XCE_HA_ARRAYRETRY_KILL_DELAY seconds slept after backgrounding the +# array submit before killing the primary +# (default 0 — immediate, to maximize "chaotic" +# uncertainty about whether the WAL append/ +# commit happened before the kill) set -u # ── Mode / flag parsing (flags may precede the positional [ROUNDS] [DEPLOY_DIR]) ─ @@ -136,6 +174,7 @@ while [ $# -gt 0 ]; do --force-snapshot) MODE="force-snapshot" ;; --factstream) MODE="factstream" ;; --array-soak) MODE="array-soak" ;; + --array-chaos-retry) MODE="array-chaos-retry" ;; --selftest) exec bash "$SELF_DIR/soak-verdict-selftest.sh" ;; *[!0-9]*|'') ARGS+=("$1") ;; # non-integer ⇒ deploy dir *) ROUNDS_CLI="$1" ;; # bare integer ⇒ rounds @@ -1471,6 +1510,128 @@ run_array_soak_round() { ARRAYSOAK_BOUND_VERDICT="$bound_verdict" } +# ── Chaotic array-retry primitives (job-array-single-wal-record task 6.2) ─── +# +# Proves the SAME cross-primary dedup mechanism single-job submission already +# relies on (bug-1419: jobid_range_exists() + the process-local in-flight +# registry inflight_range_try_add()/inflight_range_remove(), job.c) +# generalizes correctly to a job ARRAY treated as one unit. +# +# Mechanism, traced through the actual source (job.c, client.c, +# client_failover.c, test.c) rather than assumed: +# - idgen() (src/lib/idgen.c) is called EXACTLY ONCE per `jobsub` command, +# client-side, inside cmd_jobsub() (testsuite/test.c:1879) — BEFORE +# xcecli_jobsubmit() is ever invoked. The resulting base jobid is baked +# into the one xce_job_submit_t struct passed to xcecli_jobsubmit(). +# - xcecli_jobsubmit() (client.c:326) wraps the actual wire call in +# XCECLI_FAILOVER_CALL, whose internal retry loop (client_failover.c, +# max_retries=20 / max_retry_ms=5000, XCE_RETRY_DEFAULTS) reconnects and +# RESENDS THE SAME submit struct — including the SAME jobid — on a +# transport failure (a connection dying because the primary was just +# SIGKILLed lands XCE_FO_RETRY_NOW/RETRY_BACKOFF, never a fresh idgen() +# call). So every retry attempt inside ONE `jobsub` command targets the +# exact same base jobid, by construction, without this harness doing +# anything special to force it. +# - Server-side, handle_jobsubmit() (job.c:1326) checks jobid_range_exists() +# then inflight_range_try_add() BEFORE branching into the array append +# path (job.c:1388) — both checks are RANGE-shaped (the full +# [base_jobid, base_jobid+array_size) interval, task 2.2's own doc +# comment), so the array's full index range is dedup-checked as one +# unit, not per-element. +# Two legitimate outcomes, both of which must resolve to exactly one +# committed record: +# (a) the original attempt's WAL_JOB_ARRAY_SUBMIT already reached majority +# commit before the connection died — once the new primary applies it +# (restore_job_array()'s idempotent-per-index skip, Req 3.3), the +# client's automatic retry hits jobid_range_exists()==true on the new +# primary, is rejected -EEXIST, and xce_failover_classify()'s +# "Idempotent-replay rule" (client_failover.c:47, is_write && attempt>0) +# reclassifies that as SUCCESS — no second record is ever appended. +# (b) the original attempt never reached majority commit before the +# connection died — the retry is a genuinely fresh, first commit for +# that exact jobid range on the new primary — also exactly one record. +# Unlike task 6.1's clean single-kill-STRICTLY-AFTER-commit pattern, this +# scenario kills the primary the moment the array submission is +# backgrounded, deliberately BEFORE it is known which of (a)/(b) will occur — +# that uncertainty is the whole point ("chaotic"). Hard-fails (fail(), not +# count-and-tolerate) on any violation, matching --array-soak's convention: +# "exactly one committed record, full index range addressable" is a +# correctness invariant here, not a racy/sometimes-expected outcome class. + +# Background-safe wrapper around submit_array() (task 6.1, reused unmodified +# above): writes "OK " or "FAIL" as its only stdout line so the caller +# can parse the outcome after `wait`-ing on the backgrounded job. +submit_array_bg() { + local n="$1" base + if base="$(submit_array "$n")"; then + echo "OK $base" + else + echo "FAIL" + fi +} + +# One chaotic kill-during-submit-then-retry round. $1 = round label (logging/ +# filenames only), $2 = array size. Sets ARRAYRETRY_LAST_BASE for the caller. +run_array_chaos_retry_round() { + local round="$1" n="$2" + local outfile="$TOP/logs/arrayretry-round$round.out" + local rec_before rec_after delta prev_term newp result base + + PRIMARY="$(discover_primary)"; [ -n "$PRIMARY" ] || fail "array-chaos-retry round $round: no serving primary before submit" + prev_term="$(latest_primary_term)"; prev_term="${prev_term:-0}" + rec_before="$(count_array_wal_records "$PRIMARY")" + + log "array-chaos-retry round $round: backgrounding a $n-element array submit against $PRIMARY (term=$prev_term), then killing it — chaotic: commit-before-kill is NOT guaranteed either way" + : >"$outfile" + submit_array_bg "$n" >"$outfile" 2>&1 & + local submit_pid=$! + + local kill_delay="${XCE_HA_ARRAYRETRY_KILL_DELAY:-0}" + case "$kill_delay" in + 0|0.0|"") : ;; + *) sleep "$kill_delay" ;; + esac + kill_node "$PRIMARY" || fail "array-chaos-retry round $round: could not kill $PRIMARY" + DOWN="$PRIMARY" + + newp="$(wait_for_new_primary "$prev_term")" || fail "array-chaos-retry round $round: no new primary elected after the kill-during-submit" + log "array-chaos-retry round $round: new primary $newp (term=$(latest_primary_term)) — waiting for the client's own cross-primary failover retry to resolve" + record_primary "$newp" + + wait "$submit_pid" 2>/dev/null + result="$(cat "$outfile" 2>/dev/null)" + case "$result" in + "OK "*) + base="${result#OK }" + log "array-chaos-retry round $round: client-side retry resolved OK (base=$base)" + ;; + *) + fail "array-chaos-retry round $round: the array submission never resolved to a clean ack even after the client's own cross-primary failover retry (output: '$result') — see $outfile" + ;; + esac + + start_node "$DOWN" + wait_for_full_convergence || fail "array-chaos-retry round $round: cluster did not reconverge to 3-of-3 after the kill-during-submit" + DOWN="" + + rec_after="$(count_array_wal_records "$PRIMARY")" + delta=$((rec_after - rec_before)) + log "array-chaos-retry round $round: WAL_JOB_ARRAY_SUBMIT record delta = $delta (before=$rec_before after=$rec_after, node=$PRIMARY, base=$base)" + if [ "$delta" -eq 0 ]; then + fail "array-chaos-retry round $round: ZERO WAL_JOB_ARRAY_SUBMIT records committed for base=$base despite the client reporting a clean ack — the write was silently lost across the failover" + elif [ "$delta" -gt 1 ]; then + fail "array-chaos-retry round $round: $delta WAL_JOB_ARRAY_SUBMIT records committed for ONE logical submission (base=$base) — DOUBLE-SUBMIT: the cross-primary dedup mechanism did not treat the retry as the array's full index range as one unit" + fi + log "array-chaos-retry round $round: CONFIRMED exactly ONE committed WAL_JOB_ARRAY_SUBMIT record for base=$base (delta=1) — retry deduped correctly" + + verify_array_elements "$base" $(seq 0 $((n - 1))) \ + || fail "array-chaos-retry round $round: not every element of base=$base's FULL index range [0,$n) is addressable after the chaotic kill-and-retry — partial duplication/loss" + log "array-chaos-retry round $round: all $n elements of base=$base's FULL index range individually addressable — no partial duplication" + + walcheck_count "0" + ARRAYRETRY_LAST_BASE="$base" +} + # Drive a SPECIFIC node to primary — used to exercise leadership on the # lowest-priority node (ctrl-3), which the natural single-failover loop NEVER # elects: the election timeout is priority-staggered (my_priority * step_ms, p0 @@ -1679,6 +1840,32 @@ if [ "$MODE" = "array-soak" ]; then exit 0 fi +# ── Array chaos-retry mode: job-array-single-wal-record task 6.2 ──────────── +# Opt-in (--array-chaos-retry / XCE_HA_MODE=array-chaos-retry). See +# run_array_chaos_retry_round's own header comment for the full mechanism +# trace and rationale. Repeats XCE_HA_ARRAYRETRY_ROUNDS times (default 5 — +# independent of the general --chaotic ROUNDS knob) since a kill-during- +# submit's exact timing relative to the commit is inherently racy and a +# single lucky/unlucky interleaving should not decide the verdict; each round +# is nonetheless individually hard-failed (fail()), matching --array-soak's +# convention, since "exactly one committed record, full range addressable" +# is a correctness invariant, not a chaotic/racy outcome class. Does NOT run +# the sequential loop or leadership-coverage maneuver. +if [ "$MODE" = "array-chaos-retry" ]; then + AR_ROUNDS="${XCE_HA_ARRAYRETRY_ROUNDS:-5}" + AR_SIZE="${XCE_HA_ARRAYRETRY_SIZE:-6}" + log "MODE=array-chaos-retry — chaotic kill-during-array-submit retry soak ($AR_ROUNDS rounds, size=$AR_SIZE)" + for ar_round in $(seq 1 "$AR_ROUNDS"); do + log "=== array-chaos-retry round $ar_round/$AR_ROUNDS ===" + run_array_chaos_retry_round "$ar_round" "$AR_SIZE" + done + verdict_emit_arrayretry "$AR_ROUNDS" "$AR_SIZE" + log "PASS (array-chaos-retry): $AR_ROUNDS/$AR_ROUNDS rounds each resolved to exactly one committed WAL_JOB_ARRAY_SUBMIT record (no double-submit, no silent loss) with the full $AR_SIZE-element index range addressable, across a chaotic kill-during-submit failover" + cleanup + trap - EXIT INT TERM + exit 0 +fi + # ── Chaotic mode: overlapping-kill rounds → isolated verdict ────────────────── # Opt-in (--chaotic / XCE_HA_MODE=chaotic). Kills land inside election/catch-up # windows; outcomes are COUNTED (committed divergence + committed job loss) with diff --git a/.claude/skills/xce-test-ha/soak-verdict.sh b/.claude/skills/xce-test-ha/soak-verdict.sh index d7ddbcd..838d309 100644 --- a/.claude/skills/xce-test-ha/soak-verdict.sh +++ b/.claude/skills/xce-test-ha/soak-verdict.sh @@ -186,3 +186,20 @@ verdict_emit_arraysoak() { "replay_bound_verdict=$bound_verdict result=PASS" return 0 } + +# Emit the machine-parseable verdict line for the array-chaos-retry +# (job-array-single-wal-record task 6.2) scenario. Like verdict_emit_ +# arraysoak, this is NOT a count-and-classify verdict: every hard invariant +# (exactly one committed WAL_JOB_ARRAY_SUBMIT record per round, never zero, +# never two-or-more; the array's full index range addressable exactly once) +# is asserted with an immediate fail() inside run_array_chaos_retry_round() +# (run-ha-soak.sh) — a violation aborts the run before this line is ever +# reached, so result is unconditionally PASS here. This function's job is to +# report how many rounds ran and at what array size, for visibility, not to +# gate pass/fail itself. Args: rounds size +verdict_emit_arrayretry() { + local rounds="$1" size="$2" + echo "[xce-test-ha] VERDICT mode=array-chaos-retry rounds=$rounds array_size=$size" \ + "double_submit=0 silent_loss=0 partial_duplication=0 result=PASS" + return 0 +} diff --git a/.kiro/specs/job-array-single-wal-record/tasks.md b/.kiro/specs/job-array-single-wal-record/tasks.md index 6c0102c..819eb27 100644 --- a/.kiro/specs/job-array-single-wal-record/tasks.md +++ b/.kiro/specs/job-array-single-wal-record/tasks.md @@ -173,7 +173,7 @@ - _Boundary: Non-Regression & Testing_ - _Depends: 6.0a_ -- [ ] 6.2 Chaotic-failover array retry scenario +- [x] 6.2 Chaotic-failover array retry scenario - Extend the soak harness with a scenario submitting an array immediately before killing the primary, then retrying the identical submission against the new primary - Assert the retry follows the same cross-primary dedup mechanism as single-job submission, extended to the array's full index range as one unit, and does not double-submit or partially duplicate - Observable: a soak run showing the retried array resolves to exactly one committed instance of the array, with no partial duplication @@ -206,3 +206,4 @@ - **Task 5.5**: Cross-daemon integration test, test-only (zero net production diff, confirmed on `request.c` and every task 5.1-5.4 file). Drives real `xce_encode()`/`xce_decode()` wire bytes through the REAL production dispatch chain on both daemons — `extract_data()` → `scheduler_req_handler()` → `sched_handle_dataset()` → `sched_on_job_array()` (C, via `#include "../src/handler.c"` in the test) and `go_push_handler()` → `handleDataset()` → `cJobArrayToModels()`/`dispatchJobArray()` (Go/CGO) — skipping only `vtask_entry()`'s ~10 lines of `static` glue, which the test mirrors verbatim (independently verified line-by-line accurate on the one behavior that mattered: `if (callnext)` treating any nonzero `extract_data()` return, including a negative errno, as "proceed"). Redelivery is tested via genuinely independent fresh decodes of separately-encoded bytes, not a cheaper decode-once-call-twice shortcut. Bug-catching capability independently proven twice (implementer, then reviewer): temporarily reverting task 5.3b's fix makes both the C and Go tests fail meaningfully; restoring returns both to green. Go/cgo toolchain constraint discovered: `import "C"` is disallowed in any `_test.go` file, forcing a two-file split (`wire_dispatch_support.go` for cgo-preamble helpers, `wire_dispatch_test.go` for the actual tests) — logged as bug-1440/1441/1442. One low-severity, non-blocking nit (not fixed): the C test file's own header comment misdescribes which assertion fails first under the RED-phase revert (claims the `callnext != 0` check; actually a downstream `>= 0` check) — cosmetic only, the test's real bug-catching property is independently confirmed accurate. - **Task 6.0a**: Fixed `replay_wal()`'s job-count tally (persist.c ~3902) to add a `WAL_JOB_ARRAY_SUBMIT` branch that tallies `apply_rc` (the genuine per-record element count `apply_committed_entry()`'s array case already returns) when `apply_rc > 0`. Independently traced the FULL return-value space of `restore_job_array()`/`apply_committed_entry()`'s array case (6 distinct cases: 3 negative pre-mutation failures, `-EEXIST`→`apply_rc=0` idempotent no-op, the task-3.1 bulk-insert partial-failure case (stays negative, correctly still NOT tallied — a pre-existing, out-of-boundary undercount that still surfaces as a detectable anomaly rather than masking one), and positive `created` on genuine success) and confirmed every positive `apply_rc` this fix tallies is PROVABLY equal to real jobmgr growth for that record — the fix cannot overcount, and therefore cannot weaken `persist_verify_apply_correctness()`'s ability to catch a genuine divergence. Verified end-to-end via a from-scratch live 3-node HA soak re-run (`--array-soak`, both by the implementer and independently by the reviewer): zero `APPLY_DIVERGENT` across all three of the original bug's trigger paths (restart-mid-small-array, standby election-path recovery fallback, restart-mid-N=10000-array), one-WAL-record/one-commit-barrier confirmed for both N=2 and N=10000, per-element post-failover status/cancel/completion confirmed. This soak run also directly satisfies task 6.1's own Observable — its results should be reused rather than re-run from scratch. **New non-blocking finding (pre-existing, NOT introduced by this task)**: `controller_unit_test`'s single outer `UNITY_BEGIN()` is silently reset by ~19 of its called test-group functions' OWN nested `UNITY_BEGIN()/UNITY_END()` calls, making the process exit code / meson `Ok:/Fail:` summary line unreliable for detecting a regression in any group running before the last such nested call — both the implementer and reviewer worked around this by grepping raw `testlog.txt` for `:FAIL:` lines directly rather than trusting the summary, and this should be the standard practice going forward for this binary until a dedicated follow-up fixes the harness itself. - **Task 6.1**: Extended `.claude/skills/xce-test-ha/run-ha-soak.sh`/`soak-verdict.sh` with a new `--array-soak` mode (small N=2 + large N=10000 sub-rounds, one clean kill/failover per sub-round — deliberately single-kill, not chaotic, matching this task's correctness-not-chaos scope; chaos-under-array-submission is task 6.2's separate job). `waldump.c` gained a `WAL_JOB_ARRAY_SUBMIT` case in `wal_type_name()` so the scenario can grep an exact type name for its live before/after WAL-record-count delta check. **Round 1 REJECTED**: the replay-time measurement used whole-second (`date +%s`) resolution — both sub-rounds' timings collapsed to `0s`/`n/a` on a real run — and, more fundamentally, nothing ever asserted on the comparison (`verdict_emit_arraysoak()` always returned PASS regardless of measured values), leaving Requirement 4.3 ("replay time governed by record size, not element count") with zero actual evidence and unable to catch a genuine O(N) regression. **Round 2 fixed**: switched to `$EPOCHREALTIME` (millisecond resolution; discovered and worked around a genuine environment trap — this sandbox's BusyBox `date` precedes GNU coreutils on `$PATH` and silently passes `%N`/`%3N` through as a literal string suffix rather than expanding or erroring, which would have poisoned any `date +%s%3N`-based approach silently); added `verdict_check_arraysoak_replay_bound()` with a real `bound = max(t_small*ratio_max, floor_ms) + margin_ms` assertion that `run_array_soak_round()` hard-`fail()`s on when exceeded — independently verified to have real teeth (both the implementer's and reviewer's own hand-picked regression cases correctly trip `BOUND_EXCEEDED`). Live-reproduced 3 times total (once round 1, twice round 2) with genuinely discriminating millisecond numbers each time (final: small=210ms, large=163ms — large array replay was NOT slower, live evidence of the feature's O(1) claim). Also fixed opportunistically: status/cancel/completion checks spread across low/mid/high indices (previously clustered at 1,2,3) matching the already-good basic-addressability spread; selftest extended 17→28 cases covering the new bound-check logic; `SKILL.md` documented. "One commit barrier" remains an inference from the WAL-record-count delta (not independently observed live) — defensible given task 2.1/2.3's unit-tested 1:1 append↔continuation coupling, but flagged as an inference, not a direct check, should a future task want to close that gap. +- **Task 6.2**: Extended the same harness with `--array-chaos-retry` (`submit_array_bg()`, `run_array_chaos_retry_round()`, `verdict_emit_arrayretry()`). Traced the full cross-primary dedup chain end-to-end (bug-1419's mechanism): client-side `idgen()` mints the base jobid once, `xcecli_jobsubmit()`'s `XCECLI_FAILOVER_CALL` retry loop resends the IDENTICAL submit (same jobid, never re-minted); server-side `handle_jobsubmit()` checks `jobid_range_exists()`/`inflight_range_try_add()` — both genuinely RANGE-shaped over `[base,base+array_size)`, run BEFORE the array-append branch, so the whole array is dedup-checked as one unit, not per-element; `xce_failover_classify()`'s "idempotent-replay rule" reclassifies a resulting `-EEXIST` as success on retry. Reviewer went further and confirmed a subtle apply-lag race (freshly-promoted primary's process-local in-flight registry starting empty) is closed by a PRE-EXISTING, unrelated mechanism (`raft_cb_become_leader`'s synchronous apply-pump drain before the promotion callback returns, added 2026-07-10 for an unrelated bug) — not something this task needed to build, just confirmed already-sound. No production code touched or needed — the existing single-job dedup mechanism generalizes correctly to arrays with zero changes. No mutation-test RED phase was performed against production code (correctly declined per this task's explicit constraint); reviewer compensated by extracting the harness's own assertion logic into a standalone mutation-harness with stubbed collaborators and confirming all four scenarios (clean/silent-loss/double-submit/partial-duplication) correctly trigger or don't trigger `fail()` as expected. Live-verified across 4 independent fresh cluster deployments total (2 by implementer, 2 by reviewer) spanning array sizes 6, 10, and 15, 17 total chaotic kill-during-submit rounds, zero double-submits, zero silent losses, zero partial duplications, genuine election/term diversity confirmed each time. `walcheck_count`'s divergence check is informational-only here, matching task 6.1's own established convention exactly (not a new decision) — judged acceptable since `wait_for_full_convergence` already requires matching `committedLsn` across all 3 nodes before this scenario's checks run.