docs(job-array-single-wal-record): record /kiro-validate-impl GO decision

Feature-level validation ran 4 parallel dimensions: requirements
coverage (38 ACs, 33 full, 5 partial -- all tracing to the already-known
bulk-insert partial-visibility gap plus an admitted commit-barrier-count
inference), design/boundary alignment (ALIGNED, no drift), cross-task
integration (SEAMS_SOUND, wire format/arithmetic/contracts verified
consistent end to end), and mechanical checks including a live smoke
boot (fresh single-node deploy, real array submission, one WAL record,
all elements queryable) -- PASS.

The coverage pass sharpened restore_job_array()'s known bulk-insert
gap: it's a literal, code-proven violation of Requirements 2.3 and 9.4,
not just a soft nuance -- a resource-exhaustion failure during the
bulk-insert pass can permanently orphan array elements that exist and
are addressable but never get scheduled. Filed as bug-1443
(.wolf/buglog.json) for tracking. User decision: GO, ship with the
gap tracked rather than blocking on a fast-follow fix -- the trigger
condition is narrow and every other validation dimension is clean.
Closing bug-1443 needs a new kv.c/kvmgr.c primitive or a re-derive-
and-renotify path, explicitly out of this spec's approved scope.

spec.json phase -> implementation-complete.

.kiro/specs/job-array-single-wal-record/spec.json

@@ -1,23 +1,30 @@
 {
   "feature_name": "job-array-single-wal-record",
   "created_at": "2026-07-09T17:20:00Z",
-  "updated_at": "2026-07-09T17:20:00Z",
+  "updated_at": "2026-07-16T02:10:00Z",
   "language": "en",
-  "phase": "requirements-generated",
+  "phase": "implementation-complete",
   "approvals": {
     "requirements": {
       "generated": true,
-      "approved": false
+      "approved": true
     },
     "design": {
-      "generated": false,
-      "approved": false
+      "generated": true,
+      "approved": true
     },
     "tasks": {
-      "generated": false,
-      "approved": false
+      "generated": true,
+      "approved": true
     }
   },
-  "ready_for_implementation": false,
-  "blocked_on": []
+  "ready_for_implementation": true,
+  "blocked_on": [],
+  "validate_impl": {
+    "decision": "GO",
+    "validated_at": "2026-07-16T02:10:00Z",
+    "known_limitations": [
+      "bug-1443: restore_job_array()'s bulk-insert-phase failure can permanently orphan array elements (violates Req 2.3/9.4 literally); tracked in .wolf/buglog.json, not fixed in this spec"
+    ]
+  }
 }

.kiro/specs/job-array-single-wal-record/tasks.md

@@ -208,3 +208,5 @@
 - **Task 6.1**: Extended `.claude/skills/xce-test-ha/run-ha-soak.sh`/`soak-verdict.sh` with a new `--array-soak` mode (small N=2 + large N=10000 sub-rounds, one clean kill/failover per sub-round — deliberately single-kill, not chaotic, matching this task's correctness-not-chaos scope; chaos-under-array-submission is task 6.2's separate job). `waldump.c` gained a `WAL_JOB_ARRAY_SUBMIT` case in `wal_type_name()` so the scenario can grep an exact type name for its live before/after WAL-record-count delta check. **Round 1 REJECTED**: the replay-time measurement used whole-second (`date +%s`) resolution — both sub-rounds' timings collapsed to `0s`/`n/a` on a real run — and, more fundamentally, nothing ever asserted on the comparison (`verdict_emit_arraysoak()` always returned PASS regardless of measured values), leaving Requirement 4.3 ("replay time governed by record size, not element count") with zero actual evidence and unable to catch a genuine O(N) regression. **Round 2 fixed**: switched to `$EPOCHREALTIME` (millisecond resolution; discovered and worked around a genuine environment trap — this sandbox's BusyBox `date` precedes GNU coreutils on `$PATH` and silently passes `%N`/`%3N` through as a literal string suffix rather than expanding or erroring, which would have poisoned any `date +%s%3N`-based approach silently); added `verdict_check_arraysoak_replay_bound()` with a real `bound = max(t_small*ratio_max, floor_ms) + margin_ms` assertion that `run_array_soak_round()` hard-`fail()`s on when exceeded — independently verified to have real teeth (both the implementer's and reviewer's own hand-picked regression cases correctly trip `BOUND_EXCEEDED`). Live-reproduced 3 times total (once round 1, twice round 2) with genuinely discriminating millisecond numbers each time (final: small=210ms, large=163ms — large array replay was NOT slower, live evidence of the feature's O(1) claim). Also fixed opportunistically: status/cancel/completion checks spread across low/mid/high indices (previously clustered at 1,2,3) matching the already-good basic-addressability spread; selftest extended 17→28 cases covering the new bound-check logic; `SKILL.md` documented. "One commit barrier" remains an inference from the WAL-record-count delta (not independently observed live) — defensible given task 2.1/2.3's unit-tested 1:1 append↔continuation coupling, but flagged as an inference, not a direct check, should a future task want to close that gap.
 - **Task 6.2**: Extended the same harness with `--array-chaos-retry` (`submit_array_bg()`, `run_array_chaos_retry_round()`, `verdict_emit_arrayretry()`). Traced the full cross-primary dedup chain end-to-end (bug-1419's mechanism): client-side `idgen()` mints the base jobid once, `xcecli_jobsubmit()`'s `XCECLI_FAILOVER_CALL` retry loop resends the IDENTICAL submit (same jobid, never re-minted); server-side `handle_jobsubmit()` checks `jobid_range_exists()`/`inflight_range_try_add()` — both genuinely RANGE-shaped over `[base,base+array_size)`, run BEFORE the array-append branch, so the whole array is dedup-checked as one unit, not per-element; `xce_failover_classify()`'s "idempotent-replay rule" reclassifies a resulting `-EEXIST` as success on retry. Reviewer went further and confirmed a subtle apply-lag race (freshly-promoted primary's process-local in-flight registry starting empty) is closed by a PRE-EXISTING, unrelated mechanism (`raft_cb_become_leader`'s synchronous apply-pump drain before the promotion callback returns, added 2026-07-10 for an unrelated bug) — not something this task needed to build, just confirmed already-sound. No production code touched or needed — the existing single-job dedup mechanism generalizes correctly to arrays with zero changes. No mutation-test RED phase was performed against production code (correctly declined per this task's explicit constraint); reviewer compensated by extracting the harness's own assertion logic into a standalone mutation-harness with stubbed collaborators and confirming all four scenarios (clean/silent-loss/double-submit/partial-duplication) correctly trigger or don't trigger `fail()` as expected. Live-verified across 4 independent fresh cluster deployments total (2 by implementer, 2 by reviewer) spanning array sizes 6, 10, and 15, 17 total chaotic kill-during-submit rounds, zero double-submits, zero silent losses, zero partial duplications, genuine election/term diversity confirmed each time. `walcheck_count`'s divergence check is informational-only here, matching task 6.1's own established convention exactly (not a new decision) — judged acceptable since `wait_for_full_convergence` already requires matching `committedLsn` across all 3 nodes before this scenario's checks run.
 - **Task 6.3**: Pure verification task, zero file changes. Ran the complete suite: `controller_unit_test` (299 tests, 0 failures, verified via raw-log `:FAIL:` grep, not the exit code/summary — per task 6.0a's nested-`UNITY_BEGIN`/`END` counter-reset finding), all scheduler binaries, `xcelib_test` (0 failures with the registered `detect_leaks=0` config; the one pre-existing task-1.3 rtti leak still reproduces under leak-detection-on, unchanged), Go apiserver tests under both `CGO_ENABLED=0` and `=1`, and every soak mode this spec added (`--selftest` 28/28, sequential baseline, `--chaotic`, `--array-soak`, `--array-chaos-retry`) — all clean. Designed and executed a genuine empirical byte-identical baseline comparison for the non-array path (Requirement 8.1): built commit `895a204` (immediately pre-dating this spec's first commit) in an isolated, cleanly-removed `git worktree add --detach`, deployed independent clusters from both the baseline and current-tree binaries, submitted one identical single-job request to each, and did a byte-for-byte `cmp -l` of the raw on-disk WAL entries. Independently reproduced end-to-end by the reviewer (separate build, separate clusters, separate comparison) with a consistent result: every differing byte mapped cleanly to `timestamp`/`crc`/`prev_crc` (the WAL header's inherently run-specific fields) or the payload's `jobid`/`sub_time`/`queue_time` fields — zero differences anywhere in `term`/`lsn`/`type`/`payload_len` or the remaining ~6485 payload bytes. This was further corroborated by a structural code-diff argument (the entire single-job WAL-encode path is untouched between `895a204` and HEAD, purely-additive diff only) and by the pre-existing task 2.3 unit test asserting the same invariant. Confirmed the three self-flagged residual gaps (task 3.1 bulk-insert orphan prefix, task 4.2 boot-replay fence, task 5.3b bounded leak) are the complete and only set of open items across all 24 completed tasks — no new regression, job loss, divergence, or liveness stall found anywhere. **Overall verdict: GO.**
+- **`/kiro-validate-impl` feature-level gate (2026-07-16)**: Ran 4 parallel validation dimensions. Requirements coverage: 38 ACs, 33 FULL, 5 PARTIAL (2.3, 8.3, 9.4, 9.5, 9.6 — all tracing to task 3.1's already-known bulk-insert partial-visibility gap plus 8.3's admitted commit-barrier-count inference), 0 outright unclaimed gaps. Design/boundary alignment: ALIGNED, no drift, no boundary violations, all 5 Out-of-Boundary items confirmed untouched, task 5.3b/6.0a's mid-run insertions confirmed surgically scoped to their own narrow fixes. Cross-task integration: SEAMS_SOUND — wire format, `base_jobid + i` arithmetic, `apply_rc` contract, and fence naming all independently verified consistent across every producer/consumer in the chain (5 wire consumers, 3 independent arithmetic implementations in C controller/C scheduler/Go apiserver, 3 tasks sharing the `apply_rc` contract). Mechanical checks: full suite clean, TBD/secrets grep clean, and a genuine live smoke boot (fresh single-node deploy, real `jobsub -a 5` via `xce_test`, exactly one `JOB_ARRAY_SUBMIT` WAL record, all 5 elements individually queryable) — PASS.
+  The requirements-coverage pass sharpened task 3.1's known gap: it is a literal, code-proven violation of Requirement 2.3 and 9.4 (not just a soft nuance) — a bulk-insert-phase resource-exhaustion failure can permanently orphan array elements that exist and are individually addressable but never reach `HOOK_DETAIL_JOB_NEW`, so they are never scheduled or pushed, not on this attempt or any later successful retry of the same LSN. User decision on final GO/NO-GO: **GO, with the gap filed as a tracked bug (`bug-1443`, `.wolf/buglog.json`)** rather than blocking the spec on a fast-follow fix or holding NO-GO — the trigger condition is narrow (a resource-exhaustion failure at the exact moment of the bulk-insert pass) and every other validation dimension came back clean. Closing `bug-1443` requires either a new `key_del`/capacity-reservation primitive in `kv.c`/`kvmgr.c` (shared library, its own small design effort) or a re-derive-and-renotify recovery path — explicitly out of this spec's approved task list, not a quick follow-up task.