# Implementation Plan: job-array-single-wal-record

- [ ] 1. Foundation: WAL/schema additions and jobj.c hardening
- [x] 1.1 (P) Add the array WAL record type
  - Add a new `WAL_JOB_ARRAY_SUBMIT` value to the WAL entry type set, distinct from `WAL_JOB_SUBMIT`
  - Classify it as a WAIT-mode (requester-awaiting) commit, same as the existing single-job type
  - Observable: the new type is recognized by the WAL commit-mode classifier and compiles cleanly; the existing single-job type's classification is unchanged
  - _Requirements: 1.1, 1.2, 1.3_
  - _Boundary: WAL Array Submission_

- [x] 1.2 (P) Add the compact array push wire variant
  - Add one new additive variant to the existing controller-to-scheduler/apiserver push message, reusing the client submit request's existing (base jobid, array size, one spec) shape verbatim — no new struct
  - Observable: the schema compiles and generates encode/decode functions for the new variant without touching any existing variant's wire layout
  - _Requirements: 6.1, 6.2_
  - _Boundary: Compact Scheduler/API-Server Push_

- [x] 1.3 (P) jobj.c hardening
  - Make job construction check the interning/string-construction return value and propagate failure instead of silently storing a NULL success
  - Root-cause and fix the previously observed construction-table initialization-ordering gap; if root-causing shows the actual gap is outside this file's own init/call ordering (e.g. controller boot sequencing), stop and flag it as a separate follow-up rather than forcing a fix into this task's boundary
  - Verify (with a test) that the destructive-move fields on a freshly-built job payload are always NULL, so repeated use against a shared template is provably benign
  - Fix the known leak on job teardown for the one field that was never freed
  - Verify (with a test) that the field claimed to be an undocumented leak risk actually has no wire representation to leak in the first place
  - Observable: a new construction-failure path returns an error instead of a silently-broken success; the leak-fix is visible under a leak-detection test run; all five items have a passing test each
  - _Requirements: 10.1, 10.2, 10.3, 10.4, 10.5_
  - _Boundary: jobj.c Hardening_

- [ ] 2. WAL Array Submission: encode, append, commit
- [x] 2.1 Implement the single-record array append and commit path
  - Encode one compact payload carrying (base jobid, array size, one shared job spec) — the same shape as the client's own submit request
  - Append it as exactly one `WAL_JOB_ARRAY_SUBMIT` WAL entry and register exactly one commit continuation for it, regardless of array size
  - Observable: submitting an array produces one WAL append call and one registered commit continuation in a controlled test, not one per element
  - _Requirements: 1.1, 1.2, 1.3, 4.2, 9.2_
  - _Boundary: WAL Array Submission_

- [x] 2.2 Wire array submissions into the request handler
  - Branch the submission handler on array size: exactly 1 continues through today's existing single-job code path completely unmodified; greater than 1 routes to 2.1's single-append path
  - Preserve the existing pre-commit range validation (collision check plus in-flight claim) unchanged and in the same order, still rejecting strictly before any WAL append
  - Reuse the existing batched-reply mechanism, setting it to expect exactly one outcome instead of one per element
  - Observable: a single-job submission (array size 1) is byte-identical in behavior to before this feature; an array submission returns exactly one reply after exactly one outcome resolves
  - _Requirements: 1.1, 1.2, 1.3, 8.1, 9.1, 9.2_
  - _Boundary: WAL Array Submission_
  - _Depends: 1.1_

- [x] 2.3 Unit tests for the array append/commit path
  - Verify exactly one WAL entry and one commit continuation are produced for an array submission, for several array sizes
  - Verify the encoded payload's size does not grow with array size (stays constant regardless of element count)
  - Verify a single-job (array size 1) submission produces a byte-identical WAL record, commit-barrier count, and reply compared to the pre-feature baseline
  - Observable: a passing test suite demonstrating all three properties above
  - _Requirements: 1.1, 1.2, 1.3, 7.1, 7.2, 8.1_
  - _Boundary: WAL Array Submission_

- [ ] 3. Array Apply Expansion: shared N-way expansion and dispatch wiring
- [x] 3.1 Implement the shared array expansion primitive
  - Decode the committed array payload and, for each index in the array's range, build a job entry without yet making it visible to any reader
  - Skip building an entry for any index that already exists (idempotent re-apply); if every index already exists, treat the whole record as an already-applied no-op
  - Once every index has successfully built, make the whole set visible in one atomic step; if any index fails to build, discard everything built in this attempt without ever making any of it visible — a retry starts over from the first index
  - Include a test-only, forceable failure point (compiled out in production) that a test can use to force a failure at a chosen index
  - Observable: a test can force a mid-array failure and confirm zero elements are visible afterward, then force success and confirm the full set becomes visible in one step
  - _Requirements: 2.1, 2.2, 2.3, 3.3, 4.1, 5.1, 5.2, 5.3, 9.3, 9.6_
  - _Boundary: Array Apply Expansion_
  - _Depends: 1.3_

- [x] 3.2 Wire array expansion into live apply
  - Call 3.1's primitive from the live-apply/boot-replay dispatch path for the new WAL record type
  - After a successful expansion, fire the existing job-created notification exactly once, carrying every newly created element in one list — never once per element
  - Observable: applying one committed array record produces exactly one notification event carrying all N elements
  - _Requirements: 3.1, 3.4_
  - _Boundary: Array Apply Expansion_

- [x] 3.3 Wire array expansion into standby catch-up (defensive parity — see note below)
  - Call 3.1's same primitive from the separate standby catch-up/replicated-apply dispatch path for the new WAL record type, without firing any notification (matching that path's existing hook-free behavior for the single-job case)
  - Observable: a test applies the same committed array record through both the live-apply dispatch and the standby catch-up dispatch and confirms byte-identical resulting job state either way
  - _Requirements: 3.1, 3.2, 4.1 — delivered by task 3.2 alone (see Implementation Notes); this task is defensive parity, not an independent delivery path_
  - _Boundary: Array Apply Expansion_

- [x] 3.4 Unit tests for array expansion
  - Verify N-way expansion produces the correct contiguous set of element identities for several array sizes
  - Verify idempotent re-apply of an already-fully-applied record is a no-op that creates nothing new
  - Verify, using 3.1's forceable failure point, that a failure partway through leaves nothing visible, and that a subsequent successful retry produces the complete, non-duplicated set
  - Observable: a passing test suite covering all three properties above
  - _Requirements: 2.1, 2.2, 2.3, 3.3, 5.1, 5.2, 5.3, 9.4, 9.6_
  - _Boundary: Array Apply Expansion_

- [ ] 4. Live Apply Resource Fence
- [x] 4.1 Make the apply pump respect apply failures
  - Stop discarding the array-expansion (and every other WAL type's) apply outcome; on a failure, halt the apply walk at that position without advancing past it, and log a loud, operator-visible error naming the stuck position
  - Observable: a simulated apply failure at a given position causes the walk to stop exactly there and not proceed to later entries
  - _Requirements: 9.4_
  - _Boundary: Live Apply Resource Fence_

- [x] 4.2 Add the live, self-healing fence
  - Add a new piece of state (distinct from the existing boot-only unrecoverable-fence flag) recording "a position is currently stuck failing to apply"
  - Engage it when 4.1's halt condition triggers; clear it automatically, from inside the apply walk itself, the moment a retry of that same stuck position succeeds
  - Observable: after a forced failure the new state reads "fenced"; after the same position later succeeds on retry, the state reads "clear" without any external action
  - _Requirements: 9.4, 9.5_
  - _Boundary: Live Apply Resource Fence_

- [x] 4.3 Gate promotion on the live fence
  - Add the new fence state as a required condition at both existing points where a node's eligibility for promotion is decided (the periodic release check and the election-time check)
  - Observable: a node with the fence engaged is refused promotion at both points; once the fence clears, the node is eligible again without any other change
  - _Requirements: 9.4, 9.5_
  - _Boundary: Live Apply Resource Fence_

- [x] 4.4 Fault-injection test for the live fence
  - Using 3.1's forceable failure point, force a resource-exhaustion-style failure during array expansion and verify: the node fences itself and is refused promotion, constructs no partial or divergent set of elements, does not affect any other (simulated) node's independent progress, and self-heals — clearing the fence and completing the apply — once the forced condition is removed and the retry succeeds
  - Observable: a single passing test exercising the full engage-refuse-clear-recover sequence
  - _Requirements: 9.4, 9.5, 9.6_
  - _Boundary: Live Apply Resource Fence_
  - _Depends: 3.1_

- [ ] 5. Compact Scheduler/API-Server Push
- [x] 5.1 (P) Build the compact scheduler push
  - Add a function that encodes the compact (base jobid, array size, one spec) message for the scheduler and branch the existing job-created push handler to call it, instead of the existing per-job push, whenever the notification carries more than one element
  - Leave the single-element (ordinary single-job) branch completely unmodified
  - Observable: a synthetic multi-element notification produces exactly one compact push call, not N individually-encoded job structs; a synthetic single-element notification is unaffected
  - _Requirements: 6.1, 6.3, 6.4_
  - _Boundary: Compact Scheduler/API-Server Push (controller-side scheduler push)_

- [x] 5.2 (P) Build the compact API-server push
  - Same construction as 5.1, for the API-server push handler and its own existing per-job push
  - Observable: a synthetic multi-element notification produces exactly one compact push per connected API server, not N individually-encoded job structs; the single-element branch is unaffected
  - _Requirements: 6.2, 6.4_
  - _Boundary: Compact Scheduler/API-Server Push (controller-side apiserver push)_

- [x] 5.3 (P) Scheduler-side local expansion
  - Decode the compact push and expand it locally into N ordinary schedulable tasks using the same base-jobid-plus-index arithmetic the controller itself uses
  - Deduplicate by element identity so a redelivered push (e.g. after a scheduler reconnect) does not create duplicate tasks
  - Log one identifiable confirmation line (base jobid plus element count) once expansion completes
  - Observable: feeding the same compact push twice to the receiver produces the task set once, not twice, and exactly one confirmation log line per successful expansion
  - _Requirements: 6.3, 6.4, 6.5, 8.4_
  - _Boundary: Compact Scheduler/API-Server Push (scheduler daemon receive path)_
  - _Depends: 5.1_

- [x] 5.3b Fix the shared wire dispatcher to carry the compact array push
  - `extract_data()` (`src/lib/request.c`) is the shared reassembly step every inbound push (scheduler and API-server alike) passes through inside `vtask_entry()` (`src/lib/xcectx.c`) before a receiver's handler ever runs. Its dispatch switch has no case for the `Data_data_jobArray` variant (added task 1.2) and falls into `default: rc = -EINVAL`. Because `vtask_entry()`'s `if (callnext)` treats ANY nonzero return — including a negative errno — as "reassembly complete, proceed," this doesn't surface as an error: the real payload is freed and the receiver is silently handed an empty, zeroed `dataset_t` (`rows=NULL, nrows=0`), with no log anywhere in the chain (`vtask_entry()` discards the handler's own return value unconditionally)
  - Add a `case Data_data_jobArray:` to `extract_data()` that passes the single, unchunked `JobSubmit` descriptor straight through to the caller's `dataset_t` — no pagination/accumulation loop is needed (unlike the existing LIST-shaped cases), since this push is always exactly one descriptor per commit, matching the "wire shape note" both `xcesched_push_jobarray()` (task 5.1) and `apiserver_push_jobarray()` (task 5.2) already establish
  - Confirms both the scheduler (task 5.3, already landed) and the future API-server Go receiver (task 5.4) share this exact fix — this task only needs to land once
  - Observable: a synthetic `Data_data_jobArray` push driven through the REAL `vtask_entry()`/`extract_data()` path (not task 5.3's white-box `sched_handle_dataset()`-direct test) reaches the scheduler's `sched_on_job_array()` with the genuine payload intact, not an empty dataset
  - _Requirements: 6.1, 6.2, 6.4, 8.4_
  - _Boundary: shared wire transport (`src/lib/request.c`, `src/lib/xcectx.c`) — out of the Compact Scheduler/API-Server Push component's original File Structure Plan; discovered during task 5.3's review (2026-07-15), design.md corrected to record it_
  - _Depends: 5.1, 5.2_

- [x] 5.4 (P) API-server-side local expansion
  - Decode the compact push and expand it locally into N per-element records using the same base-jobid-plus-index arithmetic
  - Deduplicate by element identity so a redelivered push does not create duplicate records
  - Log one identifiable confirmation line (base jobid plus element count) once expansion completes
  - Observable: feeding the same compact push twice to the receiver produces the record set once, not twice, and exactly one confirmation log line per successful expansion
  - _Requirements: 6.2, 6.4, 6.5, 8.4_
  - _Boundary: Compact Scheduler/API-Server Push (apiserver receive path)_
  - _Depends: 5.2_

- [x] 5.5 Integration test: end-to-end compact push
  - Commit one array record end-to-end and confirm the scheduler and the API server each receive exactly one push and each expand it into N correctly-addressable entries
  - Confirm a simulated redelivery (reconnect) to either receiver does not double-expand
  - Observable: a passing integration test covering both receivers' one-push/N-entries/no-double-expand behavior, including both receivers' confirmation log lines
  - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 8.4_
  - _Boundary: Compact Scheduler/API-Server Push_

- [ ] 6. Integration & non-regression soak

- [x] 6.0a Fix replay_wal()'s job-count tally for WAL_JOB_ARRAY_SUBMIT
  - `replay_wal()` (`src/controller/persist.c` ~3902) tallies `*new_job_submits_out` only for `hdr.type == WAL_JOB_SUBMIT`. `WAL_JOB_ARRAY_SUBMIT` (task 1.1) is dispatched to `apply_committed_entry()` and correctly materializes N real jobmgr entries, but contributes 0 to this tally. `persist_verify_apply_correctness()` then compares real jobmgr growth (correct, N-sized) against the undercounted expectation, produces a false `APPLY_DIVERGENT` verdict, and engages `persist_recover_loud_fence_if_unrecoverable()` — a PERMANENT, non-self-healing fence (distinct from task 4.2's Live Apply Resource Fence, which self-heals; this one has no live-clear path short of a later restart landing after a checkpoint advances past the array's LSN)
  - Live-reproduced during task 6.1's soak attempt (2026-07-15) on 2 of 3 nodes in a single small-array (N=2) sub-round, via two different trigger paths (ordinary boot-time `persist_recover()`, and a standby's election-path recovery fallback) — confirmed deterministic, not racy, not size-dependent. A pre-existing comment at persist.c ~3863-3874 (predates this spec, from `raft-apply-decoupling` task 2.3) already flagged this exact code region as needing re-verification "before tasks 4.1/6.1/6.2/6.3 make this path broadly live" — this is that re-verification, and it failed
  - Fix: add a `WAL_JOB_ARRAY_SUBMIT` branch to the tally that adds `apply_rc` (the genuine new-element count `apply_committed_entry()`'s array case already returns) to `*new_job_submits_out` when `apply_rc > 0`, mirroring the existing dedup-safe pattern used for `WAL_JOB_SUBMIT`
  - Observable: the exact live-soak reproduction (small array submit, then restart/re-elect a node) no longer produces `APPLY_DIVERGENT`; the node boots/rejoins cleanly and remains promotable
  - _Requirements: 3.1, 3.2, 4.1, 9.4_
  - _Boundary: Array Apply Expansion (`persist.c`'s replay-side job-count accounting) — discovered during task 6.1's soak (2026-07-15), inserted per explicit user decision; blocks 6.1/6.2/6.3, all of which require a cluster that survives a restart after an array submission_

- [x] 6.1 Large-array soak scenario
  - Extend the existing HA soak harness with a large-array submission scenario asserting: one WAL record and one commit barrier regardless of array size, failover replay time governed by the record's size rather than element count, and every array element individually addressable (status, cancel, completion) after a failover
  - Observable: a soak run reporting the one-record/one-barrier assertion, a replay-time measurement independent of array size, and successful per-element status checks post-failover
  - _Requirements: 4.3, 5.2, 5.3, 8.2, 8.3_
  - _Boundary: Non-Regression & Testing_
  - _Depends: 6.0a_

- [x] 6.2 Chaotic-failover array retry scenario
  - Extend the soak harness with a scenario submitting an array immediately before killing the primary, then retrying the identical submission against the new primary
  - Assert the retry follows the same cross-primary dedup mechanism as single-job submission, extended to the array's full index range as one unit, and does not double-submit or partially duplicate
  - Observable: a soak run showing the retried array resolves to exactly one committed instance of the array, with no partial duplication
  - _Requirements: 8.2, 8.3_
  - _Boundary: Non-Regression & Testing_

- [x] 6.3 Full regression pass
  - Run the complete existing unit, integration, and soak suite (sequential and chaotic) and confirm zero new job loss, divergence, or liveness stall attributable to this feature
  - Confirm the non-array (array size 1) WAL format, commit-barrier count, and reply are byte-identical to the pre-feature baseline
  - Observable: a clean full-suite run with the byte-identical single-job baseline check passing alongside it
  - _Requirements: 8.1, 8.2, 8.4_
  - _Boundary: Non-Regression & Testing_

- [x] 7. Residual gap closure (post-GO follow-up, user-directed 2026-07-16)
- [x] 7.1 Close the bulk-insert-phase partial-visibility gap (bug-1443)
  - Add a new, purely-additive capacity-reservation primitive (`kvmgr_reserve(var kvmgr, int n)` in `kvmgr.c`, following the existing non-vtable `kvmgr_capture_keys()` template exactly — same "caller MUST hold the kvmgr write lock" contract, same bare-extern-function style rather than a new RTTI vtable slot) that pre-grows `c->jobmgr`'s backing `xzarray` for `n` additional entries via a new small non-static wrapper in `zarray.c` calling the existing, unmodified `xzarray_ensure_items(a, a->count + n)` — zero changes to any existing function in either file
  - Call `kvmgr_reserve(c->jobmgr, array_size)` in `restore_job_array()` immediately before the bulk-insert pass begins; on reservation failure, treat exactly like today's existing pre-mutation failure paths (discard the scratch list, `c->jobmgr` never touched, return negative) — this eliminates the dominant, N-scaling failure trigger (the large `realloc()` growing the value store) structurally, before any `key_add()` call in the bulk-insert loop can run
  - Document honestly (in the function's doc comment) that this narrows, not eliminates, the residual gap: `key_add()`'s own tiny per-entry hash-chain-node allocation and a content-based `-EEXIST` collision remain theoretically possible even after reservation, both far rarer and non-N-scaling than the closed failure mode
  - For that now-much-narrower residual case: make `restore_job_array()` populate `out_new_jobs` with whatever prefix was genuinely, successfully inserted before a mid-pass failure, even though the function's overall return is still negative — and make `apply_committed_entry()`'s `WAL_JOB_ARRAY_SUBMIT` case fire `HOOK_DETAIL_JOB_NEW` for that non-empty partial prefix even when `restore_job_array()`'s return is negative, while still propagating the negative return so the LSN correctly halts and the Live Apply Resource Fence still engages for the remaining un-constructed indices. Net effect: a genuinely-inserted element is never left un-pushed/un-scheduled again, even in the narrow residual failure case — closing Requirement 2.3/9.4's literal text, not just narrowing the trigger condition
  - Observable: a fault-injection test forcing the bulk-insert-phase failure (reusing task 3.1's existing bulk-insert-phase seam) shows the already-inserted prefix IS included in a fired `HOOK_DETAIL_JOB_NEW` (not silently orphaned), while the overall LSN still correctly halts/fences for the remainder; a separate test proves `kvmgr_reserve()` genuinely pre-grows capacity (no realloc occurs during the subsequent bulk-insert loop when reservation succeeded)
  - _Requirements: 2.3, 9.4, 9.6_
  - _Boundary: Array Apply Expansion (`src/lib/kvmgr.c`, `src/lib/zarray.c`, `src/controller/job.c`'s `restore_job_array()`, `src/controller/persist.c`'s `apply_committed_entry()` `WAL_JOB_ARRAY_SUBMIT` case) — touches shared library code (`kvmgr.c`/`zarray.c`) used far beyond this feature; new functions only, zero changes to existing function bodies_

- [x] 7.2 Extend the Live Apply Resource Fence's protection across a boot/restart replay (task 4.2's self-flagged gap)
  - `replay_wal()` (boot/restart replay path) currently captures `apply_committed_entry()`'s return value only for the job-count tally (task 6.0a) — it does not halt or engage the Live Apply Resource Fence on a negative return the way `persist_apply_pump_run()` (the live path, task 4.1) already does
  - Add the same halt-and-fence behavior to `replay_wal()`: on a negative `apply_committed_entry()` return during replay, stop the replay walk at that LSN without advancing past it, log loud, and call the SAME `persist_mark_apply_resource_fenced()` primitive the live path uses — reuse, do not reimplement
  - Verify (and document, not just assume) that this engages BEFORE the node could ever be considered for promotion during its own boot sequence — trace the actual boot ordering to confirm `persist_recover()`/`replay_wal()` completes before `ha.c`'s promotion-gate logic could evaluate this node as a candidate
  - `persist_init()`'s existing unconditional fence-reset on every fresh process construction remains correct and unchanged: if the same resource-exhaustion condition recurs during a later boot replay, this task's new check will immediately re-engage the fence before promotion is possible, so starting from a clear state on each fresh process is safe by construction, not a gap
  - Observable: a test simulating a node that crashes while fenced and restarts into a boot replay hitting the identical resource-exhaustion condition (reuse task 3.1's fault-injection seam) shows the node re-engages the fence during replay and is refused promotion, rather than silently booting clean and looking promotable while actually behind
  - _Requirements: 9.4, 9.5_
  - _Boundary: Live Apply Resource Fence (`src/controller/persist.c`'s `replay_wal()`) — reuses existing task 4.1/4.2 primitives verbatim, no new fence mechanism_
  - _Depends: 7.1 is not a prerequisite (independent root cause), but should land first since both touch `persist.c`_

- [x] 7.3 Fix the bounded per-commit leak on the scheduler receive path (task 5.3b's self-flagged gap)
  - `sched_handle_dataset()`'s `Data_data_jobArray` case never frees `dataset->rows` (the decoded `xce_job_submit_t*`) after `sched_on_job_array()` consumes it — `free_Dataset()` is an empty no-op for every table type, so this is the one call site responsible for freeing this specific variant's payload
  - Free `dataset->rows` (and its embedded submit spec, following whatever the existing `xce_job_submit_t`/`JobSubmitData` teardown convention already is elsewhere in this file) after `sched_on_job_array()` returns, in both the success and any early-return/error path through that case
  - Observable: a leak-detection test run (ASan) around a real array push shows zero growth attributable to this path, where it previously showed one bounded leak per delivered array push
  - _Requirements: 8.4_
  - _Boundary: Compact Scheduler/API-Server Push (`src/scheduler/src/handler.c`'s `sched_handle_dataset()`)_

## Implementation Notes

- **Task 7.3**: `sched_handle_dataset()`'s `Data_data_jobArray` case now calls `free_JobSubmit_ptr((xce_job_submit_t**)&dataset->rows)` + `dataset->nrows = 0` after `sched_on_job_array()` returns (success and error paths alike), mirroring the sibling list-shaped cases' ownership-transfer convention. `free_JobSubmit_ptr()` is the existing capnp-generated teardown for this exact struct shape (frees `submit`'s 7 strings + `start_info`'s 10 strings + both nested structs + the top-level struct) — no new free logic was needed, just the missing call site. Fixing this exposed that `test_job_array_expand.c`'s white-box fixture (`init_job_array()`) built `dataset->rows` from stack-allocated structs with string-literal fields; freeing those crashes (SEGV in `free_JobSubmitData`). The fixture was rewritten to heap-allocate + `strdup` (matching the real `decode_JobSubmit_ptr()` wire-decode shape), and `test_job_array_redelivery_does_not_duplicate` now builds a fresh descriptor per delivery (the first is consumed+freed by the handler, so reusing one pointer across two calls is no longer valid — this also matches production, where a redelivery is always a freshly-decoded wire message, never a reused pointer). Verified via direct ASAN runs of all 4 scheduler job-array test binaries (leak detection is disabled by default in the meson test harness): zero leak growth traceable to the jobArray path in any of them (confirmed by grepping leak stack traces, not just byte counts, since fixture-init debt is a separate pre-existing, unrelated leak source in the same binaries).

- **Task 1.3 (FIXED 2026-07-17, upstream)**: `meson test --suite xce`'s `libxce test` target used to SIGABRT under ASAN leak detection whenever any code path exercised a `job_init()` failure — root cause was the vendored `rtti` library's `makeInstance()` not freeing its allocation when a type's `.init()` returns negative (no `.c` source available, headers-only in the nix store). Pre-existing, independently reproduced against an unmodified pre-task `jobj.c`, not introduced by this feature. **Resolved by a `librtti` repin** (`src/flake.lock`: rev `7212ba170d8db040ac99d0fb47c7b1df70df6e11` → `53cf11e6df4e2f0bbcdf8e26feed74d011b0e9c1`, revCount 20→21) — verified via a genuinely fresh rebuild (`nix flake metadata` confirms the new rev resolved) plus a full green `libxce test` run and a standalone `test_jobj_hardening` run under `ASAN_OPTIONS=detect_leaks=1:abort_on_error=1`: `test_job_init_propagates_makeJobsub_failure` (the trigger test) now passes clean, zero LeakSanitizer output, no `makeInstance`/`jobj.c:768` leak stack. Any future task whose tests exercise a `create_job()`/`makeJob()` failure path should now see a normal PASS/FAIL Unity result, not a SIGABRT — if the SIGABRT reappears, treat it as a genuine regression (possibly a librtti downgrade), not this known-and-now-closed issue.
- **Task 1.3**: `job_obj_init()`/`job_obj_fini()` are now idempotent (atomic_flag-guarded) — safe to call more than once without an intervening `fini()`. Any future test construction can rely on this instead of coordinating a single call site.
- **Task 1.3**: `makeJobsub()`/`makeJobstartinfo()` now return `-ENODEV` (not a generic `-EINVAL`) when the interning table isn't initialized — if a later task's error handling branches on `job_init()`'s return value, `-ENODEV` is a distinct, checkable signal for "table not ready" vs. other construction failures.
- **Task 3.3**: `apply_wal_entry()` (persist.c:2574) is dead code in production — confirmed via exhaustive grep (zero call sites outside its own declaration/definition/comments) and via `ha.c:8024`'s `ha_standby_advance_committed_and_apply` (dated 2026-07-10), whose own doc comment explicitly says standby entity materialization now goes through the SAME `apply_committed_entry()`/`persist_apply_pump_kick` pump the primary uses, not `apply_wal_entry()`. The persist.c:3722 comment claiming otherwise predates that refactor and was never updated (design.md corrected accordingly). Requirement 3.1/3.2/4.1 are fully delivered by task 3.2 alone. Task 3.3's code was kept (not discarded) as defensive parity with `WAL_JOB_SUBMIT`'s existing, also-dead case in the same switch — it is tested and correct, just not currently reachable in production. If `apply_wal_entry()` is ever revived or removed entirely in a future spec, this task's case should move or be deleted with it.
- **Task 3.1 (superseded by task 7.1, see that entry below)**: `restore_job_array()`'s atomicity originally had one narrow, documented exception: the final bulk-insert-into-`c->jobmgr` pass could itself fail on allocation/growth exhaustion (`c->jobmgr` grows via `realloc()`; no capacity-reservation primitive existed anywhere in `kv.c`/`kvmgr.c`/`zarray.c` to pre-check this before mutating), and an already-inserted prefix `[0, inserted)` could be left permanently visible in `c->jobmgr` but never included in `out_new_jobs`, so `HOOK_DETAIL_JOB_NEW` never fired for it — tracked as bug-1443. **This gap is now closed** (task 7.1, commit dcf9f39, 2026-07-16): `kvmgr_reserve()`/`xzarray_reserve()` pre-grow `c->jobmgr` before the bulk-insert pass starts, structurally closing the dominant N-scaling trigger; for the narrower residual case that remains theoretically possible (a tiny per-entry hash-chain-node allocation or a content collision, non-N-scaling), `restore_job_array()` now populates `out_new_jobs` with the genuinely-inserted prefix even on failure, and `apply_committed_entry()` fires `HOOK_DETAIL_JOB_NEW` for that prefix while still propagating the negative rc so the LSN halts and the Live Apply Resource Fence engages for the remainder. See task 7.1 below for full detail.
  **UPDATE (post-4.4): this gap was NOT closed by Phase 4.** Tasks 4.1-4.4 correctly deliver the generic negative-rc halt/fence/gate/self-heal machinery — a bulk-insert-phase failure DOES still engage the fence and block promotion, same as any other failure — but none of them add the "no partial/divergent set" guarantee specifically for this seam, because that guarantee is structurally false on this path with current APIs (no `key_del`). Task 4.4's own capstone test deliberately used the (fully atomic) construction-phase seam instead, precisely because asserting "zero partial elements" against the bulk-insert seam would be dishonest. **This residual gap remains open and unowned by any task in this spec's current scope** — flag at `/kiro-validate-impl` time; closing it would require either a `key_del`/rollback primitive in `kv.c`/`kvmgr.c`, or a re-derive-and-re-notify recovery path for the orphaned prefix, either of which is design-level work beyond this spec's approved task list.
- **Task 4.1**: `persist_apply_pump_run()` now halts the walk (no `last_applied_lsn` advance) on ANY negative return from `apply_committed_entry()`, not just the new array type — this is a general fix. Investigated whether this newly exposes a hazard for the pre-existing job-state fact entries (`WAL_JOB_START/FINISH/FAIL/UPDATE`), which have their own real `-ENOENT` branch when the target jobid isn't in `c->jobmgr`: confirmed NOT a hazard, because `c->jobmgr` (`xkvmgr_type`) has no `key_del` (jobs are never removed for the process lifetime, corroborated by `job.c:428,455,645`), and this task's own strict sequential-no-skip apply means a fact's LSN is only reached after its causally-prior submit already applied. So a live `-ENOENT` on a fact can now only mean genuine corruption/divergence — halting is correct, not a new footgun. Independent reviewer confirmed this via `test_term_safety_adapter.c:2039-2043`'s synthetic (non-production-representative) trigger. design.md's "Existing Architecture Analysis" table row overclaims ("every existing WAL type... returns 0 or positive in practice") — technically inaccurate re: the fact-ENOENT branch — left as-is since the persist.c code comment at the site already documents the exception correctly; low-severity, non-blocking.
- **Task 4.2**: New `_Atomic uint64_t apply_resource_fenced_lsn` field on `persist_t` (0 = clear, nonzero = the stuck LSN) plus `persist_apply_resource_fenced()`/`persist_mark_apply_resource_fenced()`/`persist_clear_apply_resource_fence()`, wired into `persist_apply_pump_run()`'s existing halt branch (engage) and unconditionally after every successful advance (clear). The unconditional clear is provably equivalent to "clear only on the fenced LSN's own successful retry" because `last_applied_lsn` has exactly 3 writers codebase-wide, the other two (`replay_wal`, `persist_recover`) are boot-only and temporally disjoint from the live pump, and a failed walk never advances past the stuck LSN — so the next success by construction IS that LSN's retry. **Known residual gap (out of this task's boundary, flagged by review, NOT yet fixed)**: `replay_wal()` (boot/restart replay, persist.c:3875-3888) still discards `apply_committed_entry()`'s failure return the same way the live pump did before task 4.1, and `persist_init()` unconditionally zeroes the fence field on every restart. A node that crashes while fenced and re-hits the identical resource-exhaustion condition during boot replay will silently pass it and look fully promotable while actually divergent — task 4.1/4.2's protection does not yet extend to the boot-replay path. This is a pre-existing gap (not introduced by this feature) but is now inconsistent with the live path's new guarantee; worth a dedicated follow-up task before anything downstream places full trust in this fence across a restart.
- **Task 5.1**: New `xcesched_push_jobarray()` (`sched.c`) sends via direct `xce_request()`/`OPCode_push`/`Request_body_pushmsg` rather than the existing `xce_push_data_req()` helper — `xce_push_data_req()`'s dispatch switch hard-rejects (`-EINVAL`) anything outside jobs/workers/resources/resvals, so it cannot carry a single non-list `Data.jobArray` descriptor. design.md corrected to match (Dependencies + API Contract for the Compact Scheduler/API-Server Push component). Task 5.2's API-server push will hit the identical fork and should follow the same direct-`xce_request()` pattern rather than re-deriving it. One accepted, low-severity trade-off: the direct path does not go through `xce_push_data_req()`'s `checkf`-gated `check_controller`/`c->running` liveness check that every other push gets — a narrow shutdown-race behavioral difference from sibling pushes, not a new safety hazard; not fixed here since threading a liveness check into a single new call site duplicates rather than reuses the shared gate. `sched.c`/`dataex.c`/`egress.c` were never linked into `controller_unit_test` (two existing test files there ship their own fake `on_scheduler_join()`/`notify()`/`collect_confvars()` precisely because of this, and linking the real ones would collide) — task 5.1's tests live in a new standalone `sched_push_unit_test` binary instead; task 5.2 will likely face the same fork for `apiserver.c` and should check for an analogous existing-fakes collision before choosing a test-binary strategy.
- **Task 5.2**: `apiserver_push_jobarray()` (`apiserver.c`) mirrors task 5.1's `xcesched_push_jobarray()` exactly — same field derivation, same direct-`xce_request()` transport, same `l->count > 1` discriminator — and additionally, correctly replicates `apiserver_push_joblist()`'s existing per-connected-server fan-out loop (`N_APISERVER_MAX` iteration, same lock discipline), independently verified against 2 simulated connected servers each receiving exactly one push. Confirmed the same `apiserver.c`/`dataex.c` link-collision task 5.1 hit for `sched.c` (a fake `notify()` already exists in `test_commit_barrier_reply.c`) — new standalone `apiserver_push_unit_test` binary added, correctly omitting `egress.c` since `apiserver_push_joblist()`, unlike `sched.c`'s push path, never references `job_submit_committed()`/`collect_jobs_committed()`. One low-severity, non-blocking nit (not fixed): the new test file's comment justifying its `apiserver_init()` idempotency guard is factually inaccurate — it claims a second `apiserver_init()` call would double-register hooks, but `xce_add_hook()` has deduped by `(fn,data)` since bug-845, so that specific claim is wrong; the guard itself is still correct/necessary for a different, real reason (avoids re-`mylock_init()`-ing an already-initialized mutex, UB).
- **Task 5.3**: `sched_on_job_array()` (`handler.c`) decodes `Data_data_jobArray`, derives each element's jobid via `base_jobid + i`, and routes constructed elements through the unmodified per-job pipeline (`sched_on_jobs()`). Each element gets its OWN deep clone (`clone_job_submit_data`/`clone_job_startinfo`) of the shared submit spec — required because `sched_pend_update_killer()`'s per-element teardown (`free_Job_ptr`) frees each element's `submit` independently and a shared pointer across N elements would double-free; independently verified. Dedup reuses `sched_get_job()` against the same `sched->jobs` kvtab `sched_add_job()`'s `key_add` authoritatively guards — sound at the data-integrity level, but review found a narrow, non-corrupting TOCTOU: the dedup pre-check runs synchronously at receive time while the actual insert only happens later when `sched_update()` drains `pend_updates[0]`, so a redelivery landing before that drain can re-construct/re-enqueue a duplicate (harmless) object and double the confirmation log line — the same pre-existing, task-5.3-unrelated pattern as `sched_update_jobs()`'s already-ignored `key_add` `-EEXIST` return on the ordinary single-job path. Not fixed (pre-existing class of gap, narrow window, non-corrupting).
  **CRITICAL discovery during this task's review (2026-07-15), NOT a defect in task 5.3 itself**: the shared wire dispatcher `extract_data()` (`src/lib/request.c`) has no case for `Data_data_jobArray` and silently drops every real array push before ANY receiver's handler runs (see new task 5.3b, inserted immediately after this task per explicit user decision). Confirmed to affect task 5.4's future API-server receive path identically (both share the same `xcecli_ctx_init()` → `vtask_entry()` → `extract_data()` pipeline). Task 5.3's own tests are unaffected by this gap since they use the same established white-box convention as every sibling `handler.c` test (calling `sched_handle_dataset()` directly, bypassing the transport layer) — this is not a new testing shortcut, it just means task 5.3's passing tests do not, and were never intended to, prove real-wire delivery.
- **Task 5.3b**: Added `case Data_data_jobArray:` to `extract_data()` (`request.c`) — sets `dataset->rows`/`nrows=1` directly (no pagination, matching the always-one-shot wire shape) and nulls `getdata->data.job_array` after transferring it. The null is NOT cosmetic — independently verified `free_Request()`'s `Request_body_pushmsg` case calls `free_Data()`, which has a live `Data_data_jobArray` branch that would double-free/UAF without it. Proven end-to-end with a real `xce_encode()`/`xce_decode()` round-trip through `extract_data()` → `sched_handle_dataset()` → `sched_on_job_array()` (task 5.3, unmodified), not just a unit-level check. Also independently confirmed the alternate hypothesis some earlier notes floated — that `scheduler_req_handler()` itself has a `pushmsg`-rejecting branch — is dead code on the live wire path, since `vtask_entry()` always converts `body_which` to `datamsg` before invoking any handler. **New residual gap, self-flagged, NOT fixed (pre-existing since task 5.3, not introduced here, genuinely out of this task's `request.c`-only boundary)**: `dataset->rows` for the jobArray case is never deep-freed downstream — `free_Dataset()` is an empty no-op for all table types, and `sched_handle_dataset()`'s jobArray case (task 5.3) doesn't null it after consuming it. Net effect: one bounded `xce_job_submit_t` + embedded spec leaks per successfully-delivered array push commit (not per element, not unbounded) — real but small; worth a dedicated follow-up before this spec is considered fully closed, alongside the other self-flagged residual gaps (task 3.1's bulk-insert orphan prefix, task 4.2's boot-replay fence gap).
- **Task 5.4**: Go/CGO receive path — `rawJobArrayToModels()` (pure-Go N-way expansion), `cJobArrayToModels()` (CGO adapter sibling to `cJobToModel`), `dispatchJobArray()` (dedup via the SAME `store.Jobs.Has()` the ordinary `Data_data_jobs` path writes through, one confirmation log only on ≥1 newly-dispatched element), new `case C.Data_data_jobArray:` in `handleDataset`. Elements route through `OnJobUpdated` (not `OnJobCreated`), matching the ordinary single-job push's own routing — verified consistent, not a divergence. CGO boundary needed zero changes — `C.xce_job_submit_t` and `C.Data_data_jobArray` were already exposed/generated from prior tasks. A structural TOCTOU exists between the `Has()` dedup check and the eventual store write (two separate lock acquisitions, no compare-and-set primitive) but is narrower than task 5.3's accepted C-side gap (no intervening async hop) and strictly safer than the pre-existing baseline (the ordinary `Data_data_jobs` path has NO dedup check at all today) — not fixed, judged acceptable. `cJobArrayToModels()` has zero direct unit coverage at the CGO-struct level, but this exactly mirrors the pre-existing, repo-wide precedent for `cJobToModel()` itself (also never unit-tested with a real C struct) — reviewer independently reconstructed a full `CGO_ENABLED=1` build+link+test run against the real generated headers to confirm correctness beyond a compile-only check. `store.ArrayJobs` (array-level rollup aggregate) deliberately left unpopulated — confirmed genuinely out of this spec's scope per requirements.md's Boundary Context, which explicitly defers whole-array bulk query/cancel to a future spec; zero code anywhere in the codebase populates it today, including this diff.
- **Task 5.5**: Cross-daemon integration test, test-only (zero net production diff, confirmed on `request.c` and every task 5.1-5.4 file). Drives real `xce_encode()`/`xce_decode()` wire bytes through the REAL production dispatch chain on both daemons — `extract_data()` → `scheduler_req_handler()` → `sched_handle_dataset()` → `sched_on_job_array()` (C, via `#include "../src/handler.c"` in the test) and `go_push_handler()` → `handleDataset()` → `cJobArrayToModels()`/`dispatchJobArray()` (Go/CGO) — skipping only `vtask_entry()`'s ~10 lines of `static` glue, which the test mirrors verbatim (independently verified line-by-line accurate on the one behavior that mattered: `if (callnext)` treating any nonzero `extract_data()` return, including a negative errno, as "proceed"). Redelivery is tested via genuinely independent fresh decodes of separately-encoded bytes, not a cheaper decode-once-call-twice shortcut. Bug-catching capability independently proven twice (implementer, then reviewer): temporarily reverting task 5.3b's fix makes both the C and Go tests fail meaningfully; restoring returns both to green. Go/cgo toolchain constraint discovered: `import "C"` is disallowed in any `_test.go` file, forcing a two-file split (`wire_dispatch_support.go` for cgo-preamble helpers, `wire_dispatch_test.go` for the actual tests) — logged as bug-1440/1441/1442. One low-severity, non-blocking nit (not fixed): the C test file's own header comment misdescribes which assertion fails first under the RED-phase revert (claims the `callnext != 0` check; actually a downstream `>= 0` check) — cosmetic only, the test's real bug-catching property is independently confirmed accurate.
- **Task 6.0a**: Fixed `replay_wal()`'s job-count tally (persist.c ~3902) to add a `WAL_JOB_ARRAY_SUBMIT` branch that tallies `apply_rc` (the genuine per-record element count `apply_committed_entry()`'s array case already returns) when `apply_rc > 0`. Independently traced the FULL return-value space of `restore_job_array()`/`apply_committed_entry()`'s array case (6 distinct cases: 3 negative pre-mutation failures, `-EEXIST`→`apply_rc=0` idempotent no-op, the task-3.1 bulk-insert partial-failure case (stays negative, correctly still NOT tallied — a pre-existing, out-of-boundary undercount that still surfaces as a detectable anomaly rather than masking one), and positive `created` on genuine success) and confirmed every positive `apply_rc` this fix tallies is PROVABLY equal to real jobmgr growth for that record — the fix cannot overcount, and therefore cannot weaken `persist_verify_apply_correctness()`'s ability to catch a genuine divergence. Verified end-to-end via a from-scratch live 3-node HA soak re-run (`--array-soak`, both by the implementer and independently by the reviewer): zero `APPLY_DIVERGENT` across all three of the original bug's trigger paths (restart-mid-small-array, standby election-path recovery fallback, restart-mid-N=10000-array), one-WAL-record/one-commit-barrier confirmed for both N=2 and N=10000, per-element post-failover status/cancel/completion confirmed. This soak run also directly satisfies task 6.1's own Observable — its results should be reused rather than re-run from scratch. **New non-blocking finding (pre-existing, NOT introduced by this task)**: `controller_unit_test`'s single outer `UNITY_BEGIN()` is silently reset by ~19 of its called test-group functions' OWN nested `UNITY_BEGIN()/UNITY_END()` calls, making the process exit code / meson `Ok:/Fail:` summary line unreliable for detecting a regression in any group running before the last such nested call — both the implementer and reviewer worked around this by grepping raw `testlog.txt` for `:FAIL:` lines directly rather than trusting the summary, and this should be the standard practice going forward for this binary until a dedicated follow-up fixes the harness itself.
- **Task 6.1**: Extended `.claude/skills/xce-test-ha/run-ha-soak.sh`/`soak-verdict.sh` with a new `--array-soak` mode (small N=2 + large N=10000 sub-rounds, one clean kill/failover per sub-round — deliberately single-kill, not chaotic, matching this task's correctness-not-chaos scope; chaos-under-array-submission is task 6.2's separate job). `waldump.c` gained a `WAL_JOB_ARRAY_SUBMIT` case in `wal_type_name()` so the scenario can grep an exact type name for its live before/after WAL-record-count delta check. **Round 1 REJECTED**: the replay-time measurement used whole-second (`date +%s`) resolution — both sub-rounds' timings collapsed to `0s`/`n/a` on a real run — and, more fundamentally, nothing ever asserted on the comparison (`verdict_emit_arraysoak()` always returned PASS regardless of measured values), leaving Requirement 4.3 ("replay time governed by record size, not element count") with zero actual evidence and unable to catch a genuine O(N) regression. **Round 2 fixed**: switched to `$EPOCHREALTIME` (millisecond resolution; discovered and worked around a genuine environment trap — this sandbox's BusyBox `date` precedes GNU coreutils on `$PATH` and silently passes `%N`/`%3N` through as a literal string suffix rather than expanding or erroring, which would have poisoned any `date +%s%3N`-based approach silently); added `verdict_check_arraysoak_replay_bound()` with a real `bound = max(t_small*ratio_max, floor_ms) + margin_ms` assertion that `run_array_soak_round()` hard-`fail()`s on when exceeded — independently verified to have real teeth (both the implementer's and reviewer's own hand-picked regression cases correctly trip `BOUND_EXCEEDED`). Live-reproduced 3 times total (once round 1, twice round 2) with genuinely discriminating millisecond numbers each time (final: small=210ms, large=163ms — large array replay was NOT slower, live evidence of the feature's O(1) claim). Also fixed opportunistically: status/cancel/completion checks spread across low/mid/high indices (previously clustered at 1,2,3) matching the already-good basic-addressability spread; selftest extended 17→28 cases covering the new bound-check logic; `SKILL.md` documented. "One commit barrier" remains an inference from the WAL-record-count delta (not independently observed live) — defensible given task 2.1/2.3's unit-tested 1:1 append↔continuation coupling, but flagged as an inference, not a direct check, should a future task want to close that gap.
- **Task 6.2**: Extended the same harness with `--array-chaos-retry` (`submit_array_bg()`, `run_array_chaos_retry_round()`, `verdict_emit_arrayretry()`). Traced the full cross-primary dedup chain end-to-end (bug-1419's mechanism): client-side `idgen()` mints the base jobid once, `xcecli_jobsubmit()`'s `XCECLI_FAILOVER_CALL` retry loop resends the IDENTICAL submit (same jobid, never re-minted); server-side `handle_jobsubmit()` checks `jobid_range_exists()`/`inflight_range_try_add()` — both genuinely RANGE-shaped over `[base,base+array_size)`, run BEFORE the array-append branch, so the whole array is dedup-checked as one unit, not per-element; `xce_failover_classify()`'s "idempotent-replay rule" reclassifies a resulting `-EEXIST` as success on retry. Reviewer went further and confirmed a subtle apply-lag race (freshly-promoted primary's process-local in-flight registry starting empty) is closed by a PRE-EXISTING, unrelated mechanism (`raft_cb_become_leader`'s synchronous apply-pump drain before the promotion callback returns, added 2026-07-10 for an unrelated bug) — not something this task needed to build, just confirmed already-sound. No production code touched or needed — the existing single-job dedup mechanism generalizes correctly to arrays with zero changes. No mutation-test RED phase was performed against production code (correctly declined per this task's explicit constraint); reviewer compensated by extracting the harness's own assertion logic into a standalone mutation-harness with stubbed collaborators and confirming all four scenarios (clean/silent-loss/double-submit/partial-duplication) correctly trigger or don't trigger `fail()` as expected. Live-verified across 4 independent fresh cluster deployments total (2 by implementer, 2 by reviewer) spanning array sizes 6, 10, and 15, 17 total chaotic kill-during-submit rounds, zero double-submits, zero silent losses, zero partial duplications, genuine election/term diversity confirmed each time. `walcheck_count`'s divergence check is informational-only here, matching task 6.1's own established convention exactly (not a new decision) — judged acceptable since `wait_for_full_convergence` already requires matching `committedLsn` across all 3 nodes before this scenario's checks run.
- **Task 6.3**: Pure verification task, zero file changes. Ran the complete suite: `controller_unit_test` (299 tests, 0 failures, verified via raw-log `:FAIL:` grep, not the exit code/summary — per task 6.0a's nested-`UNITY_BEGIN`/`END` counter-reset finding), all scheduler binaries, `xcelib_test` (0 failures with the registered `detect_leaks=0` config; the one pre-existing task-1.3 rtti leak still reproduces under leak-detection-on, unchanged), Go apiserver tests under both `CGO_ENABLED=0` and `=1`, and every soak mode this spec added (`--selftest` 28/28, sequential baseline, `--chaotic`, `--array-soak`, `--array-chaos-retry`) — all clean. Designed and executed a genuine empirical byte-identical baseline comparison for the non-array path (Requirement 8.1): built commit `895a204` (immediately pre-dating this spec's first commit) in an isolated, cleanly-removed `git worktree add --detach`, deployed independent clusters from both the baseline and current-tree binaries, submitted one identical single-job request to each, and did a byte-for-byte `cmp -l` of the raw on-disk WAL entries. Independently reproduced end-to-end by the reviewer (separate build, separate clusters, separate comparison) with a consistent result: every differing byte mapped cleanly to `timestamp`/`crc`/`prev_crc` (the WAL header's inherently run-specific fields) or the payload's `jobid`/`sub_time`/`queue_time` fields — zero differences anywhere in `term`/`lsn`/`type`/`payload_len` or the remaining ~6485 payload bytes. This was further corroborated by a structural code-diff argument (the entire single-job WAL-encode path is untouched between `895a204` and HEAD, purely-additive diff only) and by the pre-existing task 2.3 unit test asserting the same invariant. Confirmed the three self-flagged residual gaps (task 3.1 bulk-insert orphan prefix, task 4.2 boot-replay fence, task 5.3b bounded leak) are the complete and only set of open items across all 24 completed tasks — no new regression, job loss, divergence, or liveness stall found anywhere. **Overall verdict: GO.**
- **`/kiro-validate-impl` feature-level gate (2026-07-16)**: Ran 4 parallel validation dimensions. Requirements coverage: 38 ACs, 33 FULL, 5 PARTIAL (2.3, 8.3, 9.4, 9.5, 9.6 — all tracing to task 3.1's already-known bulk-insert partial-visibility gap plus 8.3's admitted commit-barrier-count inference), 0 outright unclaimed gaps. Design/boundary alignment: ALIGNED, no drift, no boundary violations, all 5 Out-of-Boundary items confirmed untouched, task 5.3b/6.0a's mid-run insertions confirmed surgically scoped to their own narrow fixes. Cross-task integration: SEAMS_SOUND — wire format, `base_jobid + i` arithmetic, `apply_rc` contract, and fence naming all independently verified consistent across every producer/consumer in the chain (5 wire consumers, 3 independent arithmetic implementations in C controller/C scheduler/Go apiserver, 3 tasks sharing the `apply_rc` contract). Mechanical checks: full suite clean, TBD/secrets grep clean, and a genuine live smoke boot (fresh single-node deploy, real `jobsub -a 5` via `xce_test`, exactly one `JOB_ARRAY_SUBMIT` WAL record, all 5 elements individually queryable) — PASS.
  The requirements-coverage pass sharpened task 3.1's known gap: it is a literal, code-proven violation of Requirement 2.3 and 9.4 (not just a soft nuance) — a bulk-insert-phase resource-exhaustion failure can permanently orphan array elements that exist and are individually addressable but never reach `HOOK_DETAIL_JOB_NEW`, so they are never scheduled or pushed, not on this attempt or any later successful retry of the same LSN. User decision on final GO/NO-GO: **GO, with the gap filed as a tracked bug (`bug-1443`, `.wolf/buglog.json`)** rather than blocking the spec on a fast-follow fix or holding NO-GO — the trigger condition is narrow (a resource-exhaustion failure at the exact moment of the bulk-insert pass) and every other validation dimension came back clean. Closing `bug-1443` requires either a new `key_del`/capacity-reservation primitive in `kv.c`/`kvmgr.c` (shared library, its own small design effort) or a re-derive-and-renotify recovery path — explicitly out of this spec's approved task list, not a quick follow-up task.
- **Task 7.1**: Added `kvmgr_reserve()`/`xzarray_reserve()` (new, purely-additive primitives in shared `kvmgr.c`/`zarray.c`, following the existing `kvmgr_capture_keys()` template — bare extern function, no RTTI vtable slot, "caller must hold write lock" contract), wired a reservation call into `restore_job_array()`'s bulk-insert phase, and made a genuinely-inserted partial prefix always get notified (`out_new_jobs` populated + `apply_committed_entry()` fires `HOOK_DETAIL_JOB_NEW`) even when the function still fails overall. **Genuine off-by-one found in the existing, unmodified `xzarray_ensure_items()`**: its sufficiency check is strict less-than (`mincap < array->caps`), so reserving exactly `array_size` and then performing `array_size` real stores still triggers one more growth cycle on the LAST store (`mincap == caps` fails the strict check). Independently re-derived and confirmed by the reviewer with concrete arithmetic: reserving `array_size + 1` (not literally `array_size` as this task's text above says) is exactly the minimal correct compensation — not over- or under-provisioning. This is a deliberate, verified deviation from this task's literal wording, documented at the call site. Confirmed the fix narrows (does not fully eliminate) the residual gap as designed: `kvtab_add()`'s own small per-entry hash-chain-node allocation and a content-based `-EEXIST` collision remain theoretically possible even after reservation — both non-N-scaling and far rarer than the closed failure mode, honestly documented as such. Retry-double-notify risk (does a retry's own `out_new_jobs` ever re-include an already-notified prefix) independently traced and confirmed NOT present — the pre-existing per-index idempotent-skip logic excludes already-inserted indices from any subsequent attempt's `built` list. `apply_wal_entry()`'s dead-code standby path confirmed genuinely untouched (zero behavior change, still passes `NULL` for `out_new_jobs`). New build-architecture finding: `libxce.a` is compiled exactly once without `-DXCE_TESTING` and shared by every consumer, so the new test-support definitions in `zarray.c` had to be left unconditionally compiled (only their declarations in `zarray.h` stay `#ifdef XCE_TESTING`-gated) — confirmed zero production risk (force-fail flag defaults false/inert, no production translation unit can reach the ungated symbols without the gated declaration). One non-blocking coverage gap: no single test composes the full engage→notify-prefix→clear-seam→retry→no-double-notify flow end-to-end (the property is provably true by combining two already-passing tests, but not directly demonstrated in one).
- **Task 7.2**: Restructured `replay_wal()`'s per-entry loop into a three-way branch (`apply_halted`/`apply_rc<0`/success) mirroring `persist_apply_pump_run()`'s (task 4.1) halt-and-fence behavior — on a negative `apply_committed_entry()` return during boot/restart replay, logical apply (tally, `last_applied_lsn` advance, `replayed++`) halts at that LSN and `persist_mark_apply_resource_fenced()` engages (reused verbatim, no new mechanism), while PHYSICAL scan bookkeeping (`current_lsn`, CRC chain) deliberately continues unconditionally for every entry — verified safe against `persist_recover()`'s own pre-existing "Round-2 fix" precedent, which already establishes `current_lsn` and `last_applied_lsn` are expected to diverge. This closes the boot-replay half of task 4.2's self-flagged gap: a node that crashes while fenced and re-hits the identical condition on restart now correctly re-engages the fence during replay instead of silently booting clean.
  **Went through one remediation round.** Round 1 was REJECTED not on the `persist.c` fix's correctness (verified sound both rounds) but because task 7.2's own explicit deliverable — "verify and document" that the fence engages before any promotion decision — was left unmet, and the new test exercised the wrong `ha.c` gate (the periodic re-alert timer, not the actual election-time check). Round 2 fixed both: the test now drives the real election-time gate (`raft_cb_become_leader_with_cluster_for_test()`, the same seam task 4.3's own test uses for a live-pump-engaged fence — this is the first test to drive it for a fence engaged by boot replay specifically), with teeth independently reproduced twice (temporarily clearing the fence before the gate call reliably flips the test RED, both in isolation and in the full suite).
  **CONFIRMED, NOT FIXED, real first-boot promotion-ordering gap (independently verified twice — implementer, then two separate reviewers)**: `raft_cb_become_leader()`'s election-time eligibility gate (the `if (c != NULL) {...}` block, `ha.c:1303-1408`, reading `persist_apply_resource_fenced()` at `ha.c:1337`) runs strictly BEFORE the `is_first_boot` block (`ha.c:1427-1439`) that calls `cluster_init_phase2()` → `persist_recover()` → `replay_wal()` (`cluster.c:1482`) — the function this task modified. There is no re-check of `persist_apply_resource_fenced()` between `cluster_init_phase2()` returning and the unconditional `CTRL_PRIMARY` role assignment a few lines later. Net effect: this task's fix protects every promotion evaluation AFTER a node's first one in a given process's life (a later election, or the periodic re-alert timer) — but a node's very FIRST-EVER promotion evaluation, on its first election win in a freshly-started process, is decided before its own first `persist_recover()`/`replay_wal()` call has even run, so a resource-exhaustion condition that same first replay is about to hit is invisible to that one gate check. This is exactly the "all nodes restart simultaneously" race this whole task exists to guard against, and it is not fully closed by task 7.2 alone. Trigger requires BOTH (a) a node winning its very first election in a fresh process AND (b) that same node's first-ever boot replay hitting the identical resource-exhaustion condition that previously fenced it — narrow but real. Documented via a code comment in `persist.c` (citing these exact file:line references) plus this entry; fixing it requires an `ha.c` change (out of this task's `persist.c`-only boundary) — either re-checking `persist_apply_resource_fenced()` immediately after `cluster_init_phase2()` returns inside `raft_cb_become_leader()`, or reordering recovery to run and be checked before the promotion decision finalizes. Recommend a small dedicated follow-up task against `ha.c` before anything downstream places full trust in this fence across a first-ever-boot promotion race specifically (restarts/re-elections after that first one are already safe).