raft_cb_become_leader()'s eligibility gate read
persist_apply_resource_fenced() before this process's own first
cluster_init_phase2()/persist_recover()/replay_wal() call ever ran --
so a node whose first-ever replay hit a resource-exhaustion apply
failure could still become CTRL_PRIMARY, because the fence that
replay would engage didn't exist yet when the gate checked it. Every
LATER promotion evaluation was already correctly protected; this was
a narrow gap specific to a process's single very-first election.
Fix: re-check persist_apply_resource_fenced() immediately after
cluster_init_phase2() completes on first boot, refusing promotion
(same WARN + ha_arm_promotion_fence_timer() + return pattern as the
existing refusal branch) if that first replay just engaged the fence.
Confirmed ha_elect_evaluate_and_signal (the other election path) is
dead in production for the promotion outcome -- every live call site
of handle_request_vote_reply passes vote_granted=0 since bug-828
redirected real grant recording to the record-only, non-signaling
ha_record_vote_reply -- so raft_cb_become_leader is the sole live
promotion path and this fix's scope is complete.
TDD: new test test_first_boot_promotion_evaluates_own_first_replay_fence
wires ha_phase2_hook to actually call persist_recover(), reproducing
production ordering exactly. RED-confirmed against the unfixed code
at the precise predicted assertion; GREEN-confirmed after the fix.
Full controller_unit_test regression (bundles every HA-adjacent
testsuite/test_*.c) green, 0 new failures.
Originally documented (not fixed) in job-array-single-wal-record
task 7.2's Implementation Notes; logged as bug-1445 and fixed here,
outside that spec's own task list per its persist.c-only boundary.
Root cause and reproduction path were already documented inline in
tasks.md task 7.2's Implementation Notes but never had a standalone
buglog entry. Cross-referenced from spec.json's known_limitations.
Not fixed -- suggested shape captured in the bug entry: a small,
single-call-site ha.c ordering fix (move the first promotion-gate
evaluation to run after the first replay_wal(), or have that one gate
call treat "not yet replayed" as non-promotable), not a new Kiro spec.
spec.json: phase -> implementation-complete, new GO decision (2026-07-17)
covering the full feature plus phase 7's residual-gap closure; prior
decision preserved as prior_decision; bug-1443 known_limitations entry
retired (closed by task 7.1) and replaced with the one limitation that
is still genuinely open (task 7.2's first-boot promotion-ordering gap).
tasks.md: task 3.1's implementation note updated from "open, unowned
gap" to "closed by task 7.1" -- it predated that fix and had gone
stale.
Validation evidence: full C suite 9/9 green (incl. libxce test post
rtti-fix), Go suite green (88.1% backend coverage), clean TBD/secrets
grep, live array-soak smoke PASS (O(1) replay time confirmed at
2-element vs 10,000-element scale), cross-task integration verified at
every seam phase 7 touched, 10/10 requirement groups covered, zero
design drift, zero blocked tasks.
User directed (2026-07-16) that this feature's own open residual gaps
be fixed: bug-1443 (restore_job_array()'s bulk-insert partial-visibility
gap), task 4.2's boot-replay fence gap, and task 5.3b's bounded leak.
Unrelated pre-existing bugs 1437/1438 explicitly excluded from this
reopening per user's scope decision.
Adds Phase 7 (tasks 7.1-7.3) to tasks.md:
- 7.1: add a purely-additive kvmgr_reserve() capacity-reservation
primitive (mirroring the existing kvmgr_capture_keys() template) so
restore_job_array()'s bulk-insert pass can pre-grow its backing store
before mutating, closing the dominant N-scaling failure trigger; plus
make a genuinely-inserted partial prefix always get notified even in
the now-much-narrower residual failure case
- 7.2: extend the Live Apply Resource Fence's halt-and-fence protection
to replay_wal() (boot/restart replay), reusing the same primitives
task 4.1/4.2 already built for the live path
- 7.3: free dataset->rows in sched_handle_dataset()'s jobArray case
spec.json phase reverted to tasks-generated to reflect the reopening.
Feature-level validation ran 4 parallel dimensions: requirements
coverage (38 ACs, 33 full, 5 partial -- all tracing to the already-known
bulk-insert partial-visibility gap plus an admitted commit-barrier-count
inference), design/boundary alignment (ALIGNED, no drift), cross-task
integration (SEAMS_SOUND, wire format/arithmetic/contracts verified
consistent end to end), and mechanical checks including a live smoke
boot (fresh single-node deploy, real array submission, one WAL record,
all elements queryable) -- PASS.
The coverage pass sharpened restore_job_array()'s known bulk-insert
gap: it's a literal, code-proven violation of Requirements 2.3 and 9.4,
not just a soft nuance -- a resource-exhaustion failure during the
bulk-insert pass can permanently orphan array elements that exist and
are addressable but never get scheduled. Filed as bug-1443
(.wolf/buglog.json) for tracking. User decision: GO, ship with the
gap tracked rather than blocking on a fast-follow fix -- the trigger
condition is narrow and every other validation dimension is clean.
Closing bug-1443 needs a new kv.c/kvmgr.c primitive or a re-derive-
and-renotify path, explicitly out of this spec's approved scope.
spec.json phase -> implementation-complete.
