myxce/.kiro/specs/ha-consistency-invariants/research.md

Gap Analysis: ha-consistency-invariants

Date: 2026-06-10. Sources: discovery audit (brief.md), code-explorer verification pass (persist.c, repl.c, ha.c, job.c, cluster.c, config_repl.c, ha-wal-replication tasks.md).

1. Current State Investigation

Existing assets directly reusable

Asset	Location	Relevance
Per-standby acked LSN (≈ Raft matchIndex)	standby_conn_t.acked_lsn (repl.h), updated by repl_on_ack (repl.c:187)	Already exists — the bookkeeping needed for honest per-entry commit verification is in place, just unused by the wait logic
Per-standby join/applied LSN (≈ nextIndex)	standby_conn_t.join_applied_lsn	Catch-up positioning
Sync commit barrier (job submit only)	persist_wal_sync_append + repl_wait_majority_ack (persist.c:1003-1049, repl.c:386-461)	The pattern to generalize; pthread condvar wait (ack_cond) signaled by repl_on_ack
Full fan-out broadcast	repl_broadcast (repl.c:210-282)	Req 6.1 already satisfied
WAL streaming catch-up + snapshot fallback	ha_stream_wal_to_standby (config_repl.c:1450+), snapshot transfer	Req 6.3 satisfied; drive mechanism needs change for 6.2
Hook-based persistence dispatch	persist_job_hook_fun / persist_queue_hook_fun / persist_worker_hook_fun (persist.c:1107/1199/1241) on HOOK_AFTER_EVENT	The single seam through which all state-changing writes flow
Quorum gate at dispatch	ha_has_write_quorum + ha_dispatch_gate (ha.c:1507-1612)	Req 7.1 partially satisfied (gate-time only)
ha-wal-replication ACK machinery	tasks 1.1–5.5 complete; only 5.4 (live 3-node integration test) pending	Upstream dependency effectively satisfied for design

Verified behaviors (with evidence)

1. Apply-before-commit is universal — including job submit. job_submit() (job.c:518) inserts the job into the in-memory table, THEN xce_run_hooks(HOOK_AFTER_EVENT) (job.c:553) fires the WAL write + majority wait. For state transitions, mark_job_start/finish/... mutate job->job.state between HOOK_BEFORE_EVENT and HOOK_AFTER_EVENT (e.g., job.c:585-604). Queue and worker state likewise mutate before their hooks fire. On every path, in-memory state precedes the log entry; what differs is only whether the reply waits (job submit: yes; everything else: no). Req 2.4 (apply only after commit) is violated everywhere today.
2. Majority-ACK verification is not per-entry. repl_wait_majority_ack (repl.c:408-443) counts standbys with acked_lsn > 0 — i.e., "ACKed anything, ever" — not acked_lsn >= this_entry_lsn. Once a standby ACKs once, it counts toward every subsequent quorum until disconnect. Concurrent sync appends share one global ack_cond and would cross-satisfy each other. Req 4.2 needs per-LSN comparison; the acked_lsn field already supports it.
3. Threading is permissive but finite. (Corrected 2026-06-11; original pass understated the pool and overstated the risk.) The sync wait is pthread_cond_timedwait on a vtask pool thread — default 8 threads, --vcpus CLI override (cluster.c:1082). Job submit already blocks one. Generalizing to all write types multiplies concurrent blocked workers; 8 simultaneous slow commits would stall the vtask pool. This is a throughput risk, not a liveness risk: WAL ACKs are decoded and waiters signaled on the iomgr thread (peer_ctrl_msg_handler → repl_on_ack, ha.c:2770-2792), so a fully blocked pool still receives its wakeups — requests queue until a commit completes or times out, but no deadlock is possible. Steering allows vtask handlers to block briefly ("return quickly"; the iomgr never-block rule doesn't apply). Mitigation inputs in Research Needed item 3 (commit-wait admission cap; async continuation as the durable fix).
4. Job state transitions are reported facts, not requests. on_job_start/finish/fail arrive via worker notifications (do_notify, cluster.c:118-156) with no client awaiting a reply. "Rejecting" a job-finish on commit timeout cannot un-finish the job on the worker; the entry must instead be held/retried until commit. The rejection semantics in Req 2.2 map cleanly to request-shaped operations (submit, queue ctrl, config) but need a hold-and-retry interpretation for notification-shaped facts.
5. COMMITTING removal is concretely mapped. raft_cb_become_leader (ha.c:505) sets COMMITTING; removal means: set CTRL_PRIMARY directly, move hook registration from ha_promote_to_primary (ha.c:1691-1698) into the become-leader path, delete the committing watchdog (ha.c:593-616, 3082-3125), make ha_check_majority_convergence (ha.c:1313) and repl_ack_count/ha_repl_ack_received obsolete, adjust the COMMITTING guard in the FSM path (ha.c:2967) and the dispatch-gate COMMITTING branch (ha.c:1594-1612).
6. Catch-up is heartbeat-polled. ha_stream_wal_to_standby sends all batches [standby_lsn+1, committed_lsn] in one call without per-batch ACK waits, but the NEXT round is only triggered by the next heartbeat reply (3s cadence) carrying an updated applied_lsn (ha.c:2734 → 1467). Req 6.2 needs ACK-driven (or WAL-append-driven) continuation.

2. Requirement-to-Asset Map

Req	Existing asset	Gap	Tag
1.1-1.2 (direct PRIMARY, single role)	become_leader callback	Remove COMMITTING state + watchdog; move hook registration	Missing (mapped)
1.3 (no redirect from primary)	ha_should_redirect (ha.c:1655)	Returns redirect in COMMITTING; resolved by state removal	Missing
1.5 / 3.3 (no timeout bypassing majority)	—	committing_timeout_fn force-promote must be deleted	Missing
2.1 (majority barrier all ops)	sync path for WAL_JOB_SUBMIT	Generalize to JOB_* transitions, QUEUE_CTRL, WORKER_CTRL (persist.c:1169/1199/1241)	Missing
2.2 (reject on timeout)	job submit error propagation (job.c:563-570)	Worker notify is pure fire-and-forget (no reply read, no retry, no rejoin resync) — controller MUST own durability via a pending-facts queue; rejection = permanent loss (already true today). See deep-dive §A	Missing (resolved — constraint identified)
2.4 (provisional state, commit-gated egress)	Memory already rebuilt from checkpoint+WAL on restart (persist_recover); role-agnostic replay fns	DECIDED (2026-06-10): provisional-state model — mutation order unchanged; gate requester confirmation + irrevocable-action egress (dataex/scheduler export filters provisional entities by commit status); makes §C reconciliation mandatory core. See deep-dive §B	Missing (decided — egress gating + tagging)
2.5 (single outcome for timed-out entries)	WAL truncation on promotion (ha.c:471) + on standby RDHUP (ha.c:3419); safe_committed_lsn majority-median derivation (ha.c:373-432)	FIVE convergence gaps: no truncation on demotion, no in-memory phantom reconciliation, snapshot is LSN-sentinel only (no entity state), committed_lsn not persisted (restart replays uncommitted suffix), no conflict detection in apply. See deep-dive §C	Missing (resolved — 5 gaps enumerated)
3.1-3.2 (own-term commit)	—	No own-term tracking on promotion; no-op entry vs first-write decision	Missing
4.1 (count only reachable)	ha_has_write_quorum (ha.c:1507)	Excludes neither catch-up nor per-write reachability	Missing
4.2-4.4 (ACK-time per-entry verification)	acked_lsn per standby	Wait logic compares > 0 not >= entry_lsn; no per-entry wait slots	Missing
5.1-5.3 (config barrier)	push_config_to_standbys + repl_wait_majority_ack variant (config_repl.c:916-928)	Return code ignored by caller (config_repl.c:1057); error propagation	Missing (small)
6.1 (full fan-out)	repl_broadcast	None	Done
6.2 (leader-driven catch-up)	ha_stream_wal_to_standby	Continuation is heartbeat-polled, not ACK-driven	Missing
6.3-6.4 (snapshot fallback, re-include)	snapshot transfer, repl_complete_catchup	None	Done
6.5 (lag doesn't block confirmation)	majority threshold	Preserved by design	Done
7.1 (reject on quorum loss)	dispatch gate	Gate-time only; ACK-time enforcement closes the TOCTOU	Partial
7.2 (log quorum transitions)	—	Add observability logging	Missing (small)
7.4 (auto-resume)	gate re-evaluates per request	None expected	Done (verify)
7.5 (standalone self-quorum)	n_peers <= 1 paths	Preserve unchanged	Constraint

3. Implementation Approach Options

Option A: Extend in place

Generalize persist_wal_sync_append to all WAL types at the existing hook sites; fix repl_wait_majority_ack to per-LSN comparison; delete COMMITTING inline.

• ✅ Smallest diff; all infrastructure exists; follows proven job-submit pattern.
• ❌ Blocking-condvar-per-write multiplies vtask pool throughput-starvation risk (no deadlock — ACKs arrive via iomgr; see §1 item 3); concurrent sync writes still share one condvar (broadcast wakeups checking per-LSN predicates works but is crude); apply-after-commit ordering not addressed (would be documented deviation).

Option B: New commit-barrier module

New commit.c owning Raft commitIndex logic: per-entry waiter registry (LSN → waiter list), matchIndex aggregation (acked_lsn scan on each ACK → advance committed_lsn → wake waiters at or below), explicit pending-entry queue for notification-shaped facts, hold/retry. Hooks call into it; repl.c feeds it ACKs.

• ✅ Clean Raft semantics (commitIndex = median of matchIndex); supports many in-flight entries; single place to test invariants; natural home for own-term tracking and apply-after-commit sequencing.
• ❌ Larger surface; new module must integrate with persist lock ordering and the fsync thread; more migration risk in one step.

Option C: Hybrid, phased (barrier-first ordering preserved)

• Phase 1 (repl.c surgical): per-LSN wait predicate (acked_lsn >= entry_lsn count ≥ majority), remove n_active == 0 short-circuit for n_peers > 1, ACK-time quorum verification. Job submit immediately gets honest commits.
• Phase 2 (persist.c + small commit registry): generalize the barrier to QUEUE_CTRL, WORKER_CTRL, and request-shaped job ops; introduce a minimal pending-entry registry only for notification-shaped facts (hold/retry until committed); config push error propagation.
• Phase 3 (ha.c): own-term commit gate on promotion, then delete COMMITTING (+ watchdog, convergence checker, redirect fix).
• Phase 4 (config_repl.c): ACK-driven catch-up continuation; quorum-transition logging.
• ✅ Each phase independently testable and shippable; respects the brief's hard ordering (barrier before state removal); risk contained.
• ❌ Per-entry waiter logic lands in two steps; requires discipline to not stop halfway.

4. Effort & Risk

Workstream	Effort	Risk	Justification
Per-LSN ACK verification + TOCTOU closure	S-M	Low	Fields exist; localized to repl.c wait/ack path
Barrier generalization (request-shaped ops)	M	Medium	Proven pattern, but vtask pool throughput stall needs mitigation (commit-wait admission cap or async continuation — see Research Needed item 3)
Notification-shaped facts (pending-facts queue)	M	Medium	Resolved: worker provides NO redelivery (verified), so a primary-side pending queue is the required mechanism; semantics now clear, implementation moderate
Provisional-state egress gating (replaces inversion)	M	Medium	DECIDED: mutation order unchanged; tag entities with WAL LSN, filter dataex/scheduler export to committed-only (closes dispatch-of-uncommitted race); smaller than inversion but couples to §C reconciliation being real
Demotion/restart convergence (Req 2.5 gaps)	M-L	High	NEW workstream: truncate-on-demotion, persist or reconcile committed_lsn at recovery, phantom in-memory state eviction (no teardown path exists; snapshot carries no entity state)
COMMITTING removal + own-term gate	S-M	Medium	Concretely mapped (6 sites); must land after barrier
Config push error propagation	S	Low	Two call sites
Leader-driven catch-up	M	Medium	Continuation trigger redesign in streaming path

Overall: L–XL (2+ weeks across phases), Medium-High risk — dominated by the two High-risk research items.

5. Recommendations for Design Phase

Preferred approach: Option C (phased hybrid), with Phase 2's pending-entry registry designed Raft-shaped from the start (commitIndex from matchIndex aggregation) so it can absorb Phase 1's logic rather than duplicate it.

Research Needed (carry into design):

1. Apply-after-commit strategy DECIDED (user, 2026-06-10): provisional-state model — in-memory state is a provisional view, persisted majority-committed log is authoritative. Mutation order unchanged; gate confirmation + irrevocable-action egress on commit. Remaining design details: entity commit-tagging mechanism (per-entity WAL LSN vs per-type watermark), which egress points are "irrevocable" (dataex/scheduler export: yes; status queries: no, documented deviation), and the egress filter's interaction with array jobs sharing one submit entry.
   • Mechanism direction (discussed 2026-06-10, STM evaluation — REFINED same day): full STM rejected — transaction duration mismatch (commit point is a majority network ACK; STM transactions can't span syscalls), C ecosystem dead (libitm unmaintained, TSX disabled), single-writer + per-object RTTI locks already cover the concurrency. An ARIES-style undo log was briefly considered, then dropped after the recovery-oriented reframing (user): this is a failover/recovery problem, not a transaction problem. Structural fact: a single-writer leader never truncates its own log while leader (Raft leaders only append), so a provisional entry on a live primary either commits later or the node is demoted — discard events coincide exactly with role/lifecycle events (demotion, restart, rejoin), never with normal operation. Therefore: no per-entry undo; discard is handled by recovery mechanisms — truncate divergent suffix + coarse memory rebuild via the existing persist_recover/replay_wal path at demotion/rejoin (rare, node not serving writes, cost acceptable). The only piece retained from the versioning discussion is the watermark egress filter (entity WAL-LSN tag, export only lsn <= committed_lsn to dataex/scheduler) — justified as failover correctness (prevent orphan dispatch of a job the post-failover cluster never heard of), not as transaction isolation.
   • xzarray shadow evaluated (2026-06-11): the dual-zarray COW (xzarray_freeze/xzarray_thaw, zarray.c:695-757 — built and unit-tested, zero production callers; kvmgr vfreeze NULL "wired in P0-2") is slot-level STRUCTURAL COW: slots hold pointers, so in-place field mutations (job state transitions) are visible through both views. Not suitable for the committed-view egress (reads are merged active-over-shadow, single generation would collide with the future checkpoint consumer, point-in-time vs advancing watermark) nor for provisional rollback (only covers inserts, not mutations; demotion-time eviction doesn't need lock-free semantics anyway). Its real home is the entity-bearing-snapshots backlog item (its original P0-2 purpose), which indirectly strengthens §C eviction option (iii). Watermark filter stands as the egress mechanism.
2. Notification-shaped facts DECIDED (user, 2026-06-11): log-is-the-queue, Raft-pure — fact entries are appended to the WAL immediately and replicated/retried until committed with NO commit timeout (Raft leaders never time out entries; retry across quorum-loss periods per Req 2.8). No separate side-queue structure; the "pending facts" are simply the uncommitted log suffix under continuous retry. Crash-loss of uncommitted facts is accepted until worker-rejoin-resync (roadmap backstop) lands — strictly better than today's behavior. Remaining design details: idempotency guards in mark_job_* (needed for replay safety) and retry/backoff cadence.
3. Wait mechanism under load: blocking condvar per vtask worker vs async continuation (queue reply closure, complete on commit). Measure vtask pool headroom; consider dedicated commit-wait sizing.
   • Verified inputs (2026-06-11): (a) vtask pool default is ALREADY 8 with --vcpus override (cluster.c:1082; the "default 4" in ha-async-io.md was stale — fixed). (b) No wakeup deadlock: WAL ACKs are processed on the iomgr thread (peer_ctrl_msg_handler → repl_ack_decode_and_dispatch → repl_on_ack, ha.c:2770-2792, 2748) — blocked vtask workers always get woken; all-workers-blocked costs throughput, not liveness. (c) Pool size should NOT scale with n_peers (user question, analyzed): blocked-thread count ≈ in-flight write ops × commit latency (Little's law); ACKs gather in parallel so peer count barely moves latency. Write concurrency and cores are the sizing inputs. (d) Recommended interim mitigation: commit-wait admission cap — at most n_vcpus - K concurrent blocking waits (reserve K lanes for non-blocking traffic), excess write ops fail fast with NOT_READY (CP-correct under overload). Async continuation remains the durable fix. (e) Sizing ownership (user, 2026-06-11): --vcpus is a tunable operator parameter — the right value is a deployment decision driven by host capability, cluster size, and workload; 8 is only the fallback default. Design constraint: all commit-wait correctness mechanisms must be expressed relative to n_vcpus (never absolute counts) so behavior is correct at any tuning; sizing guidance belongs in operator/tuning docs, not in spec-fixed constants.
4. Own-term commit vehicle DECIDED (user, 2026-06-11): explicit no-op WAL entry at promotion — bounded post-election write-unavailability independent of traffic; an idle cluster's new primary becomes writable as soon as the no-op commits.
5. Multi-entry in-flight semantics: confirm committed_lsn advancement is contiguous (no gaps) when multiple entries await ACKs.
6. Req 2.5 verification RESOLVED into a workstream (deep-dive §C): five convergence gaps enumerated. Design must choose the phantom-eviction strategy: (i) full in-memory rebuild on demotion (needs new teardown path for jobmgr/queues/workers, then replay_wal from checkpoint after truncation), (ii) process self-restart on demotion (crude, safe, CP-consistent), or (iii) entity-bearing snapshots (depends on the deferred "Task 11" snapshot format). Also: persist committed_lsn (or reconcile it from the cluster at recovery) so restart does not replay the uncommitted suffix.

Constraints to respect: standalone n_peers <= 1 self-quorum unchanged; iomgr thread never blocks (waits live on vtask/delaymgr threads only); single-writer model; ha-wal-replication task 5.4 (live integration test) still pending — its 3-node test should be extended to cover the new barrier rather than duplicated.

---

Design Synthesis Outcomes (2026-06-11, design phase)

• Generalization: Requirements 2.x (op barrier), 3.x (own-term gate), and the per-LSN quorum fixes (4.2–4.4) are one capability — a single commit engine (commit.c) with two completion modes: WAIT (request-shaped, blocking with timeout + admission cap) and RETAIN (fact-shaped, log-is-the-queue, no timeout). Config (5.x) stays a parallel simpler track (one in-flight change, own tag) sharing the majority arithmetic helper.
• Build vs adopt: algorithm adopted from the Raft paper (§5.3 commit rule, §5.4.2 own-term no-op); machinery reuses existing assets — standby_conn_t.acked_lsn (matchIndex), repl_broadcast, catch-up streaming, persist_recover boot path (reused by demotion self-restart). No new libraries; no .capnp changes (WAL_NOOP is a controller-internal entry type).
• Simplification (locked by user decisions): no undo log, no MVCC/committed-view machinery, no side pending-facts queue (the uncommitted log suffix IS the queue), no persisted commitIndex (volatile per Raft; rejoin reconciliation instead), egress filter = one watermark comparison on submit_lsn, demotion eviction = truncate + process self-restart via the existing boot path.
• Design written to design.md; review gate passed (all 33 requirement IDs traced, no orphan components, boundary↔file alignment verified).

---

Deep-Dive Research: Requirements 2.2, 2.4, 2.5 (2026-06-10, second pass)

Three parallel code-explorer passes resolving the Unknown tags. All findings verified with file:line evidence.

§A — Req 2.2: Notification-shaped facts have NO redelivery anywhere (worker → controller)

Verdict: the controller must own durability of worker-reported job state facts. A pending-facts queue on the primary is the only option that doesn't expand scope into the worker protocol.

DECIDED (user, 2026-06-11): facts are retained and retried until committed — including across quorum-loss periods; the primary never discards an uncommitted fact while running and leading (Req 2.8). Worker-rejoin re-sync (roadmap backlog) is the designated backstop for facts that never reached the controller or were lost with it across failover.

• Send path is enqueue-and-forget: worker job start/finish/fail → xcecli_notify_* (src/lib/client.c:289/373/392) → xce_notify (xceapi.c:270) → xce_request which enqueues and returns 0 immediately (request.c:309-331). No cbuf_get anywhere in the chain — the worker never reads a reply.
• Errors are silently discarded twice: (1) the return value of xcecd_job_finished is ignored at its call sites (xcecd_worker_utils.c:297-337, with a literal /* FIXME: add error handling when msg send failure */); (2) if the controller's dispatch gate rejects the notify with -XCE_ERR_NOT_READY (cluster.c:252-258), the error reply reaches the worker but is dropped by cbuf_tryput returning -EAGAIN with no consumer (xcectx.c:82-88, 304).
• No resync on rejoin: the worker rejoin path (checking_connection, controller_connector.c:176) re-issues only hello + join; xce_join_req carries only host resource attributes (request.c:1154-1181), no running-job state. Notifications lost during disconnect/failover are never re-delivered.
• Controller side is also fire-and-forget: do_notify (cluster.c:118-157) sends no reply at all once the gate passes.
• No idempotency guards: mark_job_start/finish/fail (job.c:583/638/696) unconditionally overwrite job->job.state — duplicate or out-of-order delivery (e.g., from a future retry/replay mechanism) is unguarded.
• Scheduler-originated transitions differ: handle_jobdispatch (job.c:459) replies synchronously (status 0 at job.c:471) BEFORE processing allocations; dispatch results flow back to the scheduler via notify_dispatch (job.c:440). Retry is owned by the scheduler side (none found in scheduler.c at the reviewed path).

Pre-existing bug exposed: today, quorum-loss (or COMMITTING-window) rejection of OPCode_notify already loses job state transitions permanently — the worker never learns and never resends. The pending-facts queue fixes an existing silent-data-loss hole, not just a new-requirements need.

Design inputs: queue must persist facts arriving while quorum is absent and drain them (through the commit barrier) when quorum returns; needs bounded size + eviction policy; mark_job_* need state-machine guards so queue drain/replay is idempotent and ordered.

Companion backlog item (2026-06-10): worker-rejoin-resync added to .kiro/steering/roadmap.md — long-planned, covers the disconnect/failover window this queue cannot (facts the worker generated while not connected). Out of scope for this spec; the idempotency guards designed here are a shared prerequisite.

§B — Req 2.4: Apply-after-commit inversion is FEASIBLE (and fixes an existing race)

Verdict (superseded 2026-06-10, user decision): the PROVISIONAL-STATE MODEL is adopted instead of inversion. In-memory state is a provisional materialized view; the persisted majority-committed log is the authoritative state. Mutation order stays as-is; commit gating moves to (1) requester confirmation and (2) irrevocable-action egress points — the dataex/scheduler export must filter provisional entities (e.g., tag entities with their WAL LSN, export only lsn <= committed_lsn), closing the dispatch-of-uncommitted-job race noted below. This makes the §C reconciliation workstream mandatory core (provisional state must actually be discarded when the log disagrees). Requirements 2.4/2.7 amended accordingly. The inversion findings below are retained as evidence and fallback.

Original verdict: invert to log→commit→apply. All prerequisites verified; the rollback alternative is infeasible.

• The BEFORE-hook seam exists and is free: xce_run_hooks aborts the chain when a HOOK_BEFORE_EVENT hook returns negative (hook.c:137) — veto semantics are built in. No registered hook currently does anything on BEFORE for job/queue/worker events (sched.c:244, apiserver.c:134, persist.c:1099 all AFTER-only), and callers currently ignore the BEFORE return code — both sides of the seam are unused and available.
• WAL payloads are constructible pre-mutation: WAL_JOB_SUBMIT serializes the full xce_job_t, but every field is computable before table insertion — the job ID is assigned by the apiserver's idgen() before the RPC (idgen.c:22; submit->jobid pre-assigned, array jobs are base_id + i, job.c:34). State-transition payloads are just {jobid, new_state} where new_state is a per-transition constant (persist.c:1134-1169) — trivially known before mutation.
• Replies don't need post-mutation state: handle_jobsubmit's reply carries only jobid (job.c:575-578), known pre-insertion.
• Post-commit apply can reuse the replay functions: the standby/recovery apply path (apply_wal_entry → restore_job/restore_job_state, persist.c:691-769, job.c:71-121) is role-agnostic with an EEXIST idempotency guard (jobj.c:696-699). Caveat: apply_wal_entry skips hooks by design (persist.c:686) — primary post-commit apply must either call restore fns directly or suppress hook double-fire (sched/apiserver AFTER hooks still need to run exactly once).
• Rollback (the alternative) is infeasible: there is no job deletion path — job_obj_killer is a stub (cluster.c:759-764), key_del is never called in src/controller/. Undo would be net-new, riskier code than inversion.
• Inversion fixes an existing race: today the submitted job is visible in c->jobmgr to concurrent vtask readers (dataex/collect_jobs snapshots, up to 8 vtask threads, cluster.c:1082) for the entire majority-wait window — no per-job write lock is held during the AFTER hook (job.c:553). An uncommitted job can leak to the scheduler via a racing dataex pull. Apply-after-commit closes this.
• One trade-off: state-transition callers hold the per-job WRITE lock across BEFORE/mutate/AFTER (job.c:629-631 etc.). Naively moving the commit wait inside that region blocks readers of that job for up to the commit timeout — design should release/restructure the lock around the wait (the pending-facts queue from §A naturally decouples this).

§C — Req 2.5: Convergence currently has FIVE gaps; safe_committed_lsn derivation is sound

Verdict: the "single outcome cluster-wide" requirement is NOT met today for the demoted-ex-primary and restart cases. New workstream required.

What works:

• New leader truncates above safe_committed_lsn on promotion (ha.c:471); standby truncates on RDHUP using last_known_committed_lsn (ha.c:3416-3420); committed_lsn advances only on majority ACK (persist.c:1035).
• safe_committed_lsn = majority-median of granted voters' committed LSNs, sorted descending, index n_granted/2 (ha.c:373-432) — conservative and safe (never includes an uncommitted entry; may under-estimate and over-truncate, which is correct-but-lossy for uncommitted suffixes).

The five gaps:

1. No WAL truncation on demotion: raft_cb_become_follower (ha.c:627-690) cancels timers, closes vote endpoints, sets STANDBY — but never calls persist_truncate_wal_above. The ex-primary's divergent uncommitted suffix survives on disk until its own future RDHUP event. (Also: this production path never disables persistence hooks — ha_demote_to_standby (ha.c:1712-1724) does, but is not called here.)
2. No in-memory phantom eviction on demotion: no path rebuilds or removes in-memory jobs/queues/workers applied from uncommitted entries. Phantoms persist indefinitely on the demoted node.
3. Snapshots carry no entity state: the snapshot format is a 16-byte sentinel (magic + checkpoint_lsn, persist.c:435-437, "Task 11" placeholder); persist_load_snapshot (persist.c:680-683) only moves the LSN watermark. Snapshot transfer can therefore NEVER evict phantoms — option (iii) below depends on implementing the real format.
4. Restart resurrects phantoms: committed_lsn is in-memory only (zero-init on boot, never persisted); replay_wal (persist.c:781-951) replays everything above checkpoint_lsn with no committed_lsn gate. A crashed ex-primary replays its own uncommitted suffix back into memory.
5. No conflict detection in standby apply: apply_wal_entry advances current_lsn and applies unconditionally by type (persist.c:720-721) — entries from the new primary are applied on top of phantom state with no divergence check.

Eviction strategy options for design: (i) in-memory rebuild on demotion — new teardown for jobmgr/queues/workers, then truncate + replay_wal from checkpoint (most work, no restart); (ii) process self-restart on demotion after truncation — minimal code, CP-acceptable since the node is leaving leadership anyway; (iii) entity-bearing snapshot transfer from the new primary — depends on deferred Task 11 snapshot format. DECIDED (user, 2026-06-11): option (ii) — process self-restart on demotion (after truncating the divergent suffix). Simplest and battle-tested: recovery reuses the boot path; a demoted node's worker/scheduler connections migrate to the new primary anyway, so the restart's connection reset costs little. Rebuild-in-place (i) remains a future refinement only if restart proves operationally disruptive. For gap 4, note Raft treats commitIndex as VOLATILE (followers recover it from the leader): the Raft-conformant fix is leader-driven log reconciliation at rejoin (consistency check + truncate divergent suffix, then rebuild) rather than persisting committed_lsn locally — design should weigh reconciliation-at-rejoin vs local persistence; either must ensure the replayed-but-uncommitted suffix cannot leak through irrevocable egress before reconciliation completes.

---

Gap Re-Audit (2026-06-18)

Roll-up gap audit of the UMBRELLA contract against the landed sub-specs. The umbrella (ha-consistency-invariants) is design-generated and owns NO tasks of its own; it was decomposed (2026-06-11) into five implementing sub-specs. This pass maps every umbrella acceptance criterion to its owning sub-spec and verifies it against real code (file:line), then enumerates the invariants still OPEN with their owning future specs.

Method: codegraph + grep against src/controller/{commit.c,persist.c,ha.c,ha_convergence.c,config_repl.c,egress.c,job.c}; sub-spec tasks.md checkbox counts and spec.json; roadmap backlog (.kiro/steering/roadmap.md) for future-spec ownership.

Sub-spec landing status (verified against code, not just spec.json phase)

spec.json.phase is STALE for three sub-specs (still reads tasks-generated) and several tasks.md checkboxes were never re-ticked. The authoritative signal is the on-disk code + git history: the COMMITTING machinery is fully gone, the commit engine is live, demotion convergence is wired, and ha-commit-durability (23/23 tasks, all committed d9b43c2…1ca2d64) re-verified the whole protocol end-to-end. The audit below grades by CODE, flagging the stale-tracking discrepancy where it occurs.

Sub-spec	tasks.md ticks	Code verdict	Notes
ha-commit-engine	12/18 ticked	LANDED	commit.c present & called from persist/ha/repl; per-LSN eval, self-quorum, own-term gate, admission cap all in code. Open boxes are stale tracking.
ha-commit-barrier-rollout	23/23	LANDED	wal_classify totality (persist.c:2004) + idempotency guards (job.c:627-797) + reply-after-commit.
ha-single-leader-role	11/17 ticked	LANDED	Zero CTRL_PRIMARY_COMMITTING/committing_timeout_fn/ha_check_majority_convergence/repl_ack_count refs in src/controller (excl. testsuite). Open boxes are stale tracking.
ha-convergence-recovery	18/19	LANDED (1 E2E open)	ha_convergence.c present + demotion path wired (ha.c:1188-1232); egress filter (egress.c). Only open: task 8 = live 3-node E2E test.
ha-commit-durability	23/23	LANDED	Group-commit flusher + durable_lsn, commit≤durable invariant, durable-backed vote freshness, no-self-terminate fence.

Requirement → sub-spec → code roll-up map

Umbrella criterion	Owning sub-spec	Code evidence (file:line)	Verdict
1.1, 1.2 direct PRIMARY, single role	single-leader-role	ha_promote_to_primary ha.c:2298; no COMMITTING enum anywhere	MET
1.3 no redirect from primary	single-leader-role	ha_should_redirect ha.c:2187 (standby-only redirect)	MET
1.4 status reports primary pre-own-term	single-leader-role	role set CTRL_PRIMARY directly at promotion (ha.c:2298-2315)	MET
1.5 / 3.3 no wall-clock leadership change	single-leader-role	committing_timeout_fn deleted (zero refs); ha_check_priority_stepdown deleted (ha.c:1472)	MET
2.1 confirm only after majority ACK	commit-engine + barrier-rollout	commit_submit WAIT commit.c:284; reply-after-commit per classification	MET
2.2 requester ops fail on commit timeout	commit-engine	-ETIMEDOUT from commit_submit commit.c:340-359	MET
2.3 no commit-index advance without majority	commit-engine	commit_advance_locked (median of acked_lsn); per-LSN eval	MET
2.4 provisional state, irrevocable egress gated	convergence-recovery	job_submit_committed/collect_jobs_committed egress.c:47-109	MET
2.5 single-outcome convergence incl. memory	convergence-recovery	demotion truncate-above-committed + self-restart ha.c:1188-1232, ha_convergence.c	MET
2.6 monotonic commit index	commit-engine	monotonic max store + XCE_TESTING assert (commit.c)	MET
2.7 reconstruct from committed log	convergence-recovery + durability	boot recovery replay; recovery_truncated verdict (durability task 3.1)	MET
2.8 facts retained+retried until committed	commit-engine + barrier-rollout	RETAIN mode commit.c:296-302; log-is-the-queue; idempotency guards job.c:627-797	MET
3.1 no confirm before own-term entry commits	single-leader-role + commit-engine	ha_arm_own_term_gate ha.c:2259 + commit_set_writable_lsn commit.c:383 (writable_lsn gate)	MET
3.2 pre-own-term: barrier or NOT_READY	commit-engine	own-term gate withholds WAIT completion; admission cap → NOT_READY	MET
4.1 quorum counts only members able to receive	commit-engine	in_catch_up exclusion (ha.c:325; commit eval excludes catch-up)	MET
4.2 commit requires per-entry ACKs	commit-engine	commit_on_ack reads registry acked_lsn, per-LSN median commit.c:366	MET
4.3 post-acceptance quorum shrink → fail	commit-engine	ACK-time re-eval + WAIT timeout; commit_on_membership_change commit.c:397	MET
4.4 never zero-peer majority (n_peers>1)	commit-engine	legacy n_active==0→success removed; n_peers≤1 short-circuit only (commit.c:330)	MET
5.1–5.3 config change commit barrier	barrier-rollout (config track)	push_config_to_standbys returns -ETIMEDOUT; caller forwards (config_repl.c:941,1115-1120); committed_config_tag gated (config_repl.c:963-969)	MET
6.1 replicate to all reachable standbys	(upstream ha-wal-replication, verified)	repl_broadcast full fan-out (unchanged)	MET
6.2 catch-up without per-step heartbeat	convergence-recovery	ack-driven continuation ha_catchup_continuation_fn config_repl.c:1732 (off-iomgr delaymgr)	MET
6.3, 6.4 snapshot fallback, re-include	(upstream, verified)	snapshot transfer + repl_complete_catchup (unchanged)	MET
6.5 lag never blocks majority confirmation	commit-engine	majority-median threshold only	MET
7.1 reject state changes without quorum	commit-engine + ha gate	dispatch gate + ACK-time enforcement (TOCTOU closed)	MET
7.2 log quorum-loss enter/exit	single-leader-role	ha_log_quorum_transition (ha.c:1085, 2076, 3969, 5054)	MET
7.3 serve status while degraded	(gate allowlist, verified)	dispatch-gate allowlist (status/replication)	MET
7.4 auto-resume on quorum restore	commit-engine + ha gate	stateless gate re-eval; restore edge logged ha.c:5054	MET
7.5 standalone self-quorum	commit-engine	n_peers <= 1 self-quorum commit.c:100,330; ha.c:2037	MET

Umbrella criteria covered: 33/33 acceptance criteria MET in code. One sub-task remains unticked within scope — ha-convergence-recovery task 8 (live 3-node E2E convergence test). This is test-evidence, not a missing invariant: the convergence MECHANISM is implemented and unit/integration-tested; the standing 3-node E2E is the same pending live test inherited from ha-wal-replication task 5.4. Tag: MET (impl) / E2E-evidence pending.

Still-OPEN invariants (Deferred — owned by FUTURE specs, NOT umbrella scope)

These were declared out-of-scope by the umbrella (Boundary Context / Non-Goals) and remain unimplemented. They are Deferred, not regressions — the umbrella never promised them; it named the owning backstop.

Open invariant	Why still open	Owning future spec (roadmap)	Effort	Risk
Worker-rejoin job-state resync (facts lost across disconnect/failover, never reached controller)	Primary-side RETAIN queue only covers facts the controller already received; worker notify is fire-and-forget with zero redelivery (bug-396). Designated backstop in umbrella Req 2.8 / boundary.	worker-rejoin-resync (roadmap, planned, no effort yet)	M	Medium — touches worker join/hello protocol + reconcile-through-barrier; shares the already-landed mark_job_* idempotency guards
Automatic primary step-down on quorum loss	Umbrella satisfies CP by BLOCKING writes (Req 7.1); step-down is an availability/clarity optimization, explicitly out of scope.	primary step-down-on-quorum-loss (roadmap, optional CP follow-on)	S–M	Low — writes already blocked; step-down is additive role transition
Read consistency / leader leases	Status queries may observe provisional/stale in-memory state (documented deviation); irrevocable-action egress already commit-gated (2.4).	read-consistency / leader-leases (roadmap, Raft §8)	M	Medium — read-only lease protocol; must not regress write path
Entity-bearing snapshots ("Task 11" format)	Snapshot is a 16-byte LSN sentinel; demotion eviction uses self-restart instead. Not required by any umbrella criterion; strengthens §C option (iii) and faster recovery.	entity-bearing-snapshots (roadmap; dual-zarray COW already built/dormant)	M	Medium — serialization format + freeze/thaw wiring
Job-array single-WAL-record	Each array element = one WAL/commit/push today; amplifies commit-engine load linearly. Correctness-neutral (each element still commits); performance only.	job-array-single-wal-record (roadmap)	M	Medium — atomic array commit + scheduler local expansion

Roll-up conclusion

The umbrella contract is substantively SATISFIED: all 33 acceptance criteria are MET in landed code across the four core sub-specs (commit-engine, barrier-rollout, single-leader-role, convergence-recovery) plus the durability sub-spec, which closed bug-661 (ACK≠durable) and the no-self-terminate promotion fence. The COMMITTING transitional state is fully removed; majority ACK (now durable-majority) is the hard commit barrier for all state-changing ops; facts are retained-and-retried; egress is commit-gated; demotion converges via truncate+self-restart.

Status: PARTIAL only in the umbrella-as-program sense — the four explicitly-deferred invariants above remain owned by named future specs, and one in-scope live 3-node E2E test (convergence-recovery task 8) is still pending. No in-scope invariant is unimplemented.

Re-validation triggers still standing: worker-rejoin-resync landing → re-verify idempotency guards + fact replay ordering; entity-bearing snapshots landing → re-evaluate demotion self-restart vs snapshot rebuild; any new state-changing opcode/WAL type → must be classified (the wal_classify debug abort at persist.c:2046 enforces this).