# Gap Analysis: ha-consistency-invariants

Date: 2026-06-10. Sources: discovery audit (brief.md), code-explorer verification pass (persist.c, repl.c, ha.c, job.c, cluster.c, config_repl.c, ha-wal-replication tasks.md).

## 1. Current State Investigation

### Existing assets directly reusable

| Asset | Location | Relevance |
|---|---|---|
| Per-standby acked LSN (≈ Raft matchIndex) | `standby_conn_t.acked_lsn` (repl.h), updated by `repl_on_ack` (repl.c:187) | Already exists — the bookkeeping needed for honest per-entry commit verification is in place, just unused by the wait logic |
| Per-standby join/applied LSN (≈ nextIndex) | `standby_conn_t.join_applied_lsn` | Catch-up positioning |
| Sync commit barrier (job submit only) | `persist_wal_sync_append` + `repl_wait_majority_ack` (persist.c:1003-1049, repl.c:386-461) | The pattern to generalize; pthread condvar wait (`ack_cond`) signaled by `repl_on_ack` |
| Full fan-out broadcast | `repl_broadcast` (repl.c:210-282) | Req 6.1 already satisfied |
| WAL streaming catch-up + snapshot fallback | `ha_stream_wal_to_standby` (config_repl.c:1450+), snapshot transfer | Req 6.3 satisfied; drive mechanism needs change for 6.2 |
| Hook-based persistence dispatch | `persist_job_hook_fun` / `persist_queue_hook_fun` / `persist_worker_hook_fun` (persist.c:1107/1199/1241) on HOOK_AFTER_EVENT | The single seam through which all state-changing writes flow |
| Quorum gate at dispatch | `ha_has_write_quorum` + `ha_dispatch_gate` (ha.c:1507-1612) | Req 7.1 partially satisfied (gate-time only) |
| ha-wal-replication ACK machinery | tasks 1.1–5.5 complete; only 5.4 (live 3-node integration test) pending | Upstream dependency effectively satisfied for design |

### Verified behaviors (with evidence)

1. **Apply-before-commit is universal — including job submit.** `job_submit()` (job.c:518) inserts the job into the in-memory table, THEN `xce_run_hooks(HOOK_AFTER_EVENT)` (job.c:553) fires the WAL write + majority wait. For state transitions, `mark_job_start/finish/...` mutate `job->job.state` between HOOK_BEFORE_EVENT and HOOK_AFTER_EVENT (e.g., job.c:585-604). Queue and worker state likewise mutate before their hooks fire. On every path, in-memory state precedes the log entry; what differs is only whether the *reply* waits (job submit: yes; everything else: no). Req 2.4 (apply only after commit) is violated everywhere today.
2. **Majority-ACK verification is not per-entry.** `repl_wait_majority_ack` (repl.c:408-443) counts standbys with `acked_lsn > 0` — i.e., "ACKed anything, ever" — not `acked_lsn >= this_entry_lsn`. Once a standby ACKs once, it counts toward every subsequent quorum until disconnect. Concurrent sync appends share one global `ack_cond` and would cross-satisfy each other. Req 4.2 needs per-LSN comparison; the `acked_lsn` field already supports it.
3. **Threading is permissive but finite.** _(Corrected 2026-06-11; original pass understated the pool and overstated the risk.)_ The sync wait is `pthread_cond_timedwait` on a vtask pool thread — default **8** threads, `--vcpus` CLI override (cluster.c:1082). Job submit already blocks one. Generalizing to all write types multiplies concurrent blocked workers; 8 simultaneous slow commits would stall the vtask pool. This is a **throughput risk, not a liveness risk**: WAL ACKs are decoded and waiters signaled on the iomgr thread (`peer_ctrl_msg_handler` → `repl_on_ack`, ha.c:2770-2792), so a fully blocked pool still receives its wakeups — requests queue until a commit completes or times out, but no deadlock is possible. Steering allows vtask handlers to block briefly ("return quickly"; the iomgr never-block rule doesn't apply). Mitigation inputs in Research Needed item 3 (commit-wait admission cap; async continuation as the durable fix).
4. **Job state transitions are reported facts, not requests.** `on_job_start/finish/fail` arrive via worker notifications (`do_notify`, cluster.c:118-156) with no client awaiting a reply. "Rejecting" a job-finish on commit timeout cannot un-finish the job on the worker; the entry must instead be held/retried until commit. The rejection semantics in Req 2.2 map cleanly to request-shaped operations (submit, queue ctrl, config) but need a hold-and-retry interpretation for notification-shaped facts.
5. **COMMITTING removal is concretely mapped.** `raft_cb_become_leader` (ha.c:505) sets COMMITTING; removal means: set CTRL_PRIMARY directly, move hook registration from `ha_promote_to_primary` (ha.c:1691-1698) into the become-leader path, delete the committing watchdog (ha.c:593-616, 3082-3125), make `ha_check_majority_convergence` (ha.c:1313) and `repl_ack_count`/`ha_repl_ack_received` obsolete, adjust the COMMITTING guard in the FSM path (ha.c:2967) and the dispatch-gate COMMITTING branch (ha.c:1594-1612).
6. **Catch-up is heartbeat-polled.** `ha_stream_wal_to_standby` sends all batches `[standby_lsn+1, committed_lsn]` in one call without per-batch ACK waits, but the NEXT round is only triggered by the next heartbeat reply (3s cadence) carrying an updated `applied_lsn` (ha.c:2734 → 1467). Req 6.2 needs ACK-driven (or WAL-append-driven) continuation.

## 2. Requirement-to-Asset Map

| Req | Existing asset | Gap | Tag |
|---|---|---|---|
| 1.1-1.2 (direct PRIMARY, single role) | become_leader callback | Remove COMMITTING state + watchdog; move hook registration | Missing (mapped) |
| 1.3 (no redirect from primary) | `ha_should_redirect` (ha.c:1655) | Returns redirect in COMMITTING; resolved by state removal | Missing |
| 1.5 / 3.3 (no timeout bypassing majority) | — | `committing_timeout_fn` force-promote must be deleted | Missing |
| 2.1 (majority barrier all ops) | sync path for WAL_JOB_SUBMIT | Generalize to JOB_* transitions, QUEUE_CTRL, WORKER_CTRL (persist.c:1169/1199/1241) | Missing |
| 2.2 (reject on timeout) | job submit error propagation (job.c:563-570) | Worker notify is pure fire-and-forget (no reply read, no retry, no rejoin resync) — controller MUST own durability via a pending-facts queue; rejection = permanent loss (already true today). See deep-dive §A | Missing (resolved — constraint identified) |
| 2.4 (provisional state, commit-gated egress) | Memory already rebuilt from checkpoint+WAL on restart (persist_recover); role-agnostic replay fns | DECIDED (2026-06-10): provisional-state model — mutation order unchanged; gate requester confirmation + irrevocable-action egress (dataex/scheduler export filters provisional entities by commit status); makes §C reconciliation mandatory core. See deep-dive §B | Missing (decided — egress gating + tagging) |
| 2.5 (single outcome for timed-out entries) | WAL truncation on promotion (ha.c:471) + on standby RDHUP (ha.c:3419); safe_committed_lsn majority-median derivation (ha.c:373-432) | FIVE convergence gaps: no truncation on demotion, no in-memory phantom reconciliation, snapshot is LSN-sentinel only (no entity state), committed_lsn not persisted (restart replays uncommitted suffix), no conflict detection in apply. See deep-dive §C | Missing (resolved — 5 gaps enumerated) |
| 3.1-3.2 (own-term commit) | — | No own-term tracking on promotion; no-op entry vs first-write decision | Missing |
| 4.1 (count only reachable) | `ha_has_write_quorum` (ha.c:1507) | Excludes neither catch-up nor per-write reachability | Missing |
| 4.2-4.4 (ACK-time per-entry verification) | `acked_lsn` per standby | Wait logic compares `> 0` not `>= entry_lsn`; no per-entry wait slots | Missing |
| 5.1-5.3 (config barrier) | `push_config_to_standbys` + `repl_wait_majority_ack` variant (config_repl.c:916-928) | Return code ignored by caller (config_repl.c:1057); error propagation | Missing (small) |
| 6.1 (full fan-out) | `repl_broadcast` | None | Done |
| 6.2 (leader-driven catch-up) | `ha_stream_wal_to_standby` | Continuation is heartbeat-polled, not ACK-driven | Missing |
| 6.3-6.4 (snapshot fallback, re-include) | snapshot transfer, `repl_complete_catchup` | None | Done |
| 6.5 (lag doesn't block confirmation) | majority threshold | Preserved by design | Done |
| 7.1 (reject on quorum loss) | dispatch gate | Gate-time only; ACK-time enforcement closes the TOCTOU | Partial |
| 7.2 (log quorum transitions) | — | Add observability logging | Missing (small) |
| 7.4 (auto-resume) | gate re-evaluates per request | None expected | Done (verify) |
| 7.5 (standalone self-quorum) | `n_peers <= 1` paths | Preserve unchanged | Constraint |

## 3. Implementation Approach Options

### Option A: Extend in place
Generalize `persist_wal_sync_append` to all WAL types at the existing hook sites; fix `repl_wait_majority_ack` to per-LSN comparison; delete COMMITTING inline.
- ✅ Smallest diff; all infrastructure exists; follows proven job-submit pattern.
- ❌ Blocking-condvar-per-write multiplies vtask pool throughput-starvation risk (no deadlock — ACKs arrive via iomgr; see §1 item 3); concurrent sync writes still share one condvar (broadcast wakeups checking per-LSN predicates works but is crude); apply-after-commit ordering not addressed (would be documented deviation).

### Option B: New commit-barrier module
New `commit.c` owning Raft commitIndex logic: per-entry waiter registry (LSN → waiter list), matchIndex aggregation (`acked_lsn` scan on each ACK → advance committed_lsn → wake waiters at or below), explicit pending-entry queue for notification-shaped facts, hold/retry. Hooks call into it; repl.c feeds it ACKs.
- ✅ Clean Raft semantics (commitIndex = median of matchIndex); supports many in-flight entries; single place to test invariants; natural home for own-term tracking and apply-after-commit sequencing.
- ❌ Larger surface; new module must integrate with persist lock ordering and the fsync thread; more migration risk in one step.

### Option C: Hybrid, phased (barrier-first ordering preserved)
- Phase 1 (repl.c surgical): per-LSN wait predicate (`acked_lsn >= entry_lsn` count ≥ majority), remove `n_active == 0` short-circuit for `n_peers > 1`, ACK-time quorum verification. Job submit immediately gets honest commits.
- Phase 2 (persist.c + small commit registry): generalize the barrier to QUEUE_CTRL, WORKER_CTRL, and request-shaped job ops; introduce a minimal pending-entry registry only for notification-shaped facts (hold/retry until committed); config push error propagation.
- Phase 3 (ha.c): own-term commit gate on promotion, then delete COMMITTING (+ watchdog, convergence checker, redirect fix).
- Phase 4 (config_repl.c): ACK-driven catch-up continuation; quorum-transition logging.
- ✅ Each phase independently testable and shippable; respects the brief's hard ordering (barrier before state removal); risk contained.
- ❌ Per-entry waiter logic lands in two steps; requires discipline to not stop halfway.

## 4. Effort & Risk

| Workstream | Effort | Risk | Justification |
|---|---|---|---|
| Per-LSN ACK verification + TOCTOU closure | S-M | Low | Fields exist; localized to repl.c wait/ack path |
| Barrier generalization (request-shaped ops) | M | Medium | Proven pattern, but vtask pool throughput stall needs mitigation (commit-wait admission cap or async continuation — see Research Needed item 3) |
| Notification-shaped facts (pending-facts queue) | M | Medium | Resolved: worker provides NO redelivery (verified), so a primary-side pending queue is the required mechanism; semantics now clear, implementation moderate |
| Provisional-state egress gating (replaces inversion) | M | Medium | DECIDED: mutation order unchanged; tag entities with WAL LSN, filter dataex/scheduler export to committed-only (closes dispatch-of-uncommitted race); smaller than inversion but couples to §C reconciliation being real |
| Demotion/restart convergence (Req 2.5 gaps) | M-L | **High** | NEW workstream: truncate-on-demotion, persist or reconcile committed_lsn at recovery, phantom in-memory state eviction (no teardown path exists; snapshot carries no entity state) |
| COMMITTING removal + own-term gate | S-M | Medium | Concretely mapped (6 sites); must land after barrier |
| Config push error propagation | S | Low | Two call sites |
| Leader-driven catch-up | M | Medium | Continuation trigger redesign in streaming path |

Overall: **L–XL (2+ weeks across phases), Medium-High risk** — dominated by the two High-risk research items.

## 5. Recommendations for Design Phase

**Preferred approach: Option C** (phased hybrid), with Phase 2's pending-entry registry designed Raft-shaped from the start (commitIndex from matchIndex aggregation) so it can absorb Phase 1's logic rather than duplicate it.

**Research Needed (carry into design):**
1. ~~Apply-after-commit strategy~~ **DECIDED** (user, 2026-06-10): provisional-state model — in-memory state is a provisional view, persisted majority-committed log is authoritative. Mutation order unchanged; gate confirmation + irrevocable-action egress on commit. Remaining design details: entity commit-tagging mechanism (per-entity WAL LSN vs per-type watermark), which egress points are "irrevocable" (dataex/scheduler export: yes; status queries: no, documented deviation), and the egress filter's interaction with array jobs sharing one submit entry.
   - **Mechanism direction (discussed 2026-06-10, STM evaluation — REFINED same day)**: full STM rejected — transaction duration mismatch (commit point is a majority network ACK; STM transactions can't span syscalls), C ecosystem dead (libitm unmaintained, TSX disabled), single-writer + per-object RTTI locks already cover the concurrency. An ARIES-style undo log was briefly considered, then **dropped after the recovery-oriented reframing (user)**: this is a failover/recovery problem, not a transaction problem. Structural fact: a single-writer leader never truncates its own log while leader (Raft leaders only append), so a provisional entry on a live primary either commits later or the node is demoted — **discard events coincide exactly with role/lifecycle events** (demotion, restart, rejoin), never with normal operation. Therefore: no per-entry undo; discard is handled by recovery mechanisms — truncate divergent suffix + coarse memory rebuild via the existing `persist_recover`/`replay_wal` path at demotion/rejoin (rare, node not serving writes, cost acceptable). The only piece retained from the versioning discussion is the **watermark egress filter** (entity WAL-LSN tag, export only `lsn <= committed_lsn` to dataex/scheduler) — justified as failover correctness (prevent orphan dispatch of a job the post-failover cluster never heard of), not as transaction isolation.
   - **xzarray shadow evaluated (2026-06-11)**: the dual-zarray COW (`xzarray_freeze`/`xzarray_thaw`, zarray.c:695-757 — built and unit-tested, zero production callers; kvmgr vfreeze NULL "wired in P0-2") is slot-level STRUCTURAL COW: slots hold pointers, so in-place field mutations (job state transitions) are visible through both views. Not suitable for the committed-view egress (reads are merged active-over-shadow, single generation would collide with the future checkpoint consumer, point-in-time vs advancing watermark) nor for provisional rollback (only covers inserts, not mutations; demotion-time eviction doesn't need lock-free semantics anyway). Its real home is the entity-bearing-snapshots backlog item (its original P0-2 purpose), which indirectly strengthens §C eviction option (iii). Watermark filter stands as the egress mechanism.
2. ~~Notification-shaped facts~~ **DECIDED** (user, 2026-06-11): **log-is-the-queue, Raft-pure** — fact entries are appended to the WAL immediately and replicated/retried until committed with NO commit timeout (Raft leaders never time out entries; retry across quorum-loss periods per Req 2.8). No separate side-queue structure; the "pending facts" are simply the uncommitted log suffix under continuous retry. Crash-loss of uncommitted facts is accepted until worker-rejoin-resync (roadmap backstop) lands — strictly better than today's behavior. Remaining design details: idempotency guards in `mark_job_*` (needed for replay safety) and retry/backoff cadence.
3. **Wait mechanism under load**: blocking condvar per vtask worker vs async continuation (queue reply closure, complete on commit). Measure vtask pool headroom; consider dedicated commit-wait sizing.
   - **Verified inputs (2026-06-11)**: (a) vtask pool default is ALREADY 8 with `--vcpus` override (cluster.c:1082; the "default 4" in ha-async-io.md was stale — fixed). (b) **No wakeup deadlock**: WAL ACKs are processed on the iomgr thread (`peer_ctrl_msg_handler` → `repl_ack_decode_and_dispatch` → `repl_on_ack`, ha.c:2770-2792, 2748) — blocked vtask workers always get woken; all-workers-blocked costs throughput, not liveness. (c) **Pool size should NOT scale with n_peers** (user question, analyzed): blocked-thread count ≈ in-flight write ops × commit latency (Little's law); ACKs gather in parallel so peer count barely moves latency. Write concurrency and cores are the sizing inputs. (d) Recommended interim mitigation: commit-wait admission cap — at most `n_vcpus - K` concurrent blocking waits (reserve K lanes for non-blocking traffic), excess write ops fail fast with NOT_READY (CP-correct under overload). Async continuation remains the durable fix. (e) **Sizing ownership (user, 2026-06-11)**: `--vcpus` is a tunable operator parameter — the right value is a deployment decision driven by host capability, cluster size, and workload; 8 is only the fallback default. Design constraint: all commit-wait correctness mechanisms must be expressed relative to `n_vcpus` (never absolute counts) so behavior is correct at any tuning; sizing guidance belongs in operator/tuning docs, not in spec-fixed constants.
4. ~~Own-term commit vehicle~~ **DECIDED** (user, 2026-06-11): **explicit no-op WAL entry at promotion** — bounded post-election write-unavailability independent of traffic; an idle cluster's new primary becomes writable as soon as the no-op commits.
5. **Multi-entry in-flight semantics**: confirm `committed_lsn` advancement is contiguous (no gaps) when multiple entries await ACKs.
6. ~~Req 2.5 verification~~ **RESOLVED into a workstream** (deep-dive §C): five convergence gaps enumerated. Design must choose the phantom-eviction strategy: (i) full in-memory rebuild on demotion (needs new teardown path for jobmgr/queues/workers, then `replay_wal` from checkpoint after truncation), (ii) process self-restart on demotion (crude, safe, CP-consistent), or (iii) entity-bearing snapshots (depends on the deferred "Task 11" snapshot format). Also: persist `committed_lsn` (or reconcile it from the cluster at recovery) so restart does not replay the uncommitted suffix.

**Constraints to respect:** standalone `n_peers <= 1` self-quorum unchanged; iomgr thread never blocks (waits live on vtask/delaymgr threads only); single-writer model; ha-wal-replication task 5.4 (live integration test) still pending — its 3-node test should be extended to cover the new barrier rather than duplicated.

---

# Design Synthesis Outcomes (2026-06-11, design phase)

- **Generalization**: Requirements 2.x (op barrier), 3.x (own-term gate), and the per-LSN quorum fixes (4.2–4.4) are one capability — a single **commit engine** (`commit.c`) with two completion modes: WAIT (request-shaped, blocking with timeout + admission cap) and RETAIN (fact-shaped, log-is-the-queue, no timeout). Config (5.x) stays a parallel simpler track (one in-flight change, own tag) sharing the majority arithmetic helper.
- **Build vs adopt**: algorithm adopted from the Raft paper (§5.3 commit rule, §5.4.2 own-term no-op); machinery reuses existing assets — `standby_conn_t.acked_lsn` (matchIndex), `repl_broadcast`, catch-up streaming, `persist_recover` boot path (reused by demotion self-restart). No new libraries; no .capnp changes (WAL_NOOP is a controller-internal entry type).
- **Simplification** (locked by user decisions): no undo log, no MVCC/committed-view machinery, no side pending-facts queue (the uncommitted log suffix IS the queue), no persisted commitIndex (volatile per Raft; rejoin reconciliation instead), egress filter = one watermark comparison on `submit_lsn`, demotion eviction = truncate + process self-restart via the existing boot path.
- Design written to `design.md`; review gate passed (all 33 requirement IDs traced, no orphan components, boundary↔file alignment verified).

---

# Deep-Dive Research: Requirements 2.2, 2.4, 2.5 (2026-06-10, second pass)

Three parallel code-explorer passes resolving the Unknown tags. All findings verified with file:line evidence.

## §A — Req 2.2: Notification-shaped facts have NO redelivery anywhere (worker → controller)

**Verdict: the controller must own durability of worker-reported job state facts. A pending-facts queue on the primary is the only option that doesn't expand scope into the worker protocol.**

**DECIDED (user, 2026-06-11)**: facts are retained and retried until committed — including across quorum-loss periods; the primary never discards an uncommitted fact while running and leading (Req 2.8). Worker-rejoin re-sync (roadmap backlog) is the designated backstop for facts that never reached the controller or were lost with it across failover.

- **Send path is enqueue-and-forget**: worker job start/finish/fail → `xcecli_notify_*` (src/lib/client.c:289/373/392) → `xce_notify` (xceapi.c:270) → `xce_request` which enqueues and returns 0 immediately (request.c:309-331). No `cbuf_get` anywhere in the chain — the worker never reads a reply.
- **Errors are silently discarded twice**: (1) the return value of `xcecd_job_finished` is ignored at its call sites (xcecd_worker_utils.c:297-337, with a literal `/* FIXME: add error handling when msg send failure */`); (2) if the controller's dispatch gate rejects the notify with `-XCE_ERR_NOT_READY` (cluster.c:252-258), the error reply reaches the worker but is dropped by `cbuf_tryput` returning -EAGAIN with no consumer (xcectx.c:82-88, 304).
- **No resync on rejoin**: the worker rejoin path (`checking_connection`, controller_connector.c:176) re-issues only hello + join; `xce_join_req` carries only host resource attributes (request.c:1154-1181), no running-job state. Notifications lost during disconnect/failover are never re-delivered.
- **Controller side is also fire-and-forget**: `do_notify` (cluster.c:118-157) sends no reply at all once the gate passes.
- **No idempotency guards**: `mark_job_start/finish/fail` (job.c:583/638/696) unconditionally overwrite `job->job.state` — duplicate or out-of-order delivery (e.g., from a future retry/replay mechanism) is unguarded.
- **Scheduler-originated transitions differ**: `handle_jobdispatch` (job.c:459) replies synchronously (status 0 at job.c:471) BEFORE processing allocations; dispatch results flow back to the scheduler via `notify_dispatch` (job.c:440). Retry is owned by the scheduler side (none found in scheduler.c at the reviewed path).

**Pre-existing bug exposed**: today, quorum-loss (or COMMITTING-window) rejection of `OPCode_notify` already loses job state transitions permanently — the worker never learns and never resends. The pending-facts queue fixes an existing silent-data-loss hole, not just a new-requirements need.

**Design inputs**: queue must persist facts arriving while quorum is absent and drain them (through the commit barrier) when quorum returns; needs bounded size + eviction policy; `mark_job_*` need state-machine guards so queue drain/replay is idempotent and ordered.

**Companion backlog item (2026-06-10)**: worker-rejoin-resync added to `.kiro/steering/roadmap.md` — long-planned, covers the disconnect/failover window this queue cannot (facts the worker generated while not connected). Out of scope for this spec; the idempotency guards designed here are a shared prerequisite.

## §B — Req 2.4: Apply-after-commit inversion is FEASIBLE (and fixes an existing race)

**Verdict (superseded 2026-06-10, user decision): the PROVISIONAL-STATE MODEL is adopted instead of inversion.** In-memory state is a provisional materialized view; the persisted majority-committed log is the authoritative state. Mutation order stays as-is; commit gating moves to (1) requester confirmation and (2) irrevocable-action egress points — the dataex/scheduler export must filter provisional entities (e.g., tag entities with their WAL LSN, export only `lsn <= committed_lsn`), closing the dispatch-of-uncommitted-job race noted below. This makes the §C reconciliation workstream mandatory core (provisional state must actually be discarded when the log disagrees). Requirements 2.4/2.7 amended accordingly. The inversion findings below are retained as evidence and fallback.

**Original verdict: invert to log→commit→apply. All prerequisites verified; the rollback alternative is infeasible.**

- **The BEFORE-hook seam exists and is free**: `xce_run_hooks` aborts the chain when a HOOK_BEFORE_EVENT hook returns negative (hook.c:137) — veto semantics are built in. No registered hook currently does anything on BEFORE for job/queue/worker events (sched.c:244, apiserver.c:134, persist.c:1099 all AFTER-only), and callers currently ignore the BEFORE return code — both sides of the seam are unused and available.
- **WAL payloads are constructible pre-mutation**: WAL_JOB_SUBMIT serializes the full `xce_job_t`, but every field is computable before table insertion — the job ID is assigned by the apiserver's `idgen()` before the RPC (idgen.c:22; `submit->jobid` pre-assigned, array jobs are `base_id + i`, job.c:34). State-transition payloads are just `{jobid, new_state}` where new_state is a per-transition constant (persist.c:1134-1169) — trivially known before mutation.
- **Replies don't need post-mutation state**: `handle_jobsubmit`'s reply carries only `jobid` (job.c:575-578), known pre-insertion.
- **Post-commit apply can reuse the replay functions**: the standby/recovery apply path (`apply_wal_entry` → `restore_job`/`restore_job_state`, persist.c:691-769, job.c:71-121) is role-agnostic with an EEXIST idempotency guard (jobj.c:696-699). Caveat: `apply_wal_entry` skips hooks by design (persist.c:686) — primary post-commit apply must either call restore fns directly or suppress hook double-fire (sched/apiserver AFTER hooks still need to run exactly once).
- **Rollback (the alternative) is infeasible**: there is no job deletion path — `job_obj_killer` is a stub (cluster.c:759-764), `key_del` is never called in src/controller/. Undo would be net-new, riskier code than inversion.
- **Inversion fixes an existing race**: today the submitted job is visible in `c->jobmgr` to concurrent vtask readers (dataex/`collect_jobs` snapshots, up to 8 vtask threads, cluster.c:1082) for the entire majority-wait window — no per-job write lock is held during the AFTER hook (job.c:553). An uncommitted job can leak to the scheduler via a racing dataex pull. Apply-after-commit closes this.
- **One trade-off**: state-transition callers hold the per-job WRITE lock across BEFORE/mutate/AFTER (job.c:629-631 etc.). Naively moving the commit wait inside that region blocks readers of that job for up to the commit timeout — design should release/restructure the lock around the wait (the pending-facts queue from §A naturally decouples this).

## §C — Req 2.5: Convergence currently has FIVE gaps; safe_committed_lsn derivation is sound

**Verdict: the "single outcome cluster-wide" requirement is NOT met today for the demoted-ex-primary and restart cases. New workstream required.**

What works:
- New leader truncates above `safe_committed_lsn` on promotion (ha.c:471); standby truncates on RDHUP using `last_known_committed_lsn` (ha.c:3416-3420); `committed_lsn` advances only on majority ACK (persist.c:1035).
- `safe_committed_lsn` = majority-median of granted voters' committed LSNs, sorted descending, index `n_granted/2` (ha.c:373-432) — conservative and safe (never includes an uncommitted entry; may under-estimate and over-truncate, which is correct-but-lossy for uncommitted suffixes).

The five gaps:
1. **No WAL truncation on demotion**: `raft_cb_become_follower` (ha.c:627-690) cancels timers, closes vote endpoints, sets STANDBY — but never calls `persist_truncate_wal_above`. The ex-primary's divergent uncommitted suffix survives on disk until its own future RDHUP event. (Also: this production path never disables persistence hooks — `ha_demote_to_standby` (ha.c:1712-1724) does, but is not called here.)
2. **No in-memory phantom eviction on demotion**: no path rebuilds or removes in-memory jobs/queues/workers applied from uncommitted entries. Phantoms persist indefinitely on the demoted node.
3. **Snapshots carry no entity state**: the snapshot format is a 16-byte sentinel (magic + checkpoint_lsn, persist.c:435-437, "Task 11" placeholder); `persist_load_snapshot` (persist.c:680-683) only moves the LSN watermark. Snapshot transfer can therefore NEVER evict phantoms — option (iii) below depends on implementing the real format.
4. **Restart resurrects phantoms**: `committed_lsn` is in-memory only (zero-init on boot, never persisted); `replay_wal` (persist.c:781-951) replays everything above `checkpoint_lsn` with no committed_lsn gate. A crashed ex-primary replays its own uncommitted suffix back into memory.
5. **No conflict detection in standby apply**: `apply_wal_entry` advances `current_lsn` and applies unconditionally by type (persist.c:720-721) — entries from the new primary are applied on top of phantom state with no divergence check.

Eviction strategy options for design: (i) in-memory rebuild on demotion — new teardown for jobmgr/queues/workers, then truncate + `replay_wal` from checkpoint (most work, no restart); (ii) process self-restart on demotion after truncation — minimal code, CP-acceptable since the node is leaving leadership anyway; (iii) entity-bearing snapshot transfer from the new primary — depends on deferred Task 11 snapshot format. **DECIDED (user, 2026-06-11): option (ii) — process self-restart on demotion** (after truncating the divergent suffix). Simplest and battle-tested: recovery reuses the boot path; a demoted node's worker/scheduler connections migrate to the new primary anyway, so the restart's connection reset costs little. Rebuild-in-place (i) remains a future refinement only if restart proves operationally disruptive. For gap 4, note Raft treats commitIndex as VOLATILE (followers recover it from the leader): the Raft-conformant fix is leader-driven log reconciliation at rejoin (consistency check + truncate divergent suffix, then rebuild) rather than persisting `committed_lsn` locally — design should weigh reconciliation-at-rejoin vs local persistence; either must ensure the replayed-but-uncommitted suffix cannot leak through irrevocable egress before reconciliation completes.

---

## Gap Re-Audit (2026-06-18)

Roll-up gap audit of the UMBRELLA contract against the landed sub-specs. The umbrella
(`ha-consistency-invariants`) is design-generated and owns NO tasks of its own; it was
decomposed (2026-06-11) into five implementing sub-specs. This pass maps every umbrella
acceptance criterion to its owning sub-spec and verifies it against real code (file:line),
then enumerates the invariants still OPEN with their owning future specs.

**Method**: codegraph + grep against `src/controller/{commit.c,persist.c,ha.c,ha_convergence.c,config_repl.c,egress.c,job.c}`; sub-spec `tasks.md` checkbox counts and `spec.json`; roadmap backlog (`.kiro/steering/roadmap.md`) for future-spec ownership.

### Sub-spec landing status (verified against code, not just spec.json phase)

> `spec.json.phase` is STALE for three sub-specs (still reads `tasks-generated`) and several
> `tasks.md` checkboxes were never re-ticked. The authoritative signal is the on-disk code +
> git history: the COMMITTING machinery is **fully gone**, the commit engine is live, demotion
> convergence is wired, and `ha-commit-durability` (23/23 tasks, all committed d9b43c2…1ca2d64)
> re-verified the whole protocol end-to-end. The audit below grades by CODE, flagging the
> stale-tracking discrepancy where it occurs.

| Sub-spec | tasks.md ticks | Code verdict | Notes |
|---|---|---|---|
| ha-commit-engine | 12/18 ticked | **LANDED** | `commit.c` present & called from persist/ha/repl; per-LSN eval, self-quorum, own-term gate, admission cap all in code. Open boxes are stale tracking. |
| ha-commit-barrier-rollout | 23/23 | **LANDED** | `wal_classify` totality (persist.c:2004) + idempotency guards (job.c:627-797) + reply-after-commit. |
| ha-single-leader-role | 11/17 ticked | **LANDED** | Zero `CTRL_PRIMARY_COMMITTING`/`committing_timeout_fn`/`ha_check_majority_convergence`/`repl_ack_count` refs in src/controller (excl. testsuite). Open boxes are stale tracking. |
| ha-convergence-recovery | 18/19 | **LANDED (1 E2E open)** | `ha_convergence.c` present + demotion path wired (ha.c:1188-1232); egress filter (egress.c). Only open: task 8 = live 3-node E2E test. |
| ha-commit-durability | 23/23 | **LANDED** | Group-commit flusher + `durable_lsn`, commit≤durable invariant, durable-backed vote freshness, no-self-terminate fence. |

### Requirement → sub-spec → code roll-up map

| Umbrella criterion | Owning sub-spec | Code evidence (file:line) | Verdict |
|---|---|---|---|
| 1.1, 1.2 direct PRIMARY, single role | single-leader-role | `ha_promote_to_primary` ha.c:2298; no COMMITTING enum anywhere | MET |
| 1.3 no redirect from primary | single-leader-role | `ha_should_redirect` ha.c:2187 (standby-only redirect) | MET |
| 1.4 status reports primary pre-own-term | single-leader-role | role set CTRL_PRIMARY directly at promotion (ha.c:2298-2315) | MET |
| 1.5 / 3.3 no wall-clock leadership change | single-leader-role | `committing_timeout_fn` deleted (zero refs); `ha_check_priority_stepdown` deleted (ha.c:1472) | MET |
| 2.1 confirm only after majority ACK | commit-engine + barrier-rollout | `commit_submit` WAIT commit.c:284; reply-after-commit per classification | MET |
| 2.2 requester ops fail on commit timeout | commit-engine | `-ETIMEDOUT` from `commit_submit` commit.c:340-359 | MET |
| 2.3 no commit-index advance without majority | commit-engine | `commit_advance_locked` (median of acked_lsn); per-LSN eval | MET |
| 2.4 provisional state, irrevocable egress gated | convergence-recovery | `job_submit_committed`/`collect_jobs_committed` egress.c:47-109 | MET |
| 2.5 single-outcome convergence incl. memory | convergence-recovery | demotion truncate-above-committed + self-restart ha.c:1188-1232, ha_convergence.c | MET |
| 2.6 monotonic commit index | commit-engine | monotonic max store + XCE_TESTING assert (commit.c) | MET |
| 2.7 reconstruct from committed log | convergence-recovery + durability | boot recovery replay; `recovery_truncated` verdict (durability task 3.1) | MET |
| 2.8 facts retained+retried until committed | commit-engine + barrier-rollout | RETAIN mode commit.c:296-302; log-is-the-queue; idempotency guards job.c:627-797 | MET |
| 3.1 no confirm before own-term entry commits | single-leader-role + commit-engine | `ha_arm_own_term_gate` ha.c:2259 + `commit_set_writable_lsn` commit.c:383 (`writable_lsn` gate) | MET |
| 3.2 pre-own-term: barrier or NOT_READY | commit-engine | own-term gate withholds WAIT completion; admission cap → NOT_READY | MET |
| 4.1 quorum counts only members able to receive | commit-engine | `in_catch_up` exclusion (ha.c:325; commit eval excludes catch-up) | MET |
| 4.2 commit requires per-entry ACKs | commit-engine | `commit_on_ack` reads registry `acked_lsn`, per-LSN median commit.c:366 | MET |
| 4.3 post-acceptance quorum shrink → fail | commit-engine | ACK-time re-eval + WAIT timeout; `commit_on_membership_change` commit.c:397 | MET |
| 4.4 never zero-peer majority (n_peers>1) | commit-engine | legacy `n_active==0→success` removed; n_peers≤1 short-circuit only (commit.c:330) | MET |
| 5.1–5.3 config change commit barrier | barrier-rollout (config track) | `push_config_to_standbys` returns `-ETIMEDOUT`; caller forwards (config_repl.c:941,1115-1120); `committed_config_tag` gated (config_repl.c:963-969) | MET |
| 6.1 replicate to all reachable standbys | (upstream ha-wal-replication, verified) | `repl_broadcast` full fan-out (unchanged) | MET |
| 6.2 catch-up without per-step heartbeat | convergence-recovery | ack-driven continuation `ha_catchup_continuation_fn` config_repl.c:1732 (off-iomgr delaymgr) | MET |
| 6.3, 6.4 snapshot fallback, re-include | (upstream, verified) | snapshot transfer + `repl_complete_catchup` (unchanged) | MET |
| 6.5 lag never blocks majority confirmation | commit-engine | majority-median threshold only | MET |
| 7.1 reject state changes without quorum | commit-engine + ha gate | dispatch gate + ACK-time enforcement (TOCTOU closed) | MET |
| 7.2 log quorum-loss enter/exit | single-leader-role | `ha_log_quorum_transition` (ha.c:1085, 2076, 3969, 5054) | MET |
| 7.3 serve status while degraded | (gate allowlist, verified) | dispatch-gate allowlist (status/replication) | MET |
| 7.4 auto-resume on quorum restore | commit-engine + ha gate | stateless gate re-eval; restore edge logged ha.c:5054 | MET |
| 7.5 standalone self-quorum | commit-engine | `n_peers <= 1` self-quorum commit.c:100,330; ha.c:2037 | MET |

**Umbrella criteria covered: 33/33 acceptance criteria MET in code.** One sub-task remains
unticked within scope — `ha-convergence-recovery` task 8 (live 3-node E2E convergence test).
This is test-evidence, not a missing invariant: the convergence MECHANISM is implemented and
unit/integration-tested; the standing 3-node E2E is the same pending live test inherited from
ha-wal-replication task 5.4. Tag: **MET (impl) / E2E-evidence pending.**

### Still-OPEN invariants (Deferred — owned by FUTURE specs, NOT umbrella scope)

These were declared out-of-scope by the umbrella (Boundary Context / Non-Goals) and remain
unimplemented. They are **Deferred**, not regressions — the umbrella never promised them; it
named the owning backstop.

| Open invariant | Why still open | Owning future spec (roadmap) | Effort | Risk |
|---|---|---|---|---|
| Worker-rejoin job-state resync (facts lost across disconnect/failover, never reached controller) | Primary-side RETAIN queue only covers facts the controller already received; worker notify is fire-and-forget with zero redelivery (bug-396). Designated backstop in umbrella Req 2.8 / boundary. | **worker-rejoin-resync** (roadmap, planned, no effort yet) | M | Medium — touches worker join/hello protocol + reconcile-through-barrier; shares the already-landed `mark_job_*` idempotency guards |
| Automatic primary step-down on quorum loss | Umbrella satisfies CP by BLOCKING writes (Req 7.1); step-down is an availability/clarity optimization, explicitly out of scope. | **primary step-down-on-quorum-loss** (roadmap, optional CP follow-on) | S–M | Low — writes already blocked; step-down is additive role transition |
| Read consistency / leader leases | Status queries may observe provisional/stale in-memory state (documented deviation); irrevocable-action egress already commit-gated (2.4). | **read-consistency / leader-leases** (roadmap, Raft §8) | M | Medium — read-only lease protocol; must not regress write path |
| Entity-bearing snapshots ("Task 11" format) | Snapshot is a 16-byte LSN sentinel; demotion eviction uses self-restart instead. Not required by any umbrella criterion; strengthens §C option (iii) and faster recovery. | **entity-bearing-snapshots** (roadmap; dual-zarray COW already built/dormant) | M | Medium — serialization format + freeze/thaw wiring |
| Job-array single-WAL-record | Each array element = one WAL/commit/push today; amplifies commit-engine load linearly. Correctness-neutral (each element still commits); performance only. | **job-array-single-wal-record** (roadmap) | M | Medium — atomic array commit + scheduler local expansion |

### Roll-up conclusion

The umbrella contract is **substantively SATISFIED**: all 33 acceptance criteria are MET in
landed code across the four core sub-specs (commit-engine, barrier-rollout, single-leader-role,
convergence-recovery) plus the durability sub-spec, which closed bug-661 (ACK≠durable) and the
no-self-terminate promotion fence. The COMMITTING transitional state is fully removed; majority
ACK (now durable-majority) is the hard commit barrier for all state-changing ops; facts are
retained-and-retried; egress is commit-gated; demotion converges via truncate+self-restart.

Status: **PARTIAL** only in the umbrella-as-program sense — the four explicitly-deferred
invariants above remain owned by named future specs, and one in-scope live 3-node E2E test
(convergence-recovery task 8) is still pending. No in-scope invariant is unimplemented.

**Re-validation triggers still standing**: worker-rejoin-resync landing → re-verify idempotency
guards + fact replay ordering; entity-bearing snapshots landing → re-evaluate demotion
self-restart vs snapshot rebuild; any new state-changing opcode/WAL type → must be classified
(the `wal_classify` debug abort at persist.c:2046 enforces this).