myxce/.kiro/specs/ha-repl-hardening/research.md

Gap Analysis: HA Replication Hardening

Analysis date: 2026-06-04. Phase: requirements-generated. Purpose: identify what exists, what's missing, and viable implementation approaches.

Analysis Summary

• Scope: 7 requirements — heartbeat-driven state convergence, WAL catch-up wiring, per-standby tracking, snapshot transfer, batch WAL transfer, COMMITTING gate, observability
• Key finding: The primary already sends heartbeats (OPCode_nop) and receives replies; the CtrlHelloReply already carries committed_lsn and config_tag; persist_checkpoint already exists. The gaps are in wiring, not new infrastructure.
• Largest gaps: (1) heartbeat reply doesn't carry standby's applied_lsn/config_tag; (2) WAL catch-up handler is a stub; (3) snapshot transfer path is a stub; (4) no per-standby convergence tracking in the primary.
• Recommendation: Extend existing heartbeat reply with state fields; wire the WAL catch-up stub to apply_wal_entry(); implement snapshot transfer using existing persist_checkpoint(); add per-standby convergence tracking to ha_state_t or extend standby_conn_t.

1. Current State Investigation

1.1 Heartbeat Protocol

Current state: Asymmetric, unidirectional state push.

• Primary sends OPCode_nop via primary_heartbeat_fn (ha.c:2880) every heartbeat_interval_ms (3s default)
• Standby receives nop in myreq_handler (cluster.c:225), sends empty RequestReply, calls ha_on_heartbeat_received (ha.c:2655) to reset liveness watchdog
• CtrlHelloRequest/CtrlHelloReply (controller.capnp:835-846) carry appliedLsn, configTag, committedLsn — but only exchanged once on connect, not periodically

Gap: Heartbeat reply is empty. Standby's current (applied_lsn, config_tag) is not reported back to the primary on each heartbeat cycle.

1.2 Per-Standby State Tracking

Current state: standby_conn_t (repl.h:22-33) tracks per-standby acked_lsn and acked_config_lsn (for WAL and config quorum), plus in_catch_up and pending_ack_count. ha_state_t has standby_endpoints (zarray of endpoints) and n_peers, but no per-standby convergence tracking.

Gap: No field for the standby's self-reported applied_lsn or config_tag from heartbeat replies. The primary has no way to know whether a standby has converged without receiving explicit WAL/config ACKs.

1.3 WAL Catch-Up Handler

Current state: Stub at ha.c dispatch gate (OPCode_walCatchupEntry returns 0). The primary-side streaming loop in config_repl.c:1489-1520 reads WAL segments and sends OPCode_walCatchupEntry messages, but the standby does nothing with them.

Gap: Standby never applies WAL catch-up entries to local state. apply_wal_entry() (persist.c:685) exists and is used during startup WAL replay — can be reused.

1.4 Snapshot Transfer

Current state: handle_ctrlhello (config_repl.c:1359) checks repl_needs_snapshot() (repl.c:491) and if true, logs WARN at line 1418 ("snapshot transfer not yet implemented") and skips. persist_checkpoint() (persist.c:610) exists and creates a consistent checkpoint.

Gap: The snapshot creation and transfer path is entirely unimplemented. Without it, any standby whose LSN predates the WAL retention window is permanently dead.

1.5 WAL Entry Batching

Current state: The WAL streaming loop in config_repl.c:1481-1521 sends one xce_request per WAL entry. xce_request enqueues to the endpoint's send buffer (non-blocking), but the loop processes entries sequentially with pread() per entry + malloc/free per payload.

Gap: No batching. For a standby catching up on millions of entries, this is CPU-intensive and slow. The wire protocol needs a batch message format.

1.6 COMMITTING Gate

Current state: ha_repl_ack_received() (ha.c:2197) exists with correct quorum math, but is only called from repl_ack_decode_and_dispatch() (ha.c:1859) which processes OPCode_replCatchUpDone. No standby sends this opcode. The committing timeout (10s) is the only path to PRIMARY.

Gap: The majority-convergence check is unreachable. The committing timeout degrades the cluster on every startup.

2. Requirement-to-Asset Map

Req	Key Gap	Key Reusable Asset	Gap Tag
1 — Heartbeat carries standby state	Reply body is empty; no state fields in nop reply	OPCode_nop send/receive loop, CtrlHelloReply fields in capnp schema, xce_reply pattern in cluster.c:225-228	Missing
2 — Per-standby convergence tracking	No applied_lsn/config_tag reported per standby	standby_conn_t with acked_lsn, ha->standby_endpoints, repl_wait_majority_ack quorum math	Missing
3 — WAL catch-up handler	Stub returns 0 (ha.c:882)	apply_wal_entry() (persist.c:685), primary WAL streaming loop (config_repl.c:1489)	Stub
4 — Snapshot transfer	Stub logs WARN, skips (config_repl.c:1417-1425)	persist_checkpoint() (persist.c:610), repl_needs_snapshot() (repl.c:491), CtrlHelloReply.needsSnapshot field	Missing
5 — Batch WAL transfer	One entry per xce_request	Existing WAL streaming loop, capnp message format for walCatchupEntry	Missing
6 — COMMITTING gate	ha_repl_ack_received unreachable; timeout-only path	ha_repl_ack_received() (ha.c:2197) with correct quorum math, ha_promote_to_primary()	Missing
7 — Observability	Logs exist but need new convergence/snapshot events	INFO/WARN macros in xcelog.h	Missing (alongside)

3. Implementation Approach Options

Option A: Extend heartbeat reply with state fields (Recommended)

Extend the existing OPCode_nop reply or create a lightweight heartbeat-ack to carry (applied_lsn, config_tag):

• Standby side: In the nop handler (cluster.c:225), populate reply with current applied_lsn and config_tag before xce_reply
• Primary side: In the heartbeat send path or a reply handler, read the standby's reported state and update per-standby tracking
• Why: Minimal new code — reuses existing heartbeat loop; no new opcodes needed if we extend the reply body; follows Raft AppendEntries pattern

Trade-offs:

• ✅ Reuses existing infrastructure; minimal new opcodes
• ✅ Heartbeat already fires every 3s — convergence detected promptly
• ❌ Need capnp schema change if RequestReply needs new fields
• ❌ Primary must process heartbeat replies (currently fire-and-forget)

Option B: Use CtrlHelloReply periodically

Send CtrlHelloRequest periodically (not just on connect) and use the existing CtrlHelloReply fields:

• Already carries committedLsn, configTag in reply
• Already carries appliedLsn, configTag in request
• Why: No capnp schema changes; ctrlhello already carries all needed fields

Trade-offs:

• ✅ No schema changes
• ❌ CtrlHelloRequest triggers WAL streaming (heavy) — not suitable for periodic use
• ❌ Semantic mismatch: ctrlhello is a connection-setup message, not a state-reporting message

Option C: New lightweight opcode

Add OPCode_stateReport / OPCode_stateAck to capnp schema:

• Dedicated message type with (applied_lsn, config_tag, in_catch_up) fields
• Sent by standby in response to heartbeat, or periodically
• Why: Clean separation of concerns

Trade-offs:

• ✅ Clean protocol design
• ❌ New opcode, new capnp struct, new handler on both sides

4. Effort and Risk Summary

Req	Effort	Risk	Justification
1 — Heartbeat state report	S (1-2d)	Low	Extend existing reply; capnp schema change if needed
2 — Per-standby tracking	S (1d)	Low	New fields in ha_state_t or extend standby_conn_t
3 — WAL catch-up wiring	S (1d)	Low	Wire stub to existing apply_wal_entry()
4 — Snapshot transfer	M (3-5d)	Medium	persist_checkpoint exists; transfer protocol is new
5 — Batch WAL transfer	S (1-2d)	Low	Accumulate entries in loop; capnp schema change
6 — COMMITTING gate	S (1d)	Low	Replace ha_repl_ack_received with heartbeat-driven check
7 — Observability	S (alongside)	Low	INFO/WARN log statements

Total estimated effort: 1-2 weeks. Risk: Low-Medium (snapshot transfer is the main risk).

5. Key Reusable Assets

Asset	Location	Reuse in
OPCode_nop send/reply loop	cluster.c:225, ha.c:2880	Req 1 (heartbeat state report)
CtrlHelloReply fields	controller.capnp:835-846	Req 1 (capnp schema template)
standby_conn_t + repl_wait_majority_ack	repl.h:22, repl.c:338	Req 2 (convergence tracking)
apply_wal_entry()	persist.c:685	Req 3 (WAL catch-up)
persist_checkpoint()	persist.c:610	Req 4 (snapshot creation)
WAL streaming loop	config_repl.c:1481	Req 5 (batch send)
ha_repl_ack_received()	ha.c:2197	Req 6 (quorum math reuse)
INFO/WARN macros	xcelog.h	Req 7 (observability)

6. Research Needed (Design Phase)

• Capnp schema design: Whether to extend RequestReply with state fields, add a new stateReport struct to existing heartbeat messages, or create a new opcode. Evaluate against existing capnp codegen patterns.
• Heartbeat reply processing: Currently fire-and-forget; need to add a reply handler on the primary side. Determine best insertion point in the dispatch path.
• Snapshot wire format: How to serialize persist_checkpoint output for transfer. COW zarray vs raw file transfer.

---

Cross-Spec Gap Audit (2026-06-08)

Batch /kiro-validate-gap across 9 specs on persistent_ha branch.

Analysis Summary

• ha-repl-hardening is ~98% complete. All 6 task groups (1–6), 27 subtasks (1.1–6.7) are [x] in tasks.md. Recent commits (aab41fa through e1517bb) confirm active implementation. The git log shows all tasks through 6.7 implemented with bug fixes applied.
• Remaining gaps: Task groups 1–4 and 5–6 all show [ ] as section headers, but all subtasks within them are [x]. The section checkboxes are cosmetic.
• Deferred items: None. The original research's "Research Needed" items (capnp schema design, heartbeat reply processing, snapshot wire format) were resolved during the design phase.
• Cross-spec relationships: This spec extends ha-config-bootstrap Req 5 (majority-commit WAL) and Req 8 (majority-commit config) with heartbeat-driven convergence. It depends on ha-heartbeat-liveness for the idle-triggered NOP policy on the primary and activity-based liveness detection on the standby. It depends on ha-peer-connection-reuse for the promoted connection model — handle_ctrlhello was reduced to connection setup only (task 5.1), with all replication decisions moved to the heartbeat decision loop.
• Single-node mode: Verified working — n_peers=1, immediate convergence, CTRL_PRIMARY with no heartbeat wait.
• Recommendation: Run /xce-build controller to confirm ASAN-clean build, then mark the spec implemented.

---

Gap Analysis — Validation Pass (2026-06-09)

Validation-phase gap analysis: requirements.md vs code as on disk (including uncommitted changes to ha.c/ha.h/myapp.c/raft.c). All 27 subtasks in tasks.md are [x]; this pass verifies what the code actually does against each acceptance criterion.

Headline Finding

The control plane is complete; the data plane is not. Every protocol mechanism in this spec (heartbeat stateReport, per-standby tracking, decision loop, batch framing, snapshot chunking, convergence-gated promotion, safe commit point) is implemented and unit/integration tested. But payload bytes are never transmitted or applied:

• ha_send_snapshot_to_standby sends every chunk with chunk.payload = NULL — config_repl.c:1638 (/* encoding deferred */, explained at config_repl.c:1613-1618: the capnp Data ↔ xce_data_t tagged-union encode path is not wired).
• The standby snapshot handler passes payload_bytes=NULL, payload_len=0 — cluster.c:191-197. Consequently snap.part is never written, the is_last size check (ha.c:1094-1106) fails (or stat fails on a missing file), and persist_load_snapshot is never reached in production. Snapshot transfer works only in unit tests via xce_snapshot_chunk_hook.
• WAL catch-up applies entries header-only: ha_apply_single_entry (ha.c:974-984) builds payload_len=0 and calls apply_wal_entry(p, &hdr, NULL). apply_wal_entry (persist.c:691) then only advances the LSN chain — payload-bearing entry types (WAL_JOB_SUBMIT → capnp job, jobstate entries → wal_jobstate_t) restore nothing.

Safety implication: a standby's applied_lsn advances without the corresponding state, so it reports "converged" via stateReport, the primary promotes out of COMMITTING, and a later failover to that standby would serve from a state missing all payload-bearing entries. The Majority Consistency Invariant (design §Overview) is currently tracked but not actually established.

Cross-spec constraint: the live replication path has the same hole — repl_broadcast (repl.c:193-234) discards payload and sends nothing ("TCP send wiring deferred to task 7.6", owned by ha-config-bootstrap). Today no WAL payload bytes move between controllers on any path.

Requirement-to-Asset Map

Req / AC	Asset (file:line)	Status
1.1 heartbeat reply carries (tag, lsn)	cluster.c:273-287 OPCode_nop handler; ha_nop_reply_state_report ha.c:1140; StateReport controller.capnp:788	Implemented
1.2 primary records + evaluates	repl_ack_decode_and_dispatch ha.c:2456-2547 → ha_process_standby_state ha.c:1198	Implemented
1.3 convergence criteria (tag≥ AND lsn≥)	ha.c:1211-1212	Implemented
1.4 tag=0 before config push	reads myconf->config_version_tag (0 pre-push)	Implemented
2.1 init tracking on COMMITTING entry	Arrays zeroed only in ha_init (ha.c:816-819 comment claims the invariant); raft_cb_become_leader (ha.c:445-529) resets repl_ack_count but not standby_converged[]/standby_reported_*[] → stale convergence flags survive a demote→re-elect cycle	Partial
2.2 update + re-evaluate on reply	ha.c:1198-1223, 2525-2533	Implemented
2.3 majority threshold ceil(n/2)−1	ha_check_majority_convergence ha.c:1244-1278 requires all standbys (n_converged >= n_peers−1), not majority. Three conflicting formulas: requirements.md 2.3 = ceil(n/2)−1 (1 standby for n=3); tasks.md 5.2 = ceil(n/2) (2 for n=3); code = n−1 (2 for n=3, 4 for n=5)	Constraint / deviation (stricter than spec — safe but availability hit)
2.4 INFO on promotion	ha.c:1266-1267	Implemented
2.5 regression → not-converged	ha.c:1210-1213	Implemented
3.1 decode + apply WAL entry	Handler wired (cluster.c:161-173 → ha_apply_wal_catchup_entry ha.c:986) but applies header-only, payload dropped (ha.c:974-984)	Partial (CRITICAL)
3.2 advance applied_lsn	ha_advance_applied_lsn ha.c:960 (atomic, monotonic)	Implemented
3.3 ERROR + skip on failure	ha.c:1003-1016, 1032-1037	Implemented
3.4 stays CTRL_STANDBY, no job ops	ha_dispatch_gate CTRL_STANDBY case ha.c:1456-1483	Implemented
3.5 LSN order	sequential batch iteration	Implemented
4.1 create checkpoint + transfer	ha_send_snapshot_to_standby config_repl.c:1578 chunks an existing snap-{checkpoint_lsn}.capnp — never calls persist_checkpoint() to create a fresh one; payload=NULL	Partial
4.2 load snapshot, set applied_lsn	ha_handle_snapshot_chunk ha.c:1058-1131 complete (verify/rename/load/applied_lsn) but unreachable in production (NULL payload, cluster.c:196)	Partial
4.3 no job ops until loaded	dispatch gate (standby role unchanged)	Implemented
4.4 retry via reconnect	Restart-on-reconnect documented (ha.c:1054, cluster.c:178-180) but no production code unlinks snap.part on reconnect — stale partial file breaks the next transfer's size check	Partial
4.5 single-node: no snapshot	no standbys → decision loop never fires	Implemented
5.1 multiple entries per message	entries :List(WalCatchupEntry) controller.capnp:850; send_wal_batch config_repl.c:1409	Implemented
5.2 configurable max batch size	WAL_STREAM_BATCH_ENTRIES=256 / WAL_STREAM_BATCH_BYTES=64KB are compile-time #defines, not runtime config	Implemented (minor constraint)
5.3 apply in order, advance after batch	ha.c:992-1027 (last_good_lsn post-batch advance) — but payload gap applies	Partial
5.4 partial batch sent immediately	config_repl.c:1556-1563	Implemented
5.5 ordering + durability	Ordering yes; "durably written" weak — entities restored in-memory, no fsync on standby (applied_lsn in-memory-only is a documented design choice), and payload gap applies	Partial
6.1 heartbeats continue in COMMITTING	self-re-arming heartbeat task (fix e1517bb)	Implemented
6.2 NOT_READY for non-repl opcodes	ha_dispatch_gate CTRL_PRIMARY_COMMITTING allowlist ha.c:1436-1454	Implemented
6.3 immediate promotion on majority	ha.c:2529-2533 (before repl dispatch) — subject to 2.3 threshold deviation	Implemented (constraint)
6.4 timeout → WARN + degraded mode	committing_timeout_fn ha.c:2885-2914; default 60000 (HA_DEFAULT_COMMITTING_TIMEOUT_MS ha.h:131)	Implemented
6.5 single-node immediate	n_peers=1 → threshold 0 (ha.c:1261-1264; fixed in e10cf9a)	Implemented
7.1 catch-up start INFO	config_repl.c:1445-1447	Implemented
7.2 converged INFO	ha.c:1216-1222	Implemented
7.3 snapshot start/complete INFO	config_repl.c:1595, 1663	Implemented
7.4 majority INFO	ha.c:1266	Implemented
7.5 timeout WARN w/ counts	ha.c:2905-2908	Implemented
Flow 0 safe commit point (tasks 4.1/4.2)	ha_compute_safe_committed_lsn ha.c:373-432; truncation + committed baseline in raft_cb_become_leader ha.c:456-485	Implemented

Score: 26 of 33 acceptance criteria fully implemented. Requirements 1, 6, 7 (and Flow 0 safety) Implemented; Requirement 2 Partial (init-reset + threshold semantics); Requirements 3, 4, 5 Partial — all blocked on the same payload data-plane gap.

Uncommitted working-tree changes (ha.c/ha.h/myapp.c/raft.c/deploy-ha.sh) are election-protocol hardening adjacent to this spec: ha_init_raft one-time init (prevents raft state reset on re-election), term-scoped candidate-agnostic vote denial in raft_on_vote_request with idempotent re-grant, startup self-vote/stale-vote handling, and watchdog arming on the skip-election path. These strengthen Election Safety per raft-tla-audit.md and should be committed, but they do not close any of this spec's acceptance criteria.

Implementation Approach Options (remaining gaps)

Gap 1 — Payload encoding for WalCatchupEntry.payload and SnapshotTransfer.payload (the blocker)

Option A — Extend the typemap codegen to map capnp Data ↔ raw buffer for these structs. The schema already declares payload :Data on both structs; only the C-side encode/decode of the xce_data_t tagged union is missing.

• ✅ Fixes both messages at once; follows the established schema/typemap convention; benefits any future Data-carrying message
• ❌ Touches generated-code path — regression risk across all opcodes; needs codegen expertise

Option B — Hand-rolled encode/decode for just these two opcodes. Bypass the tagged union: serialize lsn/type/len/bytes with direct capn calls (or a length-prefixed raw layout) in send_wal_batch / ha_send_snapshot_to_standby and their decode counterparts.

• ✅ Localized, no codegen risk, fastest path to a working data plane
• ❌ Two bespoke wire paths diverging from every other message; long-term maintenance debt

Option C — Phased hybrid (recommended). Phase 1: hand-wire the snapshot payload (raw file bytes — trivially a byte stream, no per-entry structure) to make Req 4 production-real. Phase 2: solve Data encoding properly in the typemap for WalCatchupEntry (structured, batched, used on the hot path) and migrate the snapshot path to it. Until both land, keep the spec phase below implemented or gate promotion on a data-plane flag.

• ✅ De-risks incrementally; snapshot (rare, big) and WAL (frequent, small) have different shapes
• ❌ Two-step migration; temporary inconsistency between the two paths

Gap 2 — Convergence threshold semantics (Req 2.3 vs code)

• Option A: change code to the requirement's majority formula (ceil(n_peers/2) − 1 standbys). Restores availability (one dead standby in a 3-node cluster no longer forces the 60s degraded timeout) and matches Raft quorum; safe because safe_committed_lsn already derives from a majority.
• Option B: keep all-standbys convergence and amend requirements.md + tasks.md with the rationale (strongest consistency at startup; degraded timeout covers minority loss).
• Either way, reconcile the three formulas (requirements 2.3, task 5.2, code) — they currently disagree for any n ≥ 5.

Gap 3 — Small correctness fixes (extend existing components)

1. Zero standby_converged[] / standby_reported_*[] in raft_cb_become_leader on COMMITTING entry (Req 2.1 — re-election staleness).
2. unlink(snap.part) on standby reconnect/ctrlhello (Req 4.4 — stale partial file).
3. Create a fresh checkpoint (via delaymgr — persist_checkpoint is blocking I/O, must not run in the vtask/heartbeat-reply context) when SEND_SNAPSHOT fires and the on-disk snapshot is older than oldest_wal_lsn coverage requires (Req 4.1).
4. Consider caching ha_compute_oldest_wal_lsn — currently an opendir/readdir scan per heartbeat reply per standby in vtask context (ha.c:2538-2543).
5. Optionally promote batch limits to ha_config_t (Req 5.2 "configurable").

Effort and Risk

Gap	Effort	Risk	Justification
1 — Payload data plane (Option C)	M–L (5–10d)	Medium	Snapshot raw-bytes path is S; typemap Data codegen + batched WAL payload apply + ASAN-clean lifetime mgmt (current (xce_data_t *)pbuf cast at config_repl.c:1529 is already suspect) + E2E re-verification is the bulk
2 — Threshold reconciliation	S (0.5–1d)	Low	One comparison + spec text alignment; existing tests pin the 3-node case
3.1 — Tracking reset on COMMITTING	S (<0.5d)	Low	memset 3 arrays at ha.c:510; add re-election unit test
3.2 — snap.part discard	S (<0.5d)	Low	unlink in handle_ctrlhello standby path
3.3 — Fresh checkpoint on SEND_SNAPSHOT	M (2–3d)	Medium	Blocking checkpoint must hop to delaymgr; interaction with concurrent WAL writes
3.4 — oldest_wal_lsn caching	S (0.5d)	Low	Cache + invalidate on segment GC
3.5 — Configurable batch limits	S (0.5d)	Low	Plumb two fields through ha_config_t

Recommendations

1. Do not mark this spec implemented. All tasks are [x] but Requirements 3, 4, 5 are only control-plane complete; the system currently reports convergence it has not achieved. Reopen targeted tasks (suggest: 7.x series) for the payload data plane rather than re-flagging 2.x/3.5.
2. Land Gap 3.1 and 3.2 immediately (S, Low) — they are latent correctness bugs independent of the payload work.
3. Decide Gap 2 (threshold) explicitly and record it in the Decision Log; today three documents disagree.
4. Sequence the payload work after coordinating with ha-config-bootstrap task 7.6 (repl_broadcast TCP wiring) — the live path and catch-up path should share one entry serialization, or the two specs will produce divergent wire formats for the same WAL entry.
5. Add an E2E assertion that verifies entity state (e.g., a submitted job visible on the promoted standby), not just applied_lsn equality — the current tests 6.5–6.7 pass despite the data-plane gap because they only observe LSN/log convergence.

Research Needed

• How the typemap codegen represents capnp Data (xce_data_t tagged union table dispatch) and what it takes to encode an arbitrary byte buffer — prerequisite for Gap 1 Option A/C phase 2.
• Whether convergence should additionally verify content (e.g., last-entry CRC chained via wal_entry_hdr_t.crc/prev_crc in stateReport) so header-only application can never report false convergence again.
• Ownership boundary with ha-config-bootstrap task 7.6: which spec owns live WAL entry serialization, and can send_wal_batch be reused for it.
• Req 5.5 "durably written" on the standby: is in-memory entity restore + in-memory applied_lsn acceptable (current documented design), or should batch apply fsync a standby-side WAL copy.

---

Gap Re-Audit (2026-06-18)

Post-implementation gap audit. spec.json phase=implemented; all 41 sub-tasks [x] (1.1–7.6 incl. the reopened payload data-plane series). This pass re-verifies every EARS criterion against the code as on disk — specifically whether the two CRITICAL data-plane gaps and the small correctness gaps from the 2026-06-09 pass are now closed, and whether the COMMITTING-gate requirements still describe the live code after ha-single-leader-role landed.

Headline Finding

Both 2026-06-09 CRITICAL blockers are CLOSED. The data plane is real.

• WAL catch-up now applies payload-bearing entries. ha_apply_single_entry (ha.c:1558) does persist-before-apply: persist_wal_write_replicated(p, e->lsn, type, e->raw_payload, len) (fsync, no re-broadcast, idempotent) THEN apply_wal_entry(p, &hdr, e->raw_payload). apply_wal_entry (persist.c:1305) restores entities by type — WAL_JOB_SUBMIT→restore_job, job-state→restore_job_state, WAL_WORKER_CTRL→worker_from_capnp, WAL_QUEUE_CTRL→restore_queue_state (persist.c:1342-1389). Header-only application is gone.
• Snapshot payload bytes are transmitted. ha_send_snapshot_to_standby (config_repl.c:1857) stats the real file, chunks 64KB, and ships each chunk via xce_snapshot_chunk_req_raw(ep, &chunk, buf, chunk_len) (config_repl.c:1928) — raw Data bytes, not NULL. Standby ha_handle_snapshot_chunk (ha.c:1683→ via handle_snapshot_transfer cluster.c:196, dispatched at cluster.c:589) writes chunks to snap.part, verifies total size on is_last, atomic-renames to snap-{lsn}.bin, loads via persist_load_snapshot, sets applied_lsn.

The Majority Consistency Invariant is now actually established, not merely tracked. A standby that reports converged holds the entity state behind its applied_lsn.

Architectural supersession (read before re-opening any 6.x task)

CTRL_PRIMARY_COMMITTING no longer exists — grep -rn CTRL_PRIMARY_COMMITTING src/controller returns zero hits. ha-single-leader-role removed the transitional COMMITTING role: raft_cb_become_leader now stores CTRL_PRIMARY directly (ha.c:1034) with the comment "there is no transitional COMMITTING state." New-leader safety is provided by a different, stronger mechanism than this spec's heartbeat-convergence gate:

1. Own-term WAL_NOOP + writable_lsn gate (ha_arm_own_term_gate ha.c:2259) — every WAIT completion is withheld until committed_lsn >= own-term noop LSN, i.e. until this term has committed an entry on a majority.
2. Per-operation write quorum (ha_dispatch_gate ha.c:2061 → ha_has_write_quorum ha.c:2031) — state-changing opcodes return -XCE_ERR_NOT_READY unless repl_active_standby_count >= n_peers/2; replication and read-only/fact-bearing opcodes are always admitted (allowlist at ha.c:2078-2106).
3. Vote-derived safe commit point (ha_compute_safe_committed_lsn ha.c:480, called from raft_cb_become_leader ha.c:908) — safe_committed_lsn = majority-threshold element of granted voters' LSNs; persist_truncate_wal_above(safe_lsn) + committed_lsn=safe_lsn (ha.c:914).

Net effect: the intent of Req 6 (don't serve un-replicated state; gate on majority) is met more strongly and continuously than the original timeout-bounded COMMITTING design — it is now enforced per-operation for the life of the term, not just at startup. The original Req 2.3 / 6.1–6.4 wording (COMMITTING role, 60s committing_timeout_ms degraded fallback, immediate COMMITTING→PRIMARY on majority) describes a design that was deliberately replaced. This is a documentation drift, not a functional gap.

Requirement-to-Asset Map (re-verified)

Req / AC	Asset (file:line)	Status
1.1 heartbeat reply carries (tag,lsn)	OPCode_nop handler cluster.c:300-301; populates from myconf->config_version_tag + applied_lsn	CLOSED
1.2 primary records + evaluates	repl_ack_decode_and_dispatch → ha_process_standby_state ha.c:1851	CLOSED
1.3 convergence criteria tag≥ AND lsn≥	ha.c:1864-1865	CLOSED
1.4 tag=0 before config push	reads live config_version_tag (0 pre-push)	CLOSED
2.1 init tracking on term entry	FIXED — raft_cb_become_leader memsets standby_reported_tag/_lsn/_converged (ha.c:1043-1045) on CTRL_PRIMARY entry. Closes 2026-06-09 Gap 3.1.	CLOSED
2.2 update + re-evaluate on reply	ha_process_standby_state ha.c:1851-1876	CLOSED
2.3 majority threshold	RECONCILED to n_peers/2 (C int div) per task 7.5 + Decision Log 2026-06-09. ha_has_write_quorum uses connected >= n_peers/2 (ha.c:2058); safe_committed_lsn uses n_granted/2 (ha.c:527). Three-doc disagreement resolved.	CLOSED
2.4 INFO on promotion/converged	ha.c:1869-1875 (per-standby converged INFO)	CLOSED
2.5 regression → not-converged	ha.c:1863-1866 (recomputed every reply)	CLOSED
3.1 decode + apply WAL entry with payload	FIXED — ha_apply_single_entry ha.c:1558-1589 passes raw_payload to apply_wal_entry; entities restored persist.c:1342-1389. Closes 2026-06-09 CRITICAL.	CLOSED
3.2 advance applied_lsn	ha_advance_applied_lsn ha.c:1541 (atomic monotonic); batch advances after full batch (ha.c:1656-1659)	CLOSED
3.3 ERROR + skip on failure	ha.c:1633-1647, 1666-1672 (NULL/failed sub-entries skipped, no crash)	CLOSED
3.4 stays STANDBY, no job ops	ha_dispatch_gate STANDBY case	CLOSED
3.5 LSN order	sequential batch iteration + persist-before-apply	CLOSED
4.1 fresh checkpoint + transfer	FIXED — ha_dispatch_repl_action_for_standby calls persist_checkpoint(&c->persist) before send (ha.c:1961), -EBUSY falls back to existing. Closes 2026-06-09 Gap 3.3.	CLOSED
4.2 load snapshot, set applied_lsn	ha_handle_snapshot_chunk ha.c:1683 (verify→rename→persist_load_snapshot→applied_lsn); reachable in production (real payload)	CLOSED
4.3 no job ops until loaded	dispatch gate (standby role)	CLOSED
4.4 retry via reconnect / discard stale	FIXED (cleaner than proposed) — chunk_index==0 unlinks any stale snap.part (ha.c:1693, 1708-1713) instead of relying on a reconnect-time unlink. Verified by test_snapshot_recv_reconnect_discards_snappart (test_config_repl.c:2045). Closes 2026-06-09 Gap 3.2.	CLOSED
4.5 single-node: no snapshot	no standbys → decision loop never fires	CLOSED
5.1 multiple entries/message	entries :List(WalCatchupEntry) + send_wal_batch (config_repl.c)	CLOSED
5.2 configurable max batch	WAL_STREAM_BATCH_ENTRIES=256 / _BYTES=64KB compile-time #defines	CLOSED (minor: limits are compile-time, not runtime ha_config_t — Req says "configurable maximum"; defensible)
5.3 apply in order, advance after batch	ha_apply_wal_catchup_entry ha.c:1615-1681 (last_good_lsn post-batch advance) — now with real payload	CLOSED
5.4 partial batch sent immediately	config_repl.c batch flush on range end	CLOSED
5.5 ordering + durability	Ordering preserved; durability now STRONGER than original design — ha_apply_single_entry fsyncs via persist_wal_write_replicated BEFORE apply/ACK (ha.c:1570). Supersedes the "in-memory only" caveat for catch-up.	CLOSED (exceeds spec)
6.1 heartbeats continue / gate on majority	superseded: continuous per-op ha_has_write_quorum (ha.c:2031) replaces COMMITTING-only check	CLOSED (via supersession)
6.2 NOT_READY for non-repl opcodes	ha_dispatch_gate ha.c:2061-2110 (allowlist + -XCE_ERR_NOT_READY default) — now enforced for the whole term, not just COMMITTING	CLOSED (stronger)
6.3 immediate promotion on majority	superseded by own-term gate + safe_committed_lsn (no COMMITTING dwell to exit)	CLOSED (via supersession)
6.4 timeout → WARN + degraded	OBSOLETE — no COMMITTING state ⇒ no committing_timeout_ms. Degraded operation handled by write-quorum gate (rejects writes, keeps serving reads) rather than a one-shot 60s promotion fallback.	N/A (superseded)
6.5 single-node immediate	ha_has_write_quorum returns true when n_peers<=1 (ha.c:2037); safe_committed_lsn single-node path ha.c:486	CLOSED
7.1 catch-up start INFO	config_repl.c (send_wal_batch start log)	CLOSED
7.2 converged INFO	ha.c:1869-1875	CLOSED
7.3 snapshot start/complete INFO	config_repl.c:1874-1876, 1939-1941	CLOSED
7.4 majority INFO	per-standby converged INFO (ha.c:1870); promotion is direct-to-PRIMARY INFO (ha.c:1036)	CLOSED
7.5 timeout WARN	OBSOLETE with COMMITTING removal; quorum-loss edge logged once via ha_log_quorum_transition (ha.c:2076) instead	N/A (superseded by quorum-transition logging)
Flow 0 safe commit point	ha_compute_safe_committed_lsn ha.c:480-540; truncate+baseline raft_cb_become_leader ha.c:898-923	CLOSED

Score: 31/33 original acceptance criteria CLOSED; 2 (6.4, 7.5) OBSOLETE by deliberate architectural supersession (CTRL_PRIMARY_COMMITTING removed by ha-single-leader-role). Counting the superseded pair as satisfied-by-replacement (their intent — gate on majority, observe degraded mode — is met more strongly), effective coverage is 33/33. No CRITICAL or HIGH functional gap remains.

Residual / Deferred Items (all LOW, none blocking)

1. Documentation drift (HIGH-doc, LOW-code). requirements.md Req 6 + design.md still describe CTRL_PRIMARY_COMMITTING, committing_timeout_ms, and the 60s degraded-mode timeout, none of which exist in code. The Introduction (requirements.md:5-7) and Adjacent-expectations also reference COMMITTING. A reader of the spec will mis-model the live FSM. Code is correct; the prose is stale.
2. Req 5.2 "configurable" batch size is compile-time (#define), not runtime config. Minor; no operator has needed to tune it. Defensible as "configurable at build."
3. ha_compute_oldest_wal_lsn is an opendir/readdir scan per heartbeat reply per standby in vtask context (config_repl.c:1946). Performance-only; fine at current cluster sizes. The 2026-06-09 caching suggestion (Gap 3.4) was never actioned and remains optional.
4. No covering tests flagged by codegraph for ha_process_standby_state, ha_evaluate_repl_action, ha_dispatch_repl_action_for_standby, ha_send_snapshot_to_standby at the unit level (decision-loop logic is exercised via hooks in test_ha_role.c / test_config_repl.c, and convergence via test_convergence_verification.c). Coverage is integration-weighted, not absent — but the pure decision functions lack direct unit pins.
5. E2E entity-state assertion (2026-06-09 Recommendation 5): tests 6.5–6.7 historically asserted LSN convergence only. With the data plane now real this is the right place to add a "submitted job visible on promoted standby" assertion to guard against future regressions to header-only application. Not done; LOW (data plane verified by unit round-trips per spec.json note).

Effort / Risk for Open Work

Item	Option	Effort	Risk
1 — Spec/design doc reconciliation (remove COMMITTING; document own-term gate + write-quorum + safe_committed_lsn as the actual Req 6 mechanism)	A (recommended): update requirements.md Req 6 + Introduction + design.md to describe the superseding mechanism and cross-link ha-single-leader-role / ha-commit-durability. B: leave a one-line "SUPERSEDED" banner pointing to those specs.	A: S (0.5–1d) · B: XS (<0.5d)	Low (docs only)
2 — Promote batch limits to ha_config_t	A: plumb two fields. B: leave as build-time.	S (0.5d)	Low
3 — Cache oldest_wal_lsn	A: cache + invalidate on segment GC. B: defer (current scale fine).	S (0.5d)	Low
4 — Direct unit tests for decision functions	A: add ha_evaluate_repl_action truth-table + ha_process_standby_state convergence-flag tests. B: rely on hook/integration coverage.	S (1d)	Low
5 — E2E entity-state assertion	A: extend test 6.7 to assert a submitted job is present post-failover. B: rely on unit payload round-trips.	S (0.5–1d)	Low

Recommendation: the spec is functionally complete and may remain implemented. The only item worth scheduling is Item 1 (doc reconciliation) — the requirements/design now misdescribe the live FSM after ha-single-leader-role, which will mislead future readers and any /kiro-verify-completion run that checks prose against code. Items 2–5 are optional hardening.

---

Gap Analysis (re-validation) — 2026-06-24

Re-validation pass. spec.json phase=implemented (all 41 sub-tasks [x]). This pass re-verifies every EARS criterion against the code as on disk after the heavy ha-term-safety rework (commits 004a40b…b77d412, tasks 1.1–4.3) plus bug-828/991/1028/1060/1067 fixes landed since the 2026-06-18 audit. Question answered: does this spec still hold, and how much of it is now owned by sibling specs?

Headline Finding

Still functionally complete; now LARGELY SUPERSEDED in mechanism, but its data-plane code is the live load-bearing path. Two distinct things are true at once:

1. The control-plane design of this spec (Req 2.3 / Req 6: CTRL_PRIMARY_COMMITTING role + 60 s committing_timeout_ms degraded fallback + COMMITTING→PRIMARY-on-majority) is GONE. grep -rn CTRL_PRIMARY_COMMITTING src/controller/*.{c,h} → 0 hits; committing_timeout → 0 hits. Removed by ha-single-leader-role and replaced by a stronger, continuous mechanism (own-term WAL_NOOP + writable_lsn gate ha_arm_own_term_gate ha.c:107/1206; per-op write quorum ha_has_write_quorum ha.c:2781 → -XCE_ERR_NOT_READY; vote-derived ha_compute_safe_committed_lsn ha.c:510). This is documentation drift, not a functional gap — identical to the 2026-06-18 finding and still unactioned.

2. The data-plane code of this spec (Req 1,3,4,5,7) is intact, hardened, and is the path every sibling spec builds on. It was NOT rewritten away — ha-term-safety routes through it:

   • WAL catch-up payload apply: ha_apply_single_entry ha.c:2044 → persist_wal_write_replicated (fsync, idempotent) ha.c:2056 THEN apply_wal_entry ha.c:2081 (entity restore). Config markers also flow here (ha.c:2168). Header-only application remains gone.
   • Batched streaming: send_wal_batch config_repl.c:2005, WAL_STREAM_BATCH_ENTRIES=256 / _BYTES=64 KB (config_repl.c:1981-1982); batch flush loop config_repl.c:2071-2173.
   • Snapshot raw payload: ha_send_snapshot_to_standby config_repl.c:2425 → xce_snapshot_chunk_req_raw config_repl.c:2496 (real bytes).
   • Per-standby convergence: ha_process_standby_state ha.c:2592 (sets standby_reported_tag/_lsn/_converged, ha.c:2600-2607); reset on term entry ha.c:1115-1117; fed from OPCode_reply stateReport at ha.c:4620.
   • safe_committed_lsn (this spec's Flow 0, tasks 4.1/4.2) is explicitly cited by code as ha-repl-hardening: ha.c:940 "Task 4.1 (ha-repl-hardening)".

   bug-1028 (FIXED 2026-06-23) was inside this very streaming path (ha_stream_wal_to_standby gained a const persist_t *p param) — confirming the path is live and maintained, not dead.

Which sibling spec now owns what

Concern	Originating in ha-repl-hardening	Now owned / superseded by
Heartbeat stateReport (Req 1)	yes	retained as-is; consumed by ha-heartbeat-committed-propagation (adds committed+term stamp)
Per-standby convergence tracking (Req 2.1/2.2/2.5)	yes	retained as-is
Majority threshold n_peers/2 (Req 2.3)	yes	reconciled + absorbed into ha_has_write_quorum / safe_committed_lsn (raft-commit-safety, ha-term-safety)
WAL catch-up payload apply (Req 3)	yes	retained; now also drives ha-term-safety config-marker apply + bug-991 lsn-advance fix
Snapshot transfer (Req 4)	yes	retained as-is
Batch WAL transfer (Req 5)	yes	retained; bug-1028 hardened the streamer signature
COMMITTING gate (Req 6)	yes	SUPERSEDED & DELETED by ha-single-leader-role (own-term gate) + ha-commit-durability (durable-commit fence)
Flow 0 safe commit point (tasks 4.1/4.2)	yes	retained; extended by raft-commit-safety (Figure-8) + ha-term-safety (per-entry term, (term,lsn) reconcile) — the LSN-only safe point is now term-aware upstream

Requirement-to-Asset Map (delta vs 2026-06-18 — only changes noted)

All 31 CLOSED criteria from 2026-06-18 remain CLOSED (assets re-verified, line numbers shifted +~860 in ha.c / +~560 in config_repl.c due to ha-term-safety growth; spot-checked above). No regression.

Req / AC	Change since 2026-06-18	Status
3.x payload apply	Path unchanged in shape; bug-991 fix made the three live-apply setters advance applied_lsn to the LAST-held LSN (correctness hardening).	CLOSED (hardened)
5.x batch streaming	bug-1028 added const persist_t *p to ha_stream_wal_to_standby (off-by-one / hdr-source fix).	CLOSED (hardened)
Flow 0 safe commit	Now also gated by ha-term-safety per-entry term + bug-1060 promotion-truncate floor max(safe_lsn,durable_tip) (prevents new leader truncating its own committed log) + bug-1067 AE header-term separation.	CLOSED (strengthened upstream)
6.4 / 7.5 COMMITTING timeout/WARN	Still OBSOLETE (no COMMITTING).	N/A (superseded)

Score: 31/33 CLOSED, 2 OBSOLETE-by-supersession (6.4, 7.5). Effective 33/33. No CRITICAL/HIGH functional gap. Unchanged from 2026-06-18, now re-confirmed post ha-term-safety.

Disposition: RESCOPE (close as data-plane-complete + supersede the control-plane prose)

This spec should not be reopened for new feature work and should not be deleted. Recommended disposition:

• Keep phase=implemented. Every acceptance criterion is satisfied or satisfied-by-replacement.
• CLOSE-as-superseded the control plane (Req 2.3 + Req 6 entirely). Their mechanism was deliberately replaced. Carry a one-line "SUPERSEDED by ha-single-leader-role / ha-commit-durability" banner on Req 6 and on the Introduction's COMMITTING references.
• RETAIN the data plane (Req 1,3,4,5,7 + Flow 0) as the canonical owner. No sibling spec re-implements WAL-catch-up payload apply, batching, snapshot transfer, or stateReport — they all consume this code. If anything, this spec is now the foundation, not the obsolete layer.

The 2026-06-09 project-memory note ("data plane MISSING, critical, bugs 335-338") is stale — it was closed by the reopened 7.x payload series (audited 2026-06-18) and is unaffected by later work.

Residual / Deferred (all LOW, none blocking — unchanged + re-confirmed)

1. Documentation drift (HIGH-doc, LOW-code) — STILL OPEN. requirements.md Req 6 + Introduction + design.md describe CTRL_PRIMARY_COMMITTING / committing_timeout_ms / 60 s degraded fallback, none of which exist. This is the only item worth scheduling. A /kiro-verify-completion run that checks prose-vs-code will flag Req 6 falsely.
2. Req 5.2 "configurable" batch size is compile-time #define, not ha_config_t. Defensible.
3. ha_compute_oldest_wal_lsn opendir/readdir per heartbeat reply per standby. Perf-only.
4. Direct unit tests for ha_process_standby_state / ha_evaluate_repl_action / ha_send_snapshot_to_standby — coverage is integration-weighted (test_config_repl.c, test_convergence_verification.c), not absent.
5. E2E entity-state assertion (submitted job visible on promoted standby post-failover) — the live HA soak (/xce-test-ha) now exercises this end-to-end, partially discharging the gap; a pinned unit/E2E assertion is still nice-to-have.

Effort / Risk for the one schedulable item

Item	Option	Effort	Risk
Doc reconciliation: remove COMMITTING from requirements.md Req 6 + Introduction + design.md; document own-term gate + write-quorum + safe_committed_lsn as the real Req 6 mechanism; cross-link ha-single-leader-role / ha-commit-durability / ha-term-safety	A (recommended): rewrite Req 6 + Introduction (S, 0.5–1d). B: one-line SUPERSEDED banner (XS, <0.5d).	A: S · B: XS	Low (docs only)

Recommendation

1. No open implementation work. Spec is functionally complete and is the live owner of the HA replication data plane. Mark project-memory note (2026-06-09 "data plane MISSING") as resolved.
2. Schedule only the doc reconciliation (Residual #1). It has been flagged in two prior audits (2026-06-09 doc-drift, 2026-06-18 Item 1) and remains unactioned. Either rewrite Req 6 (Option A) or banner it (Option B). Everything else is optional hardening.
3. Treat this spec as RESCOPED: control plane CLOSED-as-superseded; data plane RETAINED-as-owner.

Research Needed

• None new. Prior research items (typemap Data encoding, snapshot wire format, ownership boundary with ha-config-bootstrap task 7.6 / live repl_broadcast) were all resolved by the payload series and ha-term-safety task 3.4 (repl_broadcast now carries leader term, term-0 live send retired).