myxce/.kiro/specs/ha-repl-hardening/research.md

# Gap Analysis: HA Replication Hardening

> Analysis date: 2026-06-04. Phase: `requirements-generated`.
> Purpose: identify what exists, what's missing, and viable implementation approaches.

## Analysis Summary

- **Scope**: 7 requirements — heartbeat-driven state convergence, WAL catch-up wiring, per-standby tracking, snapshot transfer, batch WAL transfer, COMMITTING gate, observability
- **Key finding**: The primary already sends heartbeats (`OPCode_nop`) and receives replies; the `CtrlHelloReply` already carries `committed_lsn` and `config_tag`; `persist_checkpoint` already exists. The gaps are in wiring, not new infrastructure.
- **Largest gaps**: (1) heartbeat reply doesn't carry standby's `applied_lsn`/`config_tag`; (2) WAL catch-up handler is a stub; (3) snapshot transfer path is a stub; (4) no per-standby convergence tracking in the primary.
- **Recommendation**: Extend existing heartbeat reply with state fields; wire the WAL catch-up stub to `apply_wal_entry()`; implement snapshot transfer using existing `persist_checkpoint()`; add per-standby convergence tracking to `ha_state_t` or extend `standby_conn_t`.

## 1. Current State Investigation

### 1.1 Heartbeat Protocol

**Current state**: Asymmetric, unidirectional state push.
- Primary sends `OPCode_nop` via `primary_heartbeat_fn` (ha.c:2880) every `heartbeat_interval_ms` (3s default)
- Standby receives nop in `myreq_handler` (cluster.c:225), sends empty `RequestReply`, calls `ha_on_heartbeat_received` (ha.c:2655) to reset liveness watchdog
- `CtrlHelloRequest`/`CtrlHelloReply` (controller.capnp:835-846) carry `appliedLsn`, `configTag`, `committedLsn` — but only exchanged once on connect, not periodically

**Gap**: Heartbeat reply is empty. Standby's current `(applied_lsn, config_tag)` is not reported back to the primary on each heartbeat cycle.

### 1.2 Per-Standby State Tracking

**Current state**: `standby_conn_t` (repl.h:22-33) tracks per-standby `acked_lsn` and `acked_config_lsn` (for WAL and config quorum), plus `in_catch_up` and `pending_ack_count`. `ha_state_t` has `standby_endpoints` (zarray of endpoints) and `n_peers`, but no per-standby convergence tracking.

**Gap**: No field for the standby's self-reported `applied_lsn` or `config_tag` from heartbeat replies. The primary has no way to know whether a standby has converged without receiving explicit WAL/config ACKs.

### 1.3 WAL Catch-Up Handler

**Current state**: Stub at ha.c dispatch gate (OPCode_walCatchupEntry returns 0). The primary-side streaming loop in config_repl.c:1489-1520 reads WAL segments and sends `OPCode_walCatchupEntry` messages, but the standby does nothing with them.

**Gap**: Standby never applies WAL catch-up entries to local state. `apply_wal_entry()` (persist.c:685) exists and is used during startup WAL replay — can be reused.

### 1.4 Snapshot Transfer

**Current state**: `handle_ctrlhello` (config_repl.c:1359) checks `repl_needs_snapshot()` (repl.c:491) and if true, logs WARN at line 1418 ("snapshot transfer not yet implemented") and skips. `persist_checkpoint()` (persist.c:610) exists and creates a consistent checkpoint.

**Gap**: The snapshot creation and transfer path is entirely unimplemented. Without it, any standby whose LSN predates the WAL retention window is permanently dead.

### 1.5 WAL Entry Batching

**Current state**: The WAL streaming loop in config_repl.c:1481-1521 sends one `xce_request` per WAL entry. `xce_request` enqueues to the endpoint's send buffer (non-blocking), but the loop processes entries sequentially with `pread()` per entry + `malloc`/`free` per payload.

**Gap**: No batching. For a standby catching up on millions of entries, this is CPU-intensive and slow. The wire protocol needs a batch message format.

### 1.6 COMMITTING Gate

**Current state**: `ha_repl_ack_received()` (ha.c:2197) exists with correct quorum math, but is only called from `repl_ack_decode_and_dispatch()` (ha.c:1859) which processes `OPCode_replCatchUpDone`. No standby sends this opcode. The committing timeout (10s) is the only path to PRIMARY.

**Gap**: The majority-convergence check is unreachable. The committing timeout degrades the cluster on every startup.

## 2. Requirement-to-Asset Map

| Req | Key Gap | Key Reusable Asset | Gap Tag |
|-----|---------|-------------------|---------|
| 1 — Heartbeat carries standby state | Reply body is empty; no state fields in nop reply | `OPCode_nop` send/receive loop, `CtrlHelloReply` fields in capnp schema, `xce_reply` pattern in cluster.c:225-228 | **Missing** |
| 2 — Per-standby convergence tracking | No `applied_lsn`/`config_tag` reported per standby | `standby_conn_t` with `acked_lsn`, `ha->standby_endpoints`, `repl_wait_majority_ack` quorum math | **Missing** |
| 3 — WAL catch-up handler | Stub returns 0 (ha.c:882) | `apply_wal_entry()` (persist.c:685), primary WAL streaming loop (config_repl.c:1489) | **Stub** |
| 4 — Snapshot transfer | Stub logs WARN, skips (config_repl.c:1417-1425) | `persist_checkpoint()` (persist.c:610), `repl_needs_snapshot()` (repl.c:491), `CtrlHelloReply.needsSnapshot` field | **Missing** |
| 5 — Batch WAL transfer | One entry per `xce_request` | Existing WAL streaming loop, capnp message format for walCatchupEntry | **Missing** |
| 6 — COMMITTING gate | `ha_repl_ack_received` unreachable; timeout-only path | `ha_repl_ack_received()` (ha.c:2197) with correct quorum math, `ha_promote_to_primary()` | **Missing** |
| 7 — Observability | Logs exist but need new convergence/snapshot events | `INFO`/`WARN` macros in xcelog.h | **Missing (alongside)** |

## 3. Implementation Approach Options

### Option A: Extend heartbeat reply with state fields (Recommended)

Extend the existing `OPCode_nop` reply or create a lightweight heartbeat-ack to carry `(applied_lsn, config_tag)`:
- **Standby side**: In the nop handler (cluster.c:225), populate reply with current `applied_lsn` and `config_tag` before `xce_reply`
- **Primary side**: In the heartbeat send path or a reply handler, read the standby's reported state and update per-standby tracking
- **Why**: Minimal new code — reuses existing heartbeat loop; no new opcodes needed if we extend the reply body; follows Raft AppendEntries pattern

**Trade-offs**:
- ✅ Reuses existing infrastructure; minimal new opcodes
- ✅ Heartbeat already fires every 3s — convergence detected promptly
- ❌ Need capnp schema change if RequestReply needs new fields
- ❌ Primary must process heartbeat replies (currently fire-and-forget)

### Option B: Use CtrlHelloReply periodically

Send `CtrlHelloRequest` periodically (not just on connect) and use the existing `CtrlHelloReply` fields:
- Already carries `committedLsn`, `configTag` in reply
- Already carries `appliedLsn`, `configTag` in request
- **Why**: No capnp schema changes; ctrlhello already carries all needed fields

**Trade-offs**:
- ✅ No schema changes
- ❌ CtrlHelloRequest triggers WAL streaming (heavy) — not suitable for periodic use
- ❌ Semantic mismatch: ctrlhello is a connection-setup message, not a state-reporting message

### Option C: New lightweight opcode

Add `OPCode_stateReport` / `OPCode_stateAck` to capnp schema:
- Dedicated message type with `(applied_lsn, config_tag, in_catch_up)` fields
- Sent by standby in response to heartbeat, or periodically
- **Why**: Clean separation of concerns

**Trade-offs**:
- ✅ Clean protocol design
- ❌ New opcode, new capnp struct, new handler on both sides

## 4. Effort and Risk Summary

| Req | Effort | Risk | Justification |
|-----|--------|------|---------------|
| 1 — Heartbeat state report | S (1-2d) | Low | Extend existing reply; capnp schema change if needed |
| 2 — Per-standby tracking | S (1d) | Low | New fields in ha_state_t or extend standby_conn_t |
| 3 — WAL catch-up wiring | S (1d) | Low | Wire stub to existing apply_wal_entry() |
| 4 — Snapshot transfer | M (3-5d) | Medium | persist_checkpoint exists; transfer protocol is new |
| 5 — Batch WAL transfer | S (1-2d) | Low | Accumulate entries in loop; capnp schema change |
| 6 — COMMITTING gate | S (1d) | Low | Replace ha_repl_ack_received with heartbeat-driven check |
| 7 — Observability | S (alongside) | Low | INFO/WARN log statements |

**Total estimated effort**: 1-2 weeks. Risk: Low-Medium (snapshot transfer is the main risk).

## 5. Key Reusable Assets

| Asset | Location | Reuse in |
|-------|----------|----------|
| `OPCode_nop` send/reply loop | cluster.c:225, ha.c:2880 | Req 1 (heartbeat state report) |
| `CtrlHelloReply` fields | controller.capnp:835-846 | Req 1 (capnp schema template) |
| `standby_conn_t` + `repl_wait_majority_ack` | repl.h:22, repl.c:338 | Req 2 (convergence tracking) |
| `apply_wal_entry()` | persist.c:685 | Req 3 (WAL catch-up) |
| `persist_checkpoint()` | persist.c:610 | Req 4 (snapshot creation) |
| WAL streaming loop | config_repl.c:1481 | Req 5 (batch send) |
| `ha_repl_ack_received()` | ha.c:2197 | Req 6 (quorum math reuse) |
| `INFO`/`WARN` macros | xcelog.h | Req 7 (observability) |

## 6. Research Needed (Design Phase)

- **Capnp schema design**: Whether to extend `RequestReply` with state fields, add a new `stateReport` struct to existing heartbeat messages, or create a new opcode. Evaluate against existing capnp codegen patterns.
- **Heartbeat reply processing**: Currently fire-and-forget; need to add a reply handler on the primary side. Determine best insertion point in the dispatch path.
- **Snapshot wire format**: How to serialize `persist_checkpoint` output for transfer. COW zarray vs raw file transfer.

---

# Cross-Spec Gap Audit (2026-06-08)

> Batch `/kiro-validate-gap` across 9 specs on persistent_ha branch.

## Analysis Summary

- **ha-repl-hardening** is **~98% complete**. All 6 task groups (1–6), 27 subtasks (1.1–6.7) are `[x]` in tasks.md. Recent commits (aab41fa through e1517bb) confirm active implementation. The git log shows all tasks through 6.7 implemented with bug fixes applied.
- **Remaining gaps**: Task groups 1–4 and 5–6 all show `[ ]` as section headers, but all subtasks within them are `[x]`. The section checkboxes are cosmetic.
- **Deferred items**: None. The original research's "Research Needed" items (capnp schema design, heartbeat reply processing, snapshot wire format) were resolved during the design phase.
- **Cross-spec relationships**: This spec extends `ha-config-bootstrap` Req 5 (majority-commit WAL) and Req 8 (majority-commit config) with heartbeat-driven convergence. It depends on `ha-heartbeat-liveness` for the idle-triggered NOP policy on the primary and activity-based liveness detection on the standby. It depends on `ha-peer-connection-reuse` for the promoted connection model — `handle_ctrlhello` was reduced to connection setup only (task 5.1), with all replication decisions moved to the heartbeat decision loop.
- **Single-node mode**: Verified working — `n_peers=1`, immediate convergence, `CTRL_PRIMARY` with no heartbeat wait.
- **Recommendation**: Run `/xce-build controller` to confirm ASAN-clean build, then mark the spec `implemented`.

---

# Gap Analysis — Validation Pass (2026-06-09)

> Validation-phase gap analysis: requirements.md vs code **as on disk** (including uncommitted
> changes to ha.c/ha.h/myapp.c/raft.c). All 27 subtasks in tasks.md are `[x]`; this pass verifies
> what the code actually does against each acceptance criterion.

## Headline Finding

**The control plane is complete; the data plane is not.** Every protocol mechanism in this spec
(heartbeat stateReport, per-standby tracking, decision loop, batch framing, snapshot chunking,
convergence-gated promotion, safe commit point) is implemented and unit/integration tested. But
**payload bytes are never transmitted or applied**:

- `ha_send_snapshot_to_standby` sends every chunk with `chunk.payload = NULL` —
  `config_repl.c:1638` (`/* encoding deferred */`, explained at config_repl.c:1613-1618: the
  capnp `Data` ↔ `xce_data_t` tagged-union encode path is not wired).
- The standby snapshot handler passes `payload_bytes=NULL, payload_len=0` —
  `cluster.c:191-197`. Consequently `snap.part` is never written, the `is_last` size check
  (`ha.c:1094-1106`) fails (or `stat` fails on a missing file), and `persist_load_snapshot` is
  never reached in production. Snapshot transfer works only in unit tests via
  `xce_snapshot_chunk_hook`.
- WAL catch-up applies entries **header-only**: `ha_apply_single_entry` (ha.c:974-984) builds
  `payload_len=0` and calls `apply_wal_entry(p, &hdr, NULL)`. `apply_wal_entry`
  (persist.c:691) then only advances the LSN chain — payload-bearing entry types
  (`WAL_JOB_SUBMIT` → capnp job, jobstate entries → `wal_jobstate_t`) restore **nothing**.

**Safety implication**: a standby's `applied_lsn` advances without the corresponding state, so it
reports "converged" via stateReport, the primary promotes out of COMMITTING, and a later failover
to that standby would serve from a state missing all payload-bearing entries. The Majority
Consistency Invariant (design §Overview) is currently tracked but not actually established.

**Cross-spec constraint**: the live replication path has the same hole — `repl_broadcast`
(repl.c:193-234) discards `payload` and sends nothing ("TCP send wiring deferred to task 7.6",
owned by ha-config-bootstrap). Today no WAL payload bytes move between controllers on any path.

## Requirement-to-Asset Map

| Req / AC | Asset (file:line) | Status |
|---|---|---|
| **1.1** heartbeat reply carries (tag, lsn) | `cluster.c:273-287` OPCode_nop handler; `ha_nop_reply_state_report` ha.c:1140; `StateReport` controller.capnp:788 | Implemented |
| **1.2** primary records + evaluates | `repl_ack_decode_and_dispatch` ha.c:2456-2547 → `ha_process_standby_state` ha.c:1198 | Implemented |
| **1.3** convergence criteria (tag≥ AND lsn≥) | ha.c:1211-1212 | Implemented |
| **1.4** tag=0 before config push | reads `myconf->config_version_tag` (0 pre-push) | Implemented |
| **2.1** init tracking on COMMITTING entry | Arrays zeroed only in `ha_init` (ha.c:816-819 comment claims the invariant); `raft_cb_become_leader` (ha.c:445-529) resets `repl_ack_count` but **not** `standby_converged[]`/`standby_reported_*[]` → stale convergence flags survive a demote→re-elect cycle | **Partial** |
| **2.2** update + re-evaluate on reply | ha.c:1198-1223, 2525-2533 | Implemented |
| **2.3** majority threshold ceil(n/2)−1 | `ha_check_majority_convergence` ha.c:1244-1278 requires **all** standbys (`n_converged >= n_peers−1`), not majority. Three conflicting formulas: requirements.md 2.3 = ceil(n/2)−1 (1 standby for n=3); tasks.md 5.2 = ceil(n/2) (2 for n=3); code = n−1 (2 for n=3, 4 for n=5) | **Constraint / deviation** (stricter than spec — safe but availability hit) |
| **2.4** INFO on promotion | ha.c:1266-1267 | Implemented |
| **2.5** regression → not-converged | ha.c:1210-1213 | Implemented |
| **3.1** decode + apply WAL entry | Handler wired (`cluster.c:161-173` → `ha_apply_wal_catchup_entry` ha.c:986) but applies **header-only, payload dropped** (ha.c:974-984) | **Partial (CRITICAL)** |
| **3.2** advance applied_lsn | `ha_advance_applied_lsn` ha.c:960 (atomic, monotonic) | Implemented |
| **3.3** ERROR + skip on failure | ha.c:1003-1016, 1032-1037 | Implemented |
| **3.4** stays CTRL_STANDBY, no job ops | `ha_dispatch_gate` CTRL_STANDBY case ha.c:1456-1483 | Implemented |
| **3.5** LSN order | sequential batch iteration | Implemented |
| **4.1** create checkpoint + transfer | `ha_send_snapshot_to_standby` config_repl.c:1578 chunks an **existing** `snap-{checkpoint_lsn}.capnp` — never calls `persist_checkpoint()` to create a fresh one; payload=NULL | **Partial** |
| **4.2** load snapshot, set applied_lsn | `ha_handle_snapshot_chunk` ha.c:1058-1131 complete (verify/rename/load/applied_lsn) but unreachable in production (NULL payload, cluster.c:196) | **Partial** |
| **4.3** no job ops until loaded | dispatch gate (standby role unchanged) | Implemented |
| **4.4** retry via reconnect | Restart-on-reconnect documented (ha.c:1054, cluster.c:178-180) but **no production code unlinks `snap.part`** on reconnect — stale partial file breaks the next transfer's size check | **Partial** |
| **4.5** single-node: no snapshot | no standbys → decision loop never fires | Implemented |
| **5.1** multiple entries per message | `entries :List(WalCatchupEntry)` controller.capnp:850; `send_wal_batch` config_repl.c:1409 | Implemented |
| **5.2** configurable max batch size | `WAL_STREAM_BATCH_ENTRIES`=256 / `WAL_STREAM_BATCH_BYTES`=64KB are compile-time #defines, not runtime config | Implemented (minor constraint) |
| **5.3** apply in order, advance after batch | ha.c:992-1027 (`last_good_lsn` post-batch advance) — but payload gap applies | **Partial** |
| **5.4** partial batch sent immediately | config_repl.c:1556-1563 | Implemented |
| **5.5** ordering + durability | Ordering yes; "durably written" weak — entities restored in-memory, no fsync on standby (applied_lsn in-memory-only is a documented design choice), and payload gap applies | **Partial** |
| **6.1** heartbeats continue in COMMITTING | self-re-arming heartbeat task (fix e1517bb) | Implemented |
| **6.2** NOT_READY for non-repl opcodes | `ha_dispatch_gate` CTRL_PRIMARY_COMMITTING allowlist ha.c:1436-1454 | Implemented |
| **6.3** immediate promotion on majority | ha.c:2529-2533 (before repl dispatch) — subject to 2.3 threshold deviation | Implemented (constraint) |
| **6.4** timeout → WARN + degraded mode | `committing_timeout_fn` ha.c:2885-2914; default 60000 (`HA_DEFAULT_COMMITTING_TIMEOUT_MS` ha.h:131) | Implemented |
| **6.5** single-node immediate | n_peers=1 → threshold 0 (ha.c:1261-1264; fixed in e10cf9a) | Implemented |
| **7.1** catch-up start INFO | config_repl.c:1445-1447 | Implemented |
| **7.2** converged INFO | ha.c:1216-1222 | Implemented |
| **7.3** snapshot start/complete INFO | config_repl.c:1595, 1663 | Implemented |
| **7.4** majority INFO | ha.c:1266 | Implemented |
| **7.5** timeout WARN w/ counts | ha.c:2905-2908 | Implemented |
| **Flow 0** safe commit point (tasks 4.1/4.2) | `ha_compute_safe_committed_lsn` ha.c:373-432; truncation + committed baseline in `raft_cb_become_leader` ha.c:456-485 | Implemented |

**Score**: 26 of 33 acceptance criteria fully implemented. Requirements 1, 6, 7 (and Flow 0
safety) Implemented; Requirement 2 Partial (init-reset + threshold semantics); Requirements 3, 4,
5 Partial — all blocked on the same payload data-plane gap.

**Uncommitted working-tree changes** (ha.c/ha.h/myapp.c/raft.c/deploy-ha.sh) are election-protocol
hardening adjacent to this spec: `ha_init_raft` one-time init (prevents raft state reset on
re-election), term-scoped candidate-agnostic vote denial in `raft_on_vote_request` with idempotent
re-grant, startup self-vote/stale-vote handling, and watchdog arming on the skip-election path.
These strengthen Election Safety per raft-tla-audit.md and should be committed, but they do not
close any of this spec's acceptance criteria.

## Implementation Approach Options (remaining gaps)

### Gap 1 — Payload encoding for `WalCatchupEntry.payload` and `SnapshotTransfer.payload` (the blocker)

**Option A — Extend the typemap codegen to map capnp `Data` ↔ raw buffer for these structs.**
The schema already declares `payload :Data` on both structs; only the C-side encode/decode of the
`xce_data_t` tagged union is missing.
- ✅ Fixes both messages at once; follows the established schema/typemap convention; benefits any
  future Data-carrying message
- ❌ Touches generated-code path — regression risk across all opcodes; needs codegen expertise

**Option B — Hand-rolled encode/decode for just these two opcodes.**
Bypass the tagged union: serialize `lsn/type/len/bytes` with direct capn calls (or a length-prefixed
raw layout) in `send_wal_batch` / `ha_send_snapshot_to_standby` and their decode counterparts.
- ✅ Localized, no codegen risk, fastest path to a working data plane
- ❌ Two bespoke wire paths diverging from every other message; long-term maintenance debt

**Option C — Phased hybrid (recommended).**
Phase 1: hand-wire the snapshot payload (raw file bytes — trivially a byte stream, no per-entry
structure) to make Req 4 production-real. Phase 2: solve `Data` encoding properly in the typemap
for `WalCatchupEntry` (structured, batched, used on the hot path) and migrate the snapshot path to
it. Until both land, keep the spec phase below `implemented` or gate promotion on a data-plane
flag.
- ✅ De-risks incrementally; snapshot (rare, big) and WAL (frequent, small) have different shapes
- ❌ Two-step migration; temporary inconsistency between the two paths

### Gap 2 — Convergence threshold semantics (Req 2.3 vs code)

- **Option A**: change code to the requirement's majority formula (`ceil(n_peers/2) − 1` standbys).
  Restores availability (one dead standby in a 3-node cluster no longer forces the 60s degraded
  timeout) and matches Raft quorum; safe because safe_committed_lsn already derives from a majority.
- **Option B**: keep all-standbys convergence and amend requirements.md + tasks.md with the
  rationale (strongest consistency at startup; degraded timeout covers minority loss).
- Either way, **reconcile the three formulas** (requirements 2.3, task 5.2, code) — they currently
  disagree for any n ≥ 5.

### Gap 3 — Small correctness fixes (extend existing components)

1. Zero `standby_converged[]` / `standby_reported_*[]` in `raft_cb_become_leader` on COMMITTING
   entry (Req 2.1 — re-election staleness).
2. `unlink(snap.part)` on standby reconnect/ctrlhello (Req 4.4 — stale partial file).
3. Create a fresh checkpoint (via delaymgr — `persist_checkpoint` is blocking I/O, must not run in
   the vtask/heartbeat-reply context) when SEND_SNAPSHOT fires and the on-disk snapshot is older
   than `oldest_wal_lsn` coverage requires (Req 4.1).
4. Consider caching `ha_compute_oldest_wal_lsn` — currently an opendir/readdir scan per heartbeat
   reply per standby in vtask context (ha.c:2538-2543).
5. Optionally promote batch limits to `ha_config_t` (Req 5.2 "configurable").

## Effort and Risk

| Gap | Effort | Risk | Justification |
|---|---|---|---|
| 1 — Payload data plane (Option C) | **M–L (5–10d)** | **Medium** | Snapshot raw-bytes path is S; typemap `Data` codegen + batched WAL payload apply + ASAN-clean lifetime mgmt (current `(xce_data_t *)pbuf` cast at config_repl.c:1529 is already suspect) + E2E re-verification is the bulk |
| 2 — Threshold reconciliation | **S (0.5–1d)** | **Low** | One comparison + spec text alignment; existing tests pin the 3-node case |
| 3.1 — Tracking reset on COMMITTING | **S (<0.5d)** | **Low** | memset 3 arrays at ha.c:510; add re-election unit test |
| 3.2 — snap.part discard | **S (<0.5d)** | **Low** | unlink in handle_ctrlhello standby path |
| 3.3 — Fresh checkpoint on SEND_SNAPSHOT | **M (2–3d)** | **Medium** | Blocking checkpoint must hop to delaymgr; interaction with concurrent WAL writes |
| 3.4 — oldest_wal_lsn caching | **S (0.5d)** | **Low** | Cache + invalidate on segment GC |
| 3.5 — Configurable batch limits | **S (0.5d)** | **Low** | Plumb two fields through ha_config_t |

## Recommendations

1. **Do not mark this spec `implemented`.** All tasks are `[x]` but Requirements 3, 4, 5 are only
   control-plane complete; the system currently reports convergence it has not achieved. Reopen
   targeted tasks (suggest: 7.x series) for the payload data plane rather than re-flagging 2.x/3.5.
2. Land Gap 3.1 and 3.2 immediately (S, Low) — they are latent correctness bugs independent of the
   payload work.
3. Decide Gap 2 (threshold) explicitly and record it in the Decision Log; today three documents
   disagree.
4. Sequence the payload work after coordinating with **ha-config-bootstrap task 7.6**
   (`repl_broadcast` TCP wiring) — the live path and catch-up path should share one entry
   serialization, or the two specs will produce divergent wire formats for the same WAL entry.
5. Add an E2E assertion that verifies **entity state** (e.g., a submitted job visible on the
   promoted standby), not just `applied_lsn` equality — the current tests 6.5–6.7 pass despite the
   data-plane gap because they only observe LSN/log convergence.

## Research Needed

- How the typemap codegen represents capnp `Data` (`xce_data_t` tagged union table dispatch) and
  what it takes to encode an arbitrary byte buffer — prerequisite for Gap 1 Option A/C phase 2.
- Whether convergence should additionally verify content (e.g., last-entry CRC chained via
  `wal_entry_hdr_t.crc/prev_crc` in stateReport) so header-only application can never report
  false convergence again.
- Ownership boundary with ha-config-bootstrap task 7.6: which spec owns live WAL entry
  serialization, and can `send_wal_batch` be reused for it.
- Req 5.5 "durably written" on the standby: is in-memory entity restore + in-memory applied_lsn
  acceptable (current documented design), or should batch apply fsync a standby-side WAL copy.

---

# Gap Re-Audit (2026-06-18)

> Post-implementation gap audit. spec.json `phase=implemented`; all 41 sub-tasks `[x]`
> (1.1–7.6 incl. the reopened payload data-plane series). This pass re-verifies every EARS
> criterion against the code **as on disk** — specifically whether the two CRITICAL data-plane
> gaps and the small correctness gaps from the 2026-06-09 pass are now closed, and whether the
> COMMITTING-gate requirements still describe the live code after `ha-single-leader-role` landed.

## Headline Finding

**Both 2026-06-09 CRITICAL blockers are CLOSED. The data plane is real.**

- WAL catch-up now applies payload-bearing entries. `ha_apply_single_entry` (ha.c:1558) does
  persist-before-apply: `persist_wal_write_replicated(p, e->lsn, type, e->raw_payload, len)`
  (fsync, no re-broadcast, idempotent) THEN `apply_wal_entry(p, &hdr, e->raw_payload)`.
  `apply_wal_entry` (persist.c:1305) restores entities by type — `WAL_JOB_SUBMIT`→`restore_job`,
  job-state→`restore_job_state`, `WAL_WORKER_CTRL`→`worker_from_capnp`,
  `WAL_QUEUE_CTRL`→`restore_queue_state` (persist.c:1342-1389). Header-only application is gone.
- Snapshot payload bytes are transmitted. `ha_send_snapshot_to_standby` (config_repl.c:1857)
  stats the real file, chunks 64KB, and ships each chunk via `xce_snapshot_chunk_req_raw(ep,
  &chunk, buf, chunk_len)` (config_repl.c:1928) — raw `Data` bytes, not NULL. Standby
  `ha_handle_snapshot_chunk` (ha.c:1683→ via `handle_snapshot_transfer` cluster.c:196, dispatched
  at cluster.c:589) writes chunks to `snap.part`, verifies total size on `is_last`, atomic-renames
  to `snap-{lsn}.bin`, loads via `persist_load_snapshot`, sets `applied_lsn`.

**The Majority Consistency Invariant is now actually established, not merely tracked.** A standby
that reports converged holds the entity state behind its `applied_lsn`.

## Architectural supersession (read before re-opening any 6.x task)

`CTRL_PRIMARY_COMMITTING` **no longer exists** — `grep -rn CTRL_PRIMARY_COMMITTING src/controller`
returns zero hits. `ha-single-leader-role` removed the transitional COMMITTING role:
`raft_cb_become_leader` now stores `CTRL_PRIMARY` directly (ha.c:1034) with the comment "there is
no transitional COMMITTING state." New-leader safety is provided by a different, stronger
mechanism than this spec's heartbeat-convergence gate:

1. **Own-term WAL_NOOP + writable_lsn gate** (`ha_arm_own_term_gate` ha.c:2259) — every WAIT
   completion is withheld until `committed_lsn >= own-term noop LSN`, i.e. until this term has
   committed an entry on a majority.
2. **Per-operation write quorum** (`ha_dispatch_gate` ha.c:2061 → `ha_has_write_quorum` ha.c:2031)
   — state-changing opcodes return `-XCE_ERR_NOT_READY` unless `repl_active_standby_count >=
   n_peers/2`; replication and read-only/fact-bearing opcodes are always admitted (allowlist at
   ha.c:2078-2106).
3. **Vote-derived safe commit point** (`ha_compute_safe_committed_lsn` ha.c:480, called from
   raft_cb_become_leader ha.c:908) — `safe_committed_lsn` = majority-threshold element of granted
   voters' LSNs; `persist_truncate_wal_above(safe_lsn)` + `committed_lsn=safe_lsn` (ha.c:914).

Net effect: the *intent* of Req 6 (don't serve un-replicated state; gate on majority) is met
**more strongly and continuously** than the original timeout-bounded COMMITTING design — it is now
enforced per-operation for the life of the term, not just at startup. The original Req 2.3 / 6.1–6.4
wording (COMMITTING role, 60s `committing_timeout_ms` degraded fallback, immediate
COMMITTING→PRIMARY on majority) describes a design that was deliberately replaced. This is a
**documentation drift**, not a functional gap.

## Requirement-to-Asset Map (re-verified)

| Req / AC | Asset (file:line) | Status |
|---|---|---|
| **1.1** heartbeat reply carries (tag,lsn) | `OPCode_nop` handler cluster.c:300-301; populates from `myconf->config_version_tag` + `applied_lsn` | CLOSED |
| **1.2** primary records + evaluates | `repl_ack_decode_and_dispatch` → `ha_process_standby_state` ha.c:1851 | CLOSED |
| **1.3** convergence criteria tag≥ AND lsn≥ | ha.c:1864-1865 | CLOSED |
| **1.4** tag=0 before config push | reads live `config_version_tag` (0 pre-push) | CLOSED |
| **2.1** init tracking on term entry | **FIXED** — `raft_cb_become_leader` memsets `standby_reported_tag/_lsn/_converged` (ha.c:1043-1045) on CTRL_PRIMARY entry. Closes 2026-06-09 Gap 3.1. | CLOSED |
| **2.2** update + re-evaluate on reply | `ha_process_standby_state` ha.c:1851-1876 | CLOSED |
| **2.3** majority threshold | **RECONCILED** to `n_peers/2` (C int div) per task 7.5 + Decision Log 2026-06-09. `ha_has_write_quorum` uses `connected >= n_peers/2` (ha.c:2058); `safe_committed_lsn` uses `n_granted/2` (ha.c:527). Three-doc disagreement resolved. | CLOSED |
| **2.4** INFO on promotion/converged | ha.c:1869-1875 (per-standby converged INFO) | CLOSED |
| **2.5** regression → not-converged | ha.c:1863-1866 (recomputed every reply) | CLOSED |
| **3.1** decode + apply WAL entry **with payload** | **FIXED** — `ha_apply_single_entry` ha.c:1558-1589 passes `raw_payload` to `apply_wal_entry`; entities restored persist.c:1342-1389. Closes 2026-06-09 CRITICAL. | CLOSED |
| **3.2** advance applied_lsn | `ha_advance_applied_lsn` ha.c:1541 (atomic monotonic); batch advances after full batch (ha.c:1656-1659) | CLOSED |
| **3.3** ERROR + skip on failure | ha.c:1633-1647, 1666-1672 (NULL/failed sub-entries skipped, no crash) | CLOSED |
| **3.4** stays STANDBY, no job ops | `ha_dispatch_gate` STANDBY case | CLOSED |
| **3.5** LSN order | sequential batch iteration + persist-before-apply | CLOSED |
| **4.1** fresh checkpoint + transfer | **FIXED** — `ha_dispatch_repl_action_for_standby` calls `persist_checkpoint(&c->persist)` before send (ha.c:1961), -EBUSY falls back to existing. Closes 2026-06-09 Gap 3.3. | CLOSED |
| **4.2** load snapshot, set applied_lsn | `ha_handle_snapshot_chunk` ha.c:1683 (verify→rename→`persist_load_snapshot`→applied_lsn); reachable in production (real payload) | CLOSED |
| **4.3** no job ops until loaded | dispatch gate (standby role) | CLOSED |
| **4.4** retry via reconnect / discard stale | **FIXED (cleaner than proposed)** — `chunk_index==0` unlinks any stale `snap.part` (ha.c:1693, 1708-1713) instead of relying on a reconnect-time unlink. Verified by `test_snapshot_recv_reconnect_discards_snappart` (test_config_repl.c:2045). Closes 2026-06-09 Gap 3.2. | CLOSED |
| **4.5** single-node: no snapshot | no standbys → decision loop never fires | CLOSED |
| **5.1** multiple entries/message | `entries :List(WalCatchupEntry)` + `send_wal_batch` (config_repl.c) | CLOSED |
| **5.2** configurable max batch | `WAL_STREAM_BATCH_ENTRIES`=256 / `_BYTES`=64KB compile-time #defines | CLOSED (minor: limits are compile-time, not runtime ha_config_t — Req says "configurable maximum"; defensible) |
| **5.3** apply in order, advance after batch | `ha_apply_wal_catchup_entry` ha.c:1615-1681 (`last_good_lsn` post-batch advance) — now with real payload | CLOSED |
| **5.4** partial batch sent immediately | config_repl.c batch flush on range end | CLOSED |
| **5.5** ordering + durability | Ordering preserved; durability now STRONGER than original design — `ha_apply_single_entry` fsyncs via `persist_wal_write_replicated` BEFORE apply/ACK (ha.c:1570). Supersedes the "in-memory only" caveat for catch-up. | CLOSED (exceeds spec) |
| **6.1** heartbeats continue / gate on majority | superseded: continuous per-op `ha_has_write_quorum` (ha.c:2031) replaces COMMITTING-only check | CLOSED (via supersession) |
| **6.2** NOT_READY for non-repl opcodes | `ha_dispatch_gate` ha.c:2061-2110 (allowlist + `-XCE_ERR_NOT_READY` default) — now enforced for the whole term, not just COMMITTING | CLOSED (stronger) |
| **6.3** immediate promotion on majority | superseded by own-term gate + safe_committed_lsn (no COMMITTING dwell to exit) | CLOSED (via supersession) |
| **6.4** timeout → WARN + degraded | **OBSOLETE** — no COMMITTING state ⇒ no `committing_timeout_ms`. Degraded operation handled by write-quorum gate (rejects writes, keeps serving reads) rather than a one-shot 60s promotion fallback. | N/A (superseded) |
| **6.5** single-node immediate | `ha_has_write_quorum` returns true when `n_peers<=1` (ha.c:2037); safe_committed_lsn single-node path ha.c:486 | CLOSED |
| **7.1** catch-up start INFO | config_repl.c (send_wal_batch start log) | CLOSED |
| **7.2** converged INFO | ha.c:1869-1875 | CLOSED |
| **7.3** snapshot start/complete INFO | config_repl.c:1874-1876, 1939-1941 | CLOSED |
| **7.4** majority INFO | per-standby converged INFO (ha.c:1870); promotion is direct-to-PRIMARY INFO (ha.c:1036) | CLOSED |
| **7.5** timeout WARN | OBSOLETE with COMMITTING removal; quorum-loss edge logged once via `ha_log_quorum_transition` (ha.c:2076) instead | N/A (superseded by quorum-transition logging) |
| **Flow 0** safe commit point | `ha_compute_safe_committed_lsn` ha.c:480-540; truncate+baseline raft_cb_become_leader ha.c:898-923 | CLOSED |

**Score: 31/33 original acceptance criteria CLOSED; 2 (6.4, 7.5) OBSOLETE by deliberate
architectural supersession (CTRL_PRIMARY_COMMITTING removed by ha-single-leader-role).**
Counting the superseded pair as satisfied-by-replacement (their intent — gate on majority, observe
degraded mode — is met more strongly), effective coverage is 33/33. No CRITICAL or HIGH functional
gap remains.

## Residual / Deferred Items (all LOW, none blocking)

1. **Documentation drift (HIGH-doc, LOW-code).** requirements.md Req 6 + design.md still describe
   `CTRL_PRIMARY_COMMITTING`, `committing_timeout_ms`, and the 60s degraded-mode timeout, none of
   which exist in code. The Introduction (requirements.md:5-7) and Adjacent-expectations also
   reference COMMITTING. A reader of the spec will mis-model the live FSM. Code is correct; the
   prose is stale.
2. **Req 5.2 "configurable" batch size** is compile-time (`#define`), not runtime config. Minor;
   no operator has needed to tune it. Defensible as "configurable at build."
3. **`ha_compute_oldest_wal_lsn`** is an `opendir`/`readdir` scan per heartbeat reply per standby
   in vtask context (config_repl.c:1946). Performance-only; fine at current cluster sizes. The
   2026-06-09 caching suggestion (Gap 3.4) was never actioned and remains optional.
4. **No covering tests** flagged by codegraph for `ha_process_standby_state`,
   `ha_evaluate_repl_action`, `ha_dispatch_repl_action_for_standby`, `ha_send_snapshot_to_standby`
   at the unit level (decision-loop logic is exercised via hooks in test_ha_role.c /
   test_config_repl.c, and convergence via test_convergence_verification.c). Coverage is
   integration-weighted, not absent — but the pure decision functions lack direct unit pins.
5. **E2E entity-state assertion** (2026-06-09 Recommendation 5): tests 6.5–6.7 historically
   asserted LSN convergence only. With the data plane now real this is the right place to add a
   "submitted job visible on promoted standby" assertion to guard against future regressions to
   header-only application. Not done; LOW (data plane verified by unit round-trips per spec.json
   note).

## Effort / Risk for Open Work

| Item | Option | Effort | Risk |
|---|---|---|---|
| 1 — Spec/design doc reconciliation (remove COMMITTING; document own-term gate + write-quorum + safe_committed_lsn as the actual Req 6 mechanism) | **A (recommended):** update requirements.md Req 6 + Introduction + design.md to describe the superseding mechanism and cross-link `ha-single-leader-role` / `ha-commit-durability`. **B:** leave a one-line "SUPERSEDED" banner pointing to those specs. | A: S (0.5–1d) · B: XS (<0.5d) | Low (docs only) |
| 2 — Promote batch limits to `ha_config_t` | A: plumb two fields. B: leave as build-time. | S (0.5d) | Low |
| 3 — Cache `oldest_wal_lsn` | A: cache + invalidate on segment GC. B: defer (current scale fine). | S (0.5d) | Low |
| 4 — Direct unit tests for decision functions | A: add `ha_evaluate_repl_action` truth-table + `ha_process_standby_state` convergence-flag tests. B: rely on hook/integration coverage. | S (1d) | Low |
| 5 — E2E entity-state assertion | A: extend test 6.7 to assert a submitted job is present post-failover. B: rely on unit payload round-trips. | S (0.5–1d) | Low |

**Recommendation:** the spec is functionally complete and may remain `implemented`. The only item
worth scheduling is **Item 1 (doc reconciliation)** — the requirements/design now misdescribe the
live FSM after `ha-single-leader-role`, which will mislead future readers and any
`/kiro-verify-completion` run that checks prose against code. Items 2–5 are optional hardening.

---

# Gap Analysis (re-validation) — 2026-06-24

> Re-validation pass. spec.json `phase=implemented` (all 41 sub-tasks `[x]`). This pass re-verifies
> every EARS criterion against the code **as on disk** after the heavy `ha-term-safety` rework
> (commits 004a40b…b77d412, tasks 1.1–4.3) plus bug-828/991/1028/1060/1067 fixes landed since the
> 2026-06-18 audit. Question answered: does this spec still hold, and how much of it is now owned by
> sibling specs?

## Headline Finding

**Still functionally complete; now LARGELY SUPERSEDED in mechanism, but its data-plane code is the
live load-bearing path.** Two distinct things are true at once:

1. **The control-plane *design* of this spec (Req 2.3 / Req 6: `CTRL_PRIMARY_COMMITTING` role +
   60 s `committing_timeout_ms` degraded fallback + COMMITTING→PRIMARY-on-majority) is GONE.**
   `grep -rn CTRL_PRIMARY_COMMITTING src/controller/*.{c,h}` → 0 hits; `committing_timeout` → 0 hits.
   Removed by `ha-single-leader-role` and replaced by a stronger, continuous mechanism
   (own-term WAL_NOOP + `writable_lsn` gate `ha_arm_own_term_gate` ha.c:107/1206; per-op write
   quorum `ha_has_write_quorum` ha.c:2781 → `-XCE_ERR_NOT_READY`; vote-derived
   `ha_compute_safe_committed_lsn` ha.c:510). This is **documentation drift**, not a functional gap —
   identical to the 2026-06-18 finding and **still unactioned**.

2. **The data-plane *code* of this spec (Req 1,3,4,5,7) is intact, hardened, and is the path every
   sibling spec builds on.** It was NOT rewritten away — `ha-term-safety` routes through it:
   - WAL catch-up payload apply: `ha_apply_single_entry` ha.c:2044 → `persist_wal_write_replicated`
     (fsync, idempotent) ha.c:2056 THEN `apply_wal_entry` ha.c:2081 (entity restore). Config markers
     also flow here (ha.c:2168). Header-only application remains gone.
   - Batched streaming: `send_wal_batch` config_repl.c:2005, `WAL_STREAM_BATCH_ENTRIES`=256 /
     `_BYTES`=64 KB (config_repl.c:1981-1982); batch flush loop config_repl.c:2071-2173.
   - Snapshot raw payload: `ha_send_snapshot_to_standby` config_repl.c:2425 →
     `xce_snapshot_chunk_req_raw` config_repl.c:2496 (real bytes).
   - Per-standby convergence: `ha_process_standby_state` ha.c:2592 (sets
     `standby_reported_tag/_lsn/_converged`, ha.c:2600-2607); reset on term entry ha.c:1115-1117;
     fed from `OPCode_reply` stateReport at ha.c:4620.
   - `safe_committed_lsn` (this spec's Flow 0, tasks 4.1/4.2) is *explicitly cited by code* as
     ha-repl-hardening: ha.c:940 "Task 4.1 (ha-repl-hardening)".

   bug-1028 (FIXED 2026-06-23) was inside this very streaming path (`ha_stream_wal_to_standby` gained
   a `const persist_t *p` param) — confirming the path is live and maintained, not dead.

## Which sibling spec now owns what

| Concern | Originating in ha-repl-hardening | Now owned / superseded by |
|---|---|---|
| Heartbeat stateReport (Req 1) | yes | retained as-is; consumed by `ha-heartbeat-committed-propagation` (adds committed+term stamp) |
| Per-standby convergence tracking (Req 2.1/2.2/2.5) | yes | retained as-is |
| Majority threshold `n_peers/2` (Req 2.3) | yes | reconciled + **absorbed** into `ha_has_write_quorum` / `safe_committed_lsn` (raft-commit-safety, ha-term-safety) |
| WAL catch-up payload apply (Req 3) | yes | retained; now also drives `ha-term-safety` config-marker apply + bug-991 lsn-advance fix |
| Snapshot transfer (Req 4) | yes | retained as-is |
| Batch WAL transfer (Req 5) | yes | retained; bug-1028 hardened the streamer signature |
| COMMITTING gate (Req 6) | yes | **SUPERSEDED & DELETED** by `ha-single-leader-role` (own-term gate) + `ha-commit-durability` (durable-commit fence) |
| Flow 0 safe commit point (tasks 4.1/4.2) | yes | retained; extended by `raft-commit-safety` (Figure-8) + `ha-term-safety` (per-entry term, (term,lsn) reconcile) — the LSN-only safe point is now term-aware upstream |

## Requirement-to-Asset Map (delta vs 2026-06-18 — only changes noted)

All 31 CLOSED criteria from 2026-06-18 remain CLOSED (assets re-verified, line numbers shifted +~860
in ha.c / +~560 in config_repl.c due to ha-term-safety growth; spot-checked above). No regression.

| Req / AC | Change since 2026-06-18 | Status |
|---|---|---|
| **3.x** payload apply | Path unchanged in shape; bug-991 fix made the three live-apply setters advance applied_lsn to the LAST-held LSN (correctness hardening). | CLOSED (hardened) |
| **5.x** batch streaming | bug-1028 added `const persist_t *p` to `ha_stream_wal_to_standby` (off-by-one / hdr-source fix). | CLOSED (hardened) |
| **Flow 0** safe commit | Now also gated by ha-term-safety per-entry term + bug-1060 promotion-truncate floor `max(safe_lsn,durable_tip)` (prevents new leader truncating its own committed log) + bug-1067 AE header-term separation. | CLOSED (strengthened upstream) |
| **6.4 / 7.5** COMMITTING timeout/WARN | Still OBSOLETE (no COMMITTING). | N/A (superseded) |

**Score: 31/33 CLOSED, 2 OBSOLETE-by-supersession (6.4, 7.5). Effective 33/33. No CRITICAL/HIGH
functional gap. Unchanged from 2026-06-18, now re-confirmed post ha-term-safety.**

## Disposition: RESCOPE (close as data-plane-complete + supersede the control-plane prose)

This spec should **not** be reopened for new feature work and should **not** be deleted. Recommended
disposition:

- **Keep `phase=implemented`.** Every acceptance criterion is satisfied or satisfied-by-replacement.
- **CLOSE-as-superseded the control plane (Req 2.3 + Req 6 entirely).** Their mechanism was
  deliberately replaced. Carry a one-line "SUPERSEDED by ha-single-leader-role / ha-commit-durability"
  banner on Req 6 and on the Introduction's COMMITTING references.
- **RETAIN the data plane (Req 1,3,4,5,7 + Flow 0) as the canonical owner.** No sibling spec
  re-implements WAL-catch-up payload apply, batching, snapshot transfer, or stateReport — they all
  consume this code. If anything, this spec is now the *foundation*, not the obsolete layer.

The 2026-06-09 project-memory note ("data plane MISSING, critical, bugs 335-338") is **stale** — it
was closed by the reopened 7.x payload series (audited 2026-06-18) and is unaffected by later work.

## Residual / Deferred (all LOW, none blocking — unchanged + re-confirmed)

1. **Documentation drift (HIGH-doc, LOW-code) — STILL OPEN.** requirements.md Req 6 + Introduction +
   design.md describe `CTRL_PRIMARY_COMMITTING` / `committing_timeout_ms` / 60 s degraded fallback,
   none of which exist. This is the *only* item worth scheduling. A `/kiro-verify-completion` run that
   checks prose-vs-code will flag Req 6 falsely.
2. **Req 5.2 "configurable" batch size** is compile-time `#define`, not `ha_config_t`. Defensible.
3. **`ha_compute_oldest_wal_lsn`** opendir/readdir per heartbeat reply per standby. Perf-only.
4. **Direct unit tests** for `ha_process_standby_state` / `ha_evaluate_repl_action` /
   `ha_send_snapshot_to_standby` — coverage is integration-weighted (test_config_repl.c,
   test_convergence_verification.c), not absent.
5. **E2E entity-state assertion** (submitted job visible on promoted standby post-failover) — the
   live HA soak (`/xce-test-ha`) now exercises this end-to-end, partially discharging the gap; a
   pinned unit/E2E assertion is still nice-to-have.

## Effort / Risk for the one schedulable item

| Item | Option | Effort | Risk |
|---|---|---|---|
| Doc reconciliation: remove COMMITTING from requirements.md Req 6 + Introduction + design.md; document own-term gate + write-quorum + safe_committed_lsn as the real Req 6 mechanism; cross-link ha-single-leader-role / ha-commit-durability / ha-term-safety | **A (recommended):** rewrite Req 6 + Introduction (S, 0.5–1d). **B:** one-line SUPERSEDED banner (XS, <0.5d). | A: S · B: XS | Low (docs only) |

## Recommendation

1. **No open implementation work.** Spec is functionally complete and is the live owner of the HA
   replication data plane. Mark project-memory note (2026-06-09 "data plane MISSING") as resolved.
2. **Schedule only the doc reconciliation (Residual #1).** It has been flagged in two prior audits
   (2026-06-09 doc-drift, 2026-06-18 Item 1) and remains unactioned. Either rewrite Req 6 (Option A)
   or banner it (Option B). Everything else is optional hardening.
3. **Treat this spec as RESCOPED:** control plane CLOSED-as-superseded; data plane RETAINED-as-owner.

## Research Needed

- None new. Prior research items (typemap `Data` encoding, snapshot wire format, ownership boundary
  with ha-config-bootstrap task 7.6 / live `repl_broadcast`) were all resolved by the payload series
  and ha-term-safety task 3.4 (`repl_broadcast` now carries leader term, term-0 live send retired).