myxce/.kiro/specs/ha-config-bootstrap/research.md

---

# Cross-Spec Gap Audit (2026-06-03)

> Triggered by: `/kiro-validate-gap ha-config-bootstrap election-iomgr-refactor, check am_i_primary() still be used?`
> Scope: confirm cross-spec coverage status; answer `am_i_primary()` question definitively.

## Analysis Summary

- **ha-config-bootstrap** remains at ~95% (all unit tasks done; tasks 11.1–11.5 are manual integration tests).
- **election-iomgr-refactor** is 100% complete and already marked `implemented`.
- The two specs are consistent and non-overlapping in implementation: election-iomgr-refactor superseded the blocking-pthread election path defined in ha-config-bootstrap, keeping all requirements satisfied through a different (better) implementation path.
- `am_i_primary()` question: **answered — it is dead code, safe to delete, but not harmful to leave.**

## `am_i_primary()` Status

### What it is

`am_i_primary()` (ha.c:869) is the original **LSF priority-based election probe** — it spawns one pthread per peer and self-promotes if no higher-priority peer responds before a deadline. It was the pre-Raft election mechanism.

### Is it still called?

**No.** Confirmed by grep:

- **`myapp.c`**: comment at line 48 explicitly says `am_i_primary_quorum` (the intermediate Raft replacement) was **removed** in Req 9.3. `am_i_primary()` is never referenced in myapp.c.
- **`cluster.c` line 1015**: The line `/* Determine initial role based on am_i_primary() result set by myapp_run */` is a **stale comment** — the actual code underneath calls `ha_is_primary(&c->ha)` (the atomic role check), not `am_i_primary()`. The comment is a lie left over from the old startup sequence.
- **`ha.c`**: The function is defined but has no callers in production code. The only cross-file reference to `am_i_primary` (not `am_i_primary_quorum`) in non-test code is the cluster.c comment above.

### What the tests use

- `test_ha_role.c` (lines 241–396): three test cases test `am_i_primary()` directly — these are legacy P1-6 tests covering the old priority-based election. They still pass because the function body is intact.
- `test_ha_bootstrap_support.c` and `test_ha_election.c`: both define stubs for `am_i_primary_quorum` (the intermediate function, not `am_i_primary`), used to isolate the quorum election logic under test.

### Decision: can `am_i_primary()` be deleted?

**Yes — safe to delete in a cleanup pass.** However:
- The three test cases in `test_ha_role.c` would also need to be deleted.
- The stale comment in cluster.c line 1015 should be updated to say "role set by ha_elect_evaluate_and_signal via vtask".
- `start_heartbeat_listener()` / `stop_heartbeat_listener()` stubs (ha.c:951-960) referenced in the same comment block can also be cleaned up.

**Recommendation**: Leave `am_i_primary()` as-is during active HA development. Mark as cleanup tech debt. Its presence does not affect correctness, ASAN, or test pass rate.

## Requirement Coverage Cross-Check

The election-iomgr-refactor implemented `ha_start_election_async` which replaced both `am_i_primary()` and `am_i_primary_quorum()`. The ha-config-bootstrap tasks 4.3 and 9.4 originally called `am_i_primary_quorum()` from myapp.c — these are now superseded by `ha_start_election_async`. Both specs' requirements are fully met by the current codebase.

No new code needed for either spec.

---

# Gap Analysis — Completion Audit (2026-06-02)

> Analysis date: 2026-06-02. Phase: `tasks-generated`. All leaf implementation tasks `[x]`.
> Purpose: verify actual codebase coverage of all 10 requirements; identify
> what remains before the spec can be marked `implemented`.

## Analysis Summary

- **Scope**: 10 requirements across 8 implementation changes (A–H), covering
  Raft-style quorum election, majority-commit WAL + config replication, persistent
  control channels, config bootstrap, term durability, and client-side retry.
- **Implementation status**: **~95% complete**. All 22 leaf subtasks (1.1–10.2)
  are implemented and verified. All 9 requirements have implementation assets in
  the live codebase (verified via grep audit). The parent group headers in
  tasks.md show `[ ]` only because they are section labels, not actionable tasks.
- **Remaining work**: Integration tests 11.1–11.5 (manual cluster validation).
- **No design changes needed**: The implementation already follows the approved
  design; `xce_request_with_retry`, `repl_wait_majority_ack`, `ha_find_and_connect_to_primary`,
  `ha_persist_term`, `bootstrap_config_if_needed`, and all 9 capnp extensions are
  in the live codebase.
- **Recommendation**: Run the 5 integration tests from `manual-tests.md`
  against a 3-node cluster, then mark the spec `implemented`.

---

# Cross-Spec Gap Audit (2026-06-08)

> Batch `/kiro-validate-gap` across 9 specs on persistent_ha branch.

## Analysis Summary

- **ha-config-bootstrap** is **~95% complete**. All 10 requirements have verified implementation assets. All 22 leaf subtasks (1.1–10.2) are done.
- **Remaining work**: Tasks 11.1–11.5 (manual integration tests documented in `manual-tests.md`).
- **Cross-spec verification**: `election-iomgr-refactor` superseded the blocking-pthread election path while preserving all ha-config-bootstrap requirements. The two specs are consistent. `ha-repl-hardening` extends Req 5 (majority-commit WAL) and Req 8 (majority-commit config) with heartbeat-driven convergence — no conflicts.
- **Cleanup debt**: `am_i_primary()` (ha.c:869), `ha_wait_for_heartbeat_or_timeout`, and `start_heartbeat_listener()`/`stop_heartbeat_listener()` stubs are dead code — safe to delete in a dedicated cleanup commit (tracked in election-iomgr-refactor/research.md).

## Requirement-to-Asset Map — Final State

| Requirement | Verified Key Assets | Status |
|-------------|-------------------|--------|
| Req 1: FSM init + dispatch gate | `CTRL_INIT`, `CTRL_ELECTING` in ha.h:59-60; `ha_dispatch_gate` in ha.c; `ha_wait_for_heartbeat_or_timeout` | ✅ Implemented |
| Req 2: Quorum election | `ha_elect_evaluate_and_signal` (ha.c:1604); `am_i_primary_quorum` replaced by `raft_start_election`; `handle_request_vote` term/freshness gate | ✅ Implemented |
| Req 3: Config bootstrap | `bootstrap_config_if_needed`, `xce_probe_config_tag`, `defer_config_fetch` — all in ha.c/config_repl.c | ✅ Implemented |
| Req 4: Contiguous WAL + ASAN | WAL schema capnp, `persist_*` callbacks, `SSL_CTX_free` fix | ✅ Implemented |
| Req 5: Majority-commit WAL | `repl_broadcast`, `repl_wait_majority_ack`, `committed_lsn` in persist_t, `persist_truncate_wal_above` | ✅ Implemented |
| Req 6: Persistent control channel | `ha_find_and_connect_to_primary`, extended ctrlhello handler, `EPOLLRDHUP` failover, `ha_on_primary_ctrl_fd_rdhup` | ✅ Implemented |
| Req 7: State sync | `apply_wal_entry`, `repl_start_catchup/complete_catchup`, ctrlhello + WAL catch-up flow | ✅ Implemented |
| Req 8: Majority-commit config | `OP_upgradeConfigAck` handler, `committed_config_tag`, `ha_repl_ack_received` | ✅ Implemented |
| Req 9: Voted-for persistence | `ha_persist_term` (ha.c:1323); `ha_init` restore path for `ha-term.bin` | ✅ Implemented |
| Req 10: Client-side retry | `xce_request_with_retry` (request.c:1664); `xce_retry_config_t` | ✅ Implemented |

## Remaining Gaps

### Gap 1: Tasks 11.1–11.5 — Integration validation (IP = In Progress via manual-tests.md)

The `manual-tests.md` file contains step-by-step procedures for all 5
integration tests. They require a running 3-node cluster:

| Test | What it validates | Status |
|------|-------------------|--------|
| 11.1 | Concurrent startup + config-fresh election + config replication | Manual test doc written; not yet run |
| 11.2 | Failover (RDHUP → promote → WAL survived); WAL commit under partition | Manual test doc written; not yet run |
| 11.3 | Unconfigured node bootstrap (`config_tag=0`) | Manual test doc written; not yet run |
| 11.4 | Config upgrade majority-commit; term durability across restart | Manual test doc written; not yet run |
| 11.5 | Full ASAN build verification | Manual test doc written; not yet run |

**Note**: These are manual cluster tests by design. The automated hook-based
unit tests in `testsuite/` (tests in `test_ha_bootstrap.c`, `test_ha_durability.c`,
`test_p1_integration.c`, `test_p2_integration.c`, `test_p3_hardening.c`,
`test_p3_manual_failover.c`) cover the in-process logic. Tasks 11.1–11.5
validate end-to-end process behavior that the hook seams cannot reach.

### Gap 2: Parent group headers in tasks.md show `[ ]` (Cosmetic)

Tasks 1., 2., 3., 4., 5., 6., 7., 8., 9., 10. are section grouping
labels — all their child subtasks are `[x]`. The parent headers can be marked
`[x]` for clarity.

### Constraint: `xce_request_with_retry` not in lib header (known limitation)

The function lives in `src/lib/request.c:1664` but is not declared in
`src/lib/include/xce/request.h`. Callers must add `extern int xce_request_with_retry(...)`
locally. Intentional — the libxce `request.h` only exposes the public API;
internal retry logic is a controller-side concern.

## Effort and Risk Assessment

| Item | Effort | Risk | Justification |
|------|--------|------|---------------|
| Tasks 11.1–11.5 (integration) | M (1–2 days) | Low | Procedures documented; run cluster + verify logs |
| Fix parent checkboxes | S (1 commit) | None | Cosmetic |

## Conclusion

The `ha-config-bootstrap` spec is **functionally complete** — all 10
requirements are implemented, tested, and verified at the unit level.
The 5 remaining integration tests (11.1–11.5) are manual cluster validations
that exercise the full system behavior. The `manual-tests.md` file contains
all the procedures. Once these are run, the spec can be marked `implemented`
and closed.

*Generated: 2026-05-27. Branch: persistent_ha.*

---

## Analysis Summary

- **Scope**: 10 requirements across 8 distinct changes (A–H). The codebase has strong foundational infrastructure (iomgr, repl slot management, WAL, SSL) but the HA election core, quorum commit, and control channel are mostly absent or stubbed.
- **Largest gaps**: Raft-style quorum election (Req 2) and persistent control channel with full state sync (Req 6) are full rewrites/additions with no current equivalent logic; majority-commit WAL (Req 5) requires implementing the two stub functions at the heart of replication.
- **Strong reusable assets**: `repl_add_standby/remove/start_catchup/complete_catchup/needs_snapshot` are real implementations; the SSL connect pattern (`connect_ssl_raw`), xorshift64 PRNG, `ha_do_promote/demote`, and iomgr epoll machinery are all directly reusable.
- **Risk concentration**: Req 2 and Req 6 carry High risk; Req 5 carries High risk due to `committed_lsn` cross-cutting `persist_t`, `repl.c`, and `ha_state_t`. The remaining requirements are Medium or Low.
- **Recommended approach**: Hybrid (Option A) — implement changes A→D→B→C→E→F→G→H in dependency order, writing C tests first per TDD, one atomic commit per change.

---

## 1. Current State Investigation

### 1.1 HA Core (ha.h / ha.c)

**`ctrl_role_t` enum** — only two values: `CTRL_PRIMARY = 0`, `CTRL_STANDBY = 1`.  
Missing: `CTRL_INIT`, `CTRL_ELECTING` (required by Req 1).

**`ha_state_t`** — current fields of interest:

| Field | Status | Notes |
|---|---|---|
| `term` (_Atomic uint64_t) | Exists | Used for fencing |
| `role` (_Atomic int32_t) | Exists | Cast to ctrl_role_t |
| `my_priority` | Exists | LSF priority index — kept for compat but unused in Raft |
| `applied_lsn` (_Atomic uint64_t) | Exists | Reusable for vote freshness check |
| `primary_host[XCE_MAXNAME_LEN]` | Exists | Reusable |
| `primary_port` | Exists | Reusable |
| `last_heartbeat` (struct timespec) | Exists | Replaces with `EPOLLRDHUP` path in Req 6 |
| `in_catch_up` (_Atomic int) | Exists | Reusable for Clarification 6 vote exclusion |
| `jitter_seed` (uint64_t) | Exists | xorshift64, directly reusable |
| `ssl_ctx` (void*) | Exists | Reusable |
| `standby_endpoints` (zarray_t*) | Exists | Superseded by per-standby fd table in Req 6 |
| `voted_for_term` (uint64_t) | **MISSING** | Req 9 |
| `voted_for_name` (char[]) | **MISSING** | Req 9 |
| `primary_ctrl_fd` (int) | **MISSING** | Req 6 |
| `defer_config_fetch` (bool) | **MISSING** | Req 3 |
| `elect_ctx` (_Atomic ptr) | **MISSING** | Req 2 |
| `last_known_committed_lsn` (_Atomic uint64_t) | **MISSING** | Req 5 |
| `heartbeat_seen` (sem_t) | **MISSING** | Req 1 |

**`ha_config_t`** — missing: `commit_timeout_ms`. All election fields present (election_timeout_ms, jitter_max_ms, etc.) but values tuned for LSF model.

**`am_i_primary()`** — LSF priority probe via per-peer pthreads. No quorum, no config_tag/LSN in vote, no term/voted_for persistence. Must be replaced entirely with `am_i_primary_quorum()`.

**`ha_check_priority_stepdown()`** — permanently unreachable once term > 0; exists only for the term=0 simultaneous-boot race. Must be retired (Req 2).

**`ha_check_heartbeat_nocluster()`** — timestamp polling with configurable timeout. Replaced by `EPOLLRDHUP` fast path + heartbeat timeout vtask (Req 6). Function itself can be deleted.

**`ha_watchdog_thread()`** — sets `failover_requested` but cannot call `ha_promote_to_primary()` (no `cluster_t`). Partially functional stub. Replaced by vtask-based heartbeat monitor (Req 6).

**`start_heartbeat_listener()` / `stop_heartbeat_listener()`** — no-op stubs with comment "full implementation in cluster.c Task 7". **Missing** (Req 1).

**`ha_fini()`** — missing `SSL_CTX_free(ha->ssl_ctx)`. ASAN-visible leak (~5 KB).

### 1.2 Startup Sequence (myapp.c / main.c)

**myapp.c current order**:
```
cluster_init_phase1()
initialize_openssl() + create_ssl_context()
am_i_primary()                     ← election here
load_primary_config() / fetch_config_from_primary()
cluster_init_phase2()              ← iomgr, vtaskmgr, bind+listen inside
cluster_start()                    ← more network init
cluster_run()
```

**Bind+listen** happens inside `cluster_init_phase2()` or `cluster_start()` — **after** election. Gap: concurrent-startup nodes probe an unbound port (Req 1).

**No `bootstrap_config_if_needed()`** — standby path goes straight to `fetch_config_from_primary()` post-election without checking whether local config_tag is 0 or whether peers have config. Gap (Req 3).

**main.c**:
- `--vcpus N` arg: **MISSING**. vcpus comes from cluster config read inside phase2. This creates a dependency that prevents vtaskmgr from running before election (Req 1, 4).
- Peer count validation: **MISSING**. Even-N peer lists accepted silently (Req 4).
- Cert pre-flight `access()` check: **MISSING**. Cert paths computed but SSL context created later in myapp.c, not with early-exit error in main.c (Req 4).

### 1.3 Replication (repl.c / repl.h)

| Function | Status | Notes |
|---|---|---|
| `repl_init / fini` | Real | Lifecycle ok |
| `repl_add_standby` | Real | Slot management implemented |
| `repl_remove_standby` | Real | Implemented |
| `repl_start_catchup` | Real | Sets in_catch_up=1 |
| `repl_complete_catchup` | Real | Clears in_catch_up, broadcasts cond |
| `repl_needs_snapshot` | Real | Compares standby_lsn vs oldest_wal_lsn |
| `repl_on_ack` | Real | Updates acked_lsn, signals cond |
| `repl_trim_lsn` | Real | WAL trim guard implemented |
| `repl_broadcast` | **STUB** | Returns 0 without sending anything (Req 5, 8) |
| `repl_wait_ack` | Partial | Waits for ANY non-catch-up standby, not majority (Req 5) |
| `repl_wait_majority_ack` | **MISSING** | Req 5, 8 |

**`standby_conn_t`** — missing `pending_ack_count` for backpressure (Req 5).

### 1.4 Persistence (persist.h / persist.c)

- `current_lsn` and `checkpoint_lsn` exist. **`committed_lsn`** — **MISSING** (Req 5).
- `persist_wal_sync_append()` uses `repl_wait_ack()` (any-standby) — must upgrade to `repl_wait_majority_ack()`.
- `persist_truncate_wal_above()` — **MISSING** (Req 5, 6).
- `persist_config_t.sync_timeout_ms` exists — analogous to `commit_timeout_ms` needed for majority-commit.

### 1.5 Config Replication (config_repl.c)

- `do_one_config_fetch()` — **wired** (recent commit `2aae45f` on branch: "wire do_one_config_fetch() — full SSL roundtrip"). Not a stub.
- `bootstrap_config_if_needed()` — **MISSING** (Req 3).
- `xce_probe_config_tag()` — **MISSING**. New lightweight RPC needed (Req 3).
- `push_config_to_standbys()` — declared in config_repl.h; likely fire-and-forget (no majority wait). **Gap**: must upgrade to majority-commit (Req 8).
- `OP_probeConfigTag` Cap'n Proto schema — **MISSING** (Req 3).
- `OP_upgradeConfigAck` Cap'n Proto schema — **MISSING** (Req 8).
- `committed_config_tag` — **MISSING** (Req 8).

### 1.6 Control Channel (ha.c / cluster.c)

- No persistent standby-to-primary TCP connection. Each config fetch and WAL entry opens a new SSL connection via `connect_ssl_raw()`.
- `ha_find_and_connect_to_primary()` — **MISSING** (Req 6).
- Extended `OP_ctrlHello` with applied_lsn, config_tag — **MISSING** (Req 6).
- `OP_snapshotTransfer`, `OP_walCatchupEntry` — **MISSING** (Req 6).
- `handle_ctrlhello()` extended handler — **MISSING** (Req 6).
- Three-tier primary discovery — **MISSING** (Req 6).
- EPOLLRDHUP failover detection — **MISSING** (Req 6).
- Heartbeat timeout vtask replacing watchdog thread — **MISSING** (Req 6).

### 1.7 Election State Durability

- `ha-term.bin` file — **does not exist**.
- `ha_persist_term()` — **MISSING** (Req 9).
- `ha_init()` zeros term and has no `ha-term.bin` read path.

### 1.8 libxce Client Retry (src/lib/)

- `ha_should_redirect()` / `ha_no_primary_known()` exist in ha.c and are used by join.c.
- No transparent retry loop for `XCE_ERR_NOT_READY`, `XCE_ERR_STANDBY`, `ECONNREFUSED`.
- `xce_retry_config_t` — **MISSING** (Req 7).
- `redirect_host` / `redirect_port` fields in `XCE_ERR_STANDBY` error reply — **MISSING** (Req 7).

---

## 2. Requirement-to-Asset Map

| Req | Key Gap | Key Reusable Asset | Gap Tag |
|---|---|---|---|
| 1 — Listener before election | `CTRL_INIT`/`CTRL_ELECTING` enum values; pre-dispatch gate in vtaskmgr; extract bind+listen from phase2; `ha_wait_for_heartbeat_or_timeout()`; `heartbeat_seen` semaphore | iomgr epoll, vtaskmgr dispatch switch, xorshift64 jitter | **Missing** |
| 2 — Quorum election | Entire `am_i_primary_quorum()` and vote protocol; `ha_vote_entry_t`, `ha_elect_ctx_t`; `handle_request_vote()` / `_reply()`; `ha_elect_evaluate_and_signal()`; OP_requestVote capnp schema | `ha_do_promote/demote`, jitter PRNG, `applied_lsn`, iomgr async connect | **Missing** |
| 3 — Config bootstrap | `bootstrap_config_if_needed()`; `xce_probe_config_tag()`; OP_probeConfigTag capnp; `defer_config_fetch` flag in ha_state_t | `do_one_config_fetch()` (wired), `connect_ssl_raw()` | **Missing** |
| 4 — Startup validation | `--vcpus N` CLI arg; even-N peer count check; cert pre-flight `access()`; `SSL_CTX_free` ASAN fix | argp option parsing pattern in main.c | **Missing** |
| 5 — Majority-commit WAL | `repl_broadcast()` implementation; `repl_wait_majority_ack()`; `committed_lsn` in persist_t; `persist_truncate_wal_above()`; `committed_lsn` in heartbeat; `last_known_committed_lsn` in ha_state_t | `repl_wait_ack()` logic, `standby_conn_t`, `repl_start/complete_catchup`, WAL sync path | **Missing** / **Stub** |
| 6 — Persistent channel | Persistent TCP fd per standby; `ha_find_and_connect_to_primary()`; EPOLLRDHUP failover; extended OP_ctrlHello; OP_snapshotTransfer; OP_walCatchupEntry; WAL streaming loop; heartbeat vtask; three-tier discovery | iomgr EPOLLIN/EPOLLRDHUP, `repl_needs_snapshot()`, `repl_start/complete_catchup()` | **Missing** |
| 7 — Client retry | Retry loop in libxce; `xce_retry_config_t`; redirect fields in error reply | `ha_should_redirect()`, `ha_no_primary_known()` | **Missing** |
| 8 — Majority-commit config | `repl_wait_majority_ack()` (shared with Req 5); OP_upgradeConfigAck; `committed_config_tag`; heartbeat piggybacking | `push_config_to_standbys()` (fire-and-forget → upgrade) | **Missing** |
| 9 — State durability | `ha_persist_term()`; `ha-term.bin` read in `ha_init()`; `voted_for_term`/`voted_for_name` in ha_state_t | Atomic write+rename pattern (existing in config_repl.c) | **Missing** |
| 10 — Observability | Structured log statements throughout all new code paths | Existing INFO/WARN/ERROR macros (xcelog.h) | **Missing** (alongside) |

---

## 3. Implementation Approach Options

### Option A: Change-at-a-time (recommended)

Implement changes in dependency order: D → A → B → C → E → H → F → G → (observability woven in).

**Rationale for ordering**:
1. D first: `--vcpus N` removes vtaskmgr's config dependency, unblocking A.
2. A before B: listener must be open before election dispatch works.
3. B before C: config bootstrap needs the proven election winner from quorum election.
4. E and H together: both use `repl_broadcast()` + `repl_wait_majority_ack()`; implement infrastructure once.
5. F after E: persistent channel carries WAL entries; broadcast must send over that channel.
6. G last: retry makes sense only after the controller states it retries against are implemented.

**Files touched per change**:
- D: `main.c`, `cluster.c` (remove vcpus from config), `ha.c` (`SSL_CTX_free`)
- A: `ha.h` (enum + semaphore + fields), `myapp.c` (startup reorder), `cluster.c` (extract listener), `ha.c` (wait function, dispatch gate)
- B: `ha.h` (new types), `ha.c` (replace `am_i_primary`), capnp schema, `cluster.c` (dispatch handler)
- C: `config_repl.c` (`bootstrap_config_if_needed`, `xce_probe_config_tag`), capnp schema, `myapp.c` (integration)
- E: `repl.c` (`repl_broadcast`, `repl_wait_majority_ack`), `persist.h`/`persist.c` (`committed_lsn`, `persist_truncate_wal_above`), `ha.h`/`ha.c` (`last_known_committed_lsn`)
- H: `config_repl.c` (upgrade majority-commit), capnp schema, `ha.h` (`committed_config_tag`)
- F: `ha.c`/`cluster.c` (channel connect, extended hello, streaming, EPOLLRDHUP)
- G: `src/lib/` (retry loop, xce_retry_config_t, redirect fields)

**Trade-offs**:
- ✅ Each atomic commit is testable before the next starts
- ✅ Follows existing TDD convention (failing test first)
- ❌ Long sequential dependency chain; F and G can only start late

### Option B: Infrastructure-first parallel tracks

Track 1 (election+bootstrap): D → A → B → C  
Track 2 (replication): E → H  
Join (channel): F using Track 1 + Track 2 artifacts  
Final: G

**Trade-offs**:
- ✅ Track 1 and Track 2 are mostly independent until F
- ❌ Track 2 cannot be fully validated without Track 1 (no quorum without election)
- ❌ More coordination overhead between developers on shared ha.h

### Option C: Stub-forward then wire

Add all new stubs with `-ENOSYS` first, then wire each one. Matches the existing codebase pattern (many stubs already present).

**Trade-offs**:
- ✅ Compiles throughout; no long broken-build windows
- ❌ Stub-first delays real testability; TDD mandate disfavors this

**Recommendation**: Option A. The dependency chain is well-defined and single-developer serial implementation is the realistic mode for this C codebase.

---

## 4. Effort and Risk Summary

| Req | Effort | Risk | Justification |
|---|---|---|---|
| 1 — Listener before election | M (3–5d) | Medium | Refactor startup order; must remove vtaskmgr/config dependency first |
| 2 — Quorum election | XL (2+ wks) | High | Full algorithm replacement; new wire protocol; FSM extension; concurrent dispatch |
| 3 — Config bootstrap | M (3–5d) | Medium | New function + probe RPC; depends on do_one_config_fetch (wired) |
| 4 — Startup validation | S (1d) | Low | Additive CLI + validation; ASAN one-liner |
| 5 — Majority-commit WAL | L (1–2 wks) | High | `committed_lsn` cross-cuts persist+repl+ha; `repl_broadcast` must become real |
| 6 — Persistent channel | XL (2+ wks) | High | New connection lifecycle; full state sync protocol; 3-tier discovery |
| 7 — Client retry | M (3d) | Low | Well-understood retry pattern; additive libxce change |
| 8 — Majority-commit config | M (3d) | Medium | Shares Req 5 infrastructure; mainly wiring push_config_to_standbys |
| 9 — State durability | S (1d) | Low | File write+read; extends ha_init |
| 10 — Observability | S (alongside) | Low | Log statements woven into each change |

**Total estimated effort**: 6–9 weeks for one engineer implementing with full TDD.

**High-risk dependencies**:
- Req 2 must complete before Req 3 can be integrated end-to-end.
- Req 5 (`repl_broadcast` implementation) must complete before Req 8 and before Req 6 (WAL streaming over the channel).
- Req 4 (vcpus CLI) must complete before Req 1 (vtaskmgr before election).

---

## 5. Key Reusable Assets

| Asset | Location | Reuse in |
|---|---|---|
| `repl_add/remove_standby` | repl.c | Req 6 (channel registration) |
| `repl_start/complete_catchup` | repl.c | Req 6 (state sync protocol) |
| `repl_needs_snapshot` | repl.c | Req 6 (snapshot decision) |
| `repl_on_ack` | repl.c | Req 5 (majority-commit ack handler) |
| `repl_trim_lsn` | repl.c | Req 5 (WAL trim with committed_lsn) |
| `connect_ssl_raw()` | netio.c/config_repl.c | Req 3 (probe RPC) |
| `do_one_config_fetch()` | config_repl.c | Req 3 (deferred fetch) |
| `xorshift64_next + ha_seed_jitter` | ha.h / ha.c | Req 1, 2 (jitter) |
| `ha_do_promote / ha_do_demote` | ha.c | Req 2, 6 (role transitions) |
| `ha_should_redirect / ha_no_primary_known` | ha.c | Req 7 (retry redirect) |
| iomgr epoll + EPOLLIN/EPOLLRDHUP | iomgr (cluster.c) | Req 1, 6 |
| Atomic write+rename pattern | config_repl.c | Req 9 (ha-term.bin) |
| INFO/WARN/ERROR macros | xcelog.h | Req 10 |

---

## 6. Constraints for Design Phase

- **C gnu17, no C++, no new external dependencies**.
- ASAN build (`nix build .#xcecore_debug`) must pass after every change.
- **Static peer membership**: `--peers` list is fixed at startup; no dynamic join/leave of controller peers. Quorum denominator is fixed for cluster lifetime.
- **TDD mandatory**: failing test in `src/controller/testsuite/` before each implementation change.
- `make test` must pass without a live server at every commit.
- `commit_timeout_ms` must be configurable (default 5000ms).
- **Single-node mode** (no `--peers`): quorum=1 everywhere; all new majority-commit paths must short-circuit to self-only.
- **WAL write stall window** (operational constraint, not a bug): during standby catch-up after election, `repl_wait_majority_ack()` will time out on every write until at least one standby completes catch-up. This is correct CP behavior; operators must be warned in docs.

---

## 7. Research Needed (Design Phase)

- **vtaskmgr API for pre-election dispatch**: confirm that `vtaskmgr_init(vcpus)` can be called before `cluster_config` is loaded (after removing vcpus from config struct). Verify no hidden config dependency.
- **iomgr async connect pattern for vote requests**: confirm `iomgr_async_vote_request()` can be implemented using existing async connect infrastructure without new socket primitives.
- **capnp schema extension**: confirm the capnp code-generation toolchain can add `OP_requestVote`, `OP_probeConfigTag`, `OP_walAck`, `OP_upgradeConfigAck`, `OP_snapshotTransfer`, `OP_walCatchupEntry` without breaking existing decode paths.
- **`persist_truncate_wal_above()` safety**: verify WAL segment file structure supports efficient truncation at arbitrary LSN boundary without full segment rewrite.
- **libxce client API surface**: identify the exact function in src/lib/ that wraps xce_encode()+send+recv+xce_decode() to locate the correct insertion point for retry logic.

---

# Gap Analysis (Updated — Brownfield Audit)

> Analysis date: 2026-06-01. Branch: persistent_ha. Phase: tasks-generated.
> This section audits current implementation state vs. all 11 task groups.

## Analysis Summary

- **Scope**: 10 requirements across 11 task groups. 30 subtasks checked `[x]`; only section headers and integration tasks 11.1–11.5 remain `[ ]`.
- **Implementation status**: **~95% complete**. All core subtasks (1.1–10.2) are marked done and confirmed implemented: `CTRL_INIT`/`CTRL_ELECTING` enum, ha-term.bin durability, `repl_wait_majority_ack`, `persist_truncate_wal_above`, `xce_request_with_retry`, `xce_probe_config_tag`, `repl_broadcast` (real, not stub).
- **Key architectural shift since original research**: The `election-iomgr-refactor` spec superseded parts of this spec. `bootstrap_config_if_needed` and `ha_find_and_connect_to_primary` are no longer called from `myapp_run` — standby connection is inbound (primary dials standby via `on_standby_connected`). `ha_find_and_connect_to_primary` exists only as a test hook reference in ha.h.
- **Remaining work**: Tasks 11.1–11.5 are manual/integration tests documented in `manual-tests.md`. The build passes with all unit tests green.
- **Effort remaining**: S (1–2 days for manual integration test execution + result recording). Risk: Low.

## Requirement-to-Asset Map (Current State)

| Req | Status | Key Evidence |
|-----|--------|-------------|
| 1 — Listener before election | ✅ Done | `CTRL_INIT`=2, `CTRL_ELECTING`=3 in ha.h; `cluster_open_listener()` called before election in myapp.c; `ha_wait_for_heartbeat_or_timeout()` implemented |
| 2 — Quorum election | ✅ Done | `am_i_primary_quorum` → superseded by `ha_start_election_async` + raft library (election-iomgr-refactor); `handle_request_vote`, `ha_elect_evaluate_and_signal` implemented |
| 3 — Config bootstrap | ✅ Done (changed arch) | `xce_probe_config_tag()` implemented in config_repl.c:1140; `bootstrap_config_if_needed` removed from myapp_run per election-iomgr-refactor task 5.3; `defer_config_fetch` field exists in ha_state_t; post-election config push via `ha_send_ctrlhello_to_all_standbys` covers deferred fetch |
| 4 — Startup validation | ✅ Done | `startup_validation.c` exists; cert pre-flight, peer count validation, vcpus CLI implemented; `SSL_CTX_free` ASAN fix in ha_fini |
| 5 — Majority-commit WAL | ✅ Done | `repl_broadcast()` real implementation (repl.c:193+); `repl_wait_majority_ack()` implemented (repl.c:338+); `committed_lsn` in persist_t (persist.c:1029); `persist_truncate_wal_above()` implemented (persist.c:1293+) |
| 6 — Persistent control channel | ✅ Done (changed arch) | `primary_ctrl_fd` in ha_state_t; `ha_on_primary_ctrl_fd_rdhup` triggers re-election; inbound channel via `on_standby_connected` in ha.c; `ha_arm_heartbeat_timeout_vtask` for timeout detection; `ha_find_and_connect_to_primary` is a hook-only reference now (CTRL_STANDBY never dials out) |
| 7 — Client retry | ✅ Done | `xce_request_with_retry()` + `xce_retry_config_t` in request.c/request.h (request.c:1664+) |
| 8 — Majority-commit config | ✅ Done | `OP_upgradeConfigAck` handler; `committed_config_tag` in ha_state_t; `push_config_to_standbys` upgraded; `repl_wait_majority_ack` with `OP_upgradeConfigAck` discriminant |
| 9 — State durability | ✅ Done | `ha_persist_term()` / `ha_restore_term()` in ha.h:317-340; ha-term.bin read in `cluster_init_phase1` (cluster.c:809); `voted_for_name` field in ha_state_t |
| 10 — Observability | ✅ Done | Structured INFO/WARN log statements throughout all paths |

## Gaps: Tasks 11.1–11.5 (Manual Integration Tests)

### Gap 1: Task 11.1 — Concurrent startup and config-fresh election
- **What's needed**: 3-node cluster starts simultaneously; verify only one elects primary; config-fresh node wins.
- **How to run**: See `manual-tests.md` — Test 1 (3-node clean start, verify ctrl-1 primary).
- **Infrastructure**: `deploy-ha.sh` + `start.sh` scripts; log grep for `CTRL_PRIMARY`.
- **Effort**: ~30 min.

### Gap 2: Task 11.2 — Failover and WAL durability
- **What's needed**: Kill primary; verify standby detects via heartbeat timeout → promotes; committed WAL survives.
- **How to run**: `manual-tests.md` Test 2.
- **Effort**: ~30 min.

### Gap 3: Task 11.3 — Unconfigured node bootstrap
- **What's needed**: Start node with `config_tag=0`; verify it receives config via primary's ctrlhello push after election.
- **How to run**: `manual-tests.md` Test 3.
- **Note**: Architecture changed — the deferred fetch now happens via `ha_send_ctrlhello_to_all_standbys` (primary pushes), not `do_one_config_fetch` (standby pulls). This is simpler and already works by default.
- **Effort**: ~30 min.

### Gap 4: Task 11.4 — Config upgrade majority-commit and term durability
- **What's needed**: Config upgrade, partition, timeout; restart with persisted term.
- **How to run**: `manual-tests.md` Test 4.
- **Effort**: ~30 min.

### Gap 5: Task 11.5 — Full ASAN build verification
- **What's needed**: `nix build .#xcecore_debug` clean.
- **Status**: Unit tests pass under `/xce-build controller` (which uses debug+ASAN). Full nix build is the final check.
- **Effort**: ~15 min build time.

## Constraints Carried Forward

- **`ha_find_and_connect_to_primary` is dead code**: Declared in ha.h with test hook but never called in production. CTRL_STANDBY waits for primary to dial in. Safe to delete in cleanup pass.
- **`bootstrap_config_if_needed` removed from myapp_run**: Covered by primary-push model in election-iomgr-refactor. Tasks 5.2–5.3 `[x]` reflect the original design; the implementation pivoted to push-on-ctrlhello.
- **Single-node path**: Verified via test_ha_election.c single-node shortcircuit.

## Implementation Approach Recommendation

No new code needed. Proceed directly to manual integration test execution:

1. Run `/xce-build` to confirm full build clean.
2. Execute `manual-tests.md` tests 1–5 in order.
3. Record results in the Results Template at the bottom of `manual-tests.md`.
4. Mark tasks 11.1–11.5 `[x]` on pass.
5. Run `nix build .#xcecore_debug` for 11.5 ASAN verification.

---

# Gap Analysis — Validation Pass (2026-06-09)

## Scope and Method

Validation pass against the working tree on branch `persistent_ha` (including uncommitted changes to `ha.c`, `ha.h`, `myapp.c`, `raft.c`). All 50 acceptance criteria across the 10 requirements were checked against on-disk code. Note that the architecture has evolved past this spec's original design via four successor specs (`raft-library`, `election-iomgr-refactor`, `ha-heartbeat-liveness`, `ha-repl-hardening`, `ha-peer-connection-reuse`): the blocking election (`am_i_primary_quorum` + semaphore wait) was replaced by a fully async raft-library-driven election inside iomgr, and the standby-pull config bootstrap was replaced by a primary-push model. Several "gaps" below are therefore *sanctioned deviations*, not defects — they need requirements-doc reconciliation, not code.

All checkboxes in `tasks.md` (1.1–11.5) are marked `[x]`, including 9.3+, and manual integration tests 11.1–11.4 are recorded as executed 2026-06-03. The memory note "9.3+ remaining" is stale.

## 1. Requirement-to-Asset Map

Status legend: **Implemented** / **Partial** / **Missing** / **Unknown** / **Constraint** (sanctioned architectural deviation needing spec reconciliation).

### Requirement 1 — Listener and Connection Framework Before Election

| AC | Status | Assets / Evidence |
|----|--------|-------------------|
| 1.1 listener bound before election | Implemented | `myapp.c:113` `cluster_open_listener()` before `ha_start_election_async()`; `cluster_register_listener` at `myapp.c:138` |
| 1.2 iomgr/vtaskmgr before election | Implemented | `myapp.c:122-131` iomgr+vtaskmgr init/start precede election (Step 3/4) |
| 1.3 not-ready rejection in INIT/ELECTING | Implemented | `ha_dispatch_gate()` `ha.c:1412` returns `-XCE_ERR_NOT_READY`; wired at vtask dispatch entry `cluster.c:226` |
| 1.4 standby error includes redirect host/port | **Partial** | `XCE_ERR_STANDBY` / `E_XCE_REDIRECT` returned (`ha.c:1481`, `join.c:241`), `ha_should_redirect()` exists, capnp fields `redirectHost/redirectPort` exist (`controller.capnp:565-566`) — **but no reply path ever populates `redirect_host/redirect_port`**; replies are sent with `data_which = nodata` |

### Requirement 2 — Quorum-Based Controller Election

| AC | Status | Assets / Evidence |
|----|--------|-------------------|
| 2.1 strict-majority quorum | Implemented | `raft.c:223-234` quorum = `n_peers/2 + 1`; `ha_elect_evaluate_and_signal` `ha.c:1995` |
| 2.2 lower-term deny | Implemented | `raft_on_vote_request` Step 1 (`raft.c:110`) |
| 2.3 higher-term update + clear vote | Implemented | `raft.c:121-130` (also steps a live leader down to follower) |
| 2.4 freshness gate deny | Implemented | `raft_cb_vote_eligible` `ha.c:202`; `handle_request_vote` `ha.c:1791` |
| 2.5 catch-up standby vote denied | Implemented | `in_catch_up` check `ha.c:218`, `ha.c:1845` |
| 2.6 one vote per term | Implemented | `raft.c:157-164` (with idempotent re-grant to same candidate — safe extension) |
| 2.7 yield to fresher peer | Implemented | yield gate `ha.c:2017-2031` (`max_resp_tag/max_resp_lsn`) |
| 2.8 tag-zero candidate vs configured peer → fatal exit | **Constraint** | Implemented as *voter-only standby*, not fatal exit: `ha.c:3133-3145` (tag==0 && n_peers>1 → CTRL_STANDBY, wait for push). Strictly safer for ops (node self-heals via config push) but deviates from the EARS text "exit with a fatal error" |
| 2.9 single-node immediate self-elect | Implemented | `raft.c:83-88` `RAFT_ELECTED_IMMEDIATELY`; zero-peer guard `ha.c:1985` |
| 2.10/2.11 split vote → one retry w/ jitter+term++, then standby | **Constraint** | raft library becomes FOLLOWER on no-quorum/timeout (`raft.c:237-244`, `raft_on_election_timeout`); retries are driven indefinitely by the standby liveness watchdog → `raft_on_leader_lost` → delayed `raft_check_and_start_election` (`raft.c:268-290`). Net behavior: unbounded jittered retry loop instead of exactly-once retry. Liveness outcome equivalent or better; spec text outdated |
| 2.12 identical-freshness tiebreak by jitter, not priority | **Partial** | Winner choice never uses priority values; but election *timing* uses deterministic priority-based stagger (`myapp.c:229-239` `priority * election_priority_step_ms`; `raft.c:272-273` `priority * election_timeout_ms`). Priority is repurposed as stagger index, not vote weight — spec wording should be reconciled |

### Requirement 3 — Config Bootstrap for Unconfigured Nodes

| AC | Status | Assets / Evidence |
|----|--------|-------------------|
| 3.1 probe all peers' config tag before candidacy | **Missing (as specified)** | `xce_probe_config_tag()` fully implemented (`config_repl.c:1140`, single-RTT SSL probe) and unit-tested, but **never called in production** — `bootstrap_config_if_needed()` was removed from `myapp_run` (see Constraints in prior pass). Replacement: unconditional voter-only gate when `my_config_tag==0 && n_peers>1` (`ha.c:3133`) |
| 3.2 voter-only participation, deferred fetch | Implemented (pivoted) | voter-only path `ha.c:3133-3145`; config arrives via primary push (`push_config_to_standbys` from `raft_cb_become_leader`/COMMITTING driver) |
| 3.3 all peers tag-zero → normal candidacy | **Partial / Unknown** | No probe means a tag-0 node cannot distinguish "peers have config" from "all-zero cold start"; it always goes voter-only first. Recovery relies on the heartbeat watchdog firing → `ha_on_primary_ctrl_fd_rdhup` → `raft_on_leader_lost` re-election, which **bypasses** the tag-0 gate (`ha.c:3183-3291`). All-zero cold start thus converges, but only after `heartbeat_timeout_ms` (~10s) delay per round; not covered by an automated test |
| 3.4 exponential-backoff probe retry 500ms→10s | **Missing (as specified)** | No probe retry loop exists; the passive watchdog loop replaces it. Spec text obsolete under the push model |
| 3.5 deferred fetch from proven winner | Implemented (pivoted) | Primary pushes config + ctrlhello to all standbys on COMMITTING (`ha.c:2087-2103`); `fetch_config_from_primary`/`do_one_config_fetch` standby-pull removed by design (CLAUDE.md: standbys never dial out) |

### Requirement 4 — Startup Validation and Safety Checks

| AC | Status | Assets / Evidence |
|----|--------|-------------------|
| 4.1 even-N peer rejection | Implemented | `validate_peer_count()` `startup_validation.c:90`; double-enforced in `raft_init` (`raft.c:21`) |
| 4.2 cert/key pre-flight with full path | Implemented | `validate_cert_files()` `startup_validation.c:106` (-ENOENT, path in message), wired in `main.c` |
| 4.3 vcpus >= 1 | Implemented | `validate_vcpus()` `startup_validation.c:25`; `--vcpus` argp option in `main.c` |
| 4.4 config-file vcpus deprecation WARN | Implemented | `validate_startup_config()` `startup_validation.c:46`; warned at `myapp.c:80-86` |

### Requirement 5 — Majority-Commit WAL Replication

| AC | Status | Assets / Evidence |
|----|--------|-------------------|
| 5.1 job submit / state transitions / queue changes majority-committed | **Partial** | Only `WAL_JOB_SUBMIT` routes through `persist_wal_sync_append()` (`persist.c:1118`). Job state transitions (`WAL_JOB_START/FINISH/FAIL/...`, `persist.c:1163`) and queue changes (`WAL_QUEUE_CTRL`, `persist.c:1194`) use `persist_wal_async()` — no majority wait |
| 5.2 host state / checkpoint refs async | Implemented | worker/host events → `persist_wal_async`; checkpoint markers never replicated (`persist.c:395`) |
| 5.3 standby durable write + ACK with LSN | Implemented | `OPCode_walAck` flow, `repl_on_ack` (`repl.c:178`), `acked_lsn` tracking |
| 5.4 lag eviction + catch-up trigger | Implemented | `pending_ack_count > max` eviction with inline catch-up start, `repl.c:214-222` |
| 5.5 commit timeout → error to caller | **Constraint** | Deliberate *degraded mode*: `persist_wal_sync_append` reports `-ETIMEDOUT/-ENODEV` via `sync_rc_out` but returns 0 and the operation proceeds with local-only durability (`persist.c:1003-1043`; hook comment `persist.c:1114`). This is an AP-leaning deviation from the spec's CP statement — documented in code, not in requirements |
| 5.6 truncate above committed LSN on promotion | Implemented | `ha_compute_safe_committed_lsn()` from voter replies (`ha.c:373-437`) → `persist_truncate_wal_above(safe_lsn)` in `raft_cb_become_leader` (`ha.c:471`); per controller CLAUDE.md this is the verified-against-TLA+ design |
| 5.7 truncate on catch-up reconnect | **Partial** | RDHUP/watchdog failover path truncates to `last_known_committed_lsn` before re-election (`ha.c:3204-3207`). A standby that *restarts* (rather than detecting RDHUP) replays its full WAL including uncommitted tail; no truncation occurs before it rejoins and receives catch-up entries — divergent-tail handling on the restart path is unverified |
| 5.8 single-node local write satisfies majority | **Partial** | `repl_wait_majority_ack` returns 0 when `n_active == 0` (`repl.c:384`), but `persist_wal_sync_append` short-circuits *before* calling it when `nstandbys == 0` and sets `sync_err = -ENODEV` **without advancing `committed_lsn`** (`persist.c:1023-1031`). Single-node `committed_lsn` appears to advance only via the election-time safe-LSN path — runtime advancement per entry is not evident. Needs verification (interacts with WAL trim guard, which uses `committed_lsn` as threshold) |

### Requirement 6 — Persistent Standby-to-Primary Control Channel

| AC | Status | Assets / Evidence |
|----|--------|-------------------|
| 6.1 standby establishes persistent TLS connection to primary | **Constraint** | Direction inverted by deliberate design (controller CLAUDE.md "Inbound vs. Outbound Connection Ownership"): the **primary dials out** to standbys; standby-side `ha_find_and_connect_to_primary()` is dead code (declared `ha.h:741`, never called in production). The functional goal (persistent channel, no per-RPC connections) is achieved |
| 6.2 disconnect → immediate election | Implemented | EPOLLRDHUP on `primary_ctrl_fd` → `ha_on_primary_ctrl_fd_rdhup()` (`ha.c:3183`) |
| 6.3 heartbeat resets timeout | Implemented | `ha_on_heartbeat_received()` on every NOP/ctrlhello (`cluster.c:249, 283`) |
| 6.4 heartbeat timeout + jitter → election | Implemented | `ha_arm_heartbeat_timeout_vtask` liveness poll; manual test 11.2 measured ~11s detection (configured timeout, not sub-second — sub-second applies to the RDHUP path) |
| 6.5 sync missing state before admitting to replication set | Implemented | heartbeat-reply-driven driver: standby `stateReport` → `ha_repl_decide()` (`ha.c:1157-1176`) → stream WAL / snapshot / config; `in_catch_up` gates quorum membership (`repl.c:367`) |
| 6.6 snapshot when standby older than oldest WAL segment | Implemented | `standby_lsn < oldest_wal_lsn → HA_REPL_SEND_SNAPSHOT` (`ha.c:1176`), `ha_send_snapshot_to_standby()` chunked transfer (`config_repl.c:1578`), standby handler `cluster.c:182` |
| 6.7 config push after catch-up on tag mismatch | Implemented | same driver pushes config on tag divergence; `push_config_to_standbys` |
| 6.8 winner closes outbound, accepts inbound | Implemented | role flip handled in `raft_cb_become_leader` / RDHUP path; surviving vote eps promoted to `standby_endpoints`; late joiners get fresh outbound connects from `primary_heartbeat_fn` |

### Requirement 7 — Client-Side Transparent Retry (libxce)

| AC | Status | Assets / Evidence |
|----|--------|-------------------|
| 7.1 NOT_READY → backoff retry | Implemented | `xce_request_with_retry()` `request.c:1664`; tests `test_libxce_retry.c` |
| 7.2 STANDBY + redirect → follow redirect | **Partial** | Client logic implemented and unit-tested via hook, but **no controller reply ever carries `redirect_host/port`** (see 1.4) — end-to-end the redirect-follow path can never trigger; falls back to backoff |
| 7.3 TCP refused/timeout → reconnect retry | Implemented | `request.c` retry classification |
| 7.4 non-retriable surfaced immediately | Implemented | AUTH/PERM/INVALID/NOENT surfaced without retry |
| 7.5 configurable params + defaults | Implemented | `xce_retry_config_t`, `XCE_RETRY_DEFAULTS` (`request.h:140-150`) |
| 7.6 thundering-herd jitter | Implemented | jittered exponential backoff; 50-client concurrency test |
| (adoption) | **Missing** | Zero production call sites: worker, scheduler, and apiserver backend never call `xce_request_with_retry` — the wrapper is shipped but unused |

### Requirement 8 — Majority-Commit Config Replication

| AC | Status | Assets / Evidence |
|----|--------|-------------------|
| 8.1 majority commit for config push | Implemented | `push_config_to_standbys` → `repl_wait_majority_ack(..., OPCode_upgradeConfigAck)` (`config_repl.c:905`) → `committed_config_tag` advanced (`config_repl.c:908`) |
| 8.2 standby durable write + ACK | Implemented | `handle_repl_config_push` + `OP_upgradeConfigAck`; `acked_config_lsn` tracked separately from WAL acks (`repl.c:375-380`) |
| 8.3 timeout → error | Implemented | `-ETIMEDOUT` from `repl_wait_majority_ack`, WARN logged |
| 8.4 promoted standby serves only committed config tag | **Partial** | `last_known_committed_config_tag` captured from CtrlHelloReply (`cluster.c:261`) and `committedConfigTag` piggybacked on heartbeat (capnp); `committed_config_tag` consulted at `ha.c:2514-2521` and safe-tag computed on promotion (`ha.c:485`). Enforcement at *serve time* (refusing to serve a fresher-but-uncommitted config) not explicitly verified |
| 8.5 single-node local write = majority | Implemented | safe-tag path on become-leader; `committed_config_tag` advanced on push without standbys |

### Requirement 9 — Election State Durability

| AC | Status | Assets / Evidence |
|----|--------|-------------------|
| 9.1 persist term+vote before reply | Implemented | raft `on_persist` fires before any grant/RPC (`raft.c:77, 176`); `ha_persist_term()` atomic write+rename with magic+CRC32 (`ha.c:1647-1681`) |
| 9.2 restore on startup | Implemented | `ha_restore_term()` (`ha.c:1683+`); manual test 11.4 confirmed `restored term=2` on restart |
| 9.3 missing file → term 0, no vote | Implemented | missing/short-read/bad-magic/CRC-mismatch all degrade safely to term 0 with WARN (`ha.c:1715-1737`) |

### Requirement 10 — Observability

| AC | Status | Assets / Evidence |
|----|--------|-------------------|
| 10.1 election outcome: term + votes granted + quorum size | **Partial** | FSM transition INFO logs carry term (`ha.c:507, 659, 870, 881, 2080, 2158`); majority-ack log carries acks+quorum (`ha.c:2929`); but no single election-complete record logs *votes granted* and *quorum size* together |
| 10.2 bootstrap probe finding | **Partial** | voter-only decision logged (`ha.c:3134`); per-peer probe logs exist in `xce_probe_config_tag` but that path is unused in production |
| 10.3 WAL quorum INFO / timeout WARN | Implemented | `repl.c:408-410` |
| 10.4 config quorum INFO / timeout WARN | Implemented | `config_repl.c` push path logs both |
| 10.5 FSM transition INFO | Implemented | all transitions logged, including RDHUP (`ha.c:3199`) |

### Summary Tally (by requirement)

| Req | Verdict |
|-----|---------|
| R1 | Implemented except redirect payload (1.4 Partial) |
| R2 | Implemented; 2.8/2.10/2.11/2.12 are sanctioned deviations needing spec reconciliation |
| R3 | Pivoted to primary-push; 3.1/3.4 Missing-as-specified, 3.3 Unknown (cold-start convergence untested) |
| R4 | Implemented (4/4) |
| R5 | Core implemented; 5.1 coverage gap, 5.5 CP→degraded-mode deviation, 5.7 restart-path gap, 5.8 needs verification |
| R6 | Implemented; 6.1 direction inverted by design (Constraint) |
| R7 | Library implemented; redirect payload + production adoption Missing |
| R8 | Implemented; 8.4 serve-time enforcement Partial |
| R9 | Implemented (3/3) |
| R10 | Implemented except consolidated election-outcome record (10.1) |

**Net: ~40 of 50 acceptance criteria fully implemented; 7 partial; 3 missing-as-specified (all three superseded or deliberately pivoted).** No requirement is wholly unimplemented.

## 2. Remaining Gaps — Concrete List

1. **G1 — Redirect payload never populated** (R1.4, R7.2): capnp fields exist; standby replies send `nodata`. Client redirect-follow logic is dead end-to-end.
2. **G2 — WAL majority-commit covers only job submission** (R5.1): state transitions and queue changes are async-only; a failover can lose acknowledged state changes.
3. **G3 — Commit timeout is degraded-mode, not an error** (R5.5): spec promises CP (timeout error to caller); code chooses availability. Needs an explicit, documented decision.
4. **G4 — Single-node `committed_lsn` runtime advancement unverified** (R5.8): `-ENODEV` short-circuit skips the advance; interacts with the WAL trim guard.
5. **G5 — Standby restart path skips divergent-tail truncation** (R5.7): truncation happens only in the RDHUP/watchdog path.
6. **G6 — Config probe path unwired** (R3.1/3.3/3.4): `xce_probe_config_tag` is production-dead; all-zero cold-start converges only via watchdog timeout and lacks a test.
7. **G7 — libxce retry has zero production adopters** (R7 objective): worker/scheduler/apiserver still have whatever ad-hoc behavior they had.
8. **G8 — Observability**: no consolidated election-outcome log with vote tally + quorum size (R10.1).
9. **G9 — Spec/code reconciliation debt**: 2.8 (standby instead of fatal exit), 2.10/2.11 (watchdog retry loop instead of retry-once), 2.12 (priority-based stagger), 3.x (push model), 5.5 (degraded mode), 6.1 (inverted channel direction) — all stable code behavior contradicting EARS text.
10. **G10 — Cleanup**: `ha_find_and_connect_to_primary` dead code; bounded per-RDHUP `elect_ctx` leak (`ha.c:3252-3258`, documented UAF-avoidance).

## 3. Implementation Approach Options

### Option A — Extend existing components in place (code-first)
Close G1–G8 directly in `ha.c` / `persist.c` / `repl.c` / `join.c` / worker+scheduler clients; patch requirements.md text afterward.
- Files: `join.c`+`reply.c` (G1), `persist.c` hook routing (G2/G4), `cluster.c` startup replay (G5), `myapp.c`+`config_repl.c` (G6), worker/scheduler request sites (G7), `ha.c` logging (G8).
- ✅ No new modules; follows established patterns (every gap has an existing seam and a testsuite file).
- ✅ Fastest path to closing the spec.
- ❌ G2/G3 are *policy* changes with latency and availability consequences — landing them silently inside this aging spec hides a significant decision.
- ❌ `ha.c` is already 3,818 lines; further growth degrades maintainability.

### Option B — Spec-first reconciliation, then targeted follow-up spec
Amend requirements.md/design.md to match the as-built architecture (push bootstrap, inverted channel, watchdog retry, degraded mode), close only the unambiguous small defects (G1, G4, G8, G10) here, and move G2/G3/G5/G7 into the active `ha-repl-hardening` spec (or a new `ha-cp-commit` spec) with their own requirements review.
- ✅ Honest record: the EARS text stops contradicting shipped behavior; the CP-vs-degraded decision gets explicit operator-facing review.
- ✅ Keeps this spec closeable; G2/G3 get proper TDD treatment in a spec that already owns repl hardening.
- ❌ More process; durability gap (G2) stays open until the follow-up lands.
- ❌ Risk that "deferred" becomes "dropped" without tracking.

### Option C — Hybrid (recommended)
Phase 1 (this spec, S-size code): G1 redirect payload, G4 single-node commit advance verification/fix, G8 election-outcome log, G10 cleanup, plus requirements.md deviation annotations (G9) and a cold-start integration test for G6's watchdog fallback.
Phase 2 (`ha-repl-hardening` spec): G2 (extend sync-append coverage to state transitions/queue changes — likely with group-commit batching to bound latency), G3 (explicit CP-vs-degraded decision + config knob), G5 (restart-path truncation), G7 (retry adoption across daemons).
- ✅ Ships every low-risk fix now; routes every policy-bearing change through review.
- ✅ Matches the existing spec topology (repl-hardening already owns quorum/commit semantics).
- ❌ Requires coordinating two specs; G6's probe-vs-watchdog question must be answered during reconciliation (recommend: delete the probe path, document the watchdog fallback, keep the test).

## 4. Effort & Risk (remaining work)

| Gap | Effort | Risk | Justification |
|-----|--------|------|---------------|
| G1 redirect payload | S | Low | Fields exist in schema/typemap; populate in 2 reply sites + 1 e2e test |
| G2 sync-append coverage | M | Medium | Mechanical routing change but puts majority-RTT on hot job-state path; needs batching design + latency measurement |
| G3 CP-vs-degraded decision | S (code) / M (decision) | High | One-line behavior flip, but it trades availability for consistency cluster-wide; needs operator sign-off and failure-mode tests |
| G4 single-node committed_lsn | S | Medium | Small fix, but wrong answer corrupts WAL trim guard invariant (`committed_lsn <= applied_lsn`) |
| G5 restart-path truncation | M | Medium | Touches startup replay ordering; must honor TLA+ safe-commit-point rule (never truncate via stale ctrlhello value) |
| G6 cold-start test + probe removal | S | Low | Behavior already converges via watchdog; work is test + dead-code deletion + doc |
| G7 retry adoption | M | Medium | Multi-daemon change; behavioral shift in error surfacing for worker/scheduler/apiserver |
| G8 election-outcome log | S | Low | Data already in `ha_elect_ctx_t`/raft state at decision points |
| G9 spec reconciliation | S | Low | Documentation only; six identified deltas |
| G10 cleanup | S | Low | Dead-code deletion + bounded-leak fix behind existing tests |

Aggregate remaining: **~M total for Phase 1 (all S items), ~L for Phase 2** (G2+G3+G5+G7 under ha-repl-hardening).

## 5. Recommendations

1. **Adopt Option C.** Close this spec with Phase 1; transfer G2/G3/G5/G7 into `ha-repl-hardening` as explicit requirements so they get their own approval gate.
2. **Reconcile requirements.md now** (G9): rewrite AC 2.8, 2.10–2.12, 3.1–3.5, 5.5, 6.1 to describe the as-built push/watchdog/outbound-primary architecture. The controller CLAUDE.md and TLA+ mapping are already correct — the spec is the lagging artifact.
3. **Decide G3 deliberately.** The spec's headline promise is "zero data loss on failover" (CP); the code ships degraded-mode availability. Either add a `commit_strict` config knob defaulting per the decision, or amend the spec's CP claim. Do not leave the contradiction implicit.
4. **Fix G1 immediately** — it is the cheapest fix with end-to-end user impact (failover redirect for clients), and it unblocks the already-tested libxce retry path.
5. **Delete dead code** (`ha_find_and_connect_to_primary`, production `xce_probe_config_tag` path if reconciliation confirms the push model) rather than leaving two competing bootstrap narratives in the tree.

### Research Needed

- **RN1**: Does single-node `committed_lsn` ever advance at runtime (G4)? Trace `persist_wal_sync_append` + WAL trim guard in a single-node soak; check whether `ha-repl-hardening` task 6.5 already addressed this.
- **RN2**: Latency budget for extending majority-commit to job state transitions (G2) — measure per-entry RTT on 3-node LAN; evaluate group commit / pipelined acks before committing to a design.
- **RN3**: Restart-path divergence (G5) — construct the failure: standby with uncommitted WAL tail restarts, rejoins, receives catch-up; verify whether `apply_wal_entry` overwrite semantics mask or corrupt the divergent tail.
- **RN4**: All-zero cold-start convergence time and liveness (G6) — 3-node cluster, no configs anywhere; confirm watchdog-driven election converges and measure time-to-primary.
- **RN5**: Serve-time enforcement for 8.4 — can a promoted standby serve a config tag above `committed_config_tag` in any window between promotion and first quorum push?

---

## Gap Re-Audit (2026-06-18)

> Trigger: post-implementation gap audit. Tasks 46/46 `[x]`, `phase=implemented`.
> Method: re-verified the 2026-06-09 gap list (G1–G10) against the current working
> tree on `persistent_ha`. Since the prior pass, two successor specs landed and
> closed several of the gaps this spec recorded: **`libxce-failover`** (client-side
> transparent failover/redirect) and **`ha-commit-durability`** (bug-661, landed
> 2026-06-18). The earlier majority-commit model was also superseded by a
> `wal_classify()` WAIT/RETAIN scheme from `ha-commit-barrier-rollout`. Line numbers
> below reflect the current tree (ha.c is now 5,338 lines; persist.c 2,587).

### Headline change vs 2026-06-09

The 2026-06-09 pass found ~40/50 ACs fully implemented with 10 open gaps (G1–G10).
**Five of those gaps are now CLOSED** by successor-spec work; the residual set is
smaller and almost entirely *spec-text reconciliation debt* rather than missing code.

### Requirement-to-Asset Map — Delta Since 2026-06-09

| Gap (2026-06-09) | Req | New Status (2026-06-18) | Evidence |
|---|---|---|---|
| **G1** Redirect payload never populated | R1.4, R7.2 | **CLOSED** | `fill_redirect_reply()` populates `reply->redirect_host/redirect_port` from `ha->primary_host/port` (`join.c:28-32`); called on the `-E_XCE_REDIRECT` path (`join.c:282`); covered by `test_join_redirect.c`. Comment cites `libxce-failover Req 8.1`. |
| **G7** libxce retry has zero adopters | R7 obj. | **CLOSED (re-homed)** | Orphaned no-op `xce_request_with_retry()` + `xce_retry_send_hook` were **removed** (`request.c:1923-1936`); superseded by the `xcecli_*` failover loop in `src/lib/client.c` + `src/lib/client_failover.c`. `XCE_RETRY_DEFAULTS` retained as the shared tunable. Production redirect-follow is now live end-to-end (G1 + client_failover). |
| **G10** `ha_find_and_connect_to_primary` dead code | cleanup | **CLOSED** | No production `.c` reference remains (grep over `src/controller/*.c` returns nothing). Dead-code deletion done. |
| **G2** WAL majority-commit covers only job submit | R5.1 | **SUPERSEDED (model changed)** | Replaced by `wal_classify()` WAIT/RETAIN scheme (`persist.c:2004`). `WAL_JOB_SUBMIT` and `WAL_QUEUE_CTRL` are `WAL_MODE_WAIT` (commit-gated before requester reply); `WAL_WORKER_CTRL` JOIN/UPDATE are WAIT; job state transitions (`WAL_JOB_START/FINISH/FAIL/...`) are `WAL_MODE_RETAIN` — replicated and retained-until-committed but not blocking the worker-fact report. The request-shaped side of dual-origin events is realized by the handler barrier in `do_jobcontrol_op`. The original "everything sync-blocks" framing was deliberately not adopted (latency); coverage is now classification-driven, owned by `ha-commit-barrier-rollout` / `ha-commit-durability`. |
| **G3** Commit timeout degraded-mode vs CP | R5.5 | **SUPERSEDED (bug-661 / ha-commit-durability)** | Durability invariant now GUARANTEED: acknowledged ⇒ `fdatasync`'d on a fault-tolerant majority. `majority_candidate_locked` caps the primary self-contribution at `persist_durable_lsn`; `fsync_worker` group-commit flusher advances `durable_lsn` only on `fdatasync` success (fail-loud rc-gate); standby `fdatasync`s before WalAck. Verified by `test_commit_durability.c`, the ack-before-fdatasync E2E, and the rolling-failover soak (zero acked-job loss). Controller CLAUDE.md checklist item flipped to GUARANTEED 2026-06-18. The old "silent local-only degrade" path is gone. |

### Residual Gaps (still open after 2026-06-18)

| ID | Req | Status | Detail |
|---|---|---|---|
| **R1 (was G6)** Config probe path unwired | R3.1, R3.3, R3.4 | **Missing-as-specified (stable)** | `xce_probe_config_tag()` is fully implemented + unit-tested (`config_repl.c:1219`) but **still not called in production** — `myapp.c` has no probe call (grep confirms). The tag-0 voter-only gate (`ha.c:4495`) uses static `my_config_tag`/`n_peers`, not probe results. The comment at `ha.c:4493` ("xce_probe_config_tag path already called in myapp_run") is **stale/incorrect** — no such call exists. All-zero cold-start converges only via watchdog-driven re-election (~heartbeat_timeout), and that path lacks an automated convergence test (prior RN4). |
| **R2 (was G9)** Spec/code reconciliation debt | 2.8, 2.10–2.12, 3.1–3.5, 5.5, 6.1 | **Open (docs only)** | requirements.md EARS text still describes: fatal-exit for tag-0 candidate (code: voter-only standby), exactly-once split-vote retry (code: unbounded watchdog retry loop), exponential-backoff probe (code: passive watchdog), standby-dials-primary (code: primary dials out — inverted by design per controller CLAUDE.md), CP-timeout-error (code: now GUARANTEED durability, but worded as the old model). spec.json `note` still lists the now-closed G1/G3 framings. The TLA+ mapping + controller CLAUDE.md are authoritative and correct; the spec is the lagging artifact. |
| **R3 (was part of G4)** Single-node `committed_lsn` runtime advance | R5.8 | **Likely closed — verify** | `test_single_node_committed_lsn_advances` (`test_p2_integration.c:251`) and `test_legacy_sync_append_holds_committed_at_durable` (`test_commit_durability.c:660`) now exist, suggesting ha-commit-durability addressed the prior `-ENODEV` short-circuit concern. Recommend a quick read of those tests to confirm RN1 is retired before closing. |
| **R4 (was G5)** Standby restart-path divergent-tail truncation | R5.7 | **Re-scoped to recovery** | The RDHUP/watchdog failover path truncates to `last_known_committed_lsn`. The *restart* path is now governed by `ha-commit-durability` recovery (`recover_verdict_t` clean-vs-corruption, durable-floor re-derivation, pre-promotion fence). Whether a restarted standby with an uncommitted tail is correctly fenced/caught-up before rejoin is owned by that spec's recovery tests, not this one. Verify via `ha-commit-durability` rather than re-opening here. |
| **R5 (was G8)** Consolidated election-outcome log | R10.1 | **Open (minor)** | FSM-transition INFO logs carry term, and majority-ack logs carry acks+quorum, but there is still no single election-complete record emitting *votes-granted AND quorum-size* together. Data is available in `ha_elect_ctx_t`/raft state at the decision point. Low value, low effort. |

### Residual Tally

- **Closed since 2026-06-09:** G1, G7, G10 (outright), G2, G3 (superseded by a better model in successor specs).
- **Still open:** R1 (probe unwired + stale comment), R2 (spec-text reconciliation), R5 (consolidated election log). R3 likely-closed-pending-verify, R4 re-homed to ha-commit-durability recovery.
- **Net AC posture:** the durability and redirect deltas mean the *functional* CP promise of this spec is now met or exceeded; the open items are documentation honesty (R2), one unwired-but-replaced bootstrap path (R1), and one cosmetic log (R5).

### Implementation Approach Options (for the residual work)

#### Option A — Close-and-reconcile in place (recommended)
Phase-1-only, all S-size, no successor spec needed:
1. **R2** rewrite requirements.md AC 2.8 / 2.10–2.12 / 3.1–3.5 / 5.5 / 6.1 to describe the as-built push/watchdog/outbound-primary/guaranteed-durability architecture; refresh the spec.json `note` (G1/G3 are no longer "known deviations"). Docs only.
2. **R1** delete the stale `ha.c:4493` comment; either (a) delete the unused `xce_probe_config_tag` production path and document the static-gate + watchdog-convergence model, or (b) add a cold-start convergence integration test (3-node, all configs absent) so the watchdog fallback is covered. Recommend (a)+test.
3. **R5** add a single INFO "election complete: term=T votes=G/Q role=…" record at the become-leader / become-follower decision point.
4. **R3** read the two new single-node durability tests; if they cover runtime advance, retire RN1 and mark R5.8 closed.
- ✅ Closes the spec honestly with no policy decisions left implicit; smallest surface.
- ✅ Every item has an existing seam + testsuite file.
- ❌ Touches requirements.md (aging artifact) — but that is the point.
- **Effort: S total. Risk: Low** (docs + dead-code delete + one log line + one test).

#### Option B — Defer reconciliation, close only code defects
Fix R1 comment + R5 log now; leave requirements.md as-is with an expanded spec.json deviation note pointing at controller CLAUDE.md as the source of truth.
- ✅ Even smaller; no requirements.md churn.
- ❌ Leaves EARS text contradicting shipped behavior — the recurring complaint across every prior pass. Reconciliation debt compounds.
- **Effort: XS. Risk: Low (but accrues doc debt).**

#### Option C — Fold residuals into successor specs and close this one verbatim
Mark ha-config-bootstrap closed; track R1/R2/R5 under `ha-commit-durability` (R3/R4 already there) or a small `ha-spec-reconcile` doc spec.
- ✅ This spec stops being touched; clean lineage.
- ❌ R2 spans this spec's own EARS text — odd to reconcile it from another spec. R5 is trivially this spec's.
- **Effort: S (mostly bookkeeping). Risk: Low; tracking-loss risk medium.**

### Effort & Risk — Residual

| Item | Effort | Risk | Notes |
|---|---|---|---|
| R1 probe path + stale comment + cold-start test | S | Low | Behavior already converges via watchdog; delete dead path or add test |
| R2 requirements.md / spec.json reconciliation | S | Low | Docs only; 6 EARS deltas; CLAUDE.md is the correct source |
| R3 single-node commit verify | XS | Low | Read 2 existing tests; likely already closed |
| R4 restart-path truncation | — | — | Re-homed to ha-commit-durability recovery; verify there, not here |
| R5 consolidated election log | S | Low | One INFO line; data already in elect_ctx |

**Recommendation:** Option A. The heavy lifting (durability, redirect, retry adoption, dead-code) is done by successor specs; the only thing standing between this spec and an honest `implemented`/closed state is requirements-text reconciliation (R2), one unwired-but-superseded path (R1, delete + document), and a cosmetic log (R5). All S/XS, all Low risk.