# Gap Analysis: ha-config-bootstrap

*Generated: 2026-05-27. Branch: persistent_ha.*

---

## Analysis Summary

- **Scope**: 10 requirements across 8 distinct changes (A–H). The codebase has strong foundational infrastructure (iomgr, repl slot management, WAL, SSL) but the HA election core, quorum commit, and control channel are mostly absent or stubbed.
- **Largest gaps**: Raft-style quorum election (Req 2) and persistent control channel with full state sync (Req 6) are full rewrites/additions with no current equivalent logic; majority-commit WAL (Req 5) requires implementing the two stub functions at the heart of replication.
- **Strong reusable assets**: `repl_add_standby/remove/start_catchup/complete_catchup/needs_snapshot` are real implementations; the SSL connect pattern (`connect_ssl_raw`), xorshift64 PRNG, `ha_do_promote/demote`, and iomgr epoll machinery are all directly reusable.
- **Risk concentration**: Req 2 and Req 6 carry High risk; Req 5 carries High risk due to `committed_lsn` cross-cutting `persist_t`, `repl.c`, and `ha_state_t`. The remaining requirements are Medium or Low.
- **Recommended approach**: Hybrid (Option A) — implement changes A→D→B→C→E→F→G→H in dependency order, writing C tests first per TDD, one atomic commit per change.

---

## 1. Current State Investigation

### 1.1 HA Core (ha.h / ha.c)

**`ctrl_role_t` enum** — only two values: `CTRL_PRIMARY = 0`, `CTRL_STANDBY = 1`.  
Missing: `CTRL_INIT`, `CTRL_ELECTING` (required by Req 1).

**`ha_state_t`** — current fields of interest:

| Field | Status | Notes |
|---|---|---|
| `term` (_Atomic uint64_t) | Exists | Used for fencing |
| `role` (_Atomic int32_t) | Exists | Cast to ctrl_role_t |
| `my_priority` | Exists | LSF priority index — kept for compat but unused in Raft |
| `applied_lsn` (_Atomic uint64_t) | Exists | Reusable for vote freshness check |
| `primary_host[XCE_MAXNAME_LEN]` | Exists | Reusable |
| `primary_port` | Exists | Reusable |
| `last_heartbeat` (struct timespec) | Exists | Replaces with `EPOLLRDHUP` path in Req 6 |
| `in_catch_up` (_Atomic int) | Exists | Reusable for Clarification 6 vote exclusion |
| `jitter_seed` (uint64_t) | Exists | xorshift64, directly reusable |
| `ssl_ctx` (void*) | Exists | Reusable |
| `standby_endpoints` (zarray_t*) | Exists | Superseded by per-standby fd table in Req 6 |
| `voted_for_term` (uint64_t) | **MISSING** | Req 9 |
| `voted_for_name` (char[]) | **MISSING** | Req 9 |
| `primary_ctrl_fd` (int) | **MISSING** | Req 6 |
| `defer_config_fetch` (bool) | **MISSING** | Req 3 |
| `elect_ctx` (_Atomic ptr) | **MISSING** | Req 2 |
| `last_known_committed_lsn` (_Atomic uint64_t) | **MISSING** | Req 5 |
| `heartbeat_seen` (sem_t) | **MISSING** | Req 1 |

**`ha_config_t`** — missing: `commit_timeout_ms`. All election fields present (election_timeout_ms, jitter_max_ms, etc.) but values tuned for LSF model.

**`am_i_primary()`** — LSF priority probe via per-peer pthreads. No quorum, no config_tag/LSN in vote, no term/voted_for persistence. Must be replaced entirely with `am_i_primary_quorum()`.

**`ha_check_priority_stepdown()`** — permanently unreachable once term > 0; exists only for the term=0 simultaneous-boot race. Must be retired (Req 2).

**`ha_check_heartbeat_nocluster()`** — timestamp polling with configurable timeout. Replaced by `EPOLLRDHUP` fast path + heartbeat timeout vtask (Req 6). Function itself can be deleted.

**`ha_watchdog_thread()`** — sets `failover_requested` but cannot call `ha_promote_to_primary()` (no `cluster_t`). Partially functional stub. Replaced by vtask-based heartbeat monitor (Req 6).

**`start_heartbeat_listener()` / `stop_heartbeat_listener()`** — no-op stubs with comment "full implementation in cluster.c Task 7". **Missing** (Req 1).

**`ha_fini()`** — missing `SSL_CTX_free(ha->ssl_ctx)`. ASAN-visible leak (~5 KB).

### 1.2 Startup Sequence (myapp.c / main.c)

**myapp.c current order**:
```
cluster_init_phase1()
initialize_openssl() + create_ssl_context()
am_i_primary()                     ← election here
load_primary_config() / fetch_config_from_primary()
cluster_init_phase2()              ← iomgr, vtaskmgr, bind+listen inside
cluster_start()                    ← more network init
cluster_run()
```

**Bind+listen** happens inside `cluster_init_phase2()` or `cluster_start()` — **after** election. Gap: concurrent-startup nodes probe an unbound port (Req 1).

**No `bootstrap_config_if_needed()`** — standby path goes straight to `fetch_config_from_primary()` post-election without checking whether local config_tag is 0 or whether peers have config. Gap (Req 3).

**main.c**:
- `--vcpus N` arg: **MISSING**. vcpus comes from cluster config read inside phase2. This creates a dependency that prevents vtaskmgr from running before election (Req 1, 4).
- Peer count validation: **MISSING**. Even-N peer lists accepted silently (Req 4).
- Cert pre-flight `access()` check: **MISSING**. Cert paths computed but SSL context created later in myapp.c, not with early-exit error in main.c (Req 4).

### 1.3 Replication (repl.c / repl.h)

| Function | Status | Notes |
|---|---|---|
| `repl_init / fini` | Real | Lifecycle ok |
| `repl_add_standby` | Real | Slot management implemented |
| `repl_remove_standby` | Real | Implemented |
| `repl_start_catchup` | Real | Sets in_catch_up=1 |
| `repl_complete_catchup` | Real | Clears in_catch_up, broadcasts cond |
| `repl_needs_snapshot` | Real | Compares standby_lsn vs oldest_wal_lsn |
| `repl_on_ack` | Real | Updates acked_lsn, signals cond |
| `repl_trim_lsn` | Real | WAL trim guard implemented |
| `repl_broadcast` | **STUB** | Returns 0 without sending anything (Req 5, 8) |
| `repl_wait_ack` | Partial | Waits for ANY non-catch-up standby, not majority (Req 5) |
| `repl_wait_majority_ack` | **MISSING** | Req 5, 8 |

**`standby_conn_t`** — missing `pending_ack_count` for backpressure (Req 5).

### 1.4 Persistence (persist.h / persist.c)

- `current_lsn` and `checkpoint_lsn` exist. **`committed_lsn`** — **MISSING** (Req 5).
- `persist_wal_sync_append()` uses `repl_wait_ack()` (any-standby) — must upgrade to `repl_wait_majority_ack()`.
- `persist_truncate_wal_above()` — **MISSING** (Req 5, 6).
- `persist_config_t.sync_timeout_ms` exists — analogous to `commit_timeout_ms` needed for majority-commit.

### 1.5 Config Replication (config_repl.c)

- `do_one_config_fetch()` — **wired** (recent commit `2aae45f` on branch: "wire do_one_config_fetch() — full SSL roundtrip"). Not a stub.
- `bootstrap_config_if_needed()` — **MISSING** (Req 3).
- `xce_probe_config_tag()` — **MISSING**. New lightweight RPC needed (Req 3).
- `push_config_to_standbys()` — declared in config_repl.h; likely fire-and-forget (no majority wait). **Gap**: must upgrade to majority-commit (Req 8).
- `OP_probeConfigTag` Cap'n Proto schema — **MISSING** (Req 3).
- `OP_upgradeConfigAck` Cap'n Proto schema — **MISSING** (Req 8).
- `committed_config_tag` — **MISSING** (Req 8).

### 1.6 Control Channel (ha.c / cluster.c)

- No persistent standby-to-primary TCP connection. Each config fetch and WAL entry opens a new SSL connection via `connect_ssl_raw()`.
- `ha_find_and_connect_to_primary()` — **MISSING** (Req 6).
- Extended `OP_ctrlHello` with applied_lsn, config_tag — **MISSING** (Req 6).
- `OP_snapshotTransfer`, `OP_walCatchupEntry` — **MISSING** (Req 6).
- `handle_ctrlhello()` extended handler — **MISSING** (Req 6).
- Three-tier primary discovery — **MISSING** (Req 6).
- EPOLLRDHUP failover detection — **MISSING** (Req 6).
- Heartbeat timeout vtask replacing watchdog thread — **MISSING** (Req 6).

### 1.7 Election State Durability

- `ha-term.bin` file — **does not exist**.
- `ha_persist_term()` — **MISSING** (Req 9).
- `ha_init()` zeros term and has no `ha-term.bin` read path.

### 1.8 libxce Client Retry (src/lib/)

- `ha_should_redirect()` / `ha_no_primary_known()` exist in ha.c and are used by join.c.
- No transparent retry loop for `XCE_ERR_NOT_READY`, `XCE_ERR_STANDBY`, `ECONNREFUSED`.
- `xce_retry_config_t` — **MISSING** (Req 7).
- `redirect_host` / `redirect_port` fields in `XCE_ERR_STANDBY` error reply — **MISSING** (Req 7).

---

## 2. Requirement-to-Asset Map

| Req | Key Gap | Key Reusable Asset | Gap Tag |
|---|---|---|---|
| 1 — Listener before election | `CTRL_INIT`/`CTRL_ELECTING` enum values; pre-dispatch gate in vtaskmgr; extract bind+listen from phase2; `ha_wait_for_heartbeat_or_timeout()`; `heartbeat_seen` semaphore | iomgr epoll, vtaskmgr dispatch switch, xorshift64 jitter | **Missing** |
| 2 — Quorum election | Entire `am_i_primary_quorum()` and vote protocol; `ha_vote_entry_t`, `ha_elect_ctx_t`; `handle_request_vote()` / `_reply()`; `ha_elect_evaluate_and_signal()`; OP_requestVote capnp schema | `ha_do_promote/demote`, jitter PRNG, `applied_lsn`, iomgr async connect | **Missing** |
| 3 — Config bootstrap | `bootstrap_config_if_needed()`; `xce_probe_config_tag()`; OP_probeConfigTag capnp; `defer_config_fetch` flag in ha_state_t | `do_one_config_fetch()` (wired), `connect_ssl_raw()` | **Missing** |
| 4 — Startup validation | `--vcpus N` CLI arg; even-N peer count check; cert pre-flight `access()`; `SSL_CTX_free` ASAN fix | argp option parsing pattern in main.c | **Missing** |
| 5 — Majority-commit WAL | `repl_broadcast()` implementation; `repl_wait_majority_ack()`; `committed_lsn` in persist_t; `persist_truncate_wal_above()`; `committed_lsn` in heartbeat; `last_known_committed_lsn` in ha_state_t | `repl_wait_ack()` logic, `standby_conn_t`, `repl_start/complete_catchup`, WAL sync path | **Missing** / **Stub** |
| 6 — Persistent channel | Persistent TCP fd per standby; `ha_find_and_connect_to_primary()`; EPOLLRDHUP failover; extended OP_ctrlHello; OP_snapshotTransfer; OP_walCatchupEntry; WAL streaming loop; heartbeat vtask; three-tier discovery | iomgr EPOLLIN/EPOLLRDHUP, `repl_needs_snapshot()`, `repl_start/complete_catchup()` | **Missing** |
| 7 — Client retry | Retry loop in libxce; `xce_retry_config_t`; redirect fields in error reply | `ha_should_redirect()`, `ha_no_primary_known()` | **Missing** |
| 8 — Majority-commit config | `repl_wait_majority_ack()` (shared with Req 5); OP_upgradeConfigAck; `committed_config_tag`; heartbeat piggybacking | `push_config_to_standbys()` (fire-and-forget → upgrade) | **Missing** |
| 9 — State durability | `ha_persist_term()`; `ha-term.bin` read in `ha_init()`; `voted_for_term`/`voted_for_name` in ha_state_t | Atomic write+rename pattern (existing in config_repl.c) | **Missing** |
| 10 — Observability | Structured log statements throughout all new code paths | Existing INFO/WARN/ERROR macros (xcelog.h) | **Missing** (alongside) |

---

## 3. Implementation Approach Options

### Option A: Change-at-a-time (recommended)

Implement changes in dependency order: D → A → B → C → E → H → F → G → (observability woven in).

**Rationale for ordering**:
1. D first: `--vcpus N` removes vtaskmgr's config dependency, unblocking A.
2. A before B: listener must be open before election dispatch works.
3. B before C: config bootstrap needs the proven election winner from quorum election.
4. E and H together: both use `repl_broadcast()` + `repl_wait_majority_ack()`; implement infrastructure once.
5. F after E: persistent channel carries WAL entries; broadcast must send over that channel.
6. G last: retry makes sense only after the controller states it retries against are implemented.

**Files touched per change**:
- D: `main.c`, `cluster.c` (remove vcpus from config), `ha.c` (`SSL_CTX_free`)
- A: `ha.h` (enum + semaphore + fields), `myapp.c` (startup reorder), `cluster.c` (extract listener), `ha.c` (wait function, dispatch gate)
- B: `ha.h` (new types), `ha.c` (replace `am_i_primary`), capnp schema, `cluster.c` (dispatch handler)
- C: `config_repl.c` (`bootstrap_config_if_needed`, `xce_probe_config_tag`), capnp schema, `myapp.c` (integration)
- E: `repl.c` (`repl_broadcast`, `repl_wait_majority_ack`), `persist.h`/`persist.c` (`committed_lsn`, `persist_truncate_wal_above`), `ha.h`/`ha.c` (`last_known_committed_lsn`)
- H: `config_repl.c` (upgrade majority-commit), capnp schema, `ha.h` (`committed_config_tag`)
- F: `ha.c`/`cluster.c` (channel connect, extended hello, streaming, EPOLLRDHUP)
- G: `src/lib/` (retry loop, xce_retry_config_t, redirect fields)

**Trade-offs**:
- ✅ Each atomic commit is testable before the next starts
- ✅ Follows existing TDD convention (failing test first)
- ❌ Long sequential dependency chain; F and G can only start late

### Option B: Infrastructure-first parallel tracks

Track 1 (election+bootstrap): D → A → B → C  
Track 2 (replication): E → H  
Join (channel): F using Track 1 + Track 2 artifacts  
Final: G

**Trade-offs**:
- ✅ Track 1 and Track 2 are mostly independent until F
- ❌ Track 2 cannot be fully validated without Track 1 (no quorum without election)
- ❌ More coordination overhead between developers on shared ha.h

### Option C: Stub-forward then wire

Add all new stubs with `-ENOSYS` first, then wire each one. Matches the existing codebase pattern (many stubs already present).

**Trade-offs**:
- ✅ Compiles throughout; no long broken-build windows
- ❌ Stub-first delays real testability; TDD mandate disfavors this

**Recommendation**: Option A. The dependency chain is well-defined and single-developer serial implementation is the realistic mode for this C codebase.

---

## 4. Effort and Risk Summary

| Req | Effort | Risk | Justification |
|---|---|---|---|
| 1 — Listener before election | M (3–5d) | Medium | Refactor startup order; must remove vtaskmgr/config dependency first |
| 2 — Quorum election | XL (2+ wks) | High | Full algorithm replacement; new wire protocol; FSM extension; concurrent dispatch |
| 3 — Config bootstrap | M (3–5d) | Medium | New function + probe RPC; depends on do_one_config_fetch (wired) |
| 4 — Startup validation | S (1d) | Low | Additive CLI + validation; ASAN one-liner |
| 5 — Majority-commit WAL | L (1–2 wks) | High | `committed_lsn` cross-cuts persist+repl+ha; `repl_broadcast` must become real |
| 6 — Persistent channel | XL (2+ wks) | High | New connection lifecycle; full state sync protocol; 3-tier discovery |
| 7 — Client retry | M (3d) | Low | Well-understood retry pattern; additive libxce change |
| 8 — Majority-commit config | M (3d) | Medium | Shares Req 5 infrastructure; mainly wiring push_config_to_standbys |
| 9 — State durability | S (1d) | Low | File write+read; extends ha_init |
| 10 — Observability | S (alongside) | Low | Log statements woven into each change |

**Total estimated effort**: 6–9 weeks for one engineer implementing with full TDD.

**High-risk dependencies**:
- Req 2 must complete before Req 3 can be integrated end-to-end.
- Req 5 (`repl_broadcast` implementation) must complete before Req 8 and before Req 6 (WAL streaming over the channel).
- Req 4 (vcpus CLI) must complete before Req 1 (vtaskmgr before election).

---

## 5. Key Reusable Assets

| Asset | Location | Reuse in |
|---|---|---|
| `repl_add/remove_standby` | repl.c | Req 6 (channel registration) |
| `repl_start/complete_catchup` | repl.c | Req 6 (state sync protocol) |
| `repl_needs_snapshot` | repl.c | Req 6 (snapshot decision) |
| `repl_on_ack` | repl.c | Req 5 (majority-commit ack handler) |
| `repl_trim_lsn` | repl.c | Req 5 (WAL trim with committed_lsn) |
| `connect_ssl_raw()` | netio.c/config_repl.c | Req 3 (probe RPC) |
| `do_one_config_fetch()` | config_repl.c | Req 3 (deferred fetch) |
| `xorshift64_next + ha_seed_jitter` | ha.h / ha.c | Req 1, 2 (jitter) |
| `ha_do_promote / ha_do_demote` | ha.c | Req 2, 6 (role transitions) |
| `ha_should_redirect / ha_no_primary_known` | ha.c | Req 7 (retry redirect) |
| iomgr epoll + EPOLLIN/EPOLLRDHUP | iomgr (cluster.c) | Req 1, 6 |
| Atomic write+rename pattern | config_repl.c | Req 9 (ha-term.bin) |
| INFO/WARN/ERROR macros | xcelog.h | Req 10 |

---

## 6. Constraints for Design Phase

- **C gnu17, no C++, no new external dependencies**.
- ASAN build (`nix build .#xcecore_debug`) must pass after every change.
- **Static peer membership**: `--peers` list is fixed at startup; no dynamic join/leave of controller peers. Quorum denominator is fixed for cluster lifetime.
- **TDD mandatory**: failing test in `src/controller/testsuite/` before each implementation change.
- `make test` must pass without a live server at every commit.
- `commit_timeout_ms` must be configurable (default 5000ms).
- **Single-node mode** (no `--peers`): quorum=1 everywhere; all new majority-commit paths must short-circuit to self-only.
- **WAL write stall window** (operational constraint, not a bug): during standby catch-up after election, `repl_wait_majority_ack()` will time out on every write until at least one standby completes catch-up. This is correct CP behavior; operators must be warned in docs.

---

## 7. Research Needed (Design Phase)

- **vtaskmgr API for pre-election dispatch**: confirm that `vtaskmgr_init(vcpus)` can be called before `cluster_config` is loaded (after removing vcpus from config struct). Verify no hidden config dependency.
- **iomgr async connect pattern for vote requests**: confirm `iomgr_async_vote_request()` can be implemented using existing async connect infrastructure without new socket primitives.
- **capnp schema extension**: confirm the capnp code-generation toolchain can add `OP_requestVote`, `OP_probeConfigTag`, `OP_walAck`, `OP_upgradeConfigAck`, `OP_snapshotTransfer`, `OP_walCatchupEntry` without breaking existing decode paths.
- **`persist_truncate_wal_above()` safety**: verify WAL segment file structure supports efficient truncation at arbitrary LSN boundary without full segment rewrite.
- **libxce client API surface**: identify the exact function in src/lib/ that wraps xce_encode()+send+recv+xce_decode() to locate the correct insertion point for retry logic.