myxce/.kiro/specs/ha-config-bootstrap/research.md

Gap Analysis: ha-config-bootstrap

Generated: 2026-05-27. Branch: persistent_ha.

---

Analysis Summary

• Scope: 10 requirements across 8 distinct changes (A–H). The codebase has strong foundational infrastructure (iomgr, repl slot management, WAL, SSL) but the HA election core, quorum commit, and control channel are mostly absent or stubbed.
• Largest gaps: Raft-style quorum election (Req 2) and persistent control channel with full state sync (Req 6) are full rewrites/additions with no current equivalent logic; majority-commit WAL (Req 5) requires implementing the two stub functions at the heart of replication.
• Strong reusable assets: repl_add_standby/remove/start_catchup/complete_catchup/needs_snapshot are real implementations; the SSL connect pattern (connect_ssl_raw), xorshift64 PRNG, ha_do_promote/demote, and iomgr epoll machinery are all directly reusable.
• Risk concentration: Req 2 and Req 6 carry High risk; Req 5 carries High risk due to committed_lsn cross-cutting persist_t, repl.c, and ha_state_t. The remaining requirements are Medium or Low.
• Recommended approach: Hybrid (Option A) — implement changes A→D→B→C→E→F→G→H in dependency order, writing C tests first per TDD, one atomic commit per change.

---

1. Current State Investigation

1.1 HA Core (ha.h / ha.c)

ctrl_role_t enum — only two values: CTRL_PRIMARY = 0, CTRL_STANDBY = 1.
Missing: CTRL_INIT, CTRL_ELECTING (required by Req 1).

ha_state_t — current fields of interest:

Field	Status	Notes
term (_Atomic uint64_t)	Exists	Used for fencing
role (_Atomic int32_t)	Exists	Cast to ctrl_role_t
my_priority	Exists	LSF priority index — kept for compat but unused in Raft
applied_lsn (_Atomic uint64_t)	Exists	Reusable for vote freshness check
primary_host[XCE_MAXNAME_LEN]	Exists	Reusable
primary_port	Exists	Reusable
last_heartbeat (struct timespec)	Exists	Replaces with EPOLLRDHUP path in Req 6
in_catch_up (_Atomic int)	Exists	Reusable for Clarification 6 vote exclusion
jitter_seed (uint64_t)	Exists	xorshift64, directly reusable
ssl_ctx (void*)	Exists	Reusable
standby_endpoints (zarray_t*)	Exists	Superseded by per-standby fd table in Req 6
voted_for_term (uint64_t)	MISSING	Req 9
voted_for_name (char[])	MISSING	Req 9
primary_ctrl_fd (int)	MISSING	Req 6
defer_config_fetch (bool)	MISSING	Req 3
elect_ctx (_Atomic ptr)	MISSING	Req 2
last_known_committed_lsn (_Atomic uint64_t)	MISSING	Req 5
heartbeat_seen (sem_t)	MISSING	Req 1

ha_config_t — missing: commit_timeout_ms. All election fields present (election_timeout_ms, jitter_max_ms, etc.) but values tuned for LSF model.

am_i_primary() — LSF priority probe via per-peer pthreads. No quorum, no config_tag/LSN in vote, no term/voted_for persistence. Must be replaced entirely with am_i_primary_quorum().

ha_check_priority_stepdown() — permanently unreachable once term > 0; exists only for the term=0 simultaneous-boot race. Must be retired (Req 2).

ha_check_heartbeat_nocluster() — timestamp polling with configurable timeout. Replaced by EPOLLRDHUP fast path + heartbeat timeout vtask (Req 6). Function itself can be deleted.

ha_watchdog_thread() — sets failover_requested but cannot call ha_promote_to_primary() (no cluster_t). Partially functional stub. Replaced by vtask-based heartbeat monitor (Req 6).

start_heartbeat_listener() / stop_heartbeat_listener() — no-op stubs with comment "full implementation in cluster.c Task 7". Missing (Req 1).

ha_fini() — missing SSL_CTX_free(ha->ssl_ctx). ASAN-visible leak (~5 KB).

1.2 Startup Sequence (myapp.c / main.c)

myapp.c current order:

cluster_init_phase1()
initialize_openssl() + create_ssl_context()
am_i_primary()                     ← election here
load_primary_config() / fetch_config_from_primary()
cluster_init_phase2()              ← iomgr, vtaskmgr, bind+listen inside
cluster_start()                    ← more network init
cluster_run()

Bind+listen happens inside cluster_init_phase2() or cluster_start() — after election. Gap: concurrent-startup nodes probe an unbound port (Req 1).

No bootstrap_config_if_needed() — standby path goes straight to fetch_config_from_primary() post-election without checking whether local config_tag is 0 or whether peers have config. Gap (Req 3).

main.c:

• --vcpus N arg: MISSING. vcpus comes from cluster config read inside phase2. This creates a dependency that prevents vtaskmgr from running before election (Req 1, 4).
• Peer count validation: MISSING. Even-N peer lists accepted silently (Req 4).
• Cert pre-flight access() check: MISSING. Cert paths computed but SSL context created later in myapp.c, not with early-exit error in main.c (Req 4).

1.3 Replication (repl.c / repl.h)

Function	Status	Notes
repl_init / fini	Real	Lifecycle ok
repl_add_standby	Real	Slot management implemented
repl_remove_standby	Real	Implemented
repl_start_catchup	Real	Sets in_catch_up=1
repl_complete_catchup	Real	Clears in_catch_up, broadcasts cond
repl_needs_snapshot	Real	Compares standby_lsn vs oldest_wal_lsn
repl_on_ack	Real	Updates acked_lsn, signals cond
repl_trim_lsn	Real	WAL trim guard implemented
repl_broadcast	STUB	Returns 0 without sending anything (Req 5, 8)
repl_wait_ack	Partial	Waits for ANY non-catch-up standby, not majority (Req 5)
repl_wait_majority_ack	MISSING	Req 5, 8

standby_conn_t — missing pending_ack_count for backpressure (Req 5).

1.4 Persistence (persist.h / persist.c)

• current_lsn and checkpoint_lsn exist. committed_lsn — MISSING (Req 5).
• persist_wal_sync_append() uses repl_wait_ack() (any-standby) — must upgrade to repl_wait_majority_ack().
• persist_truncate_wal_above() — MISSING (Req 5, 6).
• persist_config_t.sync_timeout_ms exists — analogous to commit_timeout_ms needed for majority-commit.

1.5 Config Replication (config_repl.c)

• do_one_config_fetch() — wired (recent commit 2aae45f on branch: "wire do_one_config_fetch() — full SSL roundtrip"). Not a stub.
• bootstrap_config_if_needed() — MISSING (Req 3).
• xce_probe_config_tag() — MISSING. New lightweight RPC needed (Req 3).
• push_config_to_standbys() — declared in config_repl.h; likely fire-and-forget (no majority wait). Gap: must upgrade to majority-commit (Req 8).
• OP_probeConfigTag Cap'n Proto schema — MISSING (Req 3).
• OP_upgradeConfigAck Cap'n Proto schema — MISSING (Req 8).
• committed_config_tag — MISSING (Req 8).

1.6 Control Channel (ha.c / cluster.c)

• No persistent standby-to-primary TCP connection. Each config fetch and WAL entry opens a new SSL connection via connect_ssl_raw().
• ha_find_and_connect_to_primary() — MISSING (Req 6).
• Extended OP_ctrlHello with applied_lsn, config_tag — MISSING (Req 6).
• OP_snapshotTransfer, OP_walCatchupEntry — MISSING (Req 6).
• handle_ctrlhello() extended handler — MISSING (Req 6).
• Three-tier primary discovery — MISSING (Req 6).
• EPOLLRDHUP failover detection — MISSING (Req 6).
• Heartbeat timeout vtask replacing watchdog thread — MISSING (Req 6).

1.7 Election State Durability

• ha-term.bin file — does not exist.
• ha_persist_term() — MISSING (Req 9).
• ha_init() zeros term and has no ha-term.bin read path.

1.8 libxce Client Retry (src/lib/)

• ha_should_redirect() / ha_no_primary_known() exist in ha.c and are used by join.c.
• No transparent retry loop for XCE_ERR_NOT_READY, XCE_ERR_STANDBY, ECONNREFUSED.
• xce_retry_config_t — MISSING (Req 7).
• redirect_host / redirect_port fields in XCE_ERR_STANDBY error reply — MISSING (Req 7).

---

2. Requirement-to-Asset Map

Req	Key Gap	Key Reusable Asset	Gap Tag
1 — Listener before election	CTRL_INIT/CTRL_ELECTING enum values; pre-dispatch gate in vtaskmgr; extract bind+listen from phase2; ha_wait_for_heartbeat_or_timeout(); heartbeat_seen semaphore	iomgr epoll, vtaskmgr dispatch switch, xorshift64 jitter	Missing
2 — Quorum election	Entire am_i_primary_quorum() and vote protocol; ha_vote_entry_t, ha_elect_ctx_t; handle_request_vote() / _reply(); ha_elect_evaluate_and_signal(); OP_requestVote capnp schema	ha_do_promote/demote, jitter PRNG, applied_lsn, iomgr async connect	Missing
3 — Config bootstrap	bootstrap_config_if_needed(); xce_probe_config_tag(); OP_probeConfigTag capnp; defer_config_fetch flag in ha_state_t	do_one_config_fetch() (wired), connect_ssl_raw()	Missing
4 — Startup validation	--vcpus N CLI arg; even-N peer count check; cert pre-flight access(); SSL_CTX_free ASAN fix	argp option parsing pattern in main.c	Missing
5 — Majority-commit WAL	repl_broadcast() implementation; repl_wait_majority_ack(); committed_lsn in persist_t; persist_truncate_wal_above(); committed_lsn in heartbeat; last_known_committed_lsn in ha_state_t	repl_wait_ack() logic, standby_conn_t, repl_start/complete_catchup, WAL sync path	Missing / Stub
6 — Persistent channel	Persistent TCP fd per standby; ha_find_and_connect_to_primary(); EPOLLRDHUP failover; extended OP_ctrlHello; OP_snapshotTransfer; OP_walCatchupEntry; WAL streaming loop; heartbeat vtask; three-tier discovery	iomgr EPOLLIN/EPOLLRDHUP, repl_needs_snapshot(), repl_start/complete_catchup()	Missing
7 — Client retry	Retry loop in libxce; xce_retry_config_t; redirect fields in error reply	ha_should_redirect(), ha_no_primary_known()	Missing
8 — Majority-commit config	repl_wait_majority_ack() (shared with Req 5); OP_upgradeConfigAck; committed_config_tag; heartbeat piggybacking	push_config_to_standbys() (fire-and-forget → upgrade)	Missing
9 — State durability	ha_persist_term(); ha-term.bin read in ha_init(); voted_for_term/voted_for_name in ha_state_t	Atomic write+rename pattern (existing in config_repl.c)	Missing
10 — Observability	Structured log statements throughout all new code paths	Existing INFO/WARN/ERROR macros (xcelog.h)	Missing (alongside)

---

3. Implementation Approach Options

Option A: Change-at-a-time (recommended)

Implement changes in dependency order: D → A → B → C → E → H → F → G → (observability woven in).

Rationale for ordering:

1. D first: --vcpus N removes vtaskmgr's config dependency, unblocking A.
2. A before B: listener must be open before election dispatch works.
3. B before C: config bootstrap needs the proven election winner from quorum election.
4. E and H together: both use repl_broadcast() + repl_wait_majority_ack(); implement infrastructure once.
5. F after E: persistent channel carries WAL entries; broadcast must send over that channel.
6. G last: retry makes sense only after the controller states it retries against are implemented.

Files touched per change:

• D: main.c, cluster.c (remove vcpus from config), ha.c (SSL_CTX_free)
• A: ha.h (enum + semaphore + fields), myapp.c (startup reorder), cluster.c (extract listener), ha.c (wait function, dispatch gate)
• B: ha.h (new types), ha.c (replace am_i_primary), capnp schema, cluster.c (dispatch handler)
• C: config_repl.c (bootstrap_config_if_needed, xce_probe_config_tag), capnp schema, myapp.c (integration)
• E: repl.c (repl_broadcast, repl_wait_majority_ack), persist.h/persist.c (committed_lsn, persist_truncate_wal_above), ha.h/ha.c (last_known_committed_lsn)
• H: config_repl.c (upgrade majority-commit), capnp schema, ha.h (committed_config_tag)
• F: ha.c/cluster.c (channel connect, extended hello, streaming, EPOLLRDHUP)
• G: src/lib/ (retry loop, xce_retry_config_t, redirect fields)

Trade-offs:

• ✅ Each atomic commit is testable before the next starts
• ✅ Follows existing TDD convention (failing test first)
• ❌ Long sequential dependency chain; F and G can only start late

Option B: Infrastructure-first parallel tracks

Track 1 (election+bootstrap): D → A → B → C
Track 2 (replication): E → H
Join (channel): F using Track 1 + Track 2 artifacts
Final: G

Trade-offs:

• ✅ Track 1 and Track 2 are mostly independent until F
• ❌ Track 2 cannot be fully validated without Track 1 (no quorum without election)
• ❌ More coordination overhead between developers on shared ha.h

Option C: Stub-forward then wire

Add all new stubs with -ENOSYS first, then wire each one. Matches the existing codebase pattern (many stubs already present).

Trade-offs:

• ✅ Compiles throughout; no long broken-build windows
• ❌ Stub-first delays real testability; TDD mandate disfavors this

Recommendation: Option A. The dependency chain is well-defined and single-developer serial implementation is the realistic mode for this C codebase.

---

4. Effort and Risk Summary

Req	Effort	Risk	Justification
1 — Listener before election	M (3–5d)	Medium	Refactor startup order; must remove vtaskmgr/config dependency first
2 — Quorum election	XL (2+ wks)	High	Full algorithm replacement; new wire protocol; FSM extension; concurrent dispatch
3 — Config bootstrap	M (3–5d)	Medium	New function + probe RPC; depends on do_one_config_fetch (wired)
4 — Startup validation	S (1d)	Low	Additive CLI + validation; ASAN one-liner
5 — Majority-commit WAL	L (1–2 wks)	High	committed_lsn cross-cuts persist+repl+ha; repl_broadcast must become real
6 — Persistent channel	XL (2+ wks)	High	New connection lifecycle; full state sync protocol; 3-tier discovery
7 — Client retry	M (3d)	Low	Well-understood retry pattern; additive libxce change
8 — Majority-commit config	M (3d)	Medium	Shares Req 5 infrastructure; mainly wiring push_config_to_standbys
9 — State durability	S (1d)	Low	File write+read; extends ha_init
10 — Observability	S (alongside)	Low	Log statements woven into each change

Total estimated effort: 6–9 weeks for one engineer implementing with full TDD.

High-risk dependencies:

• Req 2 must complete before Req 3 can be integrated end-to-end.
• Req 5 (repl_broadcast implementation) must complete before Req 8 and before Req 6 (WAL streaming over the channel).
• Req 4 (vcpus CLI) must complete before Req 1 (vtaskmgr before election).

---

5. Key Reusable Assets

Asset	Location	Reuse in
repl_add/remove_standby	repl.c	Req 6 (channel registration)
repl_start/complete_catchup	repl.c	Req 6 (state sync protocol)
repl_needs_snapshot	repl.c	Req 6 (snapshot decision)
repl_on_ack	repl.c	Req 5 (majority-commit ack handler)
repl_trim_lsn	repl.c	Req 5 (WAL trim with committed_lsn)
connect_ssl_raw()	netio.c/config_repl.c	Req 3 (probe RPC)
do_one_config_fetch()	config_repl.c	Req 3 (deferred fetch)
xorshift64_next + ha_seed_jitter	ha.h / ha.c	Req 1, 2 (jitter)
ha_do_promote / ha_do_demote	ha.c	Req 2, 6 (role transitions)
ha_should_redirect / ha_no_primary_known	ha.c	Req 7 (retry redirect)
iomgr epoll + EPOLLIN/EPOLLRDHUP	iomgr (cluster.c)	Req 1, 6
Atomic write+rename pattern	config_repl.c	Req 9 (ha-term.bin)
INFO/WARN/ERROR macros	xcelog.h	Req 10

---

6. Constraints for Design Phase

• C gnu17, no C++, no new external dependencies.
• ASAN build (nix build .#xcecore_debug) must pass after every change.
• Static peer membership: --peers list is fixed at startup; no dynamic join/leave of controller peers. Quorum denominator is fixed for cluster lifetime.
• TDD mandatory: failing test in src/controller/testsuite/ before each implementation change.
• make test must pass without a live server at every commit.
• commit_timeout_ms must be configurable (default 5000ms).
• Single-node mode (no --peers): quorum=1 everywhere; all new majority-commit paths must short-circuit to self-only.
• WAL write stall window (operational constraint, not a bug): during standby catch-up after election, repl_wait_majority_ack() will time out on every write until at least one standby completes catch-up. This is correct CP behavior; operators must be warned in docs.

---

7. Research Needed (Design Phase)

• vtaskmgr API for pre-election dispatch: confirm that vtaskmgr_init(vcpus) can be called before cluster_config is loaded (after removing vcpus from config struct). Verify no hidden config dependency.
• iomgr async connect pattern for vote requests: confirm iomgr_async_vote_request() can be implemented using existing async connect infrastructure without new socket primitives.
• capnp schema extension: confirm the capnp code-generation toolchain can add OP_requestVote, OP_probeConfigTag, OP_walAck, OP_upgradeConfigAck, OP_snapshotTransfer, OP_walCatchupEntry without breaking existing decode paths.
• persist_truncate_wal_above() safety: verify WAL segment file structure supports efficient truncation at arbitrary LSN boundary without full segment rewrite.
• libxce client API surface: identify the exact function in src/lib/ that wraps xce_encode()+send+recv+xce_decode() to locate the correct insertion point for retry logic.