myxce/.kiro/specs/ha-config-bootstrap/research.md

---

Cross-Spec Gap Audit (2026-06-03)

Triggered by: /kiro-validate-gap ha-config-bootstrap election-iomgr-refactor, check am_i_primary() still be used? Scope: confirm cross-spec coverage status; answer am_i_primary() question definitively.

Analysis Summary

• ha-config-bootstrap remains at ~95% (all unit tasks done; tasks 11.1–11.5 are manual integration tests).
• election-iomgr-refactor is 100% complete and already marked implemented.
• The two specs are consistent and non-overlapping in implementation: election-iomgr-refactor superseded the blocking-pthread election path defined in ha-config-bootstrap, keeping all requirements satisfied through a different (better) implementation path.
• am_i_primary() question: answered — it is dead code, safe to delete, but not harmful to leave.

am_i_primary() Status

What it is

am_i_primary() (ha.c:869) is the original LSF priority-based election probe — it spawns one pthread per peer and self-promotes if no higher-priority peer responds before a deadline. It was the pre-Raft election mechanism.

Is it still called?

No. Confirmed by grep:

• myapp.c: comment at line 48 explicitly says am_i_primary_quorum (the intermediate Raft replacement) was removed in Req 9.3. am_i_primary() is never referenced in myapp.c.
• cluster.c line 1015: The line /* Determine initial role based on am_i_primary() result set by myapp_run */ is a stale comment — the actual code underneath calls ha_is_primary(&c->ha) (the atomic role check), not am_i_primary(). The comment is a lie left over from the old startup sequence.
• ha.c: The function is defined but has no callers in production code. The only cross-file reference to am_i_primary (not am_i_primary_quorum) in non-test code is the cluster.c comment above.

What the tests use

• test_ha_role.c (lines 241–396): three test cases test am_i_primary() directly — these are legacy P1-6 tests covering the old priority-based election. They still pass because the function body is intact.
• test_ha_bootstrap_support.c and test_ha_election.c: both define stubs for am_i_primary_quorum (the intermediate function, not am_i_primary), used to isolate the quorum election logic under test.

Decision: can am_i_primary() be deleted?

Yes — safe to delete in a cleanup pass. However:

• The three test cases in test_ha_role.c would also need to be deleted.
• The stale comment in cluster.c line 1015 should be updated to say "role set by ha_elect_evaluate_and_signal via vtask".
• start_heartbeat_listener() / stop_heartbeat_listener() stubs (ha.c:951-960) referenced in the same comment block can also be cleaned up.

Recommendation: Leave am_i_primary() as-is during active HA development. Mark as cleanup tech debt. Its presence does not affect correctness, ASAN, or test pass rate.

Requirement Coverage Cross-Check

The election-iomgr-refactor implemented ha_start_election_async which replaced both am_i_primary() and am_i_primary_quorum(). The ha-config-bootstrap tasks 4.3 and 9.4 originally called am_i_primary_quorum() from myapp.c — these are now superseded by ha_start_election_async. Both specs' requirements are fully met by the current codebase.

No new code needed for either spec.

---

Gap Analysis — Completion Audit (2026-06-02)

Analysis date: 2026-06-02. Phase: tasks-generated. All leaf implementation tasks [x]. Purpose: verify actual codebase coverage of all 10 requirements; identify what remains before the spec can be marked implemented.

Analysis Summary

• Scope: 10 requirements across 8 implementation changes (A–H), covering Raft-style quorum election, majority-commit WAL + config replication, persistent control channels, config bootstrap, term durability, and client-side retry.
• Implementation status: ~95% complete. All 22 leaf subtasks (1.1–10.2) are implemented and verified. All 9 requirements have implementation assets in the live codebase (verified via grep audit). The parent group headers in tasks.md show [ ] only because they are section labels, not actionable tasks.
• Remaining work: Integration tests 11.1–11.5 (manual cluster validation).
• No design changes needed: The implementation already follows the approved design; xce_request_with_retry, repl_wait_majority_ack, ha_find_and_connect_to_primary, ha_persist_term, bootstrap_config_if_needed, and all 9 capnp extensions are in the live codebase.
• Recommendation: Run the 5 integration tests from manual-tests.md against a 3-node cluster, then mark the spec implemented.

---

Cross-Spec Gap Audit (2026-06-08)

Batch /kiro-validate-gap across 9 specs on persistent_ha branch.

Analysis Summary

• ha-config-bootstrap is ~95% complete. All 10 requirements have verified implementation assets. All 22 leaf subtasks (1.1–10.2) are done.
• Remaining work: Tasks 11.1–11.5 (manual integration tests documented in manual-tests.md).
• Cross-spec verification: election-iomgr-refactor superseded the blocking-pthread election path while preserving all ha-config-bootstrap requirements. The two specs are consistent. ha-repl-hardening extends Req 5 (majority-commit WAL) and Req 8 (majority-commit config) with heartbeat-driven convergence — no conflicts.
• Cleanup debt: am_i_primary() (ha.c:869), ha_wait_for_heartbeat_or_timeout, and start_heartbeat_listener()/stop_heartbeat_listener() stubs are dead code — safe to delete in a dedicated cleanup commit (tracked in election-iomgr-refactor/research.md).

Requirement-to-Asset Map — Final State

Requirement	Verified Key Assets	Status
Req 1: FSM init + dispatch gate	CTRL_INIT, CTRL_ELECTING in ha.h:59-60; ha_dispatch_gate in ha.c; ha_wait_for_heartbeat_or_timeout	✅ Implemented
Req 2: Quorum election	ha_elect_evaluate_and_signal (ha.c:1604); am_i_primary_quorum replaced by raft_start_election; handle_request_vote term/freshness gate	✅ Implemented
Req 3: Config bootstrap	bootstrap_config_if_needed, xce_probe_config_tag, defer_config_fetch — all in ha.c/config_repl.c	✅ Implemented
Req 4: Contiguous WAL + ASAN	WAL schema capnp, persist_* callbacks, SSL_CTX_free fix	✅ Implemented
Req 5: Majority-commit WAL	repl_broadcast, repl_wait_majority_ack, committed_lsn in persist_t, persist_truncate_wal_above	✅ Implemented
Req 6: Persistent control channel	ha_find_and_connect_to_primary, extended ctrlhello handler, EPOLLRDHUP failover, ha_on_primary_ctrl_fd_rdhup	✅ Implemented
Req 7: State sync	apply_wal_entry, repl_start_catchup/complete_catchup, ctrlhello + WAL catch-up flow	✅ Implemented
Req 8: Majority-commit config	OP_upgradeConfigAck handler, committed_config_tag, ha_repl_ack_received	✅ Implemented
Req 9: Voted-for persistence	ha_persist_term (ha.c:1323); ha_init restore path for ha-term.bin	✅ Implemented
Req 10: Client-side retry	xce_request_with_retry (request.c:1664); xce_retry_config_t	✅ Implemented

Remaining Gaps

Gap 1: Tasks 11.1–11.5 — Integration validation (IP = In Progress via manual-tests.md)

The manual-tests.md file contains step-by-step procedures for all 5 integration tests. They require a running 3-node cluster:

Test	What it validates	Status
11.1	Concurrent startup + config-fresh election + config replication	Manual test doc written; not yet run
11.2	Failover (RDHUP → promote → WAL survived); WAL commit under partition	Manual test doc written; not yet run
11.3	Unconfigured node bootstrap (config_tag=0)	Manual test doc written; not yet run
11.4	Config upgrade majority-commit; term durability across restart	Manual test doc written; not yet run
11.5	Full ASAN build verification	Manual test doc written; not yet run

Note: These are manual cluster tests by design. The automated hook-based unit tests in testsuite/ (tests in test_ha_bootstrap.c, test_ha_durability.c, test_p1_integration.c, test_p2_integration.c, test_p3_hardening.c, test_p3_manual_failover.c) cover the in-process logic. Tasks 11.1–11.5 validate end-to-end process behavior that the hook seams cannot reach.

Gap 2: Parent group headers in tasks.md show [ ] (Cosmetic)

Tasks 1., 2., 3., 4., 5., 6., 7., 8., 9., 10. are section grouping labels — all their child subtasks are [x]. The parent headers can be marked [x] for clarity.

Constraint: xce_request_with_retry not in lib header (known limitation)

The function lives in src/lib/request.c:1664 but is not declared in src/lib/include/xce/request.h. Callers must add extern int xce_request_with_retry(...) locally. Intentional — the libxce request.h only exposes the public API; internal retry logic is a controller-side concern.

Effort and Risk Assessment

Item	Effort	Risk	Justification
Tasks 11.1–11.5 (integration)	M (1–2 days)	Low	Procedures documented; run cluster + verify logs
Fix parent checkboxes	S (1 commit)	None	Cosmetic

Conclusion

The ha-config-bootstrap spec is functionally complete — all 10 requirements are implemented, tested, and verified at the unit level. The 5 remaining integration tests (11.1–11.5) are manual cluster validations that exercise the full system behavior. The manual-tests.md file contains all the procedures. Once these are run, the spec can be marked implemented and closed.

Generated: 2026-05-27. Branch: persistent_ha.

---

Analysis Summary

• Scope: 10 requirements across 8 distinct changes (A–H). The codebase has strong foundational infrastructure (iomgr, repl slot management, WAL, SSL) but the HA election core, quorum commit, and control channel are mostly absent or stubbed.
• Largest gaps: Raft-style quorum election (Req 2) and persistent control channel with full state sync (Req 6) are full rewrites/additions with no current equivalent logic; majority-commit WAL (Req 5) requires implementing the two stub functions at the heart of replication.
• Strong reusable assets: repl_add_standby/remove/start_catchup/complete_catchup/needs_snapshot are real implementations; the SSL connect pattern (connect_ssl_raw), xorshift64 PRNG, ha_do_promote/demote, and iomgr epoll machinery are all directly reusable.
• Risk concentration: Req 2 and Req 6 carry High risk; Req 5 carries High risk due to committed_lsn cross-cutting persist_t, repl.c, and ha_state_t. The remaining requirements are Medium or Low.
• Recommended approach: Hybrid (Option A) — implement changes A→D→B→C→E→F→G→H in dependency order, writing C tests first per TDD, one atomic commit per change.

---

1. Current State Investigation

1.1 HA Core (ha.h / ha.c)

ctrl_role_t enum — only two values: CTRL_PRIMARY = 0, CTRL_STANDBY = 1.
Missing: CTRL_INIT, CTRL_ELECTING (required by Req 1).

ha_state_t — current fields of interest:

Field	Status	Notes
term (_Atomic uint64_t)	Exists	Used for fencing
role (_Atomic int32_t)	Exists	Cast to ctrl_role_t
my_priority	Exists	LSF priority index — kept for compat but unused in Raft
applied_lsn (_Atomic uint64_t)	Exists	Reusable for vote freshness check
primary_host[XCE_MAXNAME_LEN]	Exists	Reusable
primary_port	Exists	Reusable
last_heartbeat (struct timespec)	Exists	Replaces with EPOLLRDHUP path in Req 6
in_catch_up (_Atomic int)	Exists	Reusable for Clarification 6 vote exclusion
jitter_seed (uint64_t)	Exists	xorshift64, directly reusable
ssl_ctx (void*)	Exists	Reusable
standby_endpoints (zarray_t*)	Exists	Superseded by per-standby fd table in Req 6
voted_for_term (uint64_t)	MISSING	Req 9
voted_for_name (char[])	MISSING	Req 9
primary_ctrl_fd (int)	MISSING	Req 6
defer_config_fetch (bool)	MISSING	Req 3
elect_ctx (_Atomic ptr)	MISSING	Req 2
last_known_committed_lsn (_Atomic uint64_t)	MISSING	Req 5
heartbeat_seen (sem_t)	MISSING	Req 1

ha_config_t — missing: commit_timeout_ms. All election fields present (election_timeout_ms, jitter_max_ms, etc.) but values tuned for LSF model.

am_i_primary() — LSF priority probe via per-peer pthreads. No quorum, no config_tag/LSN in vote, no term/voted_for persistence. Must be replaced entirely with am_i_primary_quorum().

ha_check_priority_stepdown() — permanently unreachable once term > 0; exists only for the term=0 simultaneous-boot race. Must be retired (Req 2).

ha_check_heartbeat_nocluster() — timestamp polling with configurable timeout. Replaced by EPOLLRDHUP fast path + heartbeat timeout vtask (Req 6). Function itself can be deleted.

ha_watchdog_thread() — sets failover_requested but cannot call ha_promote_to_primary() (no cluster_t). Partially functional stub. Replaced by vtask-based heartbeat monitor (Req 6).

start_heartbeat_listener() / stop_heartbeat_listener() — no-op stubs with comment "full implementation in cluster.c Task 7". Missing (Req 1).

ha_fini() — missing SSL_CTX_free(ha->ssl_ctx). ASAN-visible leak (~5 KB).

1.2 Startup Sequence (myapp.c / main.c)

myapp.c current order:

cluster_init_phase1()
initialize_openssl() + create_ssl_context()
am_i_primary()                     ← election here
load_primary_config() / fetch_config_from_primary()
cluster_init_phase2()              ← iomgr, vtaskmgr, bind+listen inside
cluster_start()                    ← more network init
cluster_run()

Bind+listen happens inside cluster_init_phase2() or cluster_start() — after election. Gap: concurrent-startup nodes probe an unbound port (Req 1).

No bootstrap_config_if_needed() — standby path goes straight to fetch_config_from_primary() post-election without checking whether local config_tag is 0 or whether peers have config. Gap (Req 3).

main.c:

• --vcpus N arg: MISSING. vcpus comes from cluster config read inside phase2. This creates a dependency that prevents vtaskmgr from running before election (Req 1, 4).
• Peer count validation: MISSING. Even-N peer lists accepted silently (Req 4).
• Cert pre-flight access() check: MISSING. Cert paths computed but SSL context created later in myapp.c, not with early-exit error in main.c (Req 4).

1.3 Replication (repl.c / repl.h)

Function	Status	Notes
repl_init / fini	Real	Lifecycle ok
repl_add_standby	Real	Slot management implemented
repl_remove_standby	Real	Implemented
repl_start_catchup	Real	Sets in_catch_up=1
repl_complete_catchup	Real	Clears in_catch_up, broadcasts cond
repl_needs_snapshot	Real	Compares standby_lsn vs oldest_wal_lsn
repl_on_ack	Real	Updates acked_lsn, signals cond
repl_trim_lsn	Real	WAL trim guard implemented
repl_broadcast	STUB	Returns 0 without sending anything (Req 5, 8)
repl_wait_ack	Partial	Waits for ANY non-catch-up standby, not majority (Req 5)
repl_wait_majority_ack	MISSING	Req 5, 8

standby_conn_t — missing pending_ack_count for backpressure (Req 5).

1.4 Persistence (persist.h / persist.c)

• current_lsn and checkpoint_lsn exist. committed_lsn — MISSING (Req 5).
• persist_wal_sync_append() uses repl_wait_ack() (any-standby) — must upgrade to repl_wait_majority_ack().
• persist_truncate_wal_above() — MISSING (Req 5, 6).
• persist_config_t.sync_timeout_ms exists — analogous to commit_timeout_ms needed for majority-commit.

1.5 Config Replication (config_repl.c)

• do_one_config_fetch() — wired (recent commit 2aae45f on branch: "wire do_one_config_fetch() — full SSL roundtrip"). Not a stub.
• bootstrap_config_if_needed() — MISSING (Req 3).
• xce_probe_config_tag() — MISSING. New lightweight RPC needed (Req 3).
• push_config_to_standbys() — declared in config_repl.h; likely fire-and-forget (no majority wait). Gap: must upgrade to majority-commit (Req 8).
• OP_probeConfigTag Cap'n Proto schema — MISSING (Req 3).
• OP_upgradeConfigAck Cap'n Proto schema — MISSING (Req 8).
• committed_config_tag — MISSING (Req 8).

1.6 Control Channel (ha.c / cluster.c)

• No persistent standby-to-primary TCP connection. Each config fetch and WAL entry opens a new SSL connection via connect_ssl_raw().
• ha_find_and_connect_to_primary() — MISSING (Req 6).
• Extended OP_ctrlHello with applied_lsn, config_tag — MISSING (Req 6).
• OP_snapshotTransfer, OP_walCatchupEntry — MISSING (Req 6).
• handle_ctrlhello() extended handler — MISSING (Req 6).
• Three-tier primary discovery — MISSING (Req 6).
• EPOLLRDHUP failover detection — MISSING (Req 6).
• Heartbeat timeout vtask replacing watchdog thread — MISSING (Req 6).

1.7 Election State Durability

• ha-term.bin file — does not exist.
• ha_persist_term() — MISSING (Req 9).
• ha_init() zeros term and has no ha-term.bin read path.

1.8 libxce Client Retry (src/lib/)

• ha_should_redirect() / ha_no_primary_known() exist in ha.c and are used by join.c.
• No transparent retry loop for XCE_ERR_NOT_READY, XCE_ERR_STANDBY, ECONNREFUSED.
• xce_retry_config_t — MISSING (Req 7).
• redirect_host / redirect_port fields in XCE_ERR_STANDBY error reply — MISSING (Req 7).

---

2. Requirement-to-Asset Map

Req	Key Gap	Key Reusable Asset	Gap Tag
1 — Listener before election	CTRL_INIT/CTRL_ELECTING enum values; pre-dispatch gate in vtaskmgr; extract bind+listen from phase2; ha_wait_for_heartbeat_or_timeout(); heartbeat_seen semaphore	iomgr epoll, vtaskmgr dispatch switch, xorshift64 jitter	Missing
2 — Quorum election	Entire am_i_primary_quorum() and vote protocol; ha_vote_entry_t, ha_elect_ctx_t; handle_request_vote() / _reply(); ha_elect_evaluate_and_signal(); OP_requestVote capnp schema	ha_do_promote/demote, jitter PRNG, applied_lsn, iomgr async connect	Missing
3 — Config bootstrap	bootstrap_config_if_needed(); xce_probe_config_tag(); OP_probeConfigTag capnp; defer_config_fetch flag in ha_state_t	do_one_config_fetch() (wired), connect_ssl_raw()	Missing
4 — Startup validation	--vcpus N CLI arg; even-N peer count check; cert pre-flight access(); SSL_CTX_free ASAN fix	argp option parsing pattern in main.c	Missing
5 — Majority-commit WAL	repl_broadcast() implementation; repl_wait_majority_ack(); committed_lsn in persist_t; persist_truncate_wal_above(); committed_lsn in heartbeat; last_known_committed_lsn in ha_state_t	repl_wait_ack() logic, standby_conn_t, repl_start/complete_catchup, WAL sync path	Missing / Stub
6 — Persistent channel	Persistent TCP fd per standby; ha_find_and_connect_to_primary(); EPOLLRDHUP failover; extended OP_ctrlHello; OP_snapshotTransfer; OP_walCatchupEntry; WAL streaming loop; heartbeat vtask; three-tier discovery	iomgr EPOLLIN/EPOLLRDHUP, repl_needs_snapshot(), repl_start/complete_catchup()	Missing
7 — Client retry	Retry loop in libxce; xce_retry_config_t; redirect fields in error reply	ha_should_redirect(), ha_no_primary_known()	Missing
8 — Majority-commit config	repl_wait_majority_ack() (shared with Req 5); OP_upgradeConfigAck; committed_config_tag; heartbeat piggybacking	push_config_to_standbys() (fire-and-forget → upgrade)	Missing
9 — State durability	ha_persist_term(); ha-term.bin read in ha_init(); voted_for_term/voted_for_name in ha_state_t	Atomic write+rename pattern (existing in config_repl.c)	Missing
10 — Observability	Structured log statements throughout all new code paths	Existing INFO/WARN/ERROR macros (xcelog.h)	Missing (alongside)

---

3. Implementation Approach Options

Option A: Change-at-a-time (recommended)

Implement changes in dependency order: D → A → B → C → E → H → F → G → (observability woven in).

Rationale for ordering:

1. D first: --vcpus N removes vtaskmgr's config dependency, unblocking A.
2. A before B: listener must be open before election dispatch works.
3. B before C: config bootstrap needs the proven election winner from quorum election.
4. E and H together: both use repl_broadcast() + repl_wait_majority_ack(); implement infrastructure once.
5. F after E: persistent channel carries WAL entries; broadcast must send over that channel.
6. G last: retry makes sense only after the controller states it retries against are implemented.

Files touched per change:

• D: main.c, cluster.c (remove vcpus from config), ha.c (SSL_CTX_free)
• A: ha.h (enum + semaphore + fields), myapp.c (startup reorder), cluster.c (extract listener), ha.c (wait function, dispatch gate)
• B: ha.h (new types), ha.c (replace am_i_primary), capnp schema, cluster.c (dispatch handler)
• C: config_repl.c (bootstrap_config_if_needed, xce_probe_config_tag), capnp schema, myapp.c (integration)
• E: repl.c (repl_broadcast, repl_wait_majority_ack), persist.h/persist.c (committed_lsn, persist_truncate_wal_above), ha.h/ha.c (last_known_committed_lsn)
• H: config_repl.c (upgrade majority-commit), capnp schema, ha.h (committed_config_tag)
• F: ha.c/cluster.c (channel connect, extended hello, streaming, EPOLLRDHUP)
• G: src/lib/ (retry loop, xce_retry_config_t, redirect fields)

Trade-offs:

• ✅ Each atomic commit is testable before the next starts
• ✅ Follows existing TDD convention (failing test first)
• ❌ Long sequential dependency chain; F and G can only start late

Option B: Infrastructure-first parallel tracks

Track 1 (election+bootstrap): D → A → B → C
Track 2 (replication): E → H
Join (channel): F using Track 1 + Track 2 artifacts
Final: G

Trade-offs:

• ✅ Track 1 and Track 2 are mostly independent until F
• ❌ Track 2 cannot be fully validated without Track 1 (no quorum without election)
• ❌ More coordination overhead between developers on shared ha.h

Option C: Stub-forward then wire

Add all new stubs with -ENOSYS first, then wire each one. Matches the existing codebase pattern (many stubs already present).

Trade-offs:

• ✅ Compiles throughout; no long broken-build windows
• ❌ Stub-first delays real testability; TDD mandate disfavors this

Recommendation: Option A. The dependency chain is well-defined and single-developer serial implementation is the realistic mode for this C codebase.

---

4. Effort and Risk Summary

Req	Effort	Risk	Justification
1 — Listener before election	M (3–5d)	Medium	Refactor startup order; must remove vtaskmgr/config dependency first
2 — Quorum election	XL (2+ wks)	High	Full algorithm replacement; new wire protocol; FSM extension; concurrent dispatch
3 — Config bootstrap	M (3–5d)	Medium	New function + probe RPC; depends on do_one_config_fetch (wired)
4 — Startup validation	S (1d)	Low	Additive CLI + validation; ASAN one-liner
5 — Majority-commit WAL	L (1–2 wks)	High	committed_lsn cross-cuts persist+repl+ha; repl_broadcast must become real
6 — Persistent channel	XL (2+ wks)	High	New connection lifecycle; full state sync protocol; 3-tier discovery
7 — Client retry	M (3d)	Low	Well-understood retry pattern; additive libxce change
8 — Majority-commit config	M (3d)	Medium	Shares Req 5 infrastructure; mainly wiring push_config_to_standbys
9 — State durability	S (1d)	Low	File write+read; extends ha_init
10 — Observability	S (alongside)	Low	Log statements woven into each change

Total estimated effort: 6–9 weeks for one engineer implementing with full TDD.

High-risk dependencies:

• Req 2 must complete before Req 3 can be integrated end-to-end.
• Req 5 (repl_broadcast implementation) must complete before Req 8 and before Req 6 (WAL streaming over the channel).
• Req 4 (vcpus CLI) must complete before Req 1 (vtaskmgr before election).

---

5. Key Reusable Assets

Asset	Location	Reuse in
repl_add/remove_standby	repl.c	Req 6 (channel registration)
repl_start/complete_catchup	repl.c	Req 6 (state sync protocol)
repl_needs_snapshot	repl.c	Req 6 (snapshot decision)
repl_on_ack	repl.c	Req 5 (majority-commit ack handler)
repl_trim_lsn	repl.c	Req 5 (WAL trim with committed_lsn)
connect_ssl_raw()	netio.c/config_repl.c	Req 3 (probe RPC)
do_one_config_fetch()	config_repl.c	Req 3 (deferred fetch)
xorshift64_next + ha_seed_jitter	ha.h / ha.c	Req 1, 2 (jitter)
ha_do_promote / ha_do_demote	ha.c	Req 2, 6 (role transitions)
ha_should_redirect / ha_no_primary_known	ha.c	Req 7 (retry redirect)
iomgr epoll + EPOLLIN/EPOLLRDHUP	iomgr (cluster.c)	Req 1, 6
Atomic write+rename pattern	config_repl.c	Req 9 (ha-term.bin)
INFO/WARN/ERROR macros	xcelog.h	Req 10

---

6. Constraints for Design Phase

• C gnu17, no C++, no new external dependencies.
• ASAN build (nix build .#xcecore_debug) must pass after every change.
• Static peer membership: --peers list is fixed at startup; no dynamic join/leave of controller peers. Quorum denominator is fixed for cluster lifetime.
• TDD mandatory: failing test in src/controller/testsuite/ before each implementation change.
• make test must pass without a live server at every commit.
• commit_timeout_ms must be configurable (default 5000ms).
• Single-node mode (no --peers): quorum=1 everywhere; all new majority-commit paths must short-circuit to self-only.
• WAL write stall window (operational constraint, not a bug): during standby catch-up after election, repl_wait_majority_ack() will time out on every write until at least one standby completes catch-up. This is correct CP behavior; operators must be warned in docs.

---

7. Research Needed (Design Phase)

• vtaskmgr API for pre-election dispatch: confirm that vtaskmgr_init(vcpus) can be called before cluster_config is loaded (after removing vcpus from config struct). Verify no hidden config dependency.
• iomgr async connect pattern for vote requests: confirm iomgr_async_vote_request() can be implemented using existing async connect infrastructure without new socket primitives.
• capnp schema extension: confirm the capnp code-generation toolchain can add OP_requestVote, OP_probeConfigTag, OP_walAck, OP_upgradeConfigAck, OP_snapshotTransfer, OP_walCatchupEntry without breaking existing decode paths.
• persist_truncate_wal_above() safety: verify WAL segment file structure supports efficient truncation at arbitrary LSN boundary without full segment rewrite.
• libxce client API surface: identify the exact function in src/lib/ that wraps xce_encode()+send+recv+xce_decode() to locate the correct insertion point for retry logic.

---

Gap Analysis (Updated — Brownfield Audit)

Analysis date: 2026-06-01. Branch: persistent_ha. Phase: tasks-generated. This section audits current implementation state vs. all 11 task groups.

Analysis Summary

• Scope: 10 requirements across 11 task groups. 30 subtasks checked [x]; only section headers and integration tasks 11.1–11.5 remain [ ].
• Implementation status: ~95% complete. All core subtasks (1.1–10.2) are marked done and confirmed implemented: CTRL_INIT/CTRL_ELECTING enum, ha-term.bin durability, repl_wait_majority_ack, persist_truncate_wal_above, xce_request_with_retry, xce_probe_config_tag, repl_broadcast (real, not stub).
• Key architectural shift since original research: The election-iomgr-refactor spec superseded parts of this spec. bootstrap_config_if_needed and ha_find_and_connect_to_primary are no longer called from myapp_run — standby connection is inbound (primary dials standby via on_standby_connected). ha_find_and_connect_to_primary exists only as a test hook reference in ha.h.
• Remaining work: Tasks 11.1–11.5 are manual/integration tests documented in manual-tests.md. The build passes with all unit tests green.
• Effort remaining: S (1–2 days for manual integration test execution + result recording). Risk: Low.

Requirement-to-Asset Map (Current State)

Req	Status	Key Evidence
1 — Listener before election	✅ Done	CTRL_INIT=2, CTRL_ELECTING=3 in ha.h; cluster_open_listener() called before election in myapp.c; ha_wait_for_heartbeat_or_timeout() implemented
2 — Quorum election	✅ Done	am_i_primary_quorum → superseded by ha_start_election_async + raft library (election-iomgr-refactor); handle_request_vote, ha_elect_evaluate_and_signal implemented
3 — Config bootstrap	✅ Done (changed arch)	xce_probe_config_tag() implemented in config_repl.c:1140; bootstrap_config_if_needed removed from myapp_run per election-iomgr-refactor task 5.3; defer_config_fetch field exists in ha_state_t; post-election config push via ha_send_ctrlhello_to_all_standbys covers deferred fetch
4 — Startup validation	✅ Done	startup_validation.c exists; cert pre-flight, peer count validation, vcpus CLI implemented; SSL_CTX_free ASAN fix in ha_fini
5 — Majority-commit WAL	✅ Done	repl_broadcast() real implementation (repl.c:193+); repl_wait_majority_ack() implemented (repl.c:338+); committed_lsn in persist_t (persist.c:1029); persist_truncate_wal_above() implemented (persist.c:1293+)
6 — Persistent control channel	✅ Done (changed arch)	primary_ctrl_fd in ha_state_t; ha_on_primary_ctrl_fd_rdhup triggers re-election; inbound channel via on_standby_connected in ha.c; ha_arm_heartbeat_timeout_vtask for timeout detection; ha_find_and_connect_to_primary is a hook-only reference now (CTRL_STANDBY never dials out)
7 — Client retry	✅ Done	xce_request_with_retry() + xce_retry_config_t in request.c/request.h (request.c:1664+)
8 — Majority-commit config	✅ Done	OP_upgradeConfigAck handler; committed_config_tag in ha_state_t; push_config_to_standbys upgraded; repl_wait_majority_ack with OP_upgradeConfigAck discriminant
9 — State durability	✅ Done	ha_persist_term() / ha_restore_term() in ha.h:317-340; ha-term.bin read in cluster_init_phase1 (cluster.c:809); voted_for_name field in ha_state_t
10 — Observability	✅ Done	Structured INFO/WARN log statements throughout all paths

Gaps: Tasks 11.1–11.5 (Manual Integration Tests)

Gap 1: Task 11.1 — Concurrent startup and config-fresh election

• What's needed: 3-node cluster starts simultaneously; verify only one elects primary; config-fresh node wins.
• How to run: See manual-tests.md — Test 1 (3-node clean start, verify ctrl-1 primary).
• Infrastructure: deploy-ha.sh + start.sh scripts; log grep for CTRL_PRIMARY.
• Effort: ~30 min.

Gap 2: Task 11.2 — Failover and WAL durability

• What's needed: Kill primary; verify standby detects via heartbeat timeout → promotes; committed WAL survives.
• How to run: manual-tests.md Test 2.
• Effort: ~30 min.

Gap 3: Task 11.3 — Unconfigured node bootstrap

• What's needed: Start node with config_tag=0; verify it receives config via primary's ctrlhello push after election.
• How to run: manual-tests.md Test 3.
• Note: Architecture changed — the deferred fetch now happens via ha_send_ctrlhello_to_all_standbys (primary pushes), not do_one_config_fetch (standby pulls). This is simpler and already works by default.
• Effort: ~30 min.

Gap 4: Task 11.4 — Config upgrade majority-commit and term durability

• What's needed: Config upgrade, partition, timeout; restart with persisted term.
• How to run: manual-tests.md Test 4.
• Effort: ~30 min.

Gap 5: Task 11.5 — Full ASAN build verification

• What's needed: nix build .#xcecore_debug clean.
• Status: Unit tests pass under /xce-build controller (which uses debug+ASAN). Full nix build is the final check.
• Effort: ~15 min build time.

Constraints Carried Forward

• ha_find_and_connect_to_primary is dead code: Declared in ha.h with test hook but never called in production. CTRL_STANDBY waits for primary to dial in. Safe to delete in cleanup pass.
• bootstrap_config_if_needed removed from myapp_run: Covered by primary-push model in election-iomgr-refactor. Tasks 5.2–5.3 [x] reflect the original design; the implementation pivoted to push-on-ctrlhello.
• Single-node path: Verified via test_ha_election.c single-node shortcircuit.

Implementation Approach Recommendation

No new code needed. Proceed directly to manual integration test execution:

1. Run /xce-build to confirm full build clean.
2. Execute manual-tests.md tests 1–5 in order.
3. Record results in the Results Template at the bottom of manual-tests.md.
4. Mark tasks 11.1–11.5 [x] on pass.
5. Run nix build .#xcecore_debug for 11.5 ASAN verification.

---

Gap Analysis — Validation Pass (2026-06-09)

Scope and Method

Validation pass against the working tree on branch persistent_ha (including uncommitted changes to ha.c, ha.h, myapp.c, raft.c). All 50 acceptance criteria across the 10 requirements were checked against on-disk code. Note that the architecture has evolved past this spec's original design via four successor specs (raft-library, election-iomgr-refactor, ha-heartbeat-liveness, ha-repl-hardening, ha-peer-connection-reuse): the blocking election (am_i_primary_quorum + semaphore wait) was replaced by a fully async raft-library-driven election inside iomgr, and the standby-pull config bootstrap was replaced by a primary-push model. Several "gaps" below are therefore sanctioned deviations, not defects — they need requirements-doc reconciliation, not code.

All checkboxes in tasks.md (1.1–11.5) are marked [x], including 9.3+, and manual integration tests 11.1–11.4 are recorded as executed 2026-06-03. The memory note "9.3+ remaining" is stale.

1. Requirement-to-Asset Map

Status legend: Implemented / Partial / Missing / Unknown / Constraint (sanctioned architectural deviation needing spec reconciliation).

Requirement 1 — Listener and Connection Framework Before Election

AC	Status	Assets / Evidence
1.1 listener bound before election	Implemented	myapp.c:113 cluster_open_listener() before ha_start_election_async(); cluster_register_listener at myapp.c:138
1.2 iomgr/vtaskmgr before election	Implemented	myapp.c:122-131 iomgr+vtaskmgr init/start precede election (Step 3/4)
1.3 not-ready rejection in INIT/ELECTING	Implemented	ha_dispatch_gate() ha.c:1412 returns -XCE_ERR_NOT_READY; wired at vtask dispatch entry cluster.c:226
1.4 standby error includes redirect host/port	Partial	XCE_ERR_STANDBY / E_XCE_REDIRECT returned (ha.c:1481, join.c:241), ha_should_redirect() exists, capnp fields redirectHost/redirectPort exist (controller.capnp:565-566) — but no reply path ever populates redirect_host/redirect_port; replies are sent with data_which = nodata

Requirement 2 — Quorum-Based Controller Election

AC	Status	Assets / Evidence
2.1 strict-majority quorum	Implemented	raft.c:223-234 quorum = n_peers/2 + 1; ha_elect_evaluate_and_signal ha.c:1995
2.2 lower-term deny	Implemented	raft_on_vote_request Step 1 (raft.c:110)
2.3 higher-term update + clear vote	Implemented	raft.c:121-130 (also steps a live leader down to follower)
2.4 freshness gate deny	Implemented	raft_cb_vote_eligible ha.c:202; handle_request_vote ha.c:1791
2.5 catch-up standby vote denied	Implemented	in_catch_up check ha.c:218, ha.c:1845
2.6 one vote per term	Implemented	raft.c:157-164 (with idempotent re-grant to same candidate — safe extension)
2.7 yield to fresher peer	Implemented	yield gate ha.c:2017-2031 (max_resp_tag/max_resp_lsn)
2.8 tag-zero candidate vs configured peer → fatal exit	Constraint	Implemented as voter-only standby, not fatal exit: ha.c:3133-3145 (tag==0 && n_peers>1 → CTRL_STANDBY, wait for push). Strictly safer for ops (node self-heals via config push) but deviates from the EARS text "exit with a fatal error"
2.9 single-node immediate self-elect	Implemented	raft.c:83-88 RAFT_ELECTED_IMMEDIATELY; zero-peer guard ha.c:1985
2.10/2.11 split vote → one retry w/ jitter+term++, then standby	Constraint	raft library becomes FOLLOWER on no-quorum/timeout (raft.c:237-244, raft_on_election_timeout); retries are driven indefinitely by the standby liveness watchdog → raft_on_leader_lost → delayed raft_check_and_start_election (raft.c:268-290). Net behavior: unbounded jittered retry loop instead of exactly-once retry. Liveness outcome equivalent or better; spec text outdated
2.12 identical-freshness tiebreak by jitter, not priority	Partial	Winner choice never uses priority values; but election timing uses deterministic priority-based stagger (myapp.c:229-239 priority * election_priority_step_ms; raft.c:272-273 priority * election_timeout_ms). Priority is repurposed as stagger index, not vote weight — spec wording should be reconciled

Requirement 3 — Config Bootstrap for Unconfigured Nodes

AC	Status	Assets / Evidence
3.1 probe all peers' config tag before candidacy	Missing (as specified)	xce_probe_config_tag() fully implemented (config_repl.c:1140, single-RTT SSL probe) and unit-tested, but never called in production — bootstrap_config_if_needed() was removed from myapp_run (see Constraints in prior pass). Replacement: unconditional voter-only gate when my_config_tag==0 && n_peers>1 (ha.c:3133)
3.2 voter-only participation, deferred fetch	Implemented (pivoted)	voter-only path ha.c:3133-3145; config arrives via primary push (push_config_to_standbys from raft_cb_become_leader/COMMITTING driver)
3.3 all peers tag-zero → normal candidacy	Partial / Unknown	No probe means a tag-0 node cannot distinguish "peers have config" from "all-zero cold start"; it always goes voter-only first. Recovery relies on the heartbeat watchdog firing → ha_on_primary_ctrl_fd_rdhup → raft_on_leader_lost re-election, which bypasses the tag-0 gate (ha.c:3183-3291). All-zero cold start thus converges, but only after heartbeat_timeout_ms (~10s) delay per round; not covered by an automated test
3.4 exponential-backoff probe retry 500ms→10s	Missing (as specified)	No probe retry loop exists; the passive watchdog loop replaces it. Spec text obsolete under the push model
3.5 deferred fetch from proven winner	Implemented (pivoted)	Primary pushes config + ctrlhello to all standbys on COMMITTING (ha.c:2087-2103); fetch_config_from_primary/do_one_config_fetch standby-pull removed by design (CLAUDE.md: standbys never dial out)

Requirement 4 — Startup Validation and Safety Checks

AC	Status	Assets / Evidence
4.1 even-N peer rejection	Implemented	validate_peer_count() startup_validation.c:90; double-enforced in raft_init (raft.c:21)
4.2 cert/key pre-flight with full path	Implemented	validate_cert_files() startup_validation.c:106 (-ENOENT, path in message), wired in main.c
4.3 vcpus >= 1	Implemented	validate_vcpus() startup_validation.c:25; --vcpus argp option in main.c
4.4 config-file vcpus deprecation WARN	Implemented	validate_startup_config() startup_validation.c:46; warned at myapp.c:80-86

Requirement 5 — Majority-Commit WAL Replication

AC	Status	Assets / Evidence
5.1 job submit / state transitions / queue changes majority-committed	Partial	Only WAL_JOB_SUBMIT routes through persist_wal_sync_append() (persist.c:1118). Job state transitions (WAL_JOB_START/FINISH/FAIL/..., persist.c:1163) and queue changes (WAL_QUEUE_CTRL, persist.c:1194) use persist_wal_async() — no majority wait
5.2 host state / checkpoint refs async	Implemented	worker/host events → persist_wal_async; checkpoint markers never replicated (persist.c:395)
5.3 standby durable write + ACK with LSN	Implemented	OPCode_walAck flow, repl_on_ack (repl.c:178), acked_lsn tracking
5.4 lag eviction + catch-up trigger	Implemented	pending_ack_count > max eviction with inline catch-up start, repl.c:214-222
5.5 commit timeout → error to caller	Constraint	Deliberate degraded mode: persist_wal_sync_append reports -ETIMEDOUT/-ENODEV via sync_rc_out but returns 0 and the operation proceeds with local-only durability (persist.c:1003-1043; hook comment persist.c:1114). This is an AP-leaning deviation from the spec's CP statement — documented in code, not in requirements
5.6 truncate above committed LSN on promotion	Implemented	ha_compute_safe_committed_lsn() from voter replies (ha.c:373-437) → persist_truncate_wal_above(safe_lsn) in raft_cb_become_leader (ha.c:471); per controller CLAUDE.md this is the verified-against-TLA+ design
5.7 truncate on catch-up reconnect	Partial	RDHUP/watchdog failover path truncates to last_known_committed_lsn before re-election (ha.c:3204-3207). A standby that restarts (rather than detecting RDHUP) replays its full WAL including uncommitted tail; no truncation occurs before it rejoins and receives catch-up entries — divergent-tail handling on the restart path is unverified
5.8 single-node local write satisfies majority	Partial	repl_wait_majority_ack returns 0 when n_active == 0 (repl.c:384), but persist_wal_sync_append short-circuits before calling it when nstandbys == 0 and sets sync_err = -ENODEV without advancing committed_lsn (persist.c:1023-1031). Single-node committed_lsn appears to advance only via the election-time safe-LSN path — runtime advancement per entry is not evident. Needs verification (interacts with WAL trim guard, which uses committed_lsn as threshold)

Requirement 6 — Persistent Standby-to-Primary Control Channel

AC	Status	Assets / Evidence
6.1 standby establishes persistent TLS connection to primary	Constraint	Direction inverted by deliberate design (controller CLAUDE.md "Inbound vs. Outbound Connection Ownership"): the primary dials out to standbys; standby-side ha_find_and_connect_to_primary() is dead code (declared ha.h:741, never called in production). The functional goal (persistent channel, no per-RPC connections) is achieved
6.2 disconnect → immediate election	Implemented	EPOLLRDHUP on primary_ctrl_fd → ha_on_primary_ctrl_fd_rdhup() (ha.c:3183)
6.3 heartbeat resets timeout	Implemented	ha_on_heartbeat_received() on every NOP/ctrlhello (cluster.c:249, 283)
6.4 heartbeat timeout + jitter → election	Implemented	ha_arm_heartbeat_timeout_vtask liveness poll; manual test 11.2 measured ~11s detection (configured timeout, not sub-second — sub-second applies to the RDHUP path)
6.5 sync missing state before admitting to replication set	Implemented	heartbeat-reply-driven driver: standby stateReport → ha_repl_decide() (ha.c:1157-1176) → stream WAL / snapshot / config; in_catch_up gates quorum membership (repl.c:367)
6.6 snapshot when standby older than oldest WAL segment	Implemented	standby_lsn < oldest_wal_lsn → HA_REPL_SEND_SNAPSHOT (ha.c:1176), ha_send_snapshot_to_standby() chunked transfer (config_repl.c:1578), standby handler cluster.c:182
6.7 config push after catch-up on tag mismatch	Implemented	same driver pushes config on tag divergence; push_config_to_standbys
6.8 winner closes outbound, accepts inbound	Implemented	role flip handled in raft_cb_become_leader / RDHUP path; surviving vote eps promoted to standby_endpoints; late joiners get fresh outbound connects from primary_heartbeat_fn

Requirement 7 — Client-Side Transparent Retry (libxce)

AC	Status	Assets / Evidence
7.1 NOT_READY → backoff retry	Implemented	xce_request_with_retry() request.c:1664; tests test_libxce_retry.c
7.2 STANDBY + redirect → follow redirect	Partial	Client logic implemented and unit-tested via hook, but no controller reply ever carries redirect_host/port (see 1.4) — end-to-end the redirect-follow path can never trigger; falls back to backoff
7.3 TCP refused/timeout → reconnect retry	Implemented	request.c retry classification
7.4 non-retriable surfaced immediately	Implemented	AUTH/PERM/INVALID/NOENT surfaced without retry
7.5 configurable params + defaults	Implemented	xce_retry_config_t, XCE_RETRY_DEFAULTS (request.h:140-150)
7.6 thundering-herd jitter	Implemented	jittered exponential backoff; 50-client concurrency test
(adoption)	Missing	Zero production call sites: worker, scheduler, and apiserver backend never call xce_request_with_retry — the wrapper is shipped but unused

Requirement 8 — Majority-Commit Config Replication

AC	Status	Assets / Evidence
8.1 majority commit for config push	Implemented	push_config_to_standbys → repl_wait_majority_ack(..., OPCode_upgradeConfigAck) (config_repl.c:905) → committed_config_tag advanced (config_repl.c:908)
8.2 standby durable write + ACK	Implemented	handle_repl_config_push + OP_upgradeConfigAck; acked_config_lsn tracked separately from WAL acks (repl.c:375-380)
8.3 timeout → error	Implemented	-ETIMEDOUT from repl_wait_majority_ack, WARN logged
8.4 promoted standby serves only committed config tag	Partial	last_known_committed_config_tag captured from CtrlHelloReply (cluster.c:261) and committedConfigTag piggybacked on heartbeat (capnp); committed_config_tag consulted at ha.c:2514-2521 and safe-tag computed on promotion (ha.c:485). Enforcement at serve time (refusing to serve a fresher-but-uncommitted config) not explicitly verified
8.5 single-node local write = majority	Implemented	safe-tag path on become-leader; committed_config_tag advanced on push without standbys

Requirement 9 — Election State Durability

AC	Status	Assets / Evidence
9.1 persist term+vote before reply	Implemented	raft on_persist fires before any grant/RPC (raft.c:77, 176); ha_persist_term() atomic write+rename with magic+CRC32 (ha.c:1647-1681)
9.2 restore on startup	Implemented	ha_restore_term() (ha.c:1683+); manual test 11.4 confirmed restored term=2 on restart
9.3 missing file → term 0, no vote	Implemented	missing/short-read/bad-magic/CRC-mismatch all degrade safely to term 0 with WARN (ha.c:1715-1737)

Requirement 10 — Observability

AC	Status	Assets / Evidence
10.1 election outcome: term + votes granted + quorum size	Partial	FSM transition INFO logs carry term (ha.c:507, 659, 870, 881, 2080, 2158); majority-ack log carries acks+quorum (ha.c:2929); but no single election-complete record logs votes granted and quorum size together
10.2 bootstrap probe finding	Partial	voter-only decision logged (ha.c:3134); per-peer probe logs exist in xce_probe_config_tag but that path is unused in production
10.3 WAL quorum INFO / timeout WARN	Implemented	repl.c:408-410
10.4 config quorum INFO / timeout WARN	Implemented	config_repl.c push path logs both
10.5 FSM transition INFO	Implemented	all transitions logged, including RDHUP (ha.c:3199)

Summary Tally (by requirement)

Req	Verdict
R1	Implemented except redirect payload (1.4 Partial)
R2	Implemented; 2.8/2.10/2.11/2.12 are sanctioned deviations needing spec reconciliation
R3	Pivoted to primary-push; 3.1/3.4 Missing-as-specified, 3.3 Unknown (cold-start convergence untested)
R4	Implemented (4/4)
R5	Core implemented; 5.1 coverage gap, 5.5 CP→degraded-mode deviation, 5.7 restart-path gap, 5.8 needs verification
R6	Implemented; 6.1 direction inverted by design (Constraint)
R7	Library implemented; redirect payload + production adoption Missing
R8	Implemented; 8.4 serve-time enforcement Partial
R9	Implemented (3/3)
R10	Implemented except consolidated election-outcome record (10.1)

Net: ~40 of 50 acceptance criteria fully implemented; 7 partial; 3 missing-as-specified (all three superseded or deliberately pivoted). No requirement is wholly unimplemented.

2. Remaining Gaps — Concrete List

1. G1 — Redirect payload never populated (R1.4, R7.2): capnp fields exist; standby replies send nodata. Client redirect-follow logic is dead end-to-end.
2. G2 — WAL majority-commit covers only job submission (R5.1): state transitions and queue changes are async-only; a failover can lose acknowledged state changes.
3. G3 — Commit timeout is degraded-mode, not an error (R5.5): spec promises CP (timeout error to caller); code chooses availability. Needs an explicit, documented decision.
4. G4 — Single-node committed_lsn runtime advancement unverified (R5.8): -ENODEV short-circuit skips the advance; interacts with the WAL trim guard.
5. G5 — Standby restart path skips divergent-tail truncation (R5.7): truncation happens only in the RDHUP/watchdog path.
6. G6 — Config probe path unwired (R3.1/3.3/3.4): xce_probe_config_tag is production-dead; all-zero cold-start converges only via watchdog timeout and lacks a test.
7. G7 — libxce retry has zero production adopters (R7 objective): worker/scheduler/apiserver still have whatever ad-hoc behavior they had.
8. G8 — Observability: no consolidated election-outcome log with vote tally + quorum size (R10.1).
9. G9 — Spec/code reconciliation debt: 2.8 (standby instead of fatal exit), 2.10/2.11 (watchdog retry loop instead of retry-once), 2.12 (priority-based stagger), 3.x (push model), 5.5 (degraded mode), 6.1 (inverted channel direction) — all stable code behavior contradicting EARS text.
10. G10 — Cleanup: ha_find_and_connect_to_primary dead code; bounded per-RDHUP elect_ctx leak (ha.c:3252-3258, documented UAF-avoidance).

3. Implementation Approach Options

Option A — Extend existing components in place (code-first)

Close G1–G8 directly in ha.c / persist.c / repl.c / join.c / worker+scheduler clients; patch requirements.md text afterward.

• Files: join.c+reply.c (G1), persist.c hook routing (G2/G4), cluster.c startup replay (G5), myapp.c+config_repl.c (G6), worker/scheduler request sites (G7), ha.c logging (G8).
• ✅ No new modules; follows established patterns (every gap has an existing seam and a testsuite file).
• ✅ Fastest path to closing the spec.
• ❌ G2/G3 are policy changes with latency and availability consequences — landing them silently inside this aging spec hides a significant decision.
• ❌ ha.c is already 3,818 lines; further growth degrades maintainability.

Option B — Spec-first reconciliation, then targeted follow-up spec

Amend requirements.md/design.md to match the as-built architecture (push bootstrap, inverted channel, watchdog retry, degraded mode), close only the unambiguous small defects (G1, G4, G8, G10) here, and move G2/G3/G5/G7 into the active ha-repl-hardening spec (or a new ha-cp-commit spec) with their own requirements review.

• ✅ Honest record: the EARS text stops contradicting shipped behavior; the CP-vs-degraded decision gets explicit operator-facing review.
• ✅ Keeps this spec closeable; G2/G3 get proper TDD treatment in a spec that already owns repl hardening.
• ❌ More process; durability gap (G2) stays open until the follow-up lands.
• ❌ Risk that "deferred" becomes "dropped" without tracking.

Option C — Hybrid (recommended)

Phase 1 (this spec, S-size code): G1 redirect payload, G4 single-node commit advance verification/fix, G8 election-outcome log, G10 cleanup, plus requirements.md deviation annotations (G9) and a cold-start integration test for G6's watchdog fallback. Phase 2 (ha-repl-hardening spec): G2 (extend sync-append coverage to state transitions/queue changes — likely with group-commit batching to bound latency), G3 (explicit CP-vs-degraded decision + config knob), G5 (restart-path truncation), G7 (retry adoption across daemons).

• ✅ Ships every low-risk fix now; routes every policy-bearing change through review.
• ✅ Matches the existing spec topology (repl-hardening already owns quorum/commit semantics).
• ❌ Requires coordinating two specs; G6's probe-vs-watchdog question must be answered during reconciliation (recommend: delete the probe path, document the watchdog fallback, keep the test).

4. Effort & Risk (remaining work)

Gap	Effort	Risk	Justification
G1 redirect payload	S	Low	Fields exist in schema/typemap; populate in 2 reply sites + 1 e2e test
G2 sync-append coverage	M	Medium	Mechanical routing change but puts majority-RTT on hot job-state path; needs batching design + latency measurement
G3 CP-vs-degraded decision	S (code) / M (decision)	High	One-line behavior flip, but it trades availability for consistency cluster-wide; needs operator sign-off and failure-mode tests
G4 single-node committed_lsn	S	Medium	Small fix, but wrong answer corrupts WAL trim guard invariant (committed_lsn <= applied_lsn)
G5 restart-path truncation	M	Medium	Touches startup replay ordering; must honor TLA+ safe-commit-point rule (never truncate via stale ctrlhello value)
G6 cold-start test + probe removal	S	Low	Behavior already converges via watchdog; work is test + dead-code deletion + doc
G7 retry adoption	M	Medium	Multi-daemon change; behavioral shift in error surfacing for worker/scheduler/apiserver
G8 election-outcome log	S	Low	Data already in ha_elect_ctx_t/raft state at decision points
G9 spec reconciliation	S	Low	Documentation only; six identified deltas
G10 cleanup	S	Low	Dead-code deletion + bounded-leak fix behind existing tests

Aggregate remaining: ~M total for Phase 1 (all S items), ~L for Phase 2 (G2+G3+G5+G7 under ha-repl-hardening).

5. Recommendations

1. Adopt Option C. Close this spec with Phase 1; transfer G2/G3/G5/G7 into ha-repl-hardening as explicit requirements so they get their own approval gate.
2. Reconcile requirements.md now (G9): rewrite AC 2.8, 2.10–2.12, 3.1–3.5, 5.5, 6.1 to describe the as-built push/watchdog/outbound-primary architecture. The controller CLAUDE.md and TLA+ mapping are already correct — the spec is the lagging artifact.
3. Decide G3 deliberately. The spec's headline promise is "zero data loss on failover" (CP); the code ships degraded-mode availability. Either add a commit_strict config knob defaulting per the decision, or amend the spec's CP claim. Do not leave the contradiction implicit.
4. Fix G1 immediately — it is the cheapest fix with end-to-end user impact (failover redirect for clients), and it unblocks the already-tested libxce retry path.
5. Delete dead code (ha_find_and_connect_to_primary, production xce_probe_config_tag path if reconciliation confirms the push model) rather than leaving two competing bootstrap narratives in the tree.

Research Needed

• RN1: Does single-node committed_lsn ever advance at runtime (G4)? Trace persist_wal_sync_append + WAL trim guard in a single-node soak; check whether ha-repl-hardening task 6.5 already addressed this.
• RN2: Latency budget for extending majority-commit to job state transitions (G2) — measure per-entry RTT on 3-node LAN; evaluate group commit / pipelined acks before committing to a design.
• RN3: Restart-path divergence (G5) — construct the failure: standby with uncommitted WAL tail restarts, rejoins, receives catch-up; verify whether apply_wal_entry overwrite semantics mask or corrupt the divergent tail.
• RN4: All-zero cold-start convergence time and liveness (G6) — 3-node cluster, no configs anywhere; confirm watchdog-driven election converges and measure time-to-primary.
• RN5: Serve-time enforcement for 8.4 — can a promoted standby serve a config tag above committed_config_tag in any window between promotion and first quorum push?

---

Gap Re-Audit (2026-06-18)

Trigger: post-implementation gap audit. Tasks 46/46 [x], phase=implemented. Method: re-verified the 2026-06-09 gap list (G1–G10) against the current working tree on persistent_ha. Since the prior pass, two successor specs landed and closed several of the gaps this spec recorded: libxce-failover (client-side transparent failover/redirect) and ha-commit-durability (bug-661, landed 2026-06-18). The earlier majority-commit model was also superseded by a wal_classify() WAIT/RETAIN scheme from ha-commit-barrier-rollout. Line numbers below reflect the current tree (ha.c is now 5,338 lines; persist.c 2,587).

Headline change vs 2026-06-09

The 2026-06-09 pass found ~40/50 ACs fully implemented with 10 open gaps (G1–G10). Five of those gaps are now CLOSED by successor-spec work; the residual set is smaller and almost entirely spec-text reconciliation debt rather than missing code.

Requirement-to-Asset Map — Delta Since 2026-06-09

Gap (2026-06-09)	Req	New Status (2026-06-18)	Evidence
G1 Redirect payload never populated	R1.4, R7.2	CLOSED	fill_redirect_reply() populates reply->redirect_host/redirect_port from ha->primary_host/port (join.c:28-32); called on the -E_XCE_REDIRECT path (join.c:282); covered by test_join_redirect.c. Comment cites libxce-failover Req 8.1.
G7 libxce retry has zero adopters	R7 obj.	CLOSED (re-homed)	Orphaned no-op xce_request_with_retry() + xce_retry_send_hook were removed (request.c:1923-1936); superseded by the xcecli_* failover loop in src/lib/client.c + src/lib/client_failover.c. XCE_RETRY_DEFAULTS retained as the shared tunable. Production redirect-follow is now live end-to-end (G1 + client_failover).
G10 ha_find_and_connect_to_primary dead code	cleanup	CLOSED	No production .c reference remains (grep over src/controller/*.c returns nothing). Dead-code deletion done.
G2 WAL majority-commit covers only job submit	R5.1	SUPERSEDED (model changed)	Replaced by wal_classify() WAIT/RETAIN scheme (persist.c:2004). WAL_JOB_SUBMIT and WAL_QUEUE_CTRL are WAL_MODE_WAIT (commit-gated before requester reply); WAL_WORKER_CTRL JOIN/UPDATE are WAIT; job state transitions (WAL_JOB_START/FINISH/FAIL/...) are WAL_MODE_RETAIN — replicated and retained-until-committed but not blocking the worker-fact report. The request-shaped side of dual-origin events is realized by the handler barrier in do_jobcontrol_op. The original "everything sync-blocks" framing was deliberately not adopted (latency); coverage is now classification-driven, owned by ha-commit-barrier-rollout / ha-commit-durability.
G3 Commit timeout degraded-mode vs CP	R5.5	SUPERSEDED (bug-661 / ha-commit-durability)	Durability invariant now GUARANTEED: acknowledged ⇒ fdatasync'd on a fault-tolerant majority. majority_candidate_locked caps the primary self-contribution at persist_durable_lsn; fsync_worker group-commit flusher advances durable_lsn only on fdatasync success (fail-loud rc-gate); standby fdatasyncs before WalAck. Verified by test_commit_durability.c, the ack-before-fdatasync E2E, and the rolling-failover soak (zero acked-job loss). Controller CLAUDE.md checklist item flipped to GUARANTEED 2026-06-18. The old "silent local-only degrade" path is gone.

Residual Gaps (still open after 2026-06-18)

ID	Req	Status	Detail
R1 (was G6) Config probe path unwired	R3.1, R3.3, R3.4	Missing-as-specified (stable)	xce_probe_config_tag() is fully implemented + unit-tested (config_repl.c:1219) but still not called in production — myapp.c has no probe call (grep confirms). The tag-0 voter-only gate (ha.c:4495) uses static my_config_tag/n_peers, not probe results. The comment at ha.c:4493 ("xce_probe_config_tag path already called in myapp_run") is stale/incorrect — no such call exists. All-zero cold-start converges only via watchdog-driven re-election (~heartbeat_timeout), and that path lacks an automated convergence test (prior RN4).
R2 (was G9) Spec/code reconciliation debt	2.8, 2.10–2.12, 3.1–3.5, 5.5, 6.1	Open (docs only)	requirements.md EARS text still describes: fatal-exit for tag-0 candidate (code: voter-only standby), exactly-once split-vote retry (code: unbounded watchdog retry loop), exponential-backoff probe (code: passive watchdog), standby-dials-primary (code: primary dials out — inverted by design per controller CLAUDE.md), CP-timeout-error (code: now GUARANTEED durability, but worded as the old model). spec.json note still lists the now-closed G1/G3 framings. The TLA+ mapping + controller CLAUDE.md are authoritative and correct; the spec is the lagging artifact.
R3 (was part of G4) Single-node committed_lsn runtime advance	R5.8	Likely closed — verify	test_single_node_committed_lsn_advances (test_p2_integration.c:251) and test_legacy_sync_append_holds_committed_at_durable (test_commit_durability.c:660) now exist, suggesting ha-commit-durability addressed the prior -ENODEV short-circuit concern. Recommend a quick read of those tests to confirm RN1 is retired before closing.
R4 (was G5) Standby restart-path divergent-tail truncation	R5.7	Re-scoped to recovery	The RDHUP/watchdog failover path truncates to last_known_committed_lsn. The restart path is now governed by ha-commit-durability recovery (recover_verdict_t clean-vs-corruption, durable-floor re-derivation, pre-promotion fence). Whether a restarted standby with an uncommitted tail is correctly fenced/caught-up before rejoin is owned by that spec's recovery tests, not this one. Verify via ha-commit-durability rather than re-opening here.
R5 (was G8) Consolidated election-outcome log	R10.1	Open (minor)	FSM-transition INFO logs carry term, and majority-ack logs carry acks+quorum, but there is still no single election-complete record emitting votes-granted AND quorum-size together. Data is available in ha_elect_ctx_t/raft state at the decision point. Low value, low effort.

Residual Tally

• Closed since 2026-06-09: G1, G7, G10 (outright), G2, G3 (superseded by a better model in successor specs).
• Still open: R1 (probe unwired + stale comment), R2 (spec-text reconciliation), R5 (consolidated election log). R3 likely-closed-pending-verify, R4 re-homed to ha-commit-durability recovery.
• Net AC posture: the durability and redirect deltas mean the functional CP promise of this spec is now met or exceeded; the open items are documentation honesty (R2), one unwired-but-replaced bootstrap path (R1), and one cosmetic log (R5).

Implementation Approach Options (for the residual work)

Option A — Close-and-reconcile in place (recommended)

Phase-1-only, all S-size, no successor spec needed:

1. R2 rewrite requirements.md AC 2.8 / 2.10–2.12 / 3.1–3.5 / 5.5 / 6.1 to describe the as-built push/watchdog/outbound-primary/guaranteed-durability architecture; refresh the spec.json note (G1/G3 are no longer "known deviations"). Docs only.
2. R1 delete the stale ha.c:4493 comment; either (a) delete the unused xce_probe_config_tag production path and document the static-gate + watchdog-convergence model, or (b) add a cold-start convergence integration test (3-node, all configs absent) so the watchdog fallback is covered. Recommend (a)+test.
3. R5 add a single INFO "election complete: term=T votes=G/Q role=…" record at the become-leader / become-follower decision point.
4. R3 read the two new single-node durability tests; if they cover runtime advance, retire RN1 and mark R5.8 closed.

• ✅ Closes the spec honestly with no policy decisions left implicit; smallest surface.
• ✅ Every item has an existing seam + testsuite file.
• ❌ Touches requirements.md (aging artifact) — but that is the point.
• Effort: S total. Risk: Low (docs + dead-code delete + one log line + one test).

Option B — Defer reconciliation, close only code defects

Fix R1 comment + R5 log now; leave requirements.md as-is with an expanded spec.json deviation note pointing at controller CLAUDE.md as the source of truth.

• ✅ Even smaller; no requirements.md churn.
• ❌ Leaves EARS text contradicting shipped behavior — the recurring complaint across every prior pass. Reconciliation debt compounds.
• Effort: XS. Risk: Low (but accrues doc debt).

Option C — Fold residuals into successor specs and close this one verbatim

Mark ha-config-bootstrap closed; track R1/R2/R5 under ha-commit-durability (R3/R4 already there) or a small ha-spec-reconcile doc spec.

• ✅ This spec stops being touched; clean lineage.
• ❌ R2 spans this spec's own EARS text — odd to reconcile it from another spec. R5 is trivially this spec's.
• Effort: S (mostly bookkeeping). Risk: Low; tracking-loss risk medium.

Effort & Risk — Residual

Item	Effort	Risk	Notes
R1 probe path + stale comment + cold-start test	S	Low	Behavior already converges via watchdog; delete dead path or add test
R2 requirements.md / spec.json reconciliation	S	Low	Docs only; 6 EARS deltas; CLAUDE.md is the correct source
R3 single-node commit verify	XS	Low	Read 2 existing tests; likely already closed
R4 restart-path truncation	—	—	Re-homed to ha-commit-durability recovery; verify there, not here
R5 consolidated election log	S	Low	One INFO line; data already in elect_ctx

Recommendation: Option A. The heavy lifting (durability, redirect, retry adoption, dead-code) is done by successor specs; the only thing standing between this spec and an honest implemented/closed state is requirements-text reconciliation (R2), one unwired-but-superseded path (R1, delete + document), and a cosmetic log (R5). All S/XS, all Low risk.