myxce/.kiro/specs/raft-library/research.md

Research & Design Decisions: raft-library

Discovery Scope

• Type: Complex new library (brownfield extraction from ha.c)
• Process: Full discovery — codebase analysis of ha.c, ha.h, test files

Key Findings

Existing Election State (ha.c)

• ha_state_t contains: term (atomic uint64), voted_for_term, voted_for_name[XCE_MAXNAME_LEN], role (atomic int32)
• ha_elect_ctx_t holds per-election: n_peers, votes_granted, n_replied, max_resp_tag, max_resp_lsn, done_posted, mutex, semaphore
• Transport coupling: ha_start_election_async directly calls endpoint_connect(), iomgr_monitor(), SSL_new()
• Post-election coupling: ha_elect_evaluate_and_signal calls cluster_init_phase2(), push_config_to_standbys()

Root Cause of Split-Brain

• Self-vote was not recorded atomically at election start; voted_for remained empty until a peer vote arrived
• This allowed node B to grant its vote to node A AND node C in the same term if A and C both started elections before B processed either request
• Fix: record voted_for = self_idx immediately in raft_start_election, before sending any RPC

Build vs. Adopt Decision

• No pure-C Raft library suitable for libxce: hashicorp/raft (Go), etcd/raft (Go), no active C alternative
• Decision: build a minimal Raft election library in src/lib/raft/
• Scope is election only (not log replication), which is simpler than full Raft

Synthesis Outcomes

1. Generalization: Requirements 2–6 are all specializations of a single state machine with callbacks. One raft_state_t + raft_callbacks_t pattern covers all cases.
2. Build vs. Adopt: Build; no suitable C library exists.
3. Simplification: No threads inside the library. Synchronous callbacks. One header, one source, one test file. raft_sim.c compiled only under XCE_TESTING.

Priority Jitter Design

• The delay formula (priority_idx * election_timeout_ms) is a protocol-level decision, not transport
• Placed inside the library as raft_on_leader_lost → on_start_election_after_delay callback
• Caller implements the actual timer/sleep

Tracing Decision

• Added on_trace callback to raft_callbacks_t for full protocol observability
• Simulator dump log enables post-mortem analysis of split-brain scenarios
• raft_snapshot() provides safe point-in-time state read for crash/failure logging

---

Cross-Spec Gap Audit (2026-06-08)

Batch /kiro-validate-gap across 9 specs on persistent_ha branch.

Analysis Summary

• raft-library is ~100% complete. All 7 task groups (1–7), all subtasks (1–2.2, 3.1–3.5, 4.1–4.2, 5.1–5.4, 6.1–6.4, 7) are [x] in tasks.md. The ready_for_implementation flag is set in spec.json.
• No remaining gaps: The library (src/lib/raft/raft.h, raft.c, raft_sim.c) exposes raft_state_t + raft_callbacks_t with 9 callback hooks. The deterministic simulator (raft_sim.c) verifies split-brain freedom across 100 randomized rounds. The controller adapter in ha.c delegates all election state management (term, voted_for, quorum evaluation) to the raft library.
• Cross-spec integration: election-iomgr-refactor uses raft_start_election instead of the old am_i_primary_quorum. The adapter callbacks in ha.c wire on_send_vote_request → iomgr async connect, on_become_leader → cluster_init_phase2 + config push, on_become_follower → CTRL_STANDBY. The on_vote_eligible callback implements the freshness gate (config_tag + applied_lsn comparison) and role gate (CTRL_PRIMARY_COMMITTING → deny).
• Test coverage: 34+ Unity test cases across 5 groups (A–H) covering basic election correctness, voted_for invariant, eligibility policy, quorum evaluation, election timeout, RDHUP failover, and split-brain prevention. All pass under ASAN.
• Recommendation: The spec can be marked implemented.

---

Gap Analysis — Validation Pass (2026-06-09)

Scope: requirements.md (9 requirement groups, 31 acceptance criteria) vs. on-disk working tree (uncommitted changes to src/lib/raft/raft.c, src/controller/ha.c, ha.h, myapp.c analyzed as-is). spec.json phase: tasks-generated, all tasks.md checkboxes [x] except parent rollups 2/3/4/5/6.

1. Current State Summary

• Library exists and is build-integrated: src/lib/raft/raft.h (276 lines), raft.c (290 lines), raft_sim.c (207 lines); wired in src/lib/meson.build (raft.c at line 108; test_raft.c + raft_sim.c under the test executable at lines 175–176, simulator guarded by XCE_TESTING).
• Test suite: src/lib/testsuite/test_raft.c (2990 lines, 47 test functions, 79 RUN_TEST invocations) covering unit tests, simulator tests, and protocol groups A–H plus F/G split-brain scenarios.
• Controller adapter live: ha_state_t embeds raft_state_t raft (ha.h:183); 9 callbacks implemented in ha.c (s_real_cbs at ha.c:735); raft_init called in ha init (ha.c:854) and re-initialized with real topology by ha_raft_reinit (ha.c:~2981–3008).
• Production dispatch fully delegated: vote requests → raft_on_vote_request (cluster.c:373), vote replies → raft_on_vote_reply (ha.c:2361), RDHUP → raft_on_leader_lost (ha.c:3290), election start → raft_start_election (ha.c:3146).

2. Requirement-to-Asset Map

Req	Criterion	Asset(s)	Status
1.1	Per-node state (term/role/voted_for/n_peers)	raft_state_t (raft.h:23–38)	Implemented
1.2	Init defaults term=0/FOLLOWER/-1	raft_init (raft.c:13–55); H2 test	Implemented (extended: restores initial_term/initial_voted_for from config)
1.3	No endpoint_t/SSL/iomgr/cluster_t/capnp in header	raft.h includes only errno/stdint/string (+stdio under XCE_TESTING)	Implemented
2.1	start_election: term++, self-vote, CANDIDATE, send RPCs	raft.c:57–104	Implemented (also persists self-vote before RPCs — stronger than spec)
2.2	Single-node immediate LEADER	raft.c:83–88; tests A1, single_node	Implemented
2.3	-EALREADY if CANDIDATE/LEADER	raft.c:60–61; tests already_candidate/leader	Implemented
3.1	term > current: upgrade, clear vote, FOLLOWER, evaluate	raft.c:121–130; B2	Implemented (extra: fires on_become_follower if was LEADER — split-brain fix, uncommitted change)
3.2	term < current: deny, return current term	raft.c:110–115; B3	Implemented
3.3	One vote per term	raft.c:157–164; B1/B4/B5	Implemented (extension: idempotent re-grant to the SAME candidate same term — beyond spec text; see Gap-3)
3.4	Freshness gate (config_tag/LSN) before grant	Split: caller-side gate in cluster.c:352–356 + on_vote_eligible hook (raft.c:167–173)	Partial — behavior present but located in the dispatcher, not in on_vote_eligible as tasks.md 6.2 states; raft_cb_vote_eligible (ha.c:202) intentionally skips freshness. Doc/code drift.
3.5	LEADER or COMMITTING deny ALL vote requests regardless of term	raft.c:135–148 (same-term deny only); COMMITTING gate in on_vote_eligible adapter	Partial (intentional deviation) — higher-term request now steps the leader down instead of denying, to prevent the COMMITTING-timeout split-brain. Requirement text is outdated.
4.1	Quorum (n/2)+1 → LEADER, on_become_leader once	raft.c:222–234; D1, A4	Implemented
4.2	Reply term > current → FOLLOWER + on_become_follower	raft.c:197–206; D4	Implemented
4.3	Reply after decided → no-op	raft.c:209–213; D3	Implemented
4.4	All replied, no quorum → FOLLOWER	raft.c:237–244; D2	Implemented
5.1	leader_lost: clear vote, FOLLOWER, delayed election (priority×timeout)	raft.c:268–276; F1	Implemented
5.2	Skip election if voted for non-self during delay	raft_check_and_start_election raft.c:280–284; F2	Implemented
5.3	Else start election internally	raft.c:286–289; F3	Implemented
6.1	start_election schedules timeout	raft.c:101; E3	Implemented
6.2	Timeout undecided → on_become_follower, cancel tracking	raft.c:260–265; E1	Implemented
6.3	Timeout after decided → on_cancel_timeout no-op	raft.c:252–257; E2	Implemented
7.1	raft_callbacks_t with 6 required fields	raft.h:66–96 (9 fields: 7 required + 2 optional)	Implemented (superset: adds on_start_election_after_delay as required, on_vote_eligible/on_trace optional)
7.2	void* ctx passed to every callback	raft_config_t.ctx → s->ctx, all call sites	Implemented
7.3	NULL required callback → error at init	raft.c:23–38 (-EINVAL); Hnull test	Implemented
8.1	Deterministic N-node simulator, synchronous routing	raft_sim.c + raft.h:197–274; sim tests	Implemented
8.2	Invariants: one-leader/3-node, dup deny, single-node, timeout re-election	A2/G1 (100 rounds), B1/B4, A1, E3 + G2 partition	Implemented
8.3	Compile + all tests pass under ASAN	tasks.md task 7 [x]; commits e10cf9a/361397a fixed ASAN issues	Unknown (stale) — last verified pre-dating current uncommitted edits to raft.c/ha.c; needs re-run
9.1	Adapter implements all callbacks via endpoint_t/iomgr	ha.c:107–744 (8 raft_cb_* + s_real_cbs)	Implemented
9.2	ha_start_election_async delegates; no direct ha->term/voted_for writes outside adapter	ha.c:3038–3161 delegates; only atomic loads + persist/restore remain	Partial — production path clean, but legacy handle_request_vote() (ha.c:1791–1873) still contains a full parallel vote-evaluation path with direct ha->voted_for_term/voted_for_name writes. It is production-dead (only test_ha_vote_gate.c calls it) but duplicates election state management contrary to the spirit of 9.2.
9.3	ha_phase2_hook / ha_rdhup_election_hook still functional	ha.c:495/631/2068/2147 (phase2), ha.c:3267–3284 (rdhup)	Implemented

Score: 27 of 31 acceptance criteria Implemented; 3 Partial (3.4, 3.5, 9.2 — all deliberate-deviation or residual-cleanup, not functional regressions); 1 Unknown (8.3 ASAN re-verification against the uncommitted working tree).

Constraints / behavior added beyond requirements

• C1 (Constraint): raft_init rejects even n_peers > 1 with -EINVAL (raft.c:21–22). Not in requirements. A 2-node or 4-node HA deployment cannot initialize raft at all — verify this matches deployment expectations (deploy-ha.sh, cluster config validation) or it becomes a runtime landmine.
• C2 (Constraint): voted_for_term is unconditionally reset to 0 in raft_init (raft.c:43) even when initial_voted_for/initial_term are restored from disk. After a restart, the duplicate-vote check (voted_for_term == term, raft.c:157) cannot match the restored term, so a node that voted for X in term T, restarted, then receives a request from Y in the same term T would fall through Step 4 and could grant a second vote in T (subject to the caller-side freshness gate). Test H1 checks term/voted_for but not voted_for_term. Potential one-vote-per-term violation across restart — needs confirmation.

3. Implementation Approach Options (for the remaining gaps)

Option A: Extend existing components (close gaps in place) — RECOMMENDED

• raft.c: initialize voted_for_term = (initial_voted_for >= 0) ? initial_term : 0 in raft_init (C2); add an H6 restart-double-vote test.
• requirements.md: amend 3.5 (higher-term step-down is the intended behavior) and 3.4 (freshness gate enforced by the dispatcher/callback split) so spec matches the as-built design; sync tasks.md 6.2 text.
• ha.c: delete or fold legacy handle_request_vote() into the raft path; retarget test_ha_vote_gate.c at raft_on_vote_request + on_vote_eligible (closes 9.2).
• Re-run /xce-build lib + /xce-build controller under ASAN to refresh 8.3.
• Trade-offs: ✅ smallest diff, no architectural change, leverages existing test harness; ❌ touches a legacy test file; requirements edit needs human re-approval per workflow.

Option B: Create new components (formal vote-record module)

• Introduce a raft_persisted_vote_t {term, voted_for} pair persisted/restored atomically, replacing the separate initial_term/initial_voted_for fields; new restart-focused test group.
• Trade-offs: ✅ makes the term-binding of a vote impossible to lose by construction; ❌ API churn in raft_config_t and ha_raft_reinit for a problem fixable with one init line; over-engineering (YAGNI).

Option C: Hybrid — fix C2 in place now (Option A), defer handle_request_vote removal to a follow-up

cleanup spec/task since it is production-dead.

• Trade-offs: ✅ de-risks the correctness fix from the test-refactor churn; ❌ leaves known duplication in tree longer; two passes of review.

4. Effort & Risk

Gap	Effort	Risk	Justification
C2 voted_for_term restore fix + test	S	Low	One-line init change + one Unity test in an established harness
Requirements/tasks doc sync (3.4, 3.5)	S	Low	Documentation only; needs approval round-trip
Legacy handle_request_vote removal (9.2)	S–M	Medium	Production-dead but test_ha_vote_gate.c must be retargeted; risk of disturbing controller test baseline
ASAN re-verification (8.3)	S	Low	Mechanical /xce-build run; fixes only if new findings
C1 even-n_peers policy confirmation	S	Medium	If 2-node deployments are expected, this is a hard init failure discovered only at runtime

Overall remaining work: S–M total, Low–Medium risk — the library and integration are functionally complete.

5. Recommendations for Design/Closure Phase

1. Adopt Option A. Fix C2 (voted_for_term restore) first — it is the only candidate correctness gap; everything else is documentation or hygiene.
2. Update requirements.md 3.5 and 3.4 to match the as-built (and better) behavior: leader steps down on higher term; freshness gate enforced caller-side before raft_on_vote_request with COMMITTING/liveness in on_vote_eligible. Re-approve, then run /kiro-verify-completion raft-library.
3. Remove or refactor legacy handle_request_vote (ha.c:1791) to eliminate the last duplicated election state path (Req 9.2), retargeting test_ha_vote_gate.c.
4. Re-run task 7 ASAN validation against the current working tree before committing (the verified state predates the uncommitted raft.c Step-3/Step-4 rewrite).

Research Needed

• R1: Confirm intended cluster-size policy — are 2-node (even) HA clusters a supported deployment? If yes, the even-n_peers -EINVAL in raft_init must be relaxed; if no, add config-load validation so the failure surfaces at startup with a clear message rather than as a silent raft_init failure.
• R2: Verify whether the restart double-vote scenario (C2) is reachable in production timing — does ha_raft_reinit run before the listener accepts vote requests? If unreachable, document why; fix anyway.
• R3: Confirm raft_check_and_start_election ignoring the raft_start_election return code (raft.c:287–289) is safe in all adapter paths (comment claims -EALREADY unreachable; verify when an inbound vote request flips role between RDHUP and the delayed check).

---

Gap Re-Audit (2026-06-18)

Post-implementation audit. Spec phase = implemented, tasks 24/24 [x]. This pass re-verifies every EARS requirement against the working tree and reconciles the prior audit's open items (C2, R1–R3). Assets are tagged: [CLOSED] verified met in code, [CLOSED-AS-BUILT] met but behavior differs from literal req text (improvement), [RESIDUAL] doc/hygiene gap, [SCOPE-DRIFT] code beyond spec.

Requirement-to-Asset Map

Req	Acceptance	Asset (file:line)	Status
1.1	per-node state fields	raft.h:28-50 raft_state_t (term/role/voted_for/voted_for_term/n_peers)	CLOSED
1.2	init term=0 role=FOLLOWER voted_for=-1	raft.c:40-56 raft_init; defaults via cfg {0,-1}	CLOSED
1.3	no transport types in header	raft.h:4-6 includes only errno/stdint/string; no endpoint_t/SSL_CTX/cluster_t	CLOSED
2.1	election: incr term + self-vote + send RPCs	raft.c:68-71,98-102	CLOSED
2.2	single-node → immediate LEADER	raft.c:87-92 returns RAFT_ELECTED_IMMEDIATELY	CLOSED
2.3	already CANDIDATE/LEADER → error, no state change	raft.c:64-65 -EALREADY	CLOSED
3.1	term>cur → update,clear vote,FOLLOWER,evaluate	raft.c:125-137 (also steps down leader via on_become_follower)	CLOSED-AS-BUILT
3.2	term<cur → deny + return cur term	raft.c:114-119	CLOSED
3.3	same term, diff voted_for → deny	raft.c:161-168 (gated on voted_for_term==term)	CLOSED
3.4	freshness gate pass → grant + persist	raft.c:170-190; freshness split: tag/lsn floor enforced caller-side (cluster.c:363), COMMITTING/liveness in on_vote_eligible (ha.c:299 raft_cb_vote_eligible)	CLOSED-AS-BUILT
3.5	LEADER/COMMITTING → deny all	raft.c:148-152 LEADER deny; COMMITTING enum removed (ha-single-leader-role 93dfc02), now an on_vote_eligible policy	CLOSED-AS-BUILT
4.1	quorum ceil((n+1)/2) → LEADER once	raft.c:372-383, quorum=(n_peers/2)+1 == ceil for odd n	CLOSED
4.2	reply term>cur → FOLLOWER + on_become_follower	raft.c:346-355	CLOSED
4.3	post-decision no-op	raft.c:358-362 election_decided guard → RAFT_PENDING	CLOSED
4.4	all replied, no quorum → FOLLOWER	raft.c:386-393 replies_received==n_peers-1	CLOSED
5.1	leader-lost → clear vote + delay(priority*timeout)	raft.c:417-425	CLOSED
5.2	voted non-self during delay → skip	raft.c:429-433 RAFT_SKIP_ELECTION	CLOSED
5.3	not voted / self → start election	raft.c:435-438 RAFT_ELECTION_STARTED	CLOSED
6.1	start election → on_schedule_timeout	raft.c:105	CLOSED
6.2	timeout undecided → on_become_follower + cancel	raft.c:409-414	CLOSED
6.3	timeout decided → on_cancel_timeout, no-op	raft.c:401-407	CLOSED
7.1	raft_callbacks_t with required fn ptrs	raft.h:79-117 (9 fields incl. on_send_pre_vote_request)	CLOSED
7.2	void* ctx passed unchanged	raft.h ctx threaded; raft.c passes s->ctx everywhere	CLOSED
7.3	NULL required callback → non-zero	raft.c:23-38 7 required-cb checks → -EINVAL	CLOSED
8.1	deterministic N-node simulator	raft_sim.c, raft.h:264-342 (XCE_TESTING)	CLOSED
8.2	4 invariant cases (a-d)	test_raft.c: A2 one-leader, B1/B4 deny-second, A1 single-node, E*/sim timeout	CLOSED
8.3	compiles + tests pass under ASAN (leaks=0)	meson raft/raft.c (l.109) + test_raft.c (l.179); 209 RUN_TEST/test fns	CLOSED (re-run, see V1)
9.1	adapter implements all callbacks, no dup state	ha.c:1309-1320 s_real_cbs wires all 10; production state in raft_state_t	CLOSED
9.2	ha delegates term/voted_for/quorum to library	ha.c:4382-4522 ha_start_election_async→raft_start_election; cluster.c:418 vote req; ha.c:3237 vote reply	CLOSED-AS-BUILT
9.3	XCE_TESTING hooks still functional	ha_rdhup_election_hook (ha.c:4528), ha_phase2 seams retained	CLOSED

Score: 24/24 acceptance-criteria groups CLOSED. No requirement is unimplemented. 5 are CLOSED-AS-BUILT (real code is a strict superset/improvement over literal req text — see residual R-DOC).

Reconciliation of Prior Audit Open Items

• C2 (voted_for_term restore double-vote) — NOW FIXED. raft.c:47 sets voted_for_term = (initial_voted_for != -1) ? initial_term : 0 at init, closing the post-restart second-vote window. Regression test test_proto_H6_restored_vote_blocks_second_vote (test_raft.c:2100, registered l.3978) guards it. Prior research's "only candidate correctness gap" is resolved (commit 9821ec4).
• R3 / Req 9.2 dual vote path — DOWNGRADED. Legacy handle_request_vote (ha.c:2608) is now #ifdef XCE_TESTING-only — the production vote path is exclusively cluster.c:418 → raft_on_vote_request. The duplicated election-state logic no longer runs in production; it remains only as a test fixture for test_ha_vote_gate.c / test_ha_election.c. No production split-brain risk from the dual path. Residual: the test-only copy can drift from raft.c semantics over time (R-DUP below).
• R1 (even-n_peers policy) — STILL OPEN. raft_init rejects even n_peers>1 with -EINVAL (raft.c:21-22), test test_raft_init_even_peers_returns_einval. Not stated in requirements; a 2-node HA deploy would silently fail raft_init. Needs requirements note + startup-time config validation.

Residual Gaps (non-blocking)

• R-DOC (Low): requirements.md 3.4/3.5 still describe COMMITTING as an in-library role gate and the freshness gate as in-library; as-built, COMMITTING enum is removed and freshness is split caller-side + on_vote_eligible. Requirements text lags the (better) implementation. Update for traceability.
• R-DUP (Low): test-only handle_request_vote duplicates vote-gate logic; risk of semantic drift from raft.c. Consider retargeting test_ha_vote_gate.c onto raft_on_vote_request directly.
• R1 (Medium): even-n_peers rejection undocumented; surfaces as silent raft_init -EINVAL rather than a clear startup error. Add config-load validation OR a requirement clause.
• R-SCOPE / SCOPE-DRIFT (informational, not a gap): PreVote (raft_begin_pre_vote, raft_on_pre_vote_request, raft_on_pre_vote_reply, on_send_pre_vote_request, RAFT_PRE_VOTE_* codes; raft.c:193-336, raft.h:160-259) lives in the raft-library files but belongs to the separate ha-prevote spec (commits 449b8e9–e6b234a). It is purely additive, mutates no persistent Raft state, and does not violate any raft-library requirement. Flag only so the raft-library boundary doc reflects the co-tenancy: raft.h/raft.c now host two specs' surfaces.

Integration Risks

• IR-1 (Low): PreVote tally arrays are bounded by RAFT_MAX_PEERS=8 (raft.h:18); raft_on_pre_vote_reply range-guards (raft.c:287). Real-vote path has no equivalent fixed array (uses scalar counters) so it is unbounded-safe. Mismatch is intentional but worth a one-line note in design.
• IR-2 (Low): raft_check_and_start_election discards raft_start_election return (raft.c:436-437); comment claims -EALREADY unreachable because raft_on_leader_lost forces FOLLOWER. Valid for the current adapter, but an inbound vote/AppendEntries that flips role between RDHUP and the delayed fire could make it reachable. Low likelihood; assert instead of (void) for defense-in-depth.

Open Work — Option Analysis

Option A — Doc-only closure (recommended). Update requirements.md 3.4/3.5/9.2 to as-built; add R1 even-n_peers clause; note PreVote co-tenancy + IR-1/IR-2 in design.md. Then /kiro-verify-completion.

• Effort: S (~1-2h, docs only). Risk: Low. No code change, no regression surface.

Option B — Doc closure + hygiene. Option A plus: (1) startup config validation for even n_peers with a loud error, (2) retarget test-only handle_request_vote onto raft_on_vote_request (kill R-DUP), (3) convert IR-2 (void)rc to an assert.

• Effort: M (~half day). Risk: Low-Medium (touches ha.c test seam + config load; rerun full controller + lib test suites).

Option C — Defer, accept as-is. Ship implemented; track R-DOC/R-DUP/R1 as backlog. The only real correctness item (C2) is already fixed and tested, so functional risk of deferring is minimal.

• Effort: none. Risk: Low functionally; Medium for spec/code traceability drift over time.

Validation To Re-Run

• V1: /xce-build lib and /xce-build controller with ASAN_OPTIONS=detect_leaks=0 — no build dir present in the worktree at audit time; the test_raft.c suite (209 cases) and controller tests should be re-run to confirm green on the current tree (which now includes the PreVote additions + C2 fix, post-dating the last recorded task-7 ASAN run).

Bottom line: raft-library is functionally complete — 24/24 met, the sole prior correctness gap (C2) is fixed and regression-tested, and the production dual-vote-path risk is eliminated (legacy path is test-only). Remaining work is documentation/hygiene (Option A, S effort, Low risk) plus a clean ASAN re-run (V1).