myxce/.kiro/specs/raft-library/research.md

Research & Design Decisions: raft-library

Discovery Scope

• Type: Complex new library (brownfield extraction from ha.c)
• Process: Full discovery — codebase analysis of ha.c, ha.h, test files

Key Findings

Existing Election State (ha.c)

• ha_state_t contains: term (atomic uint64), voted_for_term, voted_for_name[XCE_MAXNAME_LEN], role (atomic int32)
• ha_elect_ctx_t holds per-election: n_peers, votes_granted, n_replied, max_resp_tag, max_resp_lsn, done_posted, mutex, semaphore
• Transport coupling: ha_start_election_async directly calls endpoint_connect(), iomgr_monitor(), SSL_new()
• Post-election coupling: ha_elect_evaluate_and_signal calls cluster_init_phase2(), push_config_to_standbys()

Root Cause of Split-Brain

• Self-vote was not recorded atomically at election start; voted_for remained empty until a peer vote arrived
• This allowed node B to grant its vote to node A AND node C in the same term if A and C both started elections before B processed either request
• Fix: record voted_for = self_idx immediately in raft_start_election, before sending any RPC

Build vs. Adopt Decision

• No pure-C Raft library suitable for libxce: hashicorp/raft (Go), etcd/raft (Go), no active C alternative
• Decision: build a minimal Raft election library in src/lib/raft/
• Scope is election only (not log replication), which is simpler than full Raft

Synthesis Outcomes

1. Generalization: Requirements 2–6 are all specializations of a single state machine with callbacks. One raft_state_t + raft_callbacks_t pattern covers all cases.
2. Build vs. Adopt: Build; no suitable C library exists.
3. Simplification: No threads inside the library. Synchronous callbacks. One header, one source, one test file. raft_sim.c compiled only under XCE_TESTING.

Priority Jitter Design

• The delay formula (priority_idx * election_timeout_ms) is a protocol-level decision, not transport
• Placed inside the library as raft_on_leader_lost → on_start_election_after_delay callback
• Caller implements the actual timer/sleep

Tracing Decision

• Added on_trace callback to raft_callbacks_t for full protocol observability
• Simulator dump log enables post-mortem analysis of split-brain scenarios
• raft_snapshot() provides safe point-in-time state read for crash/failure logging

---

Cross-Spec Gap Audit (2026-06-08)

Batch /kiro-validate-gap across 9 specs on persistent_ha branch.

Analysis Summary

• raft-library is ~100% complete. All 7 task groups (1–7), all subtasks (1–2.2, 3.1–3.5, 4.1–4.2, 5.1–5.4, 6.1–6.4, 7) are [x] in tasks.md. The ready_for_implementation flag is set in spec.json.
• No remaining gaps: The library (src/lib/raft/raft.h, raft.c, raft_sim.c) exposes raft_state_t + raft_callbacks_t with 9 callback hooks. The deterministic simulator (raft_sim.c) verifies split-brain freedom across 100 randomized rounds. The controller adapter in ha.c delegates all election state management (term, voted_for, quorum evaluation) to the raft library.
• Cross-spec integration: election-iomgr-refactor uses raft_start_election instead of the old am_i_primary_quorum. The adapter callbacks in ha.c wire on_send_vote_request → iomgr async connect, on_become_leader → cluster_init_phase2 + config push, on_become_follower → CTRL_STANDBY. The on_vote_eligible callback implements the freshness gate (config_tag + applied_lsn comparison) and role gate (CTRL_PRIMARY_COMMITTING → deny).
• Test coverage: 34+ Unity test cases across 5 groups (A–H) covering basic election correctness, voted_for invariant, eligibility policy, quorum evaluation, election timeout, RDHUP failover, and split-brain prevention. All pass under ASAN.
• Recommendation: The spec can be marked implemented.

---

Gap Analysis — Validation Pass (2026-06-09)

Scope: requirements.md (9 requirement groups, 31 acceptance criteria) vs. on-disk working tree (uncommitted changes to src/lib/raft/raft.c, src/controller/ha.c, ha.h, myapp.c analyzed as-is). spec.json phase: tasks-generated, all tasks.md checkboxes [x] except parent rollups 2/3/4/5/6.

1. Current State Summary

• Library exists and is build-integrated: src/lib/raft/raft.h (276 lines), raft.c (290 lines), raft_sim.c (207 lines); wired in src/lib/meson.build (raft.c at line 108; test_raft.c + raft_sim.c under the test executable at lines 175–176, simulator guarded by XCE_TESTING).
• Test suite: src/lib/testsuite/test_raft.c (2990 lines, 47 test functions, 79 RUN_TEST invocations) covering unit tests, simulator tests, and protocol groups A–H plus F/G split-brain scenarios.
• Controller adapter live: ha_state_t embeds raft_state_t raft (ha.h:183); 9 callbacks implemented in ha.c (s_real_cbs at ha.c:735); raft_init called in ha init (ha.c:854) and re-initialized with real topology by ha_raft_reinit (ha.c:~2981–3008).
• Production dispatch fully delegated: vote requests → raft_on_vote_request (cluster.c:373), vote replies → raft_on_vote_reply (ha.c:2361), RDHUP → raft_on_leader_lost (ha.c:3290), election start → raft_start_election (ha.c:3146).

2. Requirement-to-Asset Map

Req	Criterion	Asset(s)	Status
1.1	Per-node state (term/role/voted_for/n_peers)	raft_state_t (raft.h:23–38)	Implemented
1.2	Init defaults term=0/FOLLOWER/-1	raft_init (raft.c:13–55); H2 test	Implemented (extended: restores initial_term/initial_voted_for from config)
1.3	No endpoint_t/SSL/iomgr/cluster_t/capnp in header	raft.h includes only errno/stdint/string (+stdio under XCE_TESTING)	Implemented
2.1	start_election: term++, self-vote, CANDIDATE, send RPCs	raft.c:57–104	Implemented (also persists self-vote before RPCs — stronger than spec)
2.2	Single-node immediate LEADER	raft.c:83–88; tests A1, single_node	Implemented
2.3	-EALREADY if CANDIDATE/LEADER	raft.c:60–61; tests already_candidate/leader	Implemented
3.1	term > current: upgrade, clear vote, FOLLOWER, evaluate	raft.c:121–130; B2	Implemented (extra: fires on_become_follower if was LEADER — split-brain fix, uncommitted change)
3.2	term < current: deny, return current term	raft.c:110–115; B3	Implemented
3.3	One vote per term	raft.c:157–164; B1/B4/B5	Implemented (extension: idempotent re-grant to the SAME candidate same term — beyond spec text; see Gap-3)
3.4	Freshness gate (config_tag/LSN) before grant	Split: caller-side gate in cluster.c:352–356 + on_vote_eligible hook (raft.c:167–173)	Partial — behavior present but located in the dispatcher, not in on_vote_eligible as tasks.md 6.2 states; raft_cb_vote_eligible (ha.c:202) intentionally skips freshness. Doc/code drift.
3.5	LEADER or COMMITTING deny ALL vote requests regardless of term	raft.c:135–148 (same-term deny only); COMMITTING gate in on_vote_eligible adapter	Partial (intentional deviation) — higher-term request now steps the leader down instead of denying, to prevent the COMMITTING-timeout split-brain. Requirement text is outdated.
4.1	Quorum (n/2)+1 → LEADER, on_become_leader once	raft.c:222–234; D1, A4	Implemented
4.2	Reply term > current → FOLLOWER + on_become_follower	raft.c:197–206; D4	Implemented
4.3	Reply after decided → no-op	raft.c:209–213; D3	Implemented
4.4	All replied, no quorum → FOLLOWER	raft.c:237–244; D2	Implemented
5.1	leader_lost: clear vote, FOLLOWER, delayed election (priority×timeout)	raft.c:268–276; F1	Implemented
5.2	Skip election if voted for non-self during delay	raft_check_and_start_election raft.c:280–284; F2	Implemented
5.3	Else start election internally	raft.c:286–289; F3	Implemented
6.1	start_election schedules timeout	raft.c:101; E3	Implemented
6.2	Timeout undecided → on_become_follower, cancel tracking	raft.c:260–265; E1	Implemented
6.3	Timeout after decided → on_cancel_timeout no-op	raft.c:252–257; E2	Implemented
7.1	raft_callbacks_t with 6 required fields	raft.h:66–96 (9 fields: 7 required + 2 optional)	Implemented (superset: adds on_start_election_after_delay as required, on_vote_eligible/on_trace optional)
7.2	void* ctx passed to every callback	raft_config_t.ctx → s->ctx, all call sites	Implemented
7.3	NULL required callback → error at init	raft.c:23–38 (-EINVAL); Hnull test	Implemented
8.1	Deterministic N-node simulator, synchronous routing	raft_sim.c + raft.h:197–274; sim tests	Implemented
8.2	Invariants: one-leader/3-node, dup deny, single-node, timeout re-election	A2/G1 (100 rounds), B1/B4, A1, E3 + G2 partition	Implemented
8.3	Compile + all tests pass under ASAN	tasks.md task 7 [x]; commits e10cf9a/361397a fixed ASAN issues	Unknown (stale) — last verified pre-dating current uncommitted edits to raft.c/ha.c; needs re-run
9.1	Adapter implements all callbacks via endpoint_t/iomgr	ha.c:107–744 (8 raft_cb_* + s_real_cbs)	Implemented
9.2	ha_start_election_async delegates; no direct ha->term/voted_for writes outside adapter	ha.c:3038–3161 delegates; only atomic loads + persist/restore remain	Partial — production path clean, but legacy handle_request_vote() (ha.c:1791–1873) still contains a full parallel vote-evaluation path with direct ha->voted_for_term/voted_for_name writes. It is production-dead (only test_ha_vote_gate.c calls it) but duplicates election state management contrary to the spirit of 9.2.
9.3	ha_phase2_hook / ha_rdhup_election_hook still functional	ha.c:495/631/2068/2147 (phase2), ha.c:3267–3284 (rdhup)	Implemented

Score: 27 of 31 acceptance criteria Implemented; 3 Partial (3.4, 3.5, 9.2 — all deliberate-deviation or residual-cleanup, not functional regressions); 1 Unknown (8.3 ASAN re-verification against the uncommitted working tree).

Constraints / behavior added beyond requirements

• C1 (Constraint): raft_init rejects even n_peers > 1 with -EINVAL (raft.c:21–22). Not in requirements. A 2-node or 4-node HA deployment cannot initialize raft at all — verify this matches deployment expectations (deploy-ha.sh, cluster config validation) or it becomes a runtime landmine.
• C2 (Constraint): voted_for_term is unconditionally reset to 0 in raft_init (raft.c:43) even when initial_voted_for/initial_term are restored from disk. After a restart, the duplicate-vote check (voted_for_term == term, raft.c:157) cannot match the restored term, so a node that voted for X in term T, restarted, then receives a request from Y in the same term T would fall through Step 4 and could grant a second vote in T (subject to the caller-side freshness gate). Test H1 checks term/voted_for but not voted_for_term. Potential one-vote-per-term violation across restart — needs confirmation.

3. Implementation Approach Options (for the remaining gaps)

Option A: Extend existing components (close gaps in place) — RECOMMENDED

• raft.c: initialize voted_for_term = (initial_voted_for >= 0) ? initial_term : 0 in raft_init (C2); add an H6 restart-double-vote test.
• requirements.md: amend 3.5 (higher-term step-down is the intended behavior) and 3.4 (freshness gate enforced by the dispatcher/callback split) so spec matches the as-built design; sync tasks.md 6.2 text.
• ha.c: delete or fold legacy handle_request_vote() into the raft path; retarget test_ha_vote_gate.c at raft_on_vote_request + on_vote_eligible (closes 9.2).
• Re-run /xce-build lib + /xce-build controller under ASAN to refresh 8.3.
• Trade-offs: ✅ smallest diff, no architectural change, leverages existing test harness; ❌ touches a legacy test file; requirements edit needs human re-approval per workflow.

Option B: Create new components (formal vote-record module)

• Introduce a raft_persisted_vote_t {term, voted_for} pair persisted/restored atomically, replacing the separate initial_term/initial_voted_for fields; new restart-focused test group.
• Trade-offs: ✅ makes the term-binding of a vote impossible to lose by construction; ❌ API churn in raft_config_t and ha_raft_reinit for a problem fixable with one init line; over-engineering (YAGNI).

Option C: Hybrid — fix C2 in place now (Option A), defer handle_request_vote removal to a follow-up

cleanup spec/task since it is production-dead.

• Trade-offs: ✅ de-risks the correctness fix from the test-refactor churn; ❌ leaves known duplication in tree longer; two passes of review.

4. Effort & Risk

Gap	Effort	Risk	Justification
C2 voted_for_term restore fix + test	S	Low	One-line init change + one Unity test in an established harness
Requirements/tasks doc sync (3.4, 3.5)	S	Low	Documentation only; needs approval round-trip
Legacy handle_request_vote removal (9.2)	S–M	Medium	Production-dead but test_ha_vote_gate.c must be retargeted; risk of disturbing controller test baseline
ASAN re-verification (8.3)	S	Low	Mechanical /xce-build run; fixes only if new findings
C1 even-n_peers policy confirmation	S	Medium	If 2-node deployments are expected, this is a hard init failure discovered only at runtime

Overall remaining work: S–M total, Low–Medium risk — the library and integration are functionally complete.

5. Recommendations for Design/Closure Phase

1. Adopt Option A. Fix C2 (voted_for_term restore) first — it is the only candidate correctness gap; everything else is documentation or hygiene.
2. Update requirements.md 3.5 and 3.4 to match the as-built (and better) behavior: leader steps down on higher term; freshness gate enforced caller-side before raft_on_vote_request with COMMITTING/liveness in on_vote_eligible. Re-approve, then run /kiro-verify-completion raft-library.
3. Remove or refactor legacy handle_request_vote (ha.c:1791) to eliminate the last duplicated election state path (Req 9.2), retargeting test_ha_vote_gate.c.
4. Re-run task 7 ASAN validation against the current working tree before committing (the verified state predates the uncommitted raft.c Step-3/Step-4 rewrite).

Research Needed

• R1: Confirm intended cluster-size policy — are 2-node (even) HA clusters a supported deployment? If yes, the even-n_peers -EINVAL in raft_init must be relaxed; if no, add config-load validation so the failure surfaces at startup with a clear message rather than as a silent raft_init failure.
• R2: Verify whether the restart double-vote scenario (C2) is reachable in production timing — does ha_raft_reinit run before the listener accepts vote requests? If unreachable, document why; fix anyway.
• R3: Confirm raft_check_and_start_election ignoring the raft_start_election return code (raft.c:287–289) is safe in all adapter paths (comment claims -EALREADY unreachable; verify when an inbound vote request flips role between RDHUP and the delayed check).