# Research & Design Decisions: raft-library ## Discovery Scope - Type: Complex new library (brownfield extraction from ha.c) - Process: Full discovery — codebase analysis of ha.c, ha.h, test files ## Key Findings ### Existing Election State (ha.c) - ha_state_t contains: term (atomic uint64), voted_for_term, voted_for_name[XCE_MAXNAME_LEN], role (atomic int32) - ha_elect_ctx_t holds per-election: n_peers, votes_granted, n_replied, max_resp_tag, max_resp_lsn, done_posted, mutex, semaphore - Transport coupling: ha_start_election_async directly calls endpoint_connect(), iomgr_monitor(), SSL_new() - Post-election coupling: ha_elect_evaluate_and_signal calls cluster_init_phase2(), push_config_to_standbys() ### Root Cause of Split-Brain - Self-vote was not recorded atomically at election start; voted_for remained empty until a peer vote arrived - This allowed node B to grant its vote to node A AND node C in the same term if A and C both started elections before B processed either request - Fix: record voted_for = self_idx immediately in raft_start_election, before sending any RPC ### Build vs. Adopt Decision - No pure-C Raft library suitable for libxce: hashicorp/raft (Go), etcd/raft (Go), no active C alternative - Decision: build a minimal Raft election library in src/lib/raft/ - Scope is election only (not log replication), which is simpler than full Raft ### Synthesis Outcomes 1. **Generalization**: Requirements 2–6 are all specializations of a single state machine with callbacks. One raft_state_t + raft_callbacks_t pattern covers all cases. 2. **Build vs. Adopt**: Build; no suitable C library exists. 3. **Simplification**: No threads inside the library. Synchronous callbacks. One header, one source, one test file. raft_sim.c compiled only under XCE_TESTING. ### Priority Jitter Design - The delay formula (priority_idx * election_timeout_ms) is a protocol-level decision, not transport - Placed inside the library as raft_on_leader_lost → on_start_election_after_delay callback - Caller implements the actual timer/sleep ### Tracing Decision - Added on_trace callback to raft_callbacks_t for full protocol observability - Simulator dump log enables post-mortem analysis of split-brain scenarios - raft_snapshot() provides safe point-in-time state read for crash/failure logging --- # Cross-Spec Gap Audit (2026-06-08) > Batch `/kiro-validate-gap` across 9 specs on persistent_ha branch. ## Analysis Summary - **raft-library** is **~100% complete**. All 7 task groups (1–7), all subtasks (1–2.2, 3.1–3.5, 4.1–4.2, 5.1–5.4, 6.1–6.4, 7) are `[x]` in tasks.md. The `ready_for_implementation` flag is set in spec.json. - **No remaining gaps**: The library (`src/lib/raft/raft.h`, `raft.c`, `raft_sim.c`) exposes `raft_state_t` + `raft_callbacks_t` with 9 callback hooks. The deterministic simulator (`raft_sim.c`) verifies split-brain freedom across 100 randomized rounds. The controller adapter in ha.c delegates all election state management (term, voted_for, quorum evaluation) to the raft library. - **Cross-spec integration**: `election-iomgr-refactor` uses `raft_start_election` instead of the old `am_i_primary_quorum`. The adapter callbacks in ha.c wire `on_send_vote_request` → iomgr async connect, `on_become_leader` → `cluster_init_phase2` + config push, `on_become_follower` → `CTRL_STANDBY`. The `on_vote_eligible` callback implements the freshness gate (config_tag + applied_lsn comparison) and role gate (`CTRL_PRIMARY_COMMITTING` → deny). - **Test coverage**: 34+ Unity test cases across 5 groups (A–H) covering basic election correctness, voted_for invariant, eligibility policy, quorum evaluation, election timeout, RDHUP failover, and split-brain prevention. All pass under ASAN. - **Recommendation**: The spec can be marked `implemented`. --- # Gap Analysis — Validation Pass (2026-06-09) > Scope: requirements.md (9 requirement groups, 31 acceptance criteria) vs. on-disk working tree > (uncommitted changes to `src/lib/raft/raft.c`, `src/controller/ha.c`, `ha.h`, `myapp.c` analyzed as-is). > spec.json phase: `tasks-generated`, all tasks.md checkboxes `[x]` except parent rollups 2/3/4/5/6. ## 1. Current State Summary - Library exists and is build-integrated: `src/lib/raft/raft.h` (276 lines), `raft.c` (290 lines), `raft_sim.c` (207 lines); wired in `src/lib/meson.build` (raft.c at line 108; test_raft.c + raft_sim.c under the test executable at lines 175–176, simulator guarded by `XCE_TESTING`). - Test suite: `src/lib/testsuite/test_raft.c` (2990 lines, 47 test functions, 79 RUN_TEST invocations) covering unit tests, simulator tests, and protocol groups A–H plus F/G split-brain scenarios. - Controller adapter live: `ha_state_t` embeds `raft_state_t raft` (`ha.h:183`); 9 callbacks implemented in `ha.c` (`s_real_cbs` at ha.c:735); `raft_init` called in ha init (ha.c:854) and re-initialized with real topology by `ha_raft_reinit` (ha.c:~2981–3008). - Production dispatch fully delegated: vote requests → `raft_on_vote_request` (cluster.c:373), vote replies → `raft_on_vote_reply` (ha.c:2361), RDHUP → `raft_on_leader_lost` (ha.c:3290), election start → `raft_start_election` (ha.c:3146). ## 2. Requirement-to-Asset Map | Req | Criterion | Asset(s) | Status | |---|---|---|---| | 1.1 | Per-node state (term/role/voted_for/n_peers) | `raft_state_t` (raft.h:23–38) | Implemented | | 1.2 | Init defaults term=0/FOLLOWER/-1 | `raft_init` (raft.c:13–55); H2 test | Implemented (extended: restores `initial_term`/`initial_voted_for` from config) | | 1.3 | No endpoint_t/SSL/iomgr/cluster_t/capnp in header | raft.h includes only errno/stdint/string (+stdio under XCE_TESTING) | Implemented | | 2.1 | start_election: term++, self-vote, CANDIDATE, send RPCs | raft.c:57–104 | Implemented (also persists self-vote before RPCs — stronger than spec) | | 2.2 | Single-node immediate LEADER | raft.c:83–88; tests A1, single_node | Implemented | | 2.3 | -EALREADY if CANDIDATE/LEADER | raft.c:60–61; tests already_candidate/leader | Implemented | | 3.1 | term > current: upgrade, clear vote, FOLLOWER, evaluate | raft.c:121–130; B2 | Implemented (extra: fires `on_become_follower` if was LEADER — split-brain fix, uncommitted change) | | 3.2 | term < current: deny, return current term | raft.c:110–115; B3 | Implemented | | 3.3 | One vote per term | raft.c:157–164; B1/B4/B5 | Implemented (extension: idempotent re-grant to the SAME candidate same term — beyond spec text; see Gap-3) | | 3.4 | Freshness gate (config_tag/LSN) before grant | Split: caller-side gate in cluster.c:352–356 + `on_vote_eligible` hook (raft.c:167–173) | Partial — behavior present but located in the dispatcher, not in `on_vote_eligible` as tasks.md 6.2 states; `raft_cb_vote_eligible` (ha.c:202) intentionally skips freshness. Doc/code drift. | | 3.5 | LEADER or COMMITTING deny ALL vote requests regardless of term | raft.c:135–148 (same-term deny only); COMMITTING gate in `on_vote_eligible` adapter | Partial (intentional deviation) — higher-term request now steps the leader down instead of denying, to prevent the COMMITTING-timeout split-brain. Requirement text is outdated. | | 4.1 | Quorum (n/2)+1 → LEADER, on_become_leader once | raft.c:222–234; D1, A4 | Implemented | | 4.2 | Reply term > current → FOLLOWER + on_become_follower | raft.c:197–206; D4 | Implemented | | 4.3 | Reply after decided → no-op | raft.c:209–213; D3 | Implemented | | 4.4 | All replied, no quorum → FOLLOWER | raft.c:237–244; D2 | Implemented | | 5.1 | leader_lost: clear vote, FOLLOWER, delayed election (priority×timeout) | raft.c:268–276; F1 | Implemented | | 5.2 | Skip election if voted for non-self during delay | `raft_check_and_start_election` raft.c:280–284; F2 | Implemented | | 5.3 | Else start election internally | raft.c:286–289; F3 | Implemented | | 6.1 | start_election schedules timeout | raft.c:101; E3 | Implemented | | 6.2 | Timeout undecided → on_become_follower, cancel tracking | raft.c:260–265; E1 | Implemented | | 6.3 | Timeout after decided → on_cancel_timeout no-op | raft.c:252–257; E2 | Implemented | | 7.1 | raft_callbacks_t with 6 required fields | raft.h:66–96 (9 fields: 7 required + 2 optional) | Implemented (superset: adds `on_start_election_after_delay` as required, `on_vote_eligible`/`on_trace` optional) | | 7.2 | void* ctx passed to every callback | raft_config_t.ctx → s->ctx, all call sites | Implemented | | 7.3 | NULL required callback → error at init | raft.c:23–38 (-EINVAL); Hnull test | Implemented | | 8.1 | Deterministic N-node simulator, synchronous routing | raft_sim.c + raft.h:197–274; sim tests | Implemented | | 8.2 | Invariants: one-leader/3-node, dup deny, single-node, timeout re-election | A2/G1 (100 rounds), B1/B4, A1, E3 + G2 partition | Implemented | | 8.3 | Compile + all tests pass under ASAN | tasks.md task 7 `[x]`; commits e10cf9a/361397a fixed ASAN issues | Unknown (stale) — last verified pre-dating current uncommitted edits to raft.c/ha.c; needs re-run | | 9.1 | Adapter implements all callbacks via endpoint_t/iomgr | ha.c:107–744 (8 raft_cb_* + s_real_cbs) | Implemented | | 9.2 | ha_start_election_async delegates; no direct ha->term/voted_for writes outside adapter | ha.c:3038–3161 delegates; only atomic loads + persist/restore remain | Partial — production path clean, but legacy `handle_request_vote()` (ha.c:1791–1873) still contains a full parallel vote-evaluation path with direct `ha->voted_for_term`/`voted_for_name` writes. It is production-dead (only `test_ha_vote_gate.c` calls it) but duplicates election state management contrary to the spirit of 9.2. | | 9.3 | ha_phase2_hook / ha_rdhup_election_hook still functional | ha.c:495/631/2068/2147 (phase2), ha.c:3267–3284 (rdhup) | Implemented | **Score: 27 of 31 acceptance criteria Implemented; 3 Partial (3.4, 3.5, 9.2 — all deliberate-deviation or residual-cleanup, not functional regressions); 1 Unknown (8.3 ASAN re-verification against the uncommitted working tree).** ### Constraints / behavior added beyond requirements - **C1 (Constraint)**: `raft_init` rejects even `n_peers > 1` with -EINVAL (raft.c:21–22). Not in requirements. A 2-node or 4-node HA deployment cannot initialize raft at all — verify this matches deployment expectations (`deploy-ha.sh`, cluster config validation) or it becomes a runtime landmine. - **C2 (Constraint)**: `voted_for_term` is unconditionally reset to 0 in `raft_init` (raft.c:43) even when `initial_voted_for`/`initial_term` are restored from disk. After a restart, the duplicate-vote check (`voted_for_term == term`, raft.c:157) cannot match the restored term, so a node that voted for X in term T, restarted, then receives a request from Y in the same term T would fall through Step 4 and could grant a second vote in T (subject to the caller-side freshness gate). Test H1 checks `term`/`voted_for` but not `voted_for_term`. Potential one-vote-per-term violation across restart — needs confirmation. ## 3. Implementation Approach Options (for the remaining gaps) ### Option A: Extend existing components (close gaps in place) — RECOMMENDED - raft.c: initialize `voted_for_term = (initial_voted_for >= 0) ? initial_term : 0` in `raft_init` (C2); add an H6 restart-double-vote test. - requirements.md: amend 3.5 (higher-term step-down is the intended behavior) and 3.4 (freshness gate enforced by the dispatcher/callback split) so spec matches the as-built design; sync tasks.md 6.2 text. - ha.c: delete or fold legacy `handle_request_vote()` into the raft path; retarget `test_ha_vote_gate.c` at `raft_on_vote_request` + `on_vote_eligible` (closes 9.2). - Re-run `/xce-build lib` + `/xce-build controller` under ASAN to refresh 8.3. - Trade-offs: ✅ smallest diff, no architectural change, leverages existing test harness; ❌ touches a legacy test file; requirements edit needs human re-approval per workflow. ### Option B: Create new components (formal vote-record module) - Introduce a `raft_persisted_vote_t {term, voted_for}` pair persisted/restored atomically, replacing the separate initial_term/initial_voted_for fields; new restart-focused test group. - Trade-offs: ✅ makes the term-binding of a vote impossible to lose by construction; ❌ API churn in raft_config_t and ha_raft_reinit for a problem fixable with one init line; over-engineering (YAGNI). ### Option C: Hybrid — fix C2 in place now (Option A), defer handle_request_vote removal to a follow-up cleanup spec/task since it is production-dead. - Trade-offs: ✅ de-risks the correctness fix from the test-refactor churn; ❌ leaves known duplication in tree longer; two passes of review. ## 4. Effort & Risk | Gap | Effort | Risk | Justification | |---|---|---|---| | C2 voted_for_term restore fix + test | S | Low | One-line init change + one Unity test in an established harness | | Requirements/tasks doc sync (3.4, 3.5) | S | Low | Documentation only; needs approval round-trip | | Legacy handle_request_vote removal (9.2) | S–M | Medium | Production-dead but test_ha_vote_gate.c must be retargeted; risk of disturbing controller test baseline | | ASAN re-verification (8.3) | S | Low | Mechanical `/xce-build` run; fixes only if new findings | | C1 even-n_peers policy confirmation | S | Medium | If 2-node deployments are expected, this is a hard init failure discovered only at runtime | Overall remaining work: **S–M total, Low–Medium risk** — the library and integration are functionally complete. ## 5. Recommendations for Design/Closure Phase 1. Adopt **Option A**. Fix C2 (`voted_for_term` restore) first — it is the only candidate correctness gap; everything else is documentation or hygiene. 2. Update requirements.md 3.5 and 3.4 to match the as-built (and better) behavior: leader steps down on higher term; freshness gate enforced caller-side before `raft_on_vote_request` with COMMITTING/liveness in `on_vote_eligible`. Re-approve, then run `/kiro-verify-completion raft-library`. 3. Remove or refactor legacy `handle_request_vote` (ha.c:1791) to eliminate the last duplicated election state path (Req 9.2), retargeting `test_ha_vote_gate.c`. 4. Re-run task 7 ASAN validation against the current working tree before committing (the verified state predates the uncommitted raft.c Step-3/Step-4 rewrite). ## Research Needed - **R1**: Confirm intended cluster-size policy — are 2-node (even) HA clusters a supported deployment? If yes, the even-n_peers -EINVAL in raft_init must be relaxed; if no, add config-load validation so the failure surfaces at startup with a clear message rather than as a silent raft_init failure. - **R2**: Verify whether the restart double-vote scenario (C2) is reachable in production timing — does `ha_raft_reinit` run before the listener accepts vote requests? If unreachable, document why; fix anyway. - **R3**: Confirm `raft_check_and_start_election` ignoring the `raft_start_election` return code (raft.c:287–289) is safe in all adapter paths (comment claims -EALREADY unreachable; verify when an inbound vote request flips role between RDHUP and the delayed check). --- ## Gap Re-Audit (2026-06-18) Post-implementation audit. Spec phase = implemented, tasks 24/24 [x]. This pass re-verifies every EARS requirement against the working tree and reconciles the prior audit's open items (C2, R1–R3). Assets are tagged: **[CLOSED]** verified met in code, **[CLOSED-AS-BUILT]** met but behavior differs from literal req text (improvement), **[RESIDUAL]** doc/hygiene gap, **[SCOPE-DRIFT]** code beyond spec. ### Requirement-to-Asset Map | Req | Acceptance | Asset (file:line) | Status | |-----|-----------|-------------------|--------| | 1.1 | per-node state fields | `raft.h:28-50` raft_state_t (term/role/voted_for/voted_for_term/n_peers) | CLOSED | | 1.2 | init term=0 role=FOLLOWER voted_for=-1 | `raft.c:40-56` raft_init; defaults via cfg {0,-1} | CLOSED | | 1.3 | no transport types in header | `raft.h:4-6` includes only errno/stdint/string; no endpoint_t/SSL_CTX/cluster_t | CLOSED | | 2.1 | election: incr term + self-vote + send RPCs | `raft.c:68-71,98-102` | CLOSED | | 2.2 | single-node → immediate LEADER | `raft.c:87-92` returns RAFT_ELECTED_IMMEDIATELY | CLOSED | | 2.3 | already CANDIDATE/LEADER → error, no state change | `raft.c:64-65` -EALREADY | CLOSED | | 3.1 | term>cur → update,clear vote,FOLLOWER,evaluate | `raft.c:125-137` (also steps down leader via on_become_follower) | CLOSED-AS-BUILT | | 3.2 | termcur → FOLLOWER + on_become_follower | `raft.c:346-355` | CLOSED | | 4.3 | post-decision no-op | `raft.c:358-362` election_decided guard → RAFT_PENDING | CLOSED | | 4.4 | all replied, no quorum → FOLLOWER | `raft.c:386-393` replies_received==n_peers-1 | CLOSED | | 5.1 | leader-lost → clear vote + delay(priority*timeout) | `raft.c:417-425` | CLOSED | | 5.2 | voted non-self during delay → skip | `raft.c:429-433` RAFT_SKIP_ELECTION | CLOSED | | 5.3 | not voted / self → start election | `raft.c:435-438` RAFT_ELECTION_STARTED | CLOSED | | 6.1 | start election → on_schedule_timeout | `raft.c:105` | CLOSED | | 6.2 | timeout undecided → on_become_follower + cancel | `raft.c:409-414` | CLOSED | | 6.3 | timeout decided → on_cancel_timeout, no-op | `raft.c:401-407` | CLOSED | | 7.1 | raft_callbacks_t with required fn ptrs | `raft.h:79-117` (9 fields incl. on_send_pre_vote_request) | CLOSED | | 7.2 | void* ctx passed unchanged | `raft.h` ctx threaded; `raft.c` passes s->ctx everywhere | CLOSED | | 7.3 | NULL required callback → non-zero | `raft.c:23-38` 7 required-cb checks → -EINVAL | CLOSED | | 8.1 | deterministic N-node simulator | `raft_sim.c`, `raft.h:264-342` (XCE_TESTING) | CLOSED | | 8.2 | 4 invariant cases (a-d) | `test_raft.c`: A2 one-leader, B1/B4 deny-second, A1 single-node, E*/sim timeout | CLOSED | | 8.3 | compiles + tests pass under ASAN (leaks=0) | meson `raft/raft.c` (l.109) + `test_raft.c` (l.179); 209 RUN_TEST/test fns | CLOSED (re-run, see V1) | | 9.1 | adapter implements all callbacks, no dup state | `ha.c:1309-1320` s_real_cbs wires all 10; production state in raft_state_t | CLOSED | | 9.2 | ha delegates term/voted_for/quorum to library | `ha.c:4382-4522` ha_start_election_async→raft_start_election; `cluster.c:418` vote req; `ha.c:3237` vote reply | CLOSED-AS-BUILT | | 9.3 | XCE_TESTING hooks still functional | `ha_rdhup_election_hook` (ha.c:4528), ha_phase2 seams retained | CLOSED | **Score: 24/24 acceptance-criteria groups CLOSED.** No requirement is unimplemented. 5 are CLOSED-AS-BUILT (real code is a strict superset/improvement over literal req text — see residual R-DOC). ### Reconciliation of Prior Audit Open Items - **C2 (voted_for_term restore double-vote) — NOW FIXED.** `raft.c:47` sets `voted_for_term = (initial_voted_for != -1) ? initial_term : 0` at init, closing the post-restart second-vote window. Regression test `test_proto_H6_restored_vote_blocks_second_vote` (test_raft.c:2100, registered l.3978) guards it. Prior research's "only candidate correctness gap" is resolved (commit 9821ec4). - **R3 / Req 9.2 dual vote path — DOWNGRADED.** Legacy `handle_request_vote` (ha.c:2608) is now `#ifdef XCE_TESTING`-only — the production vote path is exclusively `cluster.c:418 → raft_on_vote_request`. The duplicated election-state logic no longer runs in production; it remains only as a test fixture for `test_ha_vote_gate.c` / `test_ha_election.c`. No production split-brain risk from the dual path. Residual: the test-only copy can drift from raft.c semantics over time (R-DUP below). - **R1 (even-n_peers policy) — STILL OPEN.** `raft_init` rejects even n_peers>1 with `-EINVAL` (raft.c:21-22), test `test_raft_init_even_peers_returns_einval`. Not stated in requirements; a 2-node HA deploy would silently fail raft_init. Needs requirements note + startup-time config validation. ### Residual Gaps (non-blocking) - **R-DOC (Low):** requirements.md 3.4/3.5 still describe COMMITTING as an in-library role gate and the freshness gate as in-library; as-built, COMMITTING enum is removed and freshness is split caller-side + on_vote_eligible. Requirements text lags the (better) implementation. Update for traceability. - **R-DUP (Low):** test-only `handle_request_vote` duplicates vote-gate logic; risk of semantic drift from raft.c. Consider retargeting `test_ha_vote_gate.c` onto `raft_on_vote_request` directly. - **R1 (Medium):** even-n_peers rejection undocumented; surfaces as silent raft_init -EINVAL rather than a clear startup error. Add config-load validation OR a requirement clause. - **R-SCOPE / SCOPE-DRIFT (informational, not a gap):** PreVote (raft_begin_pre_vote, raft_on_pre_vote_request, raft_on_pre_vote_reply, on_send_pre_vote_request, RAFT_PRE_VOTE_* codes; raft.c:193-336, raft.h:160-259) lives in the raft-library files but belongs to the separate **ha-prevote** spec (commits 449b8e9–e6b234a). It is purely additive, mutates no persistent Raft state, and does not violate any raft-library requirement. Flag only so the raft-library boundary doc reflects the co-tenancy: raft.h/raft.c now host two specs' surfaces. ### Integration Risks - **IR-1 (Low):** PreVote tally arrays are bounded by `RAFT_MAX_PEERS=8` (raft.h:18); `raft_on_pre_vote_reply` range-guards (raft.c:287). Real-vote path has no equivalent fixed array (uses scalar counters) so it is unbounded-safe. Mismatch is intentional but worth a one-line note in design. - **IR-2 (Low):** `raft_check_and_start_election` discards `raft_start_election` return (raft.c:436-437); comment claims -EALREADY unreachable because raft_on_leader_lost forces FOLLOWER. Valid for the current adapter, but an inbound vote/AppendEntries that flips role between RDHUP and the delayed fire could make it reachable. Low likelihood; assert instead of (void) for defense-in-depth. ### Open Work — Option Analysis **Option A — Doc-only closure (recommended).** Update requirements.md 3.4/3.5/9.2 to as-built; add R1 even-n_peers clause; note PreVote co-tenancy + IR-1/IR-2 in design.md. Then `/kiro-verify-completion`. - Effort: **S** (~1-2h, docs only). Risk: **Low**. No code change, no regression surface. **Option B — Doc closure + hygiene.** Option A plus: (1) startup config validation for even n_peers with a loud error, (2) retarget test-only `handle_request_vote` onto raft_on_vote_request (kill R-DUP), (3) convert IR-2 `(void)rc` to an assert. - Effort: **M** (~half day). Risk: **Low-Medium** (touches ha.c test seam + config load; rerun full controller + lib test suites). **Option C — Defer, accept as-is.** Ship implemented; track R-DOC/R-DUP/R1 as backlog. The only real correctness item (C2) is already fixed and tested, so functional risk of deferring is minimal. - Effort: **none**. Risk: **Low** functionally; **Medium** for spec/code traceability drift over time. ### Validation To Re-Run - **V1:** `/xce-build lib` and `/xce-build controller` with `ASAN_OPTIONS=detect_leaks=0` — no build dir present in the worktree at audit time; the test_raft.c suite (209 cases) and controller tests should be re-run to confirm green on the current tree (which now includes the PreVote additions + C2 fix, post-dating the last recorded task-7 ASAN run). **Bottom line:** raft-library is functionally complete — 24/24 met, the sole prior correctness gap (C2) is fixed and regression-tested, and the production dual-vote-path risk is eliminated (legacy path is test-only). Remaining work is documentation/hygiene (Option A, S effort, Low risk) plus a clean ASAN re-run (V1).