myxce/.kiro/specs/raft-library/research.md

# Research & Design Decisions: raft-library

## Discovery Scope
- Type: Complex new library (brownfield extraction from ha.c)
- Process: Full discovery — codebase analysis of ha.c, ha.h, test files

## Key Findings

### Existing Election State (ha.c)
- ha_state_t contains: term (atomic uint64), voted_for_term, voted_for_name[XCE_MAXNAME_LEN], role (atomic int32)
- ha_elect_ctx_t holds per-election: n_peers, votes_granted, n_replied, max_resp_tag, max_resp_lsn, done_posted, mutex, semaphore
- Transport coupling: ha_start_election_async directly calls endpoint_connect(), iomgr_monitor(), SSL_new()
- Post-election coupling: ha_elect_evaluate_and_signal calls cluster_init_phase2(), push_config_to_standbys()

### Root Cause of Split-Brain
- Self-vote was not recorded atomically at election start; voted_for remained empty until a peer vote arrived
- This allowed node B to grant its vote to node A AND node C in the same term if A and C both started elections before B processed either request
- Fix: record voted_for = self_idx immediately in raft_start_election, before sending any RPC

### Build vs. Adopt Decision
- No pure-C Raft library suitable for libxce: hashicorp/raft (Go), etcd/raft (Go), no active C alternative
- Decision: build a minimal Raft election library in src/lib/raft/
- Scope is election only (not log replication), which is simpler than full Raft

### Synthesis Outcomes
1. **Generalization**: Requirements 2–6 are all specializations of a single state machine with callbacks. One raft_state_t + raft_callbacks_t pattern covers all cases.
2. **Build vs. Adopt**: Build; no suitable C library exists.
3. **Simplification**: No threads inside the library. Synchronous callbacks. One header, one source, one test file. raft_sim.c compiled only under XCE_TESTING.

### Priority Jitter Design
- The delay formula (priority_idx * election_timeout_ms) is a protocol-level decision, not transport
- Placed inside the library as raft_on_leader_lost → on_start_election_after_delay callback
- Caller implements the actual timer/sleep

### Tracing Decision
- Added on_trace callback to raft_callbacks_t for full protocol observability
- Simulator dump log enables post-mortem analysis of split-brain scenarios
- raft_snapshot() provides safe point-in-time state read for crash/failure logging

---

# Cross-Spec Gap Audit (2026-06-08)

> Batch `/kiro-validate-gap` across 9 specs on persistent_ha branch.

## Analysis Summary

- **raft-library** is **~100% complete**. All 7 task groups (1–7), all subtasks (1–2.2, 3.1–3.5, 4.1–4.2, 5.1–5.4, 6.1–6.4, 7) are `[x]` in tasks.md. The `ready_for_implementation` flag is set in spec.json.
- **No remaining gaps**: The library (`src/lib/raft/raft.h`, `raft.c`, `raft_sim.c`) exposes `raft_state_t` + `raft_callbacks_t` with 9 callback hooks. The deterministic simulator (`raft_sim.c`) verifies split-brain freedom across 100 randomized rounds. The controller adapter in ha.c delegates all election state management (term, voted_for, quorum evaluation) to the raft library.
- **Cross-spec integration**: `election-iomgr-refactor` uses `raft_start_election` instead of the old `am_i_primary_quorum`. The adapter callbacks in ha.c wire `on_send_vote_request` → iomgr async connect, `on_become_leader` → `cluster_init_phase2` + config push, `on_become_follower` → `CTRL_STANDBY`. The `on_vote_eligible` callback implements the freshness gate (config_tag + applied_lsn comparison) and role gate (`CTRL_PRIMARY_COMMITTING` → deny).
- **Test coverage**: 34+ Unity test cases across 5 groups (A–H) covering basic election correctness, voted_for invariant, eligibility policy, quorum evaluation, election timeout, RDHUP failover, and split-brain prevention. All pass under ASAN.
- **Recommendation**: The spec can be marked `implemented`.

---

# Gap Analysis — Validation Pass (2026-06-09)

> Scope: requirements.md (9 requirement groups, 31 acceptance criteria) vs. on-disk working tree
> (uncommitted changes to `src/lib/raft/raft.c`, `src/controller/ha.c`, `ha.h`, `myapp.c` analyzed as-is).
> spec.json phase: `tasks-generated`, all tasks.md checkboxes `[x]` except parent rollups 2/3/4/5/6.

## 1. Current State Summary

- Library exists and is build-integrated: `src/lib/raft/raft.h` (276 lines), `raft.c` (290 lines),
  `raft_sim.c` (207 lines); wired in `src/lib/meson.build` (raft.c at line 108; test_raft.c + raft_sim.c
  under the test executable at lines 175–176, simulator guarded by `XCE_TESTING`).
- Test suite: `src/lib/testsuite/test_raft.c` (2990 lines, 47 test functions, 79 RUN_TEST invocations)
  covering unit tests, simulator tests, and protocol groups A–H plus F/G split-brain scenarios.
- Controller adapter live: `ha_state_t` embeds `raft_state_t raft` (`ha.h:183`); 9 callbacks implemented
  in `ha.c` (`s_real_cbs` at ha.c:735); `raft_init` called in ha init (ha.c:854) and re-initialized with
  real topology by `ha_raft_reinit` (ha.c:~2981–3008).
- Production dispatch fully delegated: vote requests → `raft_on_vote_request` (cluster.c:373),
  vote replies → `raft_on_vote_reply` (ha.c:2361), RDHUP → `raft_on_leader_lost` (ha.c:3290),
  election start → `raft_start_election` (ha.c:3146).

## 2. Requirement-to-Asset Map

| Req | Criterion | Asset(s) | Status |
|---|---|---|---|
| 1.1 | Per-node state (term/role/voted_for/n_peers) | `raft_state_t` (raft.h:23–38) | Implemented |
| 1.2 | Init defaults term=0/FOLLOWER/-1 | `raft_init` (raft.c:13–55); H2 test | Implemented (extended: restores `initial_term`/`initial_voted_for` from config) |
| 1.3 | No endpoint_t/SSL/iomgr/cluster_t/capnp in header | raft.h includes only errno/stdint/string (+stdio under XCE_TESTING) | Implemented |
| 2.1 | start_election: term++, self-vote, CANDIDATE, send RPCs | raft.c:57–104 | Implemented (also persists self-vote before RPCs — stronger than spec) |
| 2.2 | Single-node immediate LEADER | raft.c:83–88; tests A1, single_node | Implemented |
| 2.3 | -EALREADY if CANDIDATE/LEADER | raft.c:60–61; tests already_candidate/leader | Implemented |
| 3.1 | term > current: upgrade, clear vote, FOLLOWER, evaluate | raft.c:121–130; B2 | Implemented (extra: fires `on_become_follower` if was LEADER — split-brain fix, uncommitted change) |
| 3.2 | term < current: deny, return current term | raft.c:110–115; B3 | Implemented |
| 3.3 | One vote per term | raft.c:157–164; B1/B4/B5 | Implemented (extension: idempotent re-grant to the SAME candidate same term — beyond spec text; see Gap-3) |
| 3.4 | Freshness gate (config_tag/LSN) before grant | Split: caller-side gate in cluster.c:352–356 + `on_vote_eligible` hook (raft.c:167–173) | Partial — behavior present but located in the dispatcher, not in `on_vote_eligible` as tasks.md 6.2 states; `raft_cb_vote_eligible` (ha.c:202) intentionally skips freshness. Doc/code drift. |
| 3.5 | LEADER or COMMITTING deny ALL vote requests regardless of term | raft.c:135–148 (same-term deny only); COMMITTING gate in `on_vote_eligible` adapter | Partial (intentional deviation) — higher-term request now steps the leader down instead of denying, to prevent the COMMITTING-timeout split-brain. Requirement text is outdated. |
| 4.1 | Quorum (n/2)+1 → LEADER, on_become_leader once | raft.c:222–234; D1, A4 | Implemented |
| 4.2 | Reply term > current → FOLLOWER + on_become_follower | raft.c:197–206; D4 | Implemented |
| 4.3 | Reply after decided → no-op | raft.c:209–213; D3 | Implemented |
| 4.4 | All replied, no quorum → FOLLOWER | raft.c:237–244; D2 | Implemented |
| 5.1 | leader_lost: clear vote, FOLLOWER, delayed election (priority×timeout) | raft.c:268–276; F1 | Implemented |
| 5.2 | Skip election if voted for non-self during delay | `raft_check_and_start_election` raft.c:280–284; F2 | Implemented |
| 5.3 | Else start election internally | raft.c:286–289; F3 | Implemented |
| 6.1 | start_election schedules timeout | raft.c:101; E3 | Implemented |
| 6.2 | Timeout undecided → on_become_follower, cancel tracking | raft.c:260–265; E1 | Implemented |
| 6.3 | Timeout after decided → on_cancel_timeout no-op | raft.c:252–257; E2 | Implemented |
| 7.1 | raft_callbacks_t with 6 required fields | raft.h:66–96 (9 fields: 7 required + 2 optional) | Implemented (superset: adds `on_start_election_after_delay` as required, `on_vote_eligible`/`on_trace` optional) |
| 7.2 | void* ctx passed to every callback | raft_config_t.ctx → s->ctx, all call sites | Implemented |
| 7.3 | NULL required callback → error at init | raft.c:23–38 (-EINVAL); Hnull test | Implemented |
| 8.1 | Deterministic N-node simulator, synchronous routing | raft_sim.c + raft.h:197–274; sim tests | Implemented |
| 8.2 | Invariants: one-leader/3-node, dup deny, single-node, timeout re-election | A2/G1 (100 rounds), B1/B4, A1, E3 + G2 partition | Implemented |
| 8.3 | Compile + all tests pass under ASAN | tasks.md task 7 `[x]`; commits e10cf9a/361397a fixed ASAN issues | Unknown (stale) — last verified pre-dating current uncommitted edits to raft.c/ha.c; needs re-run |
| 9.1 | Adapter implements all callbacks via endpoint_t/iomgr | ha.c:107–744 (8 raft_cb_* + s_real_cbs) | Implemented |
| 9.2 | ha_start_election_async delegates; no direct ha->term/voted_for writes outside adapter | ha.c:3038–3161 delegates; only atomic loads + persist/restore remain | Partial — production path clean, but legacy `handle_request_vote()` (ha.c:1791–1873) still contains a full parallel vote-evaluation path with direct `ha->voted_for_term`/`voted_for_name` writes. It is production-dead (only `test_ha_vote_gate.c` calls it) but duplicates election state management contrary to the spirit of 9.2. |
| 9.3 | ha_phase2_hook / ha_rdhup_election_hook still functional | ha.c:495/631/2068/2147 (phase2), ha.c:3267–3284 (rdhup) | Implemented |

**Score: 27 of 31 acceptance criteria Implemented; 3 Partial (3.4, 3.5, 9.2 — all deliberate-deviation or residual-cleanup, not functional regressions); 1 Unknown (8.3 ASAN re-verification against the uncommitted working tree).**

### Constraints / behavior added beyond requirements
- **C1 (Constraint)**: `raft_init` rejects even `n_peers > 1` with -EINVAL (raft.c:21–22). Not in
  requirements. A 2-node or 4-node HA deployment cannot initialize raft at all — verify this matches
  deployment expectations (`deploy-ha.sh`, cluster config validation) or it becomes a runtime landmine.
- **C2 (Constraint)**: `voted_for_term` is unconditionally reset to 0 in `raft_init` (raft.c:43) even when
  `initial_voted_for`/`initial_term` are restored from disk. After a restart, the duplicate-vote check
  (`voted_for_term == term`, raft.c:157) cannot match the restored term, so a node that voted for X in
  term T, restarted, then receives a request from Y in the same term T would fall through Step 4 and could
  grant a second vote in T (subject to the caller-side freshness gate). Test H1 checks `term`/`voted_for`
  but not `voted_for_term`. Potential one-vote-per-term violation across restart — needs confirmation.

## 3. Implementation Approach Options (for the remaining gaps)

### Option A: Extend existing components (close gaps in place) — RECOMMENDED
- raft.c: initialize `voted_for_term = (initial_voted_for >= 0) ? initial_term : 0` in `raft_init` (C2);
  add an H6 restart-double-vote test.
- requirements.md: amend 3.5 (higher-term step-down is the intended behavior) and 3.4 (freshness gate
  enforced by the dispatcher/callback split) so spec matches the as-built design; sync tasks.md 6.2 text.
- ha.c: delete or fold legacy `handle_request_vote()` into the raft path; retarget `test_ha_vote_gate.c`
  at `raft_on_vote_request` + `on_vote_eligible` (closes 9.2).
- Re-run `/xce-build lib` + `/xce-build controller` under ASAN to refresh 8.3.
- Trade-offs: ✅ smallest diff, no architectural change, leverages existing test harness; ❌ touches a
  legacy test file; requirements edit needs human re-approval per workflow.

### Option B: Create new components (formal vote-record module)
- Introduce a `raft_persisted_vote_t {term, voted_for}` pair persisted/restored atomically, replacing the
  separate initial_term/initial_voted_for fields; new restart-focused test group.
- Trade-offs: ✅ makes the term-binding of a vote impossible to lose by construction; ❌ API churn in
  raft_config_t and ha_raft_reinit for a problem fixable with one init line; over-engineering (YAGNI).

### Option C: Hybrid — fix C2 in place now (Option A), defer handle_request_vote removal to a follow-up
  cleanup spec/task since it is production-dead.
- Trade-offs: ✅ de-risks the correctness fix from the test-refactor churn; ❌ leaves known duplication
  in tree longer; two passes of review.

## 4. Effort & Risk

| Gap | Effort | Risk | Justification |
|---|---|---|---|
| C2 voted_for_term restore fix + test | S | Low | One-line init change + one Unity test in an established harness |
| Requirements/tasks doc sync (3.4, 3.5) | S | Low | Documentation only; needs approval round-trip |
| Legacy handle_request_vote removal (9.2) | S–M | Medium | Production-dead but test_ha_vote_gate.c must be retargeted; risk of disturbing controller test baseline |
| ASAN re-verification (8.3) | S | Low | Mechanical `/xce-build` run; fixes only if new findings |
| C1 even-n_peers policy confirmation | S | Medium | If 2-node deployments are expected, this is a hard init failure discovered only at runtime |

Overall remaining work: **S–M total, Low–Medium risk** — the library and integration are functionally complete.

## 5. Recommendations for Design/Closure Phase

1. Adopt **Option A**. Fix C2 (`voted_for_term` restore) first — it is the only candidate correctness
   gap; everything else is documentation or hygiene.
2. Update requirements.md 3.5 and 3.4 to match the as-built (and better) behavior: leader steps down on
   higher term; freshness gate enforced caller-side before `raft_on_vote_request` with COMMITTING/liveness
   in `on_vote_eligible`. Re-approve, then run `/kiro-verify-completion raft-library`.
3. Remove or refactor legacy `handle_request_vote` (ha.c:1791) to eliminate the last duplicated election
   state path (Req 9.2), retargeting `test_ha_vote_gate.c`.
4. Re-run task 7 ASAN validation against the current working tree before committing (the verified state
   predates the uncommitted raft.c Step-3/Step-4 rewrite).

## Research Needed

- **R1**: Confirm intended cluster-size policy — are 2-node (even) HA clusters a supported deployment?
  If yes, the even-n_peers -EINVAL in raft_init must be relaxed; if no, add config-load validation so the
  failure surfaces at startup with a clear message rather than as a silent raft_init failure.
- **R2**: Verify whether the restart double-vote scenario (C2) is reachable in production timing — does
  `ha_raft_reinit` run before the listener accepts vote requests? If unreachable, document why; fix anyway.
- **R3**: Confirm `raft_check_and_start_election` ignoring the `raft_start_election` return code
  (raft.c:287–289) is safe in all adapter paths (comment claims -EALREADY unreachable; verify when an
  inbound vote request flips role between RDHUP and the delayed check).