myxce/.kiro/specs/raft-library/research.md
Rongsong Shen 20e59d0b42 checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
2026-06-21 14:02:15 +00:00

296 lines
24 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Research & Design Decisions: raft-library
## Discovery Scope
- Type: Complex new library (brownfield extraction from ha.c)
- Process: Full discovery — codebase analysis of ha.c, ha.h, test files
## Key Findings
### Existing Election State (ha.c)
- ha_state_t contains: term (atomic uint64), voted_for_term, voted_for_name[XCE_MAXNAME_LEN], role (atomic int32)
- ha_elect_ctx_t holds per-election: n_peers, votes_granted, n_replied, max_resp_tag, max_resp_lsn, done_posted, mutex, semaphore
- Transport coupling: ha_start_election_async directly calls endpoint_connect(), iomgr_monitor(), SSL_new()
- Post-election coupling: ha_elect_evaluate_and_signal calls cluster_init_phase2(), push_config_to_standbys()
### Root Cause of Split-Brain
- Self-vote was not recorded atomically at election start; voted_for remained empty until a peer vote arrived
- This allowed node B to grant its vote to node A AND node C in the same term if A and C both started elections before B processed either request
- Fix: record voted_for = self_idx immediately in raft_start_election, before sending any RPC
### Build vs. Adopt Decision
- No pure-C Raft library suitable for libxce: hashicorp/raft (Go), etcd/raft (Go), no active C alternative
- Decision: build a minimal Raft election library in src/lib/raft/
- Scope is election only (not log replication), which is simpler than full Raft
### Synthesis Outcomes
1. **Generalization**: Requirements 2–6 are all specializations of a single state machine with callbacks. One raft_state_t + raft_callbacks_t pattern covers all cases.
2. **Build vs. Adopt**: Build; no suitable C library exists.
3. **Simplification**: No threads inside the library. Synchronous callbacks. One header, one source, one test file. raft_sim.c compiled only under XCE_TESTING.
### Priority Jitter Design
- The delay formula (priority_idx * election_timeout_ms) is a protocol-level decision, not transport
- Placed inside the library as raft_on_leader_lost → on_start_election_after_delay callback
- Caller implements the actual timer/sleep
### Tracing Decision
- Added on_trace callback to raft_callbacks_t for full protocol observability
- Simulator dump log enables post-mortem analysis of split-brain scenarios
- raft_snapshot() provides safe point-in-time state read for crash/failure logging
---
# Cross-Spec Gap Audit (2026-06-08)
> Batch `/kiro-validate-gap` across 9 specs on persistent_ha branch.
## Analysis Summary
- **raft-library** is **~100% complete**. All 7 task groups (1–7), all subtasks (1–2.2, 3.1–3.5, 4.1–4.2, 5.1–5.4, 6.1–6.4, 7) are `[x]` in tasks.md. The `ready_for_implementation` flag is set in spec.json.
- **No remaining gaps**: The library (`src/lib/raft/raft.h`, `raft.c`, `raft_sim.c`) exposes `raft_state_t` + `raft_callbacks_t` with 9 callback hooks. The deterministic simulator (`raft_sim.c`) verifies split-brain freedom across 100 randomized rounds. The controller adapter in ha.c delegates all election state management (term, voted_for, quorum evaluation) to the raft library.
- **Cross-spec integration**: `election-iomgr-refactor` uses `raft_start_election` instead of the old `am_i_primary_quorum`. The adapter callbacks in ha.c wire `on_send_vote_request` → iomgr async connect, `on_become_leader` → `cluster_init_phase2` + config push, `on_become_follower` → `CTRL_STANDBY`. The `on_vote_eligible` callback implements the freshness gate (config_tag + applied_lsn comparison) and role gate (`CTRL_PRIMARY_COMMITTING` → deny).
- **Test coverage**: 34+ Unity test cases across 5 groups (A–H) covering basic election correctness, voted_for invariant, eligibility policy, quorum evaluation, election timeout, RDHUP failover, and split-brain prevention. All pass under ASAN.
- **Recommendation**: The spec can be marked `implemented`.
---
# Gap Analysis — Validation Pass (2026-06-09)
> Scope: requirements.md (9 requirement groups, 31 acceptance criteria) vs. on-disk working tree
> (uncommitted changes to `src/lib/raft/raft.c`, `src/controller/ha.c`, `ha.h`, `myapp.c` analyzed as-is).
> spec.json phase: `tasks-generated`, all tasks.md checkboxes `[x]` except parent rollups 2/3/4/5/6.
## 1. Current State Summary
- Library exists and is build-integrated: `src/lib/raft/raft.h` (276 lines), `raft.c` (290 lines),
`raft_sim.c` (207 lines); wired in `src/lib/meson.build` (raft.c at line 108; test_raft.c + raft_sim.c
under the test executable at lines 175–176, simulator guarded by `XCE_TESTING`).
- Test suite: `src/lib/testsuite/test_raft.c` (2990 lines, 47 test functions, 79 RUN_TEST invocations)
covering unit tests, simulator tests, and protocol groups A–H plus F/G split-brain scenarios.
- Controller adapter live: `ha_state_t` embeds `raft_state_t raft` (`ha.h:183`); 9 callbacks implemented
in `ha.c` (`s_real_cbs` at ha.c:735); `raft_init` called in ha init (ha.c:854) and re-initialized with
real topology by `ha_raft_reinit` (ha.c:~2981–3008).
- Production dispatch fully delegated: vote requests → `raft_on_vote_request` (cluster.c:373),
vote replies → `raft_on_vote_reply` (ha.c:2361), RDHUP → `raft_on_leader_lost` (ha.c:3290),
election start → `raft_start_election` (ha.c:3146).
## 2. Requirement-to-Asset Map
| Req | Criterion | Asset(s) | Status |
|---|---|---|---|
| 1.1 | Per-node state (term/role/voted_for/n_peers) | `raft_state_t` (raft.h:23–38) | Implemented |
| 1.2 | Init defaults term=0/FOLLOWER/-1 | `raft_init` (raft.c:13–55); H2 test | Implemented (extended: restores `initial_term`/`initial_voted_for` from config) |
| 1.3 | No endpoint_t/SSL/iomgr/cluster_t/capnp in header | raft.h includes only errno/stdint/string (+stdio under XCE_TESTING) | Implemented |
| 2.1 | start_election: term++, self-vote, CANDIDATE, send RPCs | raft.c:57–104 | Implemented (also persists self-vote before RPCs — stronger than spec) |
| 2.2 | Single-node immediate LEADER | raft.c:83–88; tests A1, single_node | Implemented |
| 2.3 | -EALREADY if CANDIDATE/LEADER | raft.c:60–61; tests already_candidate/leader | Implemented |
| 3.1 | term > current: upgrade, clear vote, FOLLOWER, evaluate | raft.c:121–130; B2 | Implemented (extra: fires `on_become_follower` if was LEADER — split-brain fix, uncommitted change) |
| 3.2 | term < current: deny, return current term | raft.c:110–115; B3 | Implemented |
| 3.3 | One vote per term | raft.c:157–164; B1/B4/B5 | Implemented (extension: idempotent re-grant to the SAME candidate same term — beyond spec text; see Gap-3) |
| 3.4 | Freshness gate (config_tag/LSN) before grant | Split: caller-side gate in cluster.c:352–356 + `on_vote_eligible` hook (raft.c:167–173) | Partial — behavior present but located in the dispatcher, not in `on_vote_eligible` as tasks.md 6.2 states; `raft_cb_vote_eligible` (ha.c:202) intentionally skips freshness. Doc/code drift. |
| 3.5 | LEADER or COMMITTING deny ALL vote requests regardless of term | raft.c:135–148 (same-term deny only); COMMITTING gate in `on_vote_eligible` adapter | Partial (intentional deviation) — higher-term request now steps the leader down instead of denying, to prevent the COMMITTING-timeout split-brain. Requirement text is outdated. |
| 4.1 | Quorum (n/2)+1 → LEADER, on_become_leader once | raft.c:222–234; D1, A4 | Implemented |
| 4.2 | Reply term > current → FOLLOWER + on_become_follower | raft.c:197–206; D4 | Implemented |
| 4.3 | Reply after decided → no-op | raft.c:209–213; D3 | Implemented |
| 4.4 | All replied, no quorum → FOLLOWER | raft.c:237–244; D2 | Implemented |
| 5.1 | leader_lost: clear vote, FOLLOWER, delayed election (priority×timeout) | raft.c:268–276; F1 | Implemented |
| 5.2 | Skip election if voted for non-self during delay | `raft_check_and_start_election` raft.c:280–284; F2 | Implemented |
| 5.3 | Else start election internally | raft.c:286–289; F3 | Implemented |
| 6.1 | start_election schedules timeout | raft.c:101; E3 | Implemented |
| 6.2 | Timeout undecided → on_become_follower, cancel tracking | raft.c:260–265; E1 | Implemented |
| 6.3 | Timeout after decided → on_cancel_timeout no-op | raft.c:252–257; E2 | Implemented |
| 7.1 | raft_callbacks_t with 6 required fields | raft.h:66–96 (9 fields: 7 required + 2 optional) | Implemented (superset: adds `on_start_election_after_delay` as required, `on_vote_eligible`/`on_trace` optional) |
| 7.2 | void* ctx passed to every callback | raft_config_t.ctx → s->ctx, all call sites | Implemented |
| 7.3 | NULL required callback → error at init | raft.c:23–38 (-EINVAL); Hnull test | Implemented |
| 8.1 | Deterministic N-node simulator, synchronous routing | raft_sim.c + raft.h:197–274; sim tests | Implemented |
| 8.2 | Invariants: one-leader/3-node, dup deny, single-node, timeout re-election | A2/G1 (100 rounds), B1/B4, A1, E3 + G2 partition | Implemented |
| 8.3 | Compile + all tests pass under ASAN | tasks.md task 7 `[x]`; commits e10cf9a/361397a fixed ASAN issues | Unknown (stale) — last verified pre-dating current uncommitted edits to raft.c/ha.c; needs re-run |
| 9.1 | Adapter implements all callbacks via endpoint_t/iomgr | ha.c:107–744 (8 raft_cb_* + s_real_cbs) | Implemented |
| 9.2 | ha_start_election_async delegates; no direct ha->term/voted_for writes outside adapter | ha.c:3038–3161 delegates; only atomic loads + persist/restore remain | Partial — production path clean, but legacy `handle_request_vote()` (ha.c:1791–1873) still contains a full parallel vote-evaluation path with direct `ha->voted_for_term`/`voted_for_name` writes. It is production-dead (only `test_ha_vote_gate.c` calls it) but duplicates election state management contrary to the spirit of 9.2. |
| 9.3 | ha_phase2_hook / ha_rdhup_election_hook still functional | ha.c:495/631/2068/2147 (phase2), ha.c:3267–3284 (rdhup) | Implemented |
**Score: 27 of 31 acceptance criteria Implemented; 3 Partial (3.4, 3.5, 9.2 — all deliberate-deviation or residual-cleanup, not functional regressions); 1 Unknown (8.3 ASAN re-verification against the uncommitted working tree).**
### Constraints / behavior added beyond requirements
- **C1 (Constraint)**: `raft_init` rejects even `n_peers > 1` with -EINVAL (raft.c:21–22). Not in
requirements. A 2-node or 4-node HA deployment cannot initialize raft at all — verify this matches
deployment expectations (`deploy-ha.sh`, cluster config validation) or it becomes a runtime landmine.
- **C2 (Constraint)**: `voted_for_term` is unconditionally reset to 0 in `raft_init` (raft.c:43) even when
`initial_voted_for`/`initial_term` are restored from disk. After a restart, the duplicate-vote check
(`voted_for_term == term`, raft.c:157) cannot match the restored term, so a node that voted for X in
term T, restarted, then receives a request from Y in the same term T would fall through Step 4 and could
grant a second vote in T (subject to the caller-side freshness gate). Test H1 checks `term`/`voted_for`
but not `voted_for_term`. Potential one-vote-per-term violation across restart — needs confirmation.
## 3. Implementation Approach Options (for the remaining gaps)
### Option A: Extend existing components (close gaps in place) — RECOMMENDED
- raft.c: initialize `voted_for_term = (initial_voted_for >= 0) ? initial_term : 0` in `raft_init` (C2);
add an H6 restart-double-vote test.
- requirements.md: amend 3.5 (higher-term step-down is the intended behavior) and 3.4 (freshness gate
enforced by the dispatcher/callback split) so spec matches the as-built design; sync tasks.md 6.2 text.
- ha.c: delete or fold legacy `handle_request_vote()` into the raft path; retarget `test_ha_vote_gate.c`
at `raft_on_vote_request` + `on_vote_eligible` (closes 9.2).
- Re-run `/xce-build lib` + `/xce-build controller` under ASAN to refresh 8.3.
- Trade-offs: ✅ smallest diff, no architectural change, leverages existing test harness; ❌ touches a
legacy test file; requirements edit needs human re-approval per workflow.
### Option B: Create new components (formal vote-record module)
- Introduce a `raft_persisted_vote_t {term, voted_for}` pair persisted/restored atomically, replacing the
separate initial_term/initial_voted_for fields; new restart-focused test group.
- Trade-offs: ✅ makes the term-binding of a vote impossible to lose by construction; ❌ API churn in
raft_config_t and ha_raft_reinit for a problem fixable with one init line; over-engineering (YAGNI).
### Option C: Hybrid — fix C2 in place now (Option A), defer handle_request_vote removal to a follow-up
cleanup spec/task since it is production-dead.
- Trade-offs: ✅ de-risks the correctness fix from the test-refactor churn; ❌ leaves known duplication
in tree longer; two passes of review.
## 4. Effort & Risk
| Gap | Effort | Risk | Justification |
|---|---|---|---|
| C2 voted_for_term restore fix + test | S | Low | One-line init change + one Unity test in an established harness |
| Requirements/tasks doc sync (3.4, 3.5) | S | Low | Documentation only; needs approval round-trip |
| Legacy handle_request_vote removal (9.2) | S–M | Medium | Production-dead but test_ha_vote_gate.c must be retargeted; risk of disturbing controller test baseline |
| ASAN re-verification (8.3) | S | Low | Mechanical `/xce-build` run; fixes only if new findings |
| C1 even-n_peers policy confirmation | S | Medium | If 2-node deployments are expected, this is a hard init failure discovered only at runtime |
Overall remaining work: **S–M total, Low–Medium risk** — the library and integration are functionally complete.
## 5. Recommendations for Design/Closure Phase
1. Adopt **Option A**. Fix C2 (`voted_for_term` restore) first — it is the only candidate correctness
gap; everything else is documentation or hygiene.
2. Update requirements.md 3.5 and 3.4 to match the as-built (and better) behavior: leader steps down on
higher term; freshness gate enforced caller-side before `raft_on_vote_request` with COMMITTING/liveness
in `on_vote_eligible`. Re-approve, then run `/kiro-verify-completion raft-library`.
3. Remove or refactor legacy `handle_request_vote` (ha.c:1791) to eliminate the last duplicated election
state path (Req 9.2), retargeting `test_ha_vote_gate.c`.
4. Re-run task 7 ASAN validation against the current working tree before committing (the verified state
predates the uncommitted raft.c Step-3/Step-4 rewrite).
## Research Needed
- **R1**: Confirm intended cluster-size policy — are 2-node (even) HA clusters a supported deployment?
If yes, the even-n_peers -EINVAL in raft_init must be relaxed; if no, add config-load validation so the
failure surfaces at startup with a clear message rather than as a silent raft_init failure.
- **R2**: Verify whether the restart double-vote scenario (C2) is reachable in production timing — does
`ha_raft_reinit` run before the listener accepts vote requests? If unreachable, document why; fix anyway.
- **R3**: Confirm `raft_check_and_start_election` ignoring the `raft_start_election` return code
(raft.c:287–289) is safe in all adapter paths (comment claims -EALREADY unreachable; verify when an
inbound vote request flips role between RDHUP and the delayed check).
---
## Gap Re-Audit (2026-06-18)
Post-implementation audit. Spec phase = implemented, tasks 24/24 [x]. This pass re-verifies every
EARS requirement against the working tree and reconciles the prior audit's open items (C2, R1–R3).
Assets are tagged: **[CLOSED]** verified met in code, **[CLOSED-AS-BUILT]** met but behavior differs
from literal req text (improvement), **[RESIDUAL]** doc/hygiene gap, **[SCOPE-DRIFT]** code beyond spec.
### Requirement-to-Asset Map
| Req | Acceptance | Asset (file:line) | Status |
|-----|-----------|-------------------|--------|
| 1.1 | per-node state fields | `raft.h:28-50` raft_state_t (term/role/voted_for/voted_for_term/n_peers) | CLOSED |
| 1.2 | init term=0 role=FOLLOWER voted_for=-1 | `raft.c:40-56` raft_init; defaults via cfg {0,-1} | CLOSED |
| 1.3 | no transport types in header | `raft.h:4-6` includes only errno/stdint/string; no endpoint_t/SSL_CTX/cluster_t | CLOSED |
| 2.1 | election: incr term + self-vote + send RPCs | `raft.c:68-71,98-102` | CLOSED |
| 2.2 | single-node → immediate LEADER | `raft.c:87-92` returns RAFT_ELECTED_IMMEDIATELY | CLOSED |
| 2.3 | already CANDIDATE/LEADER → error, no state change | `raft.c:64-65` -EALREADY | CLOSED |
| 3.1 | term>cur → update,clear vote,FOLLOWER,evaluate | `raft.c:125-137` (also steps down leader via on_become_follower) | CLOSED-AS-BUILT |
| 3.2 | term<cur → deny + return cur term | `raft.c:114-119` | CLOSED |
| 3.3 | same term, diff voted_for → deny | `raft.c:161-168` (gated on voted_for_term==term) | CLOSED |
| 3.4 | freshness gate pass → grant + persist | `raft.c:170-190`; freshness split: tag/lsn floor enforced caller-side (cluster.c:363), COMMITTING/liveness in on_vote_eligible (`ha.c:299` raft_cb_vote_eligible) | CLOSED-AS-BUILT |
| 3.5 | LEADER/COMMITTING → deny all | `raft.c:148-152` LEADER deny; COMMITTING enum removed (ha-single-leader-role 93dfc02), now an on_vote_eligible policy | CLOSED-AS-BUILT |
| 4.1 | quorum ceil((n+1)/2) → LEADER once | `raft.c:372-383`, quorum=(n_peers/2)+1 == ceil for odd n | CLOSED |
| 4.2 | reply term>cur → FOLLOWER + on_become_follower | `raft.c:346-355` | CLOSED |
| 4.3 | post-decision no-op | `raft.c:358-362` election_decided guard → RAFT_PENDING | CLOSED |
| 4.4 | all replied, no quorum → FOLLOWER | `raft.c:386-393` replies_received==n_peers-1 | CLOSED |
| 5.1 | leader-lost → clear vote + delay(priority*timeout) | `raft.c:417-425` | CLOSED |
| 5.2 | voted non-self during delay → skip | `raft.c:429-433` RAFT_SKIP_ELECTION | CLOSED |
| 5.3 | not voted / self → start election | `raft.c:435-438` RAFT_ELECTION_STARTED | CLOSED |
| 6.1 | start election → on_schedule_timeout | `raft.c:105` | CLOSED |
| 6.2 | timeout undecided → on_become_follower + cancel | `raft.c:409-414` | CLOSED |
| 6.3 | timeout decided → on_cancel_timeout, no-op | `raft.c:401-407` | CLOSED |
| 7.1 | raft_callbacks_t with required fn ptrs | `raft.h:79-117` (9 fields incl. on_send_pre_vote_request) | CLOSED |
| 7.2 | void* ctx passed unchanged | `raft.h` ctx threaded; `raft.c` passes s->ctx everywhere | CLOSED |
| 7.3 | NULL required callback → non-zero | `raft.c:23-38` 7 required-cb checks → -EINVAL | CLOSED |
| 8.1 | deterministic N-node simulator | `raft_sim.c`, `raft.h:264-342` (XCE_TESTING) | CLOSED |
| 8.2 | 4 invariant cases (a-d) | `test_raft.c`: A2 one-leader, B1/B4 deny-second, A1 single-node, E*/sim timeout | CLOSED |
| 8.3 | compiles + tests pass under ASAN (leaks=0) | meson `raft/raft.c` (l.109) + `test_raft.c` (l.179); 209 RUN_TEST/test fns | CLOSED (re-run, see V1) |
| 9.1 | adapter implements all callbacks, no dup state | `ha.c:1309-1320` s_real_cbs wires all 10; production state in raft_state_t | CLOSED |
| 9.2 | ha delegates term/voted_for/quorum to library | `ha.c:4382-4522` ha_start_election_async→raft_start_election; `cluster.c:418` vote req; `ha.c:3237` vote reply | CLOSED-AS-BUILT |
| 9.3 | XCE_TESTING hooks still functional | `ha_rdhup_election_hook` (ha.c:4528), ha_phase2 seams retained | CLOSED |
**Score: 24/24 acceptance-criteria groups CLOSED.** No requirement is unimplemented. 5 are CLOSED-AS-BUILT
(real code is a strict superset/improvement over literal req text — see residual R-DOC).
### Reconciliation of Prior Audit Open Items
- **C2 (voted_for_term restore double-vote) — NOW FIXED.** `raft.c:47` sets
`voted_for_term = (initial_voted_for != -1) ? initial_term : 0` at init, closing the post-restart
second-vote window. Regression test `test_proto_H6_restored_vote_blocks_second_vote` (test_raft.c:2100,
registered l.3978) guards it. Prior research's "only candidate correctness gap" is resolved (commit 9821ec4).
- **R3 / Req 9.2 dual vote path — DOWNGRADED.** Legacy `handle_request_vote` (ha.c:2608) is now
`#ifdef XCE_TESTING`-only — the production vote path is exclusively `cluster.c:418 → raft_on_vote_request`.
The duplicated election-state logic no longer runs in production; it remains only as a test fixture for
`test_ha_vote_gate.c` / `test_ha_election.c`. No production split-brain risk from the dual path. Residual:
the test-only copy can drift from raft.c semantics over time (R-DUP below).
- **R1 (even-n_peers policy) — STILL OPEN.** `raft_init` rejects even n_peers>1 with `-EINVAL`
(raft.c:21-22), test `test_raft_init_even_peers_returns_einval`. Not stated in requirements; a 2-node HA
deploy would silently fail raft_init. Needs requirements note + startup-time config validation.
### Residual Gaps (non-blocking)
- **R-DOC (Low):** requirements.md 3.4/3.5 still describe COMMITTING as an in-library role gate and the
freshness gate as in-library; as-built, COMMITTING enum is removed and freshness is split caller-side +
on_vote_eligible. Requirements text lags the (better) implementation. Update for traceability.
- **R-DUP (Low):** test-only `handle_request_vote` duplicates vote-gate logic; risk of semantic drift from
raft.c. Consider retargeting `test_ha_vote_gate.c` onto `raft_on_vote_request` directly.
- **R1 (Medium):** even-n_peers rejection undocumented; surfaces as silent raft_init -EINVAL rather than a
clear startup error. Add config-load validation OR a requirement clause.
- **R-SCOPE / SCOPE-DRIFT (informational, not a gap):** PreVote (raft_begin_pre_vote,
raft_on_pre_vote_request, raft_on_pre_vote_reply, on_send_pre_vote_request, RAFT_PRE_VOTE_* codes;
raft.c:193-336, raft.h:160-259) lives in the raft-library files but belongs to the separate **ha-prevote**
spec (commits 449b8e9–e6b234a). It is purely additive, mutates no persistent Raft state, and does not
violate any raft-library requirement. Flag only so the raft-library boundary doc reflects the co-tenancy:
raft.h/raft.c now host two specs' surfaces.
### Integration Risks
- **IR-1 (Low):** PreVote tally arrays are bounded by `RAFT_MAX_PEERS=8` (raft.h:18); `raft_on_pre_vote_reply`
range-guards (raft.c:287). Real-vote path has no equivalent fixed array (uses scalar counters) so it is
unbounded-safe. Mismatch is intentional but worth a one-line note in design.
- **IR-2 (Low):** `raft_check_and_start_election` discards `raft_start_election` return (raft.c:436-437);
comment claims -EALREADY unreachable because raft_on_leader_lost forces FOLLOWER. Valid for the current
adapter, but an inbound vote/AppendEntries that flips role between RDHUP and the delayed fire could make it
reachable. Low likelihood; assert instead of (void) for defense-in-depth.
### Open Work — Option Analysis
**Option A — Doc-only closure (recommended).** Update requirements.md 3.4/3.5/9.2 to as-built; add R1
even-n_peers clause; note PreVote co-tenancy + IR-1/IR-2 in design.md. Then `/kiro-verify-completion`.
- Effort: **S** (~1-2h, docs only). Risk: **Low**. No code change, no regression surface.
**Option B — Doc closure + hygiene.** Option A plus: (1) startup config validation for even n_peers with a
loud error, (2) retarget test-only `handle_request_vote` onto raft_on_vote_request (kill R-DUP), (3) convert
IR-2 `(void)rc` to an assert.
- Effort: **M** (~half day). Risk: **Low-Medium** (touches ha.c test seam + config load; rerun full
controller + lib test suites).
**Option C — Defer, accept as-is.** Ship implemented; track R-DOC/R-DUP/R1 as backlog. The only real
correctness item (C2) is already fixed and tested, so functional risk of deferring is minimal.
- Effort: **none**. Risk: **Low** functionally; **Medium** for spec/code traceability drift over time.
### Validation To Re-Run
- **V1:** `/xce-build lib` and `/xce-build controller` with `ASAN_OPTIONS=detect_leaks=0` — no build dir
present in the worktree at audit time; the test_raft.c suite (209 cases) and controller tests should be
re-run to confirm green on the current tree (which now includes the PreVote additions + C2 fix, post-dating
the last recorded task-7 ASAN run).
**Bottom line:** raft-library is functionally complete — 24/24 met, the sole prior correctness gap (C2) is
fixed and regression-tested, and the production dual-vote-path risk is eliminated (legacy path is test-only).
Remaining work is documentation/hygiene (Option A, S effort, Low risk) plus a clean ASAN re-run (V1).