myxce/.kiro/specs/raft-library/research.md

Research & Design Decisions: raft-library

Discovery Scope

• Type: Complex new library (brownfield extraction from ha.c)
• Process: Full discovery — codebase analysis of ha.c, ha.h, test files

Key Findings

Existing Election State (ha.c)

• ha_state_t contains: term (atomic uint64), voted_for_term, voted_for_name[XCE_MAXNAME_LEN], role (atomic int32)
• ha_elect_ctx_t holds per-election: n_peers, votes_granted, n_replied, max_resp_tag, max_resp_lsn, done_posted, mutex, semaphore
• Transport coupling: ha_start_election_async directly calls endpoint_connect(), iomgr_monitor(), SSL_new()
• Post-election coupling: ha_elect_evaluate_and_signal calls cluster_init_phase2(), push_config_to_standbys()

Root Cause of Split-Brain

• Self-vote was not recorded atomically at election start; voted_for remained empty until a peer vote arrived
• This allowed node B to grant its vote to node A AND node C in the same term if A and C both started elections before B processed either request
• Fix: record voted_for = self_idx immediately in raft_start_election, before sending any RPC

Build vs. Adopt Decision

• No pure-C Raft library suitable for libxce: hashicorp/raft (Go), etcd/raft (Go), no active C alternative
• Decision: build a minimal Raft election library in src/lib/raft/
• Scope is election only (not log replication), which is simpler than full Raft

Synthesis Outcomes

1. Generalization: Requirements 2–6 are all specializations of a single state machine with callbacks. One raft_state_t + raft_callbacks_t pattern covers all cases.
2. Build vs. Adopt: Build; no suitable C library exists.
3. Simplification: No threads inside the library. Synchronous callbacks. One header, one source, one test file. raft_sim.c compiled only under XCE_TESTING.

Priority Jitter Design

• The delay formula (priority_idx * election_timeout_ms) is a protocol-level decision, not transport
• Placed inside the library as raft_on_leader_lost → on_start_election_after_delay callback
• Caller implements the actual timer/sleep

Tracing Decision

• Added on_trace callback to raft_callbacks_t for full protocol observability
• Simulator dump log enables post-mortem analysis of split-brain scenarios
• raft_snapshot() provides safe point-in-time state read for crash/failure logging

---

Cross-Spec Gap Audit (2026-06-08)

Batch /kiro-validate-gap across 9 specs on persistent_ha branch.

Analysis Summary

• raft-library is ~100% complete. All 7 task groups (1–7), all subtasks (1–2.2, 3.1–3.5, 4.1–4.2, 5.1–5.4, 6.1–6.4, 7) are [x] in tasks.md. The ready_for_implementation flag is set in spec.json.
• No remaining gaps: The library (src/lib/raft/raft.h, raft.c, raft_sim.c) exposes raft_state_t + raft_callbacks_t with 9 callback hooks. The deterministic simulator (raft_sim.c) verifies split-brain freedom across 100 randomized rounds. The controller adapter in ha.c delegates all election state management (term, voted_for, quorum evaluation) to the raft library.
• Cross-spec integration: election-iomgr-refactor uses raft_start_election instead of the old am_i_primary_quorum. The adapter callbacks in ha.c wire on_send_vote_request → iomgr async connect, on_become_leader → cluster_init_phase2 + config push, on_become_follower → CTRL_STANDBY. The on_vote_eligible callback implements the freshness gate (config_tag + applied_lsn comparison) and role gate (CTRL_PRIMARY_COMMITTING → deny).
• Test coverage: 34+ Unity test cases across 5 groups (A–H) covering basic election correctness, voted_for invariant, eligibility policy, quorum evaluation, election timeout, RDHUP failover, and split-brain prevention. All pass under ASAN.
• Recommendation: The spec can be marked implemented.