docs(specs): gap-analysis validation pass across all 9 HA-family specs

Appended "Gap Analysis — Validation Pass (2026-06-09)" to each spec's
research.md with requirement-to-asset maps, status tags, and effort/risk.

Key outcomes:
- ha-repl-hardening: REOPENED with task series 7.1-7.7 — replication
  payload data plane is missing (snapshot chunks and WAL catch-up ship
  header-only; standbys converge on LSN without holding entity state)
- worker-join-refactor, resval-id-decoupling: marked implemented
  (all criteria re-verified, full test suite green)
- buglog: 335/336 (scheduler dequeue, open), 337/338/339 (fixed in
  previous commit); cerebrum Decision Log corrected (term-only vote
  deny was recorded as intentional but was a regression)

.kiro/specs/raft-library/research.md

@@ -50,3 +50,132 @@
 - **Cross-spec integration**: `election-iomgr-refactor` uses `raft_start_election` instead of the old `am_i_primary_quorum`. The adapter callbacks in ha.c wire `on_send_vote_request` → iomgr async connect, `on_become_leader` → `cluster_init_phase2` + config push, `on_become_follower` → `CTRL_STANDBY`. The `on_vote_eligible` callback implements the freshness gate (config_tag + applied_lsn comparison) and role gate (`CTRL_PRIMARY_COMMITTING` → deny).
 - **Test coverage**: 34+ Unity test cases across 5 groups (A–H) covering basic election correctness, voted_for invariant, eligibility policy, quorum evaluation, election timeout, RDHUP failover, and split-brain prevention. All pass under ASAN.
 - **Recommendation**: The spec can be marked `implemented`.
+
+---
+
+# Gap Analysis — Validation Pass (2026-06-09)
+
+> Scope: requirements.md (9 requirement groups, 31 acceptance criteria) vs. on-disk working tree
+> (uncommitted changes to `src/lib/raft/raft.c`, `src/controller/ha.c`, `ha.h`, `myapp.c` analyzed as-is).
+> spec.json phase: `tasks-generated`, all tasks.md checkboxes `[x]` except parent rollups 2/3/4/5/6.
+
+## 1. Current State Summary
+
+- Library exists and is build-integrated: `src/lib/raft/raft.h` (276 lines), `raft.c` (290 lines),
+  `raft_sim.c` (207 lines); wired in `src/lib/meson.build` (raft.c at line 108; test_raft.c + raft_sim.c
+  under the test executable at lines 175–176, simulator guarded by `XCE_TESTING`).
+- Test suite: `src/lib/testsuite/test_raft.c` (2990 lines, 47 test functions, 79 RUN_TEST invocations)
+  covering unit tests, simulator tests, and protocol groups A–H plus F/G split-brain scenarios.
+- Controller adapter live: `ha_state_t` embeds `raft_state_t raft` (`ha.h:183`); 9 callbacks implemented
+  in `ha.c` (`s_real_cbs` at ha.c:735); `raft_init` called in ha init (ha.c:854) and re-initialized with
+  real topology by `ha_raft_reinit` (ha.c:~2981–3008).
+- Production dispatch fully delegated: vote requests → `raft_on_vote_request` (cluster.c:373),
+  vote replies → `raft_on_vote_reply` (ha.c:2361), RDHUP → `raft_on_leader_lost` (ha.c:3290),
+  election start → `raft_start_election` (ha.c:3146).
+
+## 2. Requirement-to-Asset Map
+
+| Req | Criterion | Asset(s) | Status |
+|---|---|---|---|
+| 1.1 | Per-node state (term/role/voted_for/n_peers) | `raft_state_t` (raft.h:23–38) | Implemented |
+| 1.2 | Init defaults term=0/FOLLOWER/-1 | `raft_init` (raft.c:13–55); H2 test | Implemented (extended: restores `initial_term`/`initial_voted_for` from config) |
+| 1.3 | No endpoint_t/SSL/iomgr/cluster_t/capnp in header | raft.h includes only errno/stdint/string (+stdio under XCE_TESTING) | Implemented |
+| 2.1 | start_election: term++, self-vote, CANDIDATE, send RPCs | raft.c:57–104 | Implemented (also persists self-vote before RPCs — stronger than spec) |
+| 2.2 | Single-node immediate LEADER | raft.c:83–88; tests A1, single_node | Implemented |
+| 2.3 | -EALREADY if CANDIDATE/LEADER | raft.c:60–61; tests already_candidate/leader | Implemented |
+| 3.1 | term > current: upgrade, clear vote, FOLLOWER, evaluate | raft.c:121–130; B2 | Implemented (extra: fires `on_become_follower` if was LEADER — split-brain fix, uncommitted change) |
+| 3.2 | term < current: deny, return current term | raft.c:110–115; B3 | Implemented |
+| 3.3 | One vote per term | raft.c:157–164; B1/B4/B5 | Implemented (extension: idempotent re-grant to the SAME candidate same term — beyond spec text; see Gap-3) |
+| 3.4 | Freshness gate (config_tag/LSN) before grant | Split: caller-side gate in cluster.c:352–356 + `on_vote_eligible` hook (raft.c:167–173) | Partial — behavior present but located in the dispatcher, not in `on_vote_eligible` as tasks.md 6.2 states; `raft_cb_vote_eligible` (ha.c:202) intentionally skips freshness. Doc/code drift. |
+| 3.5 | LEADER or COMMITTING deny ALL vote requests regardless of term | raft.c:135–148 (same-term deny only); COMMITTING gate in `on_vote_eligible` adapter | Partial (intentional deviation) — higher-term request now steps the leader down instead of denying, to prevent the COMMITTING-timeout split-brain. Requirement text is outdated. |
+| 4.1 | Quorum (n/2)+1 → LEADER, on_become_leader once | raft.c:222–234; D1, A4 | Implemented |
+| 4.2 | Reply term > current → FOLLOWER + on_become_follower | raft.c:197–206; D4 | Implemented |
+| 4.3 | Reply after decided → no-op | raft.c:209–213; D3 | Implemented |
+| 4.4 | All replied, no quorum → FOLLOWER | raft.c:237–244; D2 | Implemented |
+| 5.1 | leader_lost: clear vote, FOLLOWER, delayed election (priority×timeout) | raft.c:268–276; F1 | Implemented |
+| 5.2 | Skip election if voted for non-self during delay | `raft_check_and_start_election` raft.c:280–284; F2 | Implemented |
+| 5.3 | Else start election internally | raft.c:286–289; F3 | Implemented |
+| 6.1 | start_election schedules timeout | raft.c:101; E3 | Implemented |
+| 6.2 | Timeout undecided → on_become_follower, cancel tracking | raft.c:260–265; E1 | Implemented |
+| 6.3 | Timeout after decided → on_cancel_timeout no-op | raft.c:252–257; E2 | Implemented |
+| 7.1 | raft_callbacks_t with 6 required fields | raft.h:66–96 (9 fields: 7 required + 2 optional) | Implemented (superset: adds `on_start_election_after_delay` as required, `on_vote_eligible`/`on_trace` optional) |
+| 7.2 | void* ctx passed to every callback | raft_config_t.ctx → s->ctx, all call sites | Implemented |
+| 7.3 | NULL required callback → error at init | raft.c:23–38 (-EINVAL); Hnull test | Implemented |
+| 8.1 | Deterministic N-node simulator, synchronous routing | raft_sim.c + raft.h:197–274; sim tests | Implemented |
+| 8.2 | Invariants: one-leader/3-node, dup deny, single-node, timeout re-election | A2/G1 (100 rounds), B1/B4, A1, E3 + G2 partition | Implemented |
+| 8.3 | Compile + all tests pass under ASAN | tasks.md task 7 `[x]`; commits e10cf9a/361397a fixed ASAN issues | Unknown (stale) — last verified pre-dating current uncommitted edits to raft.c/ha.c; needs re-run |
+| 9.1 | Adapter implements all callbacks via endpoint_t/iomgr | ha.c:107–744 (8 raft_cb_* + s_real_cbs) | Implemented |
+| 9.2 | ha_start_election_async delegates; no direct ha->term/voted_for writes outside adapter | ha.c:3038–3161 delegates; only atomic loads + persist/restore remain | Partial — production path clean, but legacy `handle_request_vote()` (ha.c:1791–1873) still contains a full parallel vote-evaluation path with direct `ha->voted_for_term`/`voted_for_name` writes. It is production-dead (only `test_ha_vote_gate.c` calls it) but duplicates election state management contrary to the spirit of 9.2. |
+| 9.3 | ha_phase2_hook / ha_rdhup_election_hook still functional | ha.c:495/631/2068/2147 (phase2), ha.c:3267–3284 (rdhup) | Implemented |
+
+**Score: 27 of 31 acceptance criteria Implemented; 3 Partial (3.4, 3.5, 9.2 — all deliberate-deviation or residual-cleanup, not functional regressions); 1 Unknown (8.3 ASAN re-verification against the uncommitted working tree).**
+
+### Constraints / behavior added beyond requirements
+- **C1 (Constraint)**: `raft_init` rejects even `n_peers > 1` with -EINVAL (raft.c:21–22). Not in
+  requirements. A 2-node or 4-node HA deployment cannot initialize raft at all — verify this matches
+  deployment expectations (`deploy-ha.sh`, cluster config validation) or it becomes a runtime landmine.
+- **C2 (Constraint)**: `voted_for_term` is unconditionally reset to 0 in `raft_init` (raft.c:43) even when
+  `initial_voted_for`/`initial_term` are restored from disk. After a restart, the duplicate-vote check
+  (`voted_for_term == term`, raft.c:157) cannot match the restored term, so a node that voted for X in
+  term T, restarted, then receives a request from Y in the same term T would fall through Step 4 and could
+  grant a second vote in T (subject to the caller-side freshness gate). Test H1 checks `term`/`voted_for`
+  but not `voted_for_term`. Potential one-vote-per-term violation across restart — needs confirmation.
+
+## 3. Implementation Approach Options (for the remaining gaps)
+
+### Option A: Extend existing components (close gaps in place) — RECOMMENDED
+- raft.c: initialize `voted_for_term = (initial_voted_for >= 0) ? initial_term : 0` in `raft_init` (C2);
+  add an H6 restart-double-vote test.
+- requirements.md: amend 3.5 (higher-term step-down is the intended behavior) and 3.4 (freshness gate
+  enforced by the dispatcher/callback split) so spec matches the as-built design; sync tasks.md 6.2 text.
+- ha.c: delete or fold legacy `handle_request_vote()` into the raft path; retarget `test_ha_vote_gate.c`
+  at `raft_on_vote_request` + `on_vote_eligible` (closes 9.2).
+- Re-run `/xce-build lib` + `/xce-build controller` under ASAN to refresh 8.3.
+- Trade-offs: ✅ smallest diff, no architectural change, leverages existing test harness; ❌ touches a
+  legacy test file; requirements edit needs human re-approval per workflow.
+
+### Option B: Create new components (formal vote-record module)
+- Introduce a `raft_persisted_vote_t {term, voted_for}` pair persisted/restored atomically, replacing the
+  separate initial_term/initial_voted_for fields; new restart-focused test group.
+- Trade-offs: ✅ makes the term-binding of a vote impossible to lose by construction; ❌ API churn in
+  raft_config_t and ha_raft_reinit for a problem fixable with one init line; over-engineering (YAGNI).
+
+### Option C: Hybrid — fix C2 in place now (Option A), defer handle_request_vote removal to a follow-up
+  cleanup spec/task since it is production-dead.
+- Trade-offs: ✅ de-risks the correctness fix from the test-refactor churn; ❌ leaves known duplication
+  in tree longer; two passes of review.
+
+## 4. Effort & Risk
+
+| Gap | Effort | Risk | Justification |
+|---|---|---|---|
+| C2 voted_for_term restore fix + test | S | Low | One-line init change + one Unity test in an established harness |
+| Requirements/tasks doc sync (3.4, 3.5) | S | Low | Documentation only; needs approval round-trip |
+| Legacy handle_request_vote removal (9.2) | S–M | Medium | Production-dead but test_ha_vote_gate.c must be retargeted; risk of disturbing controller test baseline |
+| ASAN re-verification (8.3) | S | Low | Mechanical `/xce-build` run; fixes only if new findings |
+| C1 even-n_peers policy confirmation | S | Medium | If 2-node deployments are expected, this is a hard init failure discovered only at runtime |
+
+Overall remaining work: **S–M total, Low–Medium risk** — the library and integration are functionally complete.
+
+## 5. Recommendations for Design/Closure Phase
+
+1. Adopt **Option A**. Fix C2 (`voted_for_term` restore) first — it is the only candidate correctness
+   gap; everything else is documentation or hygiene.
+2. Update requirements.md 3.5 and 3.4 to match the as-built (and better) behavior: leader steps down on
+   higher term; freshness gate enforced caller-side before `raft_on_vote_request` with COMMITTING/liveness
+   in `on_vote_eligible`. Re-approve, then run `/kiro-verify-completion raft-library`.
+3. Remove or refactor legacy `handle_request_vote` (ha.c:1791) to eliminate the last duplicated election
+   state path (Req 9.2), retargeting `test_ha_vote_gate.c`.
+4. Re-run task 7 ASAN validation against the current working tree before committing (the verified state
+   predates the uncommitted raft.c Step-3/Step-4 rewrite).
+
+## Research Needed
+
+- **R1**: Confirm intended cluster-size policy — are 2-node (even) HA clusters a supported deployment?
+  If yes, the even-n_peers -EINVAL in raft_init must be relaxed; if no, add config-load validation so the
+  failure surfaces at startup with a clear message rather than as a silent raft_init failure.
+- **R2**: Verify whether the restart double-vote scenario (C2) is reachable in production timing — does
+  `ha_raft_reinit` run before the listener accepts vote requests? If unreachable, document why; fix anyway.
+- **R3**: Confirm `raft_check_and_start_election` ignoring the `raft_start_election` return code
+  (raft.c:287–289) is safe in all adapter paths (comment claims -EALREADY unreachable; verify when an
+  inbound vote request flips role between RDHUP and the delayed check).