research(raft-ha-core-safety): Phase-4/5 stall root cause — RDHUP-flap spurious failover

Post-Phase-2 chaotic wedge root-caused from frozen soak logs: primary stable
at term 6, but recovered standby ctrl-2 (divergent uncommitted tail 12-15)
cannot reconverge; its primary control channel RDHUP-flaps ~15s, each flap
spawns an unwinnable PreVote failover (peers see primary alive, no grant) ->
timeout -> reconnect -> repeat, so the cluster never reaches stable 3-of-3.
Fix spans Phase 4 (truncate divergent tail + catch up like ctrl-3 does) and
Phase 5 (don't launch unwinnable failover storm on transient RDHUP while a
current-term primary is reachable; reconnect+catch-up first). RED direction
recorded. (Dispatched investigator hung; reconstructed by parent.)

.kiro/specs/raft-ha-core-safety/research.md

@@ -222,3 +222,31 @@ Rebuilt the release controller at HEAD 63b9c5a (2.2 payload-aware guard + 2.4 st
 2. **Phase 4 (standby reconvergence)** — lagging members must finish catching up so the tail `MISSING` resolves once the cluster stabilizes.
 
 **Steering implication:** Phase 3 (snapshot-boundary reconcile) is defense-in-depth for a path NOT implicated in the live repro; Phases 5 and 4 are on the critical path to a clean soak verdict. Consider doing 5 (and 4) before 3.
+
+---
+
+# Phase 4/5 — Post-Phase-2 stall root cause (investigation, salvaged from a hung run)
+
+Reproduced the chaotic-soak wedge on real-fs release binaries (HEAD with Phase-2 fixes). The committed/payload prefix is clean (Phase-2 holds). The remaining wedge is a **reconvergence-triggered spurious-failover loop**, NOT a simple missing-election-restart.
+
+## Evidence (frozen `~/tmp/ha-estall` logs)
+- **ctrl-1 = stable primary at term 6**: `became_leader`, repeated `repl_broadcast` / `term=6 event=log_append`. The cluster HAS a leader.
+- **ctrl-3 reconverges (slow but works)**: `term=6 ae_recv → ae_reject_logmismatch` ×4, then `ae_already_have`. The AE log-matching backtrack converges for a short divergent suffix.
+- **ctrl-2 = the stall**. It recovered as a former primary with an uncommitted higher tail (`durable_floor=15 current_lsn=16`, entries 12-15 at an old term) vs the leader's committed 11. Its log shows a ~15s flap cycle, repeating indefinitely:
+  1. `handle_ctrlhello: primary control channel established (fd=6)` → receives heartbeats → re-attaches CTRL_STANDBY.
+  2. ~12-15s later: `ha_on_primary_ctrl_fd_rdhup: primary_ctrl_fd RDHUP → begin PreVote failover (Req 1.1)`.
+  3. `ha_start_prevote_async (prospective term=6)` → `ha_prevote_round_timeout_fn: PreVote round timed out before a decision — treating as LOSS, staying CTRL_STANDBY` (NO `pre_vote_reply_in` — peers see the primary alive and don't grant).
+  4. Primary reconnects → loop repeats. (Last heartbeat and last prevote-timeout share the same timestamp — the channel is flapping, not cleanly down.)
+
+## Root cause
+The primary↔ctrl-2 control channel keeps **RDHUP-flapping** (~15s period). Because "control-channel RDHUP = primary died → failover" is the designed trigger (CLAUDE.md), each flap spawns a spurious PreVote failover that **cannot win** (the primary is genuinely alive, so the other voters don't grant), times out, and re-arms. The likely driver of the flap: ctrl-2 cannot reconverge its **divergent uncommitted tail (12-15)** against the leader — the catch-up/AE path stalls or the leader drops the channel — so ctrl-2 never settles as a follower and the channel cycles. Net: the recovered node never rejoins → no stable 3-of-3 → soak `wait_for_full_convergence` times out (the observed wedge).
+
+## What the fix must address (Phase 4 + Phase 5, intertwined)
+1. **Reconvergence (Phase 4 / R5):** a standby that recovers with a divergent *uncommitted* tail must truncate it (follower-AE-conflict-truncate contract) and catch up to the leader — the same way ctrl-3 does for a short suffix — so the control channel stays up. Determine why ctrl-2's longer/former-primary divergent tail does not reconcile while ctrl-3's short one does (candidate: the backtrack doesn't reach far enough before the channel drops, or the former-primary recovery advertises a tip that the leader's nextIndex backtracking never crosses).
+2. **Spurious-failover suppression (Phase 5 / R6/R4):** a transient/flapping control-channel RDHUP must not spawn an unwinnable PreVote storm while the primary is demonstrably alive (peers still see it). A standby should prefer to **reconnect + catch up** on RDHUP before (or instead of) launching failover, or back off failover when its own PreVotes repeatedly get no quorum because a current-term leader exists. (The retry itself works — R6 retry is present — but it retries an election that can never win instead of rejoining.)
+
+## RED test direction (5.1 / 4.1)
+- Force a recovered standby holding a divergent uncommitted tail (former-primary shape, tip above the new leader's committed) to rejoin a cluster with a live higher-term primary, and assert it **reconverges to follower (truncates tail, catches up) without entering a PreVote-failover loop** — i.e. no repeated `ha_begin_failover`/`PreVote timed out` while a current-term primary is reachable. This is deterministically constructible with the 1.4 kill/election + apply-gap seams plus a divergent-tail fixture.
+
+## Process note
+The dispatched investigation subagent backgrounded the soak and hung (~no findings returned); this section was reconstructed by the parent directly from the frozen `~/tmp/ha-estall` controller logs.