myxce/.kiro/specs/raft-ha-core-safety/research.md

Gap Analysis — raft-ha-core-safety

Generated by /kiro-validate-gap. Information over decisions: options and findings, not final choices.

Headline Finding (reframes the spec)

The (term, lsn) commit-identity + reconcile + commit-safety machinery the requirements describe already exists and is wired at the baseline. Confirmed:

• Library half is 100% complete and validated: raft-log-replication (24/24, G1/G2/G3/G5) and raft-commit-safety (18/18, G4 Figure-8 + G6 vote freshness). All five callbacks present in src/lib/raft/raft.{c,h}.
• Controller adapter is 16/20 wired by ha-term-safety: all five callbacks implemented in s_real_cbs (ha.c:1480/1539/1656/1711/1739), primary append routed through raft_log_append (ha.c:3408+), heartbeat clean (no on_persist_entry on n_entries==0), per-LSN term index (persist.c:414/476/487/752), config markers, and walcompare (src/controller/testsuite/walcompare.c).
• Baseline 21e26a1 already contains all of it. git: 21e26a1 is itself a raft-drive-async task-2.3 commit, ancestor of HEAD, and contains s_real_cbs + walcompare. So the machinery is NOT absent at baseline.

Yet the live chaotic-failover soak still fails on this branch AND on baseline 21e26a1: walcompare FAIL: committed-prefix divergence at LSN 9 (round 3) and standby ctrl-2 rejoin failure (round 4), with intermittent round-N election stall.

Therefore raft-ha-core-safety is NOT a build-the-machinery spec. It is a residual-defect + convergence/liveness + live-verification spec. The work is: root-cause why divergence survives the existing reconcile path under overlapping chaotic failover; close the convergence and election-liveness gaps no completed spec actually closed; and finish the live chaotic soak that ha-term-safety left permanently BLOCKED on env. Do not rebuild the (term,lsn)/callback machinery — extend and correct it.

Requirement-to-Asset Map

Req	Capability	Existing asset	Gap tag
R1 Committed-prefix safety	quorum commit + Figure-8 guard	raft_advance_commit raft.c:769-821; on_commit_advanced ha.c:1739	Constraint — machinery present but live soak still diverges at LSN 9 → residual defect
R2 Per-entry commit identity	term stamped from leader; per-LSN term index; legacy-0→1	raft.c:571-580; persist_set_term persist.c:476; record_entry_term persist.c:414/752; WAL_LEGACY_TERM persist.c:456	Done (verify) — confirm no path still emits term-0 under chaos
R3 Divergence detect + repair	logOk + conflict-truncate-then-persist on AE intake	raft.c:701-765; on_truncate_from ha.c:1711; walcompare detector	Constraint — repair fires on AE-intake path only; rejoin/catch-up path may bypass it (see Research Needed #1)
R4 Single leader per term	term++ on election, vote-once, step-down, pre-vote, vote freshness	raft.c:105-249; raft_cb_vote_eligible ha.c:342	Constraint — two-leaders-per-term window under churn not yet proven closed live
R5 Standby rejoin → 3-of-3	catch-up apply, applied-lsn floor, snapshot backfill, contiguity guard	ha_apply_wal_catchup_entry ha.c:2233; ha_advance_applied_lsn ha.c:2078; ha_handle_snapshot_chunk ha.c:2492; contiguity persist.c:839	Missing (effective) — bug-1042 standby-never-converges has NO dedicated owning task; ha-convergence-recovery claims it but soak still fails round 4
R6 Election liveness under churn	election restart, candidate advertise floor	raft_start_election raft.c:105; ha_candidate_advertise_lsn ha.c:3921	Missing (effective) — round-N stall not closed; no deterministic churn test
R7 RED→GREEN harness	xce-test-ha soak, walcompare, controller_unit_test, test_ha_dispatch_integration	.claude/skills/xce-test-ha/; walcompare.c; src/xce.build/controller/controller_unit_test	Missing — no deterministic harness that forces chaotic divergence/rejoin/stall as a RED unit/integration gate (current soak is probabilistic)
R8 Acceptance soak gate	real-fs release soak; BIN/XT overridable	run-ha-soak.sh (BIN/XT env override added this branch); non-ASAN release build	Unknown — never passed at 6+ chaotic rounds; was BLOCKED on overlayfs fdatasync (bug-924)

Existing-Code Constraints (must respect)

• Reconcile is single-seam: divergence repair happens only inside raft_on_append_entries_request (logOk → on_truncate_from → re-persist, raft.c:730-765). Any code path that brings a node's log forward without routing through this function (e.g. a snapshot/catch-up stream that writes WAL directly) will not detect or repair a divergent committed prefix.
• Truncate must never cut the majority-committed prefix (raft.c monotonic commit; promotion truncate-clamp max(safe_committed_lsn, durable_tip) ha.c:999-1000, bug-1060 fix). R3.4 depends on this clamp holding under chaos.
• Persist-before-ACK durability is owned by ha-commit-durability, not this spec. R1/R2 must not weaken it. on_commit_advanced is a barrier trigger, not the durability mechanism.
• raft-drive-async is out of scope and dormant. Do not change SlowLane / peer-control dispatch / endpoint-quiescence behavior (Boundary Context).
• Client-side bug-854 (libxce primary discovery) can fail the soak harness at EXIT-1 even when server data safety passes. Must be isolated so it does not mask/forge R8 verdicts.

Implementation Approach Options

Option A — Extend & correct existing machinery (recommended)

Keep all library + adapter machinery. Root-cause the residual divergence, route the rejoin/catch-up path through the reconcile seam, close the convergence + liveness gaps with targeted fixes, add a deterministic chaotic harness, finish the live soak.

• ✅ No rebuild of consensus-critical code that already passed library + clean-failover validation
• ✅ Smallest blast radius; aligns with TLA+ already-proven invariants
• ❌ Requires precise root-cause of a hard-to-repro chaotic defect before fixing
• ❌ Risk of "fix the symptom not the invariant violation" if harness stays probabilistic

Option B — New convergence/divergence-repair subsystem

Build a standalone post-failover reconcile/repair component separate from the AE-intake seam.

• ✅ Clean place to own rejoin reconcile + unrecoverable-divergence fencing
• ❌ Duplicates logic that already lives in raft_on_append_entries_request; two reconcile paths is exactly the class of bug being fixed
• ❌ High risk of new two-leaders/two-reconcile divergence

Option C — Hybrid (Option A + new deterministic harness component)

Extend/correct in place (A), but build the deterministic chaotic harness (R7) as genuinely new test components (forced overlapping kills, injected divergent prefix, scripted round-N stall) rather than relying on the probabilistic soak.

• ✅ Keeps consensus code as A; isolates the new surface to test infrastructure where new code is safe
• ✅ Gives a RED gate that pins each fix to an invariant violation
• ❌ Harness design is itself non-trivial (deterministic chaos, no sleeps, gate/probe seams like test_ha_dispatch_integration.c)

Effort & Risk

Item	Effort	Risk	Justification
Root-cause residual LSN-9 divergence	M	High	Chaotic, intermittent; survives existing reconcile — invariant violation is subtle
Route rejoin/catch-up through reconcile seam (R3/R5)	M	High	Touches recovery + commit prefix; must not cut majority-committed
Election liveness under churn (R6)	M	Medium	Election machinery exists; needs churn-stall root cause + restart guarantee
Deterministic chaotic harness (R7)	L	Medium	New, but test-only; pattern exists (test_ha_dispatch_integration.c)
Live 6+round real-fs soak (R8)	M	High	Was permanently blocked on overlayfs fdatasync (bug-924); needs real-fs env + bug-854 isolation
Total	L–XL	High	Consensus correctness under chaos + env-gated live verification

Research Needed (carry to design phase)

1. Why does divergence survive the reconcile seam? Trace the rejoin/catch-up path (ha_apply_wal_catchup_entry ha.c:2233, snapshot ha.c:2492): does it route through raft_on_append_entries_request so logOk/on_truncate_from fire, or does it write WAL directly and bypass conflict repair? Hypothesis: catch-up streams [watermark..committed] directly (per memory bug-795/828 family) → no (term,lsn) conflict check on rejoin.
2. Is LSN-9 divergence in the committed or uncommitted prefix? If committed, the violation is upstream (two-leaders-per-term or premature commit-advance), not a repair gap. Use walcompare (lsn→term,pcrc) output to classify.
3. Two-leaders-per-term under overlapping kills — does pre-vote + vote-once fully close the window when kills overlap election timeouts? Map against raft/raft.tla Election Safety.
4. bug-1042 ownership — ha-convergence-recovery claims the rejoin path but soak still fails round 4. Determine whether its tasks are actually landed on this branch or stranded, to avoid duplicate work.
5. R8 env — confirm real-fs (ext4) soak runs now via run-ha-soak.sh BIN/XT override + non-ASAN release build; quantify how bug-924 (overlayfs) and bug-854 (client discovery) are excluded from the R8 verdict.

Recommendation for Design Phase

• Preferred: Option C (extend/correct in place + new deterministic chaotic harness). Treat the spec as residual-defect + convergence + liveness + live-verification, explicitly not a machinery rebuild.
• Key design decisions to make: (a) route rejoin/catch-up through the single reconcile seam vs. add a guarded repair at promotion; (b) classify LSN-9 as committed vs uncommitted before choosing repair vs upstream fix; (c) harness determinism mechanism (forced-kill gates/probes) for R7; (d) how R8 isolates bug-854/bug-924 so the gate measures only server-side committed-job safety.
• Carry forward: Research Needed #1–#5; verify every change against raft/raft.tla (Election Safety, Leader Completeness, State Machine Safety, Commit Ordering/Durability) per src/controller/CLAUDE.md.

---

Design-Phase Discovery (kiro-spec-design)

ROOT CAUSE (term-blind catch-up bypass) — ⚠️ SUPERSEDED / REFUTED by task 1.3

This hypothesis was tested live (task 1.3) and REFUTED. The real defect is cross-term catch-up payload loss (see §"Task 1.3 — Live divergence reproduction & bisect" below). The divergent entry carries the correct term and arrives through the reconcile seam; the term-blind path is a latent hazard but not the observed cause. Retained below for the diagnostic record.

Deep-trace of the standby catch-up/rejoin paths found a split where one branch bypasses the single reconcile seam:

Path	Entry	Routes through raft_on_append_entries_request?	logOk	conflict-truncate	term stamped
1 Term-carrying	entry->term != 0 → ha_apply_term_entry_via_library (ha.c:2173) → ha_consume_append_entries (ha.c:2383) → raft.c:646	YES	YES (raft.c:701)	YES (raft.c:735)	leader term via persist_set_term
2 Term-blind (defect)	entry->term == 0 → ha_apply_single_entry (ha.c:2097) → persist_wal_write_replicated (persist.c:802) DIRECT	NO	NO	NO	standby's own p->term (persist.c:862)
3 Snapshot	ha_handle_snapshot_chunk (ha.c:2492) → persist_load_snapshot sets applied_lsn=checkpoint	NO	N/A	N/A	N/A (canonical state)
4 Stream+ACK	ha_stream_wal_to_standby (config_repl.c:2032) feeds Path 1 or 2 by entry->term	depends	depends	depends	depends
5 ha_rejoin_reconcile	ha_convergence.c:254 — now a no-op stub (committed-watermark truncate retired; relies on library)	NO	N/A	retired	N/A

Divergence mechanism (LSN 9):

1. An entry reaches a standby via Path 2 (term-blind). persist_wal_write_replicated stamps hdr.term = standby p->term (wrong) and records it (record_entry_term, persist.c:902). No logOk, no conflict check.
2. The correct term-carrying entry for the same LSN later arrives via Path 1. The persist idempotency guard (if lsn+1 <= p->current_lsn return 0, persist.c:822) silently no-ops — the divergent entry is never overwritten/repaired.
3. walcompare (testsuite/walcompare.c:135) compares (term, payload_crc) per committed LSN; the member with the term-blind write shows a different term → WALCMP_DIVERGENT. match_term maps raw 0 → WAL_LEGACY_TERM, so legacy-0 vs genuine-term at the same LSN reads as divergence.

Why the existing (term,lsn) machinery does not catch it: all conflict detection lives inside raft_on_append_entries_request; Path 2 never enters it, and the idempotency guard treats "LSN already present" as "done" regardless of term.

Synthesis

⚠️ D1/D3 partially superseded by task 1.3. The "eliminate every bypass of the one seam / collapse Path 2 into Path 1" framing assumed the term-blind hypothesis, now refuted. The single-seam invariant (D2) and harness-first discipline (D4) still hold; but the actual fix is the cross-term catch-up streamer payload integrity + a (term,lsn,payload-crc)-aware guard (see §"Task 1.3"), not seam-bypass elimination. Term-blind retirement remains as defensive hardening only.

• Generalization (D1): R1/R2/R3/R4 are one underlying invariant — every entry that enters a member's log must pass the single (term,lsn) reconcile seam. The fix is not four mechanisms but eliminating every bypass of the one seam + making persistence (term,lsn)-conflict-aware rather than LSN-only-idempotent.
• Build vs adopt (D2): Adopt the existing library seam (raft_on_append_entries_request + on_truncate_from + persist_set_term) — already validated (raft-log-replication/raft-commit-safety). Build nothing new in the consensus core; only route into it and harden the persist guard. Reuse ha_rejoin_reconcile intake hook (currently stub) rather than a second reconcile path.
• Simplification (D3): Do not add a separate "divergence repair subsystem" (gap-analysis Option B rejected) — that would create the exact second-reconcile-path class of bug being fixed. Collapse Path 2 into Path 1.
• Harness-first (D4): The live bug is intermittent/chaotic. Make it deterministic: a RED unit test injects a term-blind/divergent entry at an LSN and asserts the seam repairs it; a chaotic soak mode forces overlapping kills. RED before any fix (R7), pins each fix to a walcompare/invariant violation.

Open risks carried to tasks

• Confirm Path 2 is the only divergence source live (harness RED test will confirm/refute; if soak still diverges after Path-2 closure, classify next via walcompare (term,lsn,pcrc)).
• Snapshot boundary (Path 3): post-install AE intake must reconcile against the snapshot's (term,lsn); verify no committed entry below the checkpoint can diverge.
• Standby reconvergence (R5) and election-liveness (R6) are separable seams from divergence-repair; phaseable but kept in one spec per the unified acceptance gate.
• bug-854 (client discovery) and bug-924 (overlayfs fdatasync) must be excluded from the R8 verdict so the gate measures server-side committed-job safety only.

---

Design Review Resolution (kiro-validate-design)

Three critical issues raised and folded into design.md (GO):

1. Repair-vs-clamp conflation (correctness). Split truncate into two contracts: promotion-truncate keeps the bug-1060 max(safe_committed_lsn, durable_tip) clamp; follower-AE-conflict-truncate obeys Raft Log Matching and truncates the conflicting suffix regardless of local durable_tip (a conflicting entry was never majority-committed → safe to cut). Applying the promotion clamp on the follower path would have converted repairable divergence into a permanent fence and failed R8. Updated: §Existing Architecture, §Unified Entry Ingest, §Persist Conflict Guard, §Error Handling, Traceability 3.x, Testing.
2. R4 two-leaders-per-term unverified. Added a falsifiable Election Safety assertion (≤1 committing-primary per term under overlapping kills, mapped to raft.tla); failure promotes a concrete adapter-gate guard from verification to required work. Updated: §Election Liveness, Testing Integration.
3. Hypothesis-validation risk. RED gate (7.1) must reproduce the live walcompare divergence and bisect to a seam before synthetic (term,lsn) injection counts; baseline 21e26a1 already retired live term-0 (ha-term-safety 3.4), so the term-blind path may not be the live trigger. Contingency: if soak still diverges after Path-2 closure, re-classify via walcompare (term,lsn,pcrc) and add a fix task — do not mark R1/R8 done on green unit tests alone. Updated: §Deterministic Chaotic Harness, Testing E2E + RED-first.

Verdict: GO (unconditional after edits). Architecture unchanged; refinements were bounded contract clarifications.

---

Task 1.3 — Live divergence reproduction & bisect

Generated by the task-1.3 investigation subagent (REPRODUCE + CLASSIFY + BISECT). Evidence-backed; no production code changed. Real-fs (ext4) ~/tmp, non-ASAN release binaries src/build.release/controller/{xce_controller,xce_test} (built 08:48 this day).

TL;DR / VERDICT

The design's primary "term-blind catch-up bypass" hypothesis is REFUTED for the reproduced live committed divergence. The divergent committed entry has the CORRECT term, was ingested through the single library reconcile seam (ae_append → on_persist_entry), and the LSN-only idempotency no-op was not the mechanism. The real defect is payload loss on cross-term catch-up: a committed JOB_SUBMIT entry is persisted on the catching-up member with payload_len=0 (empty body) while its (term, lsn) are correct. It is a sibling of the bug-795/828/1060 committed-prefix-loss family, located in the catch-up replication payload path, not in the (term,lsn)/idempotency-guard seam Phase 2 currently targets.

Commands run & reproduction status

• Two --chaotic soak attempts via run-ha-soak.sh (rounds=6, XCE_HA_BIN/XT → release binaries, dir ~/tmp/ha-1p3) both aborted at cold-start with FAIL: no primary elected within 45s. A --sequential attempt failed identically.
• This cold-start "election stall" is a HARNESS/LOG-BUFFER ARTIFACT, not a consensus stall. Evidence: in every aborted run a primary WAS elected on time (e.g. ctrl-1 → CTRL_PRIMARY (term=2) logged ~15s after start, quorum_reached), yet wait_for_primary_log never tripped. All three nodes' last log line is the identical cleanup-kill timestamp (= start+45s): the 08:48 release build block-buffers its --logfile and only flushes on signal, so the harness's log-grep primary detection (latest_primary_term) cannot observe progress live. The OLD binaries used by the prior passing sequential soaks (~/tmp/xce-soak-{rel,bl}.log, 02:56/03:01) did not have this buffering behaviour. The client (ctrlstatus) path is unaffected.
• Reproduced cleanly by driving the cluster with a client-based (log-buffer-immune) driver (scratchpad/drive.sh, fresh deploy ~/tmp/ha19, sequential kill→restart→re-elect, 6 rounds). Committed-prefix divergence appeared by round 1 and persisted: round 1 LSN 8 (present=2), round 2 LSN 8 (present=2), round 3 LSN 9 (present=3 — all members hold it but disagree). Driver wedged at round 4 on a transient quorum loss; the frozen WAL dirs were classified directly.

walcompare --report classification (frozen ~/tmp/ha19, --committed 25)

Decisive divergence — committed LSN 9, class=committed, compared=3, present=3 (every member holds the entry; they disagree on payload):

WALCMP_DIVERGENCE lsn=9 class=committed compared=3 present=3
  member 0 (ctrl-1): term=3 pcrc=00000000   <-- MINORITY / divergent
  member 1 (ctrl-2): term=3 pcrc=512dc5b5   <-- majority
  member 2 (ctrl-3): term=3 pcrc=512dc5b5   <-- majority

Same pattern at committed LSN 10 (ctrl-1 pcrc=0 vs majority 06e88dac) and LSN 11 (ctrl-1 pcrc=0 vs majority fea75587). Term is identical (3) across all three members; only the payload-CRC diverges. (ctrl-3 additionally shows status=MISSING for LSN 12-19 — a separate catch-up-lag/convergence gap from the round-4 quorum loss, bug-1042 family, not the payload defect.)

Raw waldump confirms the payload loss, not a term mismatch:

ctrl-1 (divergent):   9 | 3 | JOB_SUBMIT | PLEN=0    | 00000000 | jobid=?
ctrl-2/ctrl-3 (maj):  9 | 3 | JOB_SUBMIT | PLEN=3560 | 512dc5b5 | jobid=0-0-1782550813-0 ...

The origin is intact — both majority members hold the full 3560-byte body with a decodable jobid; ctrl-1 lost the payload bytes during catch-up while keeping the correct type (JOB_SUBMIT) and term (3).

Bisect — the concrete ingest path

ctrl-1's --logfile (fully flushed post-kill) traces exactly how it acquired the divergent entries:

1. ctrl-1 was the term-1 cold-start primary (wrote LSN 1-7), killed round 1, restarted round 2 (persist_recover done — recovered durable_floor=7 current_lsn=8). It then needed LSN 8-onward.
2. 09:00:13 — term-3 leader ctrl-3 streamed catch-up op 37 walCatchupEntry, but starting at prevIdx=8 (entry 9) while ctrl-1 only held up to 7 → library ae_reject_logmismatch, ha_consume_append_entries … success=0. Nothing written. (ctrl-3 then died — round 2 failover.)
3. 09:00:26 — new term-4 leader ctrl-2 streamed op 37 walCatchupEntry from prevIdx=7. The library accepted and APPENDED: RAFT[p0] term=4 event=ae_append, ha_consume_append_entries … prevIdx=7 n=1 → success=1 match=8, then match=9, 10, 11, 12. These were fresh appends through the reconcile seam (on_persist_entry), not idempotent no-ops (current_lsn was 8 immediately prior). LSN 9-11 landed with payload_len=0.
4. In the SAME catch-up, LSN 13-15 (stored term=4, the streaming leader's OWN term) landed with FULL payloads. Signature: same-term catch-up preserves payload; cross-term catch-up of prior-term entries (stored term 3 streamed under leader term 4) drops it.

Code-path mapping (read at the research anchors):

• Receiver ha_consume_append_entries (ha.c:2410-2440) copies the body unconditionally (if (payload && plen>0) memcpy(...), ha.c:2438) from e->raw_payload/e->raw_payload_len. So the empty body arrived empty on the wire → the loss is SEND-side / source-side, not in the seam's persist.
• Both documented send paths nominally read the payload: batch streamer ha_stream_wal_to_standby (config_repl.c:2154-2161, pread when hdr.payload_len>0, and sets entry->term = persist_entry_term(...) + entry->leader_current_term = leader_term, config_repl.c:2130-2151) and the single-entry on_send_append_entries → ha_read_wal_entry_at (ha.c:1611-1640). Neither statically drops a cross-term payload — so the loss is either (a) the streaming leader (ctrl-2) itself held an empty copy at stream time and the full payload was reconciled into ctrl-2's WAL only later (a transitive empty-propagation across the failover chain), or (b) the wire encode/decode of the body for these messages. Distinguishing (a) vs (b) requires one instrumented run capturing per-entry payload_len at the leader's send and at the receiver's on_persist_entry for a cross-term LSN — see "Implied next work".

Why the term-blind hypothesis is refuted here

• Predicted by design.md ROOT CAUSE: entry->term==0 → ha_apply_single_entry (ha.c:2097) → persist_wal_write_replicated DIRECT, wrong term (standby's p->term), idempotency no-op of the later correct entry.
• Observed: the entry's term is correct (3), it was ingested via ha_apply_term_entry_via_library → ha_consume_append_entries → on_persist_entry (the term-carrying LIBRARY seam, event=ae_append), and the catch-up was a fresh append (no idempotency-guard no-op involved). The streamer stamps a non-zero term (config_repl.c:2141), so the term==0 → ha_apply_single_entry branch (ha.c:2253/2291 gate sub->term != 0) is not taken for these entries.

VERDICT

• Term-blind catch-up bypass (wrong-term + LSN-only idempotency no-op): REFUTED as the trigger of the reproduced live committed divergence.
• Actual trigger: payload-loss on cross-term catch-up — committed JOB_SUBMIT entries persisted with payload_len=0 (term/type correct) on a member that catches up prior-term entries from a higher-term leader; the divergence is a payload-CRC mismatch at an identical (term,lsn), in the committed prefix. A member carrying this can (and in round 3 did) become primary, so it is a silent committed-state corruption in the bug-795/828/1060 family.

Implied next work (steers Phase 2)

1. Re-scope the Phase-2 fix away from the term-stamp/idempotency seam. Collapsing Path-2 into Path-1 (design D3/D1) will NOT fix this divergence — the entry already goes through Path-1 with the correct term. The fix must target the catch-up payload source-of-truth: add an instrumented capture of payload_len at ha_stream_wal_to_standby/on_send_append_entries (send) and on_persist_entry (receive) for cross-term LSNs to prove send-side empty vs transitive-empty, then fix whichever drops/propagates the empty body.
2. Harden the persist conflict guard to be (term,lsn,payload)-aware, not term-only: a committed (term,lsn) whose payload-CRC disagrees with the quorum must be detected/repaired (or the member fenced), since term-only reconcile is blind to this class.
3. Fix the divergence detector / RED gate to assert payload-CRC at committed LSN even when terms match (it already does via pcrc; keep walcompare --report (term,lsn,pcrc) as the canonical RED classifier — it caught this).
4. R6/Harness defect (secondary): the release build's log buffering defeats run-ha-soak.sh's log-based wait_for_primary_log/primary_from_log primary detection (false "election stall"). The chaotic acceptance gate (R8) needs client-based (ctrlstatus) primary detection, or the controller must line-buffer/flush its INFO log, before the soak can run unattended on release binaries.

Evidence artifacts (under ~/tmp, ephemeral)

• ~/tmp/ha19/ — frozen 3-member deploy with the reproduced LSN 9/10/11 payload divergence (controller/ctrl-{1,2,3}/data, logs/ctrl-1.log).
• ~/tmp/ha19-drive.log — driver round-by-round output (divergence first seen round 1).
• ~/tmp/ha-1p3-run{1,2}.log, ~/tmp/ha-1p3-seq.log — the cold-start log-buffer "stall" runs.

---

Phase-2 interim live validation (post-2.4, real-fs release rebuild)

Rebuilt the release controller at HEAD 63b9c5a (2.2 payload-aware guard + 2.4 streamer fail-loud) and ran a chaotic soak on real ext4 (~/tmp/ha-verify-2p4). The 4-round run wedged (timed out ~9m40s) on an election stall before reaching its own verdict — the un-fixed Phase-5 liveness defect. Froze the WALs and measured directly with walcompare --report:

• Universally-held committed prefix (LSN 1..11) is content-identical across all 3 members (walcompare: OK — committed prefixes identical).
• No present-but-empty committed divergence — the bug-1.3 signature (one member pcrc=0/PLEN=0 PRESENT where the majority holds a full payload) is absent. The pcrc=00000000 entries at LSN 1/8/12/16 are present+identical on ALL members (legitimate empty-payload marker/NOOP entries, same term+crc everywhere).
• Remaining apparent divergences are all status=MISSING tail entries (ctrl-1 tip=19, ctrl-2=15, ctrl-3=11) — catch-up lag on lagging/killed members, not committed-content corruption. Classifying these as "committed divergence" only happens when an inflated --committed boundary over-claims the committed region; at the true majority-held boundary they are uncommitted tail / in-flight catch-up.

Conclusion: the Phase-2 fix empirically eliminates the cross-term payload-loss divergence on the live repro. A clean end-to-end committed_divergence=0 soak verdict is now gated on:

1. Phase 5 (election liveness under churn) — the chaotic soak wedges on an election stall before completing, so it never reaches its verdict line. This is the top blocker for the acceptance gate (7.2).
2. Phase 4 (standby reconvergence) — lagging members must finish catching up so the tail MISSING resolves once the cluster stabilizes.

Steering implication: Phase 3 (snapshot-boundary reconcile) is defense-in-depth for a path NOT implicated in the live repro; Phases 5 and 4 are on the critical path to a clean soak verdict. Consider doing 5 (and 4) before 3.

---

Phase 4/5 — Post-Phase-2 stall root cause (investigation, salvaged from a hung run)

Reproduced the chaotic-soak wedge on real-fs release binaries (HEAD with Phase-2 fixes). The committed/payload prefix is clean (Phase-2 holds). The remaining wedge is a reconvergence-triggered spurious-failover loop, NOT a simple missing-election-restart.

Evidence (frozen ~/tmp/ha-estall logs)

• ctrl-1 = stable primary at term 6: became_leader, repeated repl_broadcast / term=6 event=log_append. The cluster HAS a leader.
• ctrl-3 reconverges (slow but works): term=6 ae_recv → ae_reject_logmismatch ×4, then ae_already_have. The AE log-matching backtrack converges for a short divergent suffix.
• ctrl-2 = the stall. It recovered as a former primary with an uncommitted higher tail (durable_floor=15 current_lsn=16, entries 12-15 at an old term) vs the leader's committed 11. Its log shows a ~15s flap cycle, repeating indefinitely:
  1. handle_ctrlhello: primary control channel established (fd=6) → receives heartbeats → re-attaches CTRL_STANDBY.
  2. ~12-15s later: ha_on_primary_ctrl_fd_rdhup: primary_ctrl_fd RDHUP → begin PreVote failover (Req 1.1).
  3. ha_start_prevote_async (prospective term=6) → ha_prevote_round_timeout_fn: PreVote round timed out before a decision — treating as LOSS, staying CTRL_STANDBY (NO pre_vote_reply_in — peers see the primary alive and don't grant).
  4. Primary reconnects → loop repeats. (Last heartbeat and last prevote-timeout share the same timestamp — the channel is flapping, not cleanly down.)

Root cause

The primary↔ctrl-2 control channel keeps RDHUP-flapping (~15s period). Because "control-channel RDHUP = primary died → failover" is the designed trigger (CLAUDE.md), each flap spawns a spurious PreVote failover that cannot win (the primary is genuinely alive, so the other voters don't grant), times out, and re-arms. The likely driver of the flap: ctrl-2 cannot reconverge its divergent uncommitted tail (12-15) against the leader — the catch-up/AE path stalls or the leader drops the channel — so ctrl-2 never settles as a follower and the channel cycles. Net: the recovered node never rejoins → no stable 3-of-3 → soak wait_for_full_convergence times out (the observed wedge).

What the fix must address (Phase 4 + Phase 5, intertwined)

1. Reconvergence (Phase 4 / R5): a standby that recovers with a divergent uncommitted tail must truncate it (follower-AE-conflict-truncate contract) and catch up to the leader — the same way ctrl-3 does for a short suffix — so the control channel stays up. Determine why ctrl-2's longer/former-primary divergent tail does not reconcile while ctrl-3's short one does (candidate: the backtrack doesn't reach far enough before the channel drops, or the former-primary recovery advertises a tip that the leader's nextIndex backtracking never crosses).
2. Spurious-failover suppression (Phase 5 / R6/R4): a transient/flapping control-channel RDHUP must not spawn an unwinnable PreVote storm while the primary is demonstrably alive (peers still see it). A standby should prefer to reconnect + catch up on RDHUP before (or instead of) launching failover, or back off failover when its own PreVotes repeatedly get no quorum because a current-term leader exists. (The retry itself works — R6 retry is present — but it retries an election that can never win instead of rejoining.)

RED test direction (5.1 / 4.1)

• Force a recovered standby holding a divergent uncommitted tail (former-primary shape, tip above the new leader's committed) to rejoin a cluster with a live higher-term primary, and assert it reconverges to follower (truncates tail, catches up) without entering a PreVote-failover loop — i.e. no repeated ha_begin_failover/PreVote timed out while a current-term primary is reachable. This is deterministically constructible with the 1.4 kill/election + apply-gap seams plus a divergent-tail fixture.

Process note

The dispatched investigation subagent backgrounded the soak and hung (~no findings returned); this section was reconstructed by the parent directly from the frozen ~/tmp/ha-estall controller logs.

---

Task 7.2 — Acceptance chaotic soak (first run with Phase-2+5 fixes): NO-GO + new finding

Ran run-ha-soak.sh --chaotic 6 on real-fs (ext4 /home/claude/tmp/ha-7p2) with the release binaries rebuilt at HEAD 01419e1 (Phase-2 payload fix + Phase-5 5.2 spurious-failover suppression). Frozen at /home/claude/tmp/ha-7p2-frozen.

What PASSED (job-safety + the landed fixes)

• Zero committed-JOB loss. Every round the job-verify confirmed all tracked jobs present on the new primary (round1 6/6, round2 9/9, round3 12/12).
• Zero TRUE committed divergence. walcompare --committed 11 over the frozen WALs: "OK — committed prefixes identical" across all 3 members. The Phase-2 payload-loss signature (present=3, one member pcrc=0 vs majority real) is ABSENT. lsn=8 is pcrc=0 identically on all three — a legit empty NOOP marker, not loss.
• The counted walcompare divergences are catch-up LAG, not corruption. The only boundary-19 divergence is lsn=12 present=2: ctrl-3 (tip=11, restarted round 3) is MISSING it; ctrl-1/ctrl-2 hold it with matching content. All "divergences" are present=2 MISSING tails of restarted-but-uncaught followers (tips ctrl-1=19, ctrl-2=15, ctrl-3=11), which an inflated --committed boundary over-claims as committed — they are uncommitted/in-flight catch-up tail, exactly the research.md Phase-2-interim note.
• 5.2 spurious-failover suppression confirmed working. Rounds 1–3 each elected a strictly higher-term primary (ctrl-3 t3 → ctrl-2 t4 → ctrl-1 t5): liveness intact, the fix did not over-suppress. The round-4 RDHUP logs show the GENUINE-loss path ("primary_ctrl_fd RDHUP → begin PreVote failover"), NOT the suppression message — correct, because the primary was truly dead (ha_heartbeat_is_fresh==0).

What FAILED — round-4 election stall (NEW residual liveness defect)

After 3 overlapping kills, round 4 kills ctrl-1 (tip 19, the only fully-up-to-date node). The two survivors are both restarted-and-lagging (ctrl-2 tip=15, ctrl-3 tip=11), and their PreVote rounds repeatedly time out before a decision (no grant quorum): each loops ha_start_prevote_async → ha_prevote_round_timeout_fn (LOSS) → RDHUP → ha_begin_failover → … indefinitely, prospective terms frozen (ctrl-2=5, ctrl-3=6, never bumped because no PreVote ever WINS). The soak wedged here until the 10-min cap.

This is NOT the spurious-failover flap (5.2 closed that) and NOT a job-safety bug. A SAFE candidate exists — ctrl-2 holds the full committed prefix [1..11] (Leader Completeness satisfied) and is the freshest survivor — yet the election never converges on it under sustained overlapping churn. Observed contributors (need a dedicated RED→bisect): freshness mutual-denial (PreVote DENIED … reason=freshness, floor lsn=11) between two lagging survivors, the in_catch_up-denies-all-votes gate (Req 2.5; obs 1240 "a catching-up node sends zero PreVote replies") with no live primary left to catch up FROM, and/or peer-to-peer vote connections not established between two nodes that each only held a control channel to the now-dead primary.

Verdict

7.2 = NO-GO. Job safety (R8.1/R8.2) holds; the acceptance liveness criterion (R8.3 — reconverge 3-of-3 + elect after EVERY kill round) fails at round 4. Per the design's chaotic-harness Contingency, this is a recorded new finding, not closure: added follow-up task 5.3 (overlapping-churn election stall — a lagging-but-safe survivor must be electable; the catching-up/freshness vote gates must not deadlock when the primary is gone). 7.2 stays blocked on 5.3. Re-run 7.2 after 5.3 lands.