myxce/.kiro/specs/raft-ha-core-safety/research.md

Gap Analysis — raft-ha-core-safety

Generated by /kiro-validate-gap. Information over decisions: options and findings, not final choices.

Headline Finding (reframes the spec)

The (term, lsn) commit-identity + reconcile + commit-safety machinery the requirements describe already exists and is wired at the baseline. Confirmed:

• Library half is 100% complete and validated: raft-log-replication (24/24, G1/G2/G3/G5) and raft-commit-safety (18/18, G4 Figure-8 + G6 vote freshness). All five callbacks present in src/lib/raft/raft.{c,h}.
• Controller adapter is 16/20 wired by ha-term-safety: all five callbacks implemented in s_real_cbs (ha.c:1480/1539/1656/1711/1739), primary append routed through raft_log_append (ha.c:3408+), heartbeat clean (no on_persist_entry on n_entries==0), per-LSN term index (persist.c:414/476/487/752), config markers, and walcompare (src/controller/testsuite/walcompare.c).
• Baseline 21e26a1 already contains all of it. git: 21e26a1 is itself a raft-drive-async task-2.3 commit, ancestor of HEAD, and contains s_real_cbs + walcompare. So the machinery is NOT absent at baseline.

Yet the live chaotic-failover soak still fails on this branch AND on baseline 21e26a1: walcompare FAIL: committed-prefix divergence at LSN 9 (round 3) and standby ctrl-2 rejoin failure (round 4), with intermittent round-N election stall.

Therefore raft-ha-core-safety is NOT a build-the-machinery spec. It is a residual-defect + convergence/liveness + live-verification spec. The work is: root-cause why divergence survives the existing reconcile path under overlapping chaotic failover; close the convergence and election-liveness gaps no completed spec actually closed; and finish the live chaotic soak that ha-term-safety left permanently BLOCKED on env. Do not rebuild the (term,lsn)/callback machinery — extend and correct it.

Requirement-to-Asset Map

Req	Capability	Existing asset	Gap tag
R1 Committed-prefix safety	quorum commit + Figure-8 guard	raft_advance_commit raft.c:769-821; on_commit_advanced ha.c:1739	Constraint — machinery present but live soak still diverges at LSN 9 → residual defect
R2 Per-entry commit identity	term stamped from leader; per-LSN term index; legacy-0→1	raft.c:571-580; persist_set_term persist.c:476; record_entry_term persist.c:414/752; WAL_LEGACY_TERM persist.c:456	Done (verify) — confirm no path still emits term-0 under chaos
R3 Divergence detect + repair	logOk + conflict-truncate-then-persist on AE intake	raft.c:701-765; on_truncate_from ha.c:1711; walcompare detector	Constraint — repair fires on AE-intake path only; rejoin/catch-up path may bypass it (see Research Needed #1)
R4 Single leader per term	term++ on election, vote-once, step-down, pre-vote, vote freshness	raft.c:105-249; raft_cb_vote_eligible ha.c:342	Constraint — two-leaders-per-term window under churn not yet proven closed live
R5 Standby rejoin → 3-of-3	catch-up apply, applied-lsn floor, snapshot backfill, contiguity guard	ha_apply_wal_catchup_entry ha.c:2233; ha_advance_applied_lsn ha.c:2078; ha_handle_snapshot_chunk ha.c:2492; contiguity persist.c:839	Missing (effective) — bug-1042 standby-never-converges has NO dedicated owning task; ha-convergence-recovery claims it but soak still fails round 4
R6 Election liveness under churn	election restart, candidate advertise floor	raft_start_election raft.c:105; ha_candidate_advertise_lsn ha.c:3921	Missing (effective) — round-N stall not closed; no deterministic churn test
R7 RED→GREEN harness	xce-test-ha soak, walcompare, controller_unit_test, test_ha_dispatch_integration	.claude/skills/xce-test-ha/; walcompare.c; src/xce.build/controller/controller_unit_test	Missing — no deterministic harness that forces chaotic divergence/rejoin/stall as a RED unit/integration gate (current soak is probabilistic)
R8 Acceptance soak gate	real-fs release soak; BIN/XT overridable	run-ha-soak.sh (BIN/XT env override added this branch); non-ASAN release build	Unknown — never passed at 6+ chaotic rounds; was BLOCKED on overlayfs fdatasync (bug-924)

Existing-Code Constraints (must respect)

• Reconcile is single-seam: divergence repair happens only inside raft_on_append_entries_request (logOk → on_truncate_from → re-persist, raft.c:730-765). Any code path that brings a node's log forward without routing through this function (e.g. a snapshot/catch-up stream that writes WAL directly) will not detect or repair a divergent committed prefix.
• Truncate must never cut the majority-committed prefix (raft.c monotonic commit; promotion truncate-clamp max(safe_committed_lsn, durable_tip) ha.c:999-1000, bug-1060 fix). R3.4 depends on this clamp holding under chaos.
• Persist-before-ACK durability is owned by ha-commit-durability, not this spec. R1/R2 must not weaken it. on_commit_advanced is a barrier trigger, not the durability mechanism.
• raft-drive-async is out of scope and dormant. Do not change SlowLane / peer-control dispatch / endpoint-quiescence behavior (Boundary Context).
• Client-side bug-854 (libxce primary discovery) can fail the soak harness at EXIT-1 even when server data safety passes. Must be isolated so it does not mask/forge R8 verdicts.

Implementation Approach Options

Option A — Extend & correct existing machinery (recommended)

Keep all library + adapter machinery. Root-cause the residual divergence, route the rejoin/catch-up path through the reconcile seam, close the convergence + liveness gaps with targeted fixes, add a deterministic chaotic harness, finish the live soak.

• ✅ No rebuild of consensus-critical code that already passed library + clean-failover validation
• ✅ Smallest blast radius; aligns with TLA+ already-proven invariants
• ❌ Requires precise root-cause of a hard-to-repro chaotic defect before fixing
• ❌ Risk of "fix the symptom not the invariant violation" if harness stays probabilistic

Option B — New convergence/divergence-repair subsystem

Build a standalone post-failover reconcile/repair component separate from the AE-intake seam.

• ✅ Clean place to own rejoin reconcile + unrecoverable-divergence fencing
• ❌ Duplicates logic that already lives in raft_on_append_entries_request; two reconcile paths is exactly the class of bug being fixed
• ❌ High risk of new two-leaders/two-reconcile divergence

Option C — Hybrid (Option A + new deterministic harness component)

Extend/correct in place (A), but build the deterministic chaotic harness (R7) as genuinely new test components (forced overlapping kills, injected divergent prefix, scripted round-N stall) rather than relying on the probabilistic soak.

• ✅ Keeps consensus code as A; isolates the new surface to test infrastructure where new code is safe
• ✅ Gives a RED gate that pins each fix to an invariant violation
• ❌ Harness design is itself non-trivial (deterministic chaos, no sleeps, gate/probe seams like test_ha_dispatch_integration.c)

Effort & Risk

Item	Effort	Risk	Justification
Root-cause residual LSN-9 divergence	M	High	Chaotic, intermittent; survives existing reconcile — invariant violation is subtle
Route rejoin/catch-up through reconcile seam (R3/R5)	M	High	Touches recovery + commit prefix; must not cut majority-committed
Election liveness under churn (R6)	M	Medium	Election machinery exists; needs churn-stall root cause + restart guarantee
Deterministic chaotic harness (R7)	L	Medium	New, but test-only; pattern exists (test_ha_dispatch_integration.c)
Live 6+round real-fs soak (R8)	M	High	Was permanently blocked on overlayfs fdatasync (bug-924); needs real-fs env + bug-854 isolation
Total	L–XL	High	Consensus correctness under chaos + env-gated live verification

Research Needed (carry to design phase)

1. Why does divergence survive the reconcile seam? Trace the rejoin/catch-up path (ha_apply_wal_catchup_entry ha.c:2233, snapshot ha.c:2492): does it route through raft_on_append_entries_request so logOk/on_truncate_from fire, or does it write WAL directly and bypass conflict repair? Hypothesis: catch-up streams [watermark..committed] directly (per memory bug-795/828 family) → no (term,lsn) conflict check on rejoin.
2. Is LSN-9 divergence in the committed or uncommitted prefix? If committed, the violation is upstream (two-leaders-per-term or premature commit-advance), not a repair gap. Use walcompare (lsn→term,pcrc) output to classify.
3. Two-leaders-per-term under overlapping kills — does pre-vote + vote-once fully close the window when kills overlap election timeouts? Map against raft/raft.tla Election Safety.
4. bug-1042 ownership — ha-convergence-recovery claims the rejoin path but soak still fails round 4. Determine whether its tasks are actually landed on this branch or stranded, to avoid duplicate work.
5. R8 env — confirm real-fs (ext4) soak runs now via run-ha-soak.sh BIN/XT override + non-ASAN release build; quantify how bug-924 (overlayfs) and bug-854 (client discovery) are excluded from the R8 verdict.

Recommendation for Design Phase

• Preferred: Option C (extend/correct in place + new deterministic chaotic harness). Treat the spec as residual-defect + convergence + liveness + live-verification, explicitly not a machinery rebuild.
• Key design decisions to make: (a) route rejoin/catch-up through the single reconcile seam vs. add a guarded repair at promotion; (b) classify LSN-9 as committed vs uncommitted before choosing repair vs upstream fix; (c) harness determinism mechanism (forced-kill gates/probes) for R7; (d) how R8 isolates bug-854/bug-924 so the gate measures only server-side committed-job safety.
• Carry forward: Research Needed #1–#5; verify every change against raft/raft.tla (Election Safety, Leader Completeness, State Machine Safety, Commit Ordering/Durability) per src/controller/CLAUDE.md.

---

Design-Phase Discovery (kiro-spec-design)

ROOT CAUSE (term-blind catch-up bypass) — ⚠️ SUPERSEDED / REFUTED by task 1.3

This hypothesis was tested live (task 1.3) and REFUTED. The real defect is cross-term catch-up payload loss (see §"Task 1.3 — Live divergence reproduction & bisect" below). The divergent entry carries the correct term and arrives through the reconcile seam; the term-blind path is a latent hazard but not the observed cause. Retained below for the diagnostic record.

Deep-trace of the standby catch-up/rejoin paths found a split where one branch bypasses the single reconcile seam:

Path	Entry	Routes through raft_on_append_entries_request?	logOk	conflict-truncate	term stamped
1 Term-carrying	entry->term != 0 → ha_apply_term_entry_via_library (ha.c:2173) → ha_consume_append_entries (ha.c:2383) → raft.c:646	YES	YES (raft.c:701)	YES (raft.c:735)	leader term via persist_set_term
2 Term-blind (defect)	entry->term == 0 → ha_apply_single_entry (ha.c:2097) → persist_wal_write_replicated (persist.c:802) DIRECT	NO	NO	NO	standby's own p->term (persist.c:862)
3 Snapshot	ha_handle_snapshot_chunk (ha.c:2492) → persist_load_snapshot sets applied_lsn=checkpoint	NO	N/A	N/A	N/A (canonical state)
4 Stream+ACK	ha_stream_wal_to_standby (config_repl.c:2032) feeds Path 1 or 2 by entry->term	depends	depends	depends	depends
5 ha_rejoin_reconcile	ha_convergence.c:254 — now a no-op stub (committed-watermark truncate retired; relies on library)	NO	N/A	retired	N/A

Divergence mechanism (LSN 9):

1. An entry reaches a standby via Path 2 (term-blind). persist_wal_write_replicated stamps hdr.term = standby p->term (wrong) and records it (record_entry_term, persist.c:902). No logOk, no conflict check.
2. The correct term-carrying entry for the same LSN later arrives via Path 1. The persist idempotency guard (if lsn+1 <= p->current_lsn return 0, persist.c:822) silently no-ops — the divergent entry is never overwritten/repaired.
3. walcompare (testsuite/walcompare.c:135) compares (term, payload_crc) per committed LSN; the member with the term-blind write shows a different term → WALCMP_DIVERGENT. match_term maps raw 0 → WAL_LEGACY_TERM, so legacy-0 vs genuine-term at the same LSN reads as divergence.

Why the existing (term,lsn) machinery does not catch it: all conflict detection lives inside raft_on_append_entries_request; Path 2 never enters it, and the idempotency guard treats "LSN already present" as "done" regardless of term.

Synthesis

⚠️ D1/D3 partially superseded by task 1.3. The "eliminate every bypass of the one seam / collapse Path 2 into Path 1" framing assumed the term-blind hypothesis, now refuted. The single-seam invariant (D2) and harness-first discipline (D4) still hold; but the actual fix is the cross-term catch-up streamer payload integrity + a (term,lsn,payload-crc)-aware guard (see §"Task 1.3"), not seam-bypass elimination. Term-blind retirement remains as defensive hardening only.

• Generalization (D1): R1/R2/R3/R4 are one underlying invariant — every entry that enters a member's log must pass the single (term,lsn) reconcile seam. The fix is not four mechanisms but eliminating every bypass of the one seam + making persistence (term,lsn)-conflict-aware rather than LSN-only-idempotent.
• Build vs adopt (D2): Adopt the existing library seam (raft_on_append_entries_request + on_truncate_from + persist_set_term) — already validated (raft-log-replication/raft-commit-safety). Build nothing new in the consensus core; only route into it and harden the persist guard. Reuse ha_rejoin_reconcile intake hook (currently stub) rather than a second reconcile path.
• Simplification (D3): Do not add a separate "divergence repair subsystem" (gap-analysis Option B rejected) — that would create the exact second-reconcile-path class of bug being fixed. Collapse Path 2 into Path 1.
• Harness-first (D4): The live bug is intermittent/chaotic. Make it deterministic: a RED unit test injects a term-blind/divergent entry at an LSN and asserts the seam repairs it; a chaotic soak mode forces overlapping kills. RED before any fix (R7), pins each fix to a walcompare/invariant violation.

Open risks carried to tasks

• Confirm Path 2 is the only divergence source live (harness RED test will confirm/refute; if soak still diverges after Path-2 closure, classify next via walcompare (term,lsn,pcrc)).
• Snapshot boundary (Path 3): post-install AE intake must reconcile against the snapshot's (term,lsn); verify no committed entry below the checkpoint can diverge.
• Standby reconvergence (R5) and election-liveness (R6) are separable seams from divergence-repair; phaseable but kept in one spec per the unified acceptance gate.
• bug-854 (client discovery) and bug-924 (overlayfs fdatasync) must be excluded from the R8 verdict so the gate measures server-side committed-job safety only.

---

Design Review Resolution (kiro-validate-design)

Three critical issues raised and folded into design.md (GO):

1. Repair-vs-clamp conflation (correctness). Split truncate into two contracts: promotion-truncate keeps the bug-1060 max(safe_committed_lsn, durable_tip) clamp; follower-AE-conflict-truncate obeys Raft Log Matching and truncates the conflicting suffix regardless of local durable_tip (a conflicting entry was never majority-committed → safe to cut). Applying the promotion clamp on the follower path would have converted repairable divergence into a permanent fence and failed R8. Updated: §Existing Architecture, §Unified Entry Ingest, §Persist Conflict Guard, §Error Handling, Traceability 3.x, Testing.
2. R4 two-leaders-per-term unverified. Added a falsifiable Election Safety assertion (≤1 committing-primary per term under overlapping kills, mapped to raft.tla); failure promotes a concrete adapter-gate guard from verification to required work. Updated: §Election Liveness, Testing Integration.
3. Hypothesis-validation risk. RED gate (7.1) must reproduce the live walcompare divergence and bisect to a seam before synthetic (term,lsn) injection counts; baseline 21e26a1 already retired live term-0 (ha-term-safety 3.4), so the term-blind path may not be the live trigger. Contingency: if soak still diverges after Path-2 closure, re-classify via walcompare (term,lsn,pcrc) and add a fix task — do not mark R1/R8 done on green unit tests alone. Updated: §Deterministic Chaotic Harness, Testing E2E + RED-first.

Verdict: GO (unconditional after edits). Architecture unchanged; refinements were bounded contract clarifications.

---

Task 1.3 — Live divergence reproduction & bisect

Generated by the task-1.3 investigation subagent (REPRODUCE + CLASSIFY + BISECT). Evidence-backed; no production code changed. Real-fs (ext4) ~/tmp, non-ASAN release binaries src/build.release/controller/{xce_controller,xce_test} (built 08:48 this day).

TL;DR / VERDICT

The design's primary "term-blind catch-up bypass" hypothesis is REFUTED for the reproduced live committed divergence. The divergent committed entry has the CORRECT term, was ingested through the single library reconcile seam (ae_append → on_persist_entry), and the LSN-only idempotency no-op was not the mechanism. The real defect is payload loss on cross-term catch-up: a committed JOB_SUBMIT entry is persisted on the catching-up member with payload_len=0 (empty body) while its (term, lsn) are correct. It is a sibling of the bug-795/828/1060 committed-prefix-loss family, located in the catch-up replication payload path, not in the (term,lsn)/idempotency-guard seam Phase 2 currently targets.

Commands run & reproduction status

• Two --chaotic soak attempts via run-ha-soak.sh (rounds=6, XCE_HA_BIN/XT → release binaries, dir ~/tmp/ha-1p3) both aborted at cold-start with FAIL: no primary elected within 45s. A --sequential attempt failed identically.
• This cold-start "election stall" is a HARNESS/LOG-BUFFER ARTIFACT, not a consensus stall. Evidence: in every aborted run a primary WAS elected on time (e.g. ctrl-1 → CTRL_PRIMARY (term=2) logged ~15s after start, quorum_reached), yet wait_for_primary_log never tripped. All three nodes' last log line is the identical cleanup-kill timestamp (= start+45s): the 08:48 release build block-buffers its --logfile and only flushes on signal, so the harness's log-grep primary detection (latest_primary_term) cannot observe progress live. The OLD binaries used by the prior passing sequential soaks (~/tmp/xce-soak-{rel,bl}.log, 02:56/03:01) did not have this buffering behaviour. The client (ctrlstatus) path is unaffected.
• Reproduced cleanly by driving the cluster with a client-based (log-buffer-immune) driver (scratchpad/drive.sh, fresh deploy ~/tmp/ha19, sequential kill→restart→re-elect, 6 rounds). Committed-prefix divergence appeared by round 1 and persisted: round 1 LSN 8 (present=2), round 2 LSN 8 (present=2), round 3 LSN 9 (present=3 — all members hold it but disagree). Driver wedged at round 4 on a transient quorum loss; the frozen WAL dirs were classified directly.

walcompare --report classification (frozen ~/tmp/ha19, --committed 25)

Decisive divergence — committed LSN 9, class=committed, compared=3, present=3 (every member holds the entry; they disagree on payload):

WALCMP_DIVERGENCE lsn=9 class=committed compared=3 present=3
  member 0 (ctrl-1): term=3 pcrc=00000000   <-- MINORITY / divergent
  member 1 (ctrl-2): term=3 pcrc=512dc5b5   <-- majority
  member 2 (ctrl-3): term=3 pcrc=512dc5b5   <-- majority

Same pattern at committed LSN 10 (ctrl-1 pcrc=0 vs majority 06e88dac) and LSN 11 (ctrl-1 pcrc=0 vs majority fea75587). Term is identical (3) across all three members; only the payload-CRC diverges. (ctrl-3 additionally shows status=MISSING for LSN 12-19 — a separate catch-up-lag/convergence gap from the round-4 quorum loss, bug-1042 family, not the payload defect.)

Raw waldump confirms the payload loss, not a term mismatch:

ctrl-1 (divergent):   9 | 3 | JOB_SUBMIT | PLEN=0    | 00000000 | jobid=?
ctrl-2/ctrl-3 (maj):  9 | 3 | JOB_SUBMIT | PLEN=3560 | 512dc5b5 | jobid=0-0-1782550813-0 ...

The origin is intact — both majority members hold the full 3560-byte body with a decodable jobid; ctrl-1 lost the payload bytes during catch-up while keeping the correct type (JOB_SUBMIT) and term (3).

Bisect — the concrete ingest path

ctrl-1's --logfile (fully flushed post-kill) traces exactly how it acquired the divergent entries:

1. ctrl-1 was the term-1 cold-start primary (wrote LSN 1-7), killed round 1, restarted round 2 (persist_recover done — recovered durable_floor=7 current_lsn=8). It then needed LSN 8-onward.
2. 09:00:13 — term-3 leader ctrl-3 streamed catch-up op 37 walCatchupEntry, but starting at prevIdx=8 (entry 9) while ctrl-1 only held up to 7 → library ae_reject_logmismatch, ha_consume_append_entries … success=0. Nothing written. (ctrl-3 then died — round 2 failover.)
3. 09:00:26 — new term-4 leader ctrl-2 streamed op 37 walCatchupEntry from prevIdx=7. The library accepted and APPENDED: RAFT[p0] term=4 event=ae_append, ha_consume_append_entries … prevIdx=7 n=1 → success=1 match=8, then match=9, 10, 11, 12. These were fresh appends through the reconcile seam (on_persist_entry), not idempotent no-ops (current_lsn was 8 immediately prior). LSN 9-11 landed with payload_len=0.
4. In the SAME catch-up, LSN 13-15 (stored term=4, the streaming leader's OWN term) landed with FULL payloads. Signature: same-term catch-up preserves payload; cross-term catch-up of prior-term entries (stored term 3 streamed under leader term 4) drops it.

Code-path mapping (read at the research anchors):

• Receiver ha_consume_append_entries (ha.c:2410-2440) copies the body unconditionally (if (payload && plen>0) memcpy(...), ha.c:2438) from e->raw_payload/e->raw_payload_len. So the empty body arrived empty on the wire → the loss is SEND-side / source-side, not in the seam's persist.
• Both documented send paths nominally read the payload: batch streamer ha_stream_wal_to_standby (config_repl.c:2154-2161, pread when hdr.payload_len>0, and sets entry->term = persist_entry_term(...) + entry->leader_current_term = leader_term, config_repl.c:2130-2151) and the single-entry on_send_append_entries → ha_read_wal_entry_at (ha.c:1611-1640). Neither statically drops a cross-term payload — so the loss is either (a) the streaming leader (ctrl-2) itself held an empty copy at stream time and the full payload was reconciled into ctrl-2's WAL only later (a transitive empty-propagation across the failover chain), or (b) the wire encode/decode of the body for these messages. Distinguishing (a) vs (b) requires one instrumented run capturing per-entry payload_len at the leader's send and at the receiver's on_persist_entry for a cross-term LSN — see "Implied next work".

Why the term-blind hypothesis is refuted here

• Predicted by design.md ROOT CAUSE: entry->term==0 → ha_apply_single_entry (ha.c:2097) → persist_wal_write_replicated DIRECT, wrong term (standby's p->term), idempotency no-op of the later correct entry.
• Observed: the entry's term is correct (3), it was ingested via ha_apply_term_entry_via_library → ha_consume_append_entries → on_persist_entry (the term-carrying LIBRARY seam, event=ae_append), and the catch-up was a fresh append (no idempotency-guard no-op involved). The streamer stamps a non-zero term (config_repl.c:2141), so the term==0 → ha_apply_single_entry branch (ha.c:2253/2291 gate sub->term != 0) is not taken for these entries.

VERDICT

• Term-blind catch-up bypass (wrong-term + LSN-only idempotency no-op): REFUTED as the trigger of the reproduced live committed divergence.
• Actual trigger: payload-loss on cross-term catch-up — committed JOB_SUBMIT entries persisted with payload_len=0 (term/type correct) on a member that catches up prior-term entries from a higher-term leader; the divergence is a payload-CRC mismatch at an identical (term,lsn), in the committed prefix. A member carrying this can (and in round 3 did) become primary, so it is a silent committed-state corruption in the bug-795/828/1060 family.

Implied next work (steers Phase 2)

1. Re-scope the Phase-2 fix away from the term-stamp/idempotency seam. Collapsing Path-2 into Path-1 (design D3/D1) will NOT fix this divergence — the entry already goes through Path-1 with the correct term. The fix must target the catch-up payload source-of-truth: add an instrumented capture of payload_len at ha_stream_wal_to_standby/on_send_append_entries (send) and on_persist_entry (receive) for cross-term LSNs to prove send-side empty vs transitive-empty, then fix whichever drops/propagates the empty body.
2. Harden the persist conflict guard to be (term,lsn,payload)-aware, not term-only: a committed (term,lsn) whose payload-CRC disagrees with the quorum must be detected/repaired (or the member fenced), since term-only reconcile is blind to this class.
3. Fix the divergence detector / RED gate to assert payload-CRC at committed LSN even when terms match (it already does via pcrc; keep walcompare --report (term,lsn,pcrc) as the canonical RED classifier — it caught this).
4. R6/Harness defect (secondary): the release build's log buffering defeats run-ha-soak.sh's log-based wait_for_primary_log/primary_from_log primary detection (false "election stall"). The chaotic acceptance gate (R8) needs client-based (ctrlstatus) primary detection, or the controller must line-buffer/flush its INFO log, before the soak can run unattended on release binaries.

Evidence artifacts (under ~/tmp, ephemeral)

• ~/tmp/ha19/ — frozen 3-member deploy with the reproduced LSN 9/10/11 payload divergence (controller/ctrl-{1,2,3}/data, logs/ctrl-1.log).
• ~/tmp/ha19-drive.log — driver round-by-round output (divergence first seen round 1).
• ~/tmp/ha-1p3-run{1,2}.log, ~/tmp/ha-1p3-seq.log — the cold-start log-buffer "stall" runs.

---

Phase-2 interim live validation (post-2.4, real-fs release rebuild)

Rebuilt the release controller at HEAD 63b9c5a (2.2 payload-aware guard + 2.4 streamer fail-loud) and ran a chaotic soak on real ext4 (~/tmp/ha-verify-2p4). The 4-round run wedged (timed out ~9m40s) on an election stall before reaching its own verdict — the un-fixed Phase-5 liveness defect. Froze the WALs and measured directly with walcompare --report:

• Universally-held committed prefix (LSN 1..11) is content-identical across all 3 members (walcompare: OK — committed prefixes identical).
• No present-but-empty committed divergence — the bug-1.3 signature (one member pcrc=0/PLEN=0 PRESENT where the majority holds a full payload) is absent. The pcrc=00000000 entries at LSN 1/8/12/16 are present+identical on ALL members (legitimate empty-payload marker/NOOP entries, same term+crc everywhere).
• Remaining apparent divergences are all status=MISSING tail entries (ctrl-1 tip=19, ctrl-2=15, ctrl-3=11) — catch-up lag on lagging/killed members, not committed-content corruption. Classifying these as "committed divergence" only happens when an inflated --committed boundary over-claims the committed region; at the true majority-held boundary they are uncommitted tail / in-flight catch-up.

Conclusion: the Phase-2 fix empirically eliminates the cross-term payload-loss divergence on the live repro. A clean end-to-end committed_divergence=0 soak verdict is now gated on:

1. Phase 5 (election liveness under churn) — the chaotic soak wedges on an election stall before completing, so it never reaches its verdict line. This is the top blocker for the acceptance gate (7.2).
2. Phase 4 (standby reconvergence) — lagging members must finish catching up so the tail MISSING resolves once the cluster stabilizes.

Steering implication: Phase 3 (snapshot-boundary reconcile) is defense-in-depth for a path NOT implicated in the live repro; Phases 5 and 4 are on the critical path to a clean soak verdict. Consider doing 5 (and 4) before 3.

---

Phase 4/5 — Post-Phase-2 stall root cause (investigation, salvaged from a hung run)

Reproduced the chaotic-soak wedge on real-fs release binaries (HEAD with Phase-2 fixes). The committed/payload prefix is clean (Phase-2 holds). The remaining wedge is a reconvergence-triggered spurious-failover loop, NOT a simple missing-election-restart.

Evidence (frozen ~/tmp/ha-estall logs)

• ctrl-1 = stable primary at term 6: became_leader, repeated repl_broadcast / term=6 event=log_append. The cluster HAS a leader.
• ctrl-3 reconverges (slow but works): term=6 ae_recv → ae_reject_logmismatch ×4, then ae_already_have. The AE log-matching backtrack converges for a short divergent suffix.
• ctrl-2 = the stall. It recovered as a former primary with an uncommitted higher tail (durable_floor=15 current_lsn=16, entries 12-15 at an old term) vs the leader's committed 11. Its log shows a ~15s flap cycle, repeating indefinitely:
  1. handle_ctrlhello: primary control channel established (fd=6) → receives heartbeats → re-attaches CTRL_STANDBY.
  2. ~12-15s later: ha_on_primary_ctrl_fd_rdhup: primary_ctrl_fd RDHUP → begin PreVote failover (Req 1.1).
  3. ha_start_prevote_async (prospective term=6) → ha_prevote_round_timeout_fn: PreVote round timed out before a decision — treating as LOSS, staying CTRL_STANDBY (NO pre_vote_reply_in — peers see the primary alive and don't grant).
  4. Primary reconnects → loop repeats. (Last heartbeat and last prevote-timeout share the same timestamp — the channel is flapping, not cleanly down.)

Root cause

The primary↔ctrl-2 control channel keeps RDHUP-flapping (~15s period). Because "control-channel RDHUP = primary died → failover" is the designed trigger (CLAUDE.md), each flap spawns a spurious PreVote failover that cannot win (the primary is genuinely alive, so the other voters don't grant), times out, and re-arms. The likely driver of the flap: ctrl-2 cannot reconverge its divergent uncommitted tail (12-15) against the leader — the catch-up/AE path stalls or the leader drops the channel — so ctrl-2 never settles as a follower and the channel cycles. Net: the recovered node never rejoins → no stable 3-of-3 → soak wait_for_full_convergence times out (the observed wedge).

What the fix must address (Phase 4 + Phase 5, intertwined)

1. Reconvergence (Phase 4 / R5): a standby that recovers with a divergent uncommitted tail must truncate it (follower-AE-conflict-truncate contract) and catch up to the leader — the same way ctrl-3 does for a short suffix — so the control channel stays up. Determine why ctrl-2's longer/former-primary divergent tail does not reconcile while ctrl-3's short one does (candidate: the backtrack doesn't reach far enough before the channel drops, or the former-primary recovery advertises a tip that the leader's nextIndex backtracking never crosses).
2. Spurious-failover suppression (Phase 5 / R6/R4): a transient/flapping control-channel RDHUP must not spawn an unwinnable PreVote storm while the primary is demonstrably alive (peers still see it). A standby should prefer to reconnect + catch up on RDHUP before (or instead of) launching failover, or back off failover when its own PreVotes repeatedly get no quorum because a current-term leader exists. (The retry itself works — R6 retry is present — but it retries an election that can never win instead of rejoining.)

RED test direction (5.1 / 4.1)

• Force a recovered standby holding a divergent uncommitted tail (former-primary shape, tip above the new leader's committed) to rejoin a cluster with a live higher-term primary, and assert it reconverges to follower (truncates tail, catches up) without entering a PreVote-failover loop — i.e. no repeated ha_begin_failover/PreVote timed out while a current-term primary is reachable. This is deterministically constructible with the 1.4 kill/election + apply-gap seams plus a divergent-tail fixture.

Process note

The dispatched investigation subagent backgrounded the soak and hung (~no findings returned); this section was reconstructed by the parent directly from the frozen ~/tmp/ha-estall controller logs.

---

Task 7.2 — Acceptance chaotic soak (first run with Phase-2+5 fixes): NO-GO + new finding

Ran run-ha-soak.sh --chaotic 6 on real-fs (ext4 /home/claude/tmp/ha-7p2) with the release binaries rebuilt at HEAD 01419e1 (Phase-2 payload fix + Phase-5 5.2 spurious-failover suppression). Frozen at /home/claude/tmp/ha-7p2-frozen.

What PASSED (job-safety + the landed fixes)

• Zero committed-JOB loss. Every round the job-verify confirmed all tracked jobs present on the new primary (round1 6/6, round2 9/9, round3 12/12).
• Zero TRUE committed divergence. walcompare --committed 11 over the frozen WALs: "OK — committed prefixes identical" across all 3 members. The Phase-2 payload-loss signature (present=3, one member pcrc=0 vs majority real) is ABSENT. lsn=8 is pcrc=0 identically on all three — a legit empty NOOP marker, not loss.
• The counted walcompare divergences are catch-up LAG, not corruption. The only boundary-19 divergence is lsn=12 present=2: ctrl-3 (tip=11, restarted round 3) is MISSING it; ctrl-1/ctrl-2 hold it with matching content. All "divergences" are present=2 MISSING tails of restarted-but-uncaught followers (tips ctrl-1=19, ctrl-2=15, ctrl-3=11), which an inflated --committed boundary over-claims as committed — they are uncommitted/in-flight catch-up tail, exactly the research.md Phase-2-interim note.
• 5.2 spurious-failover suppression confirmed working. Rounds 1–3 each elected a strictly higher-term primary (ctrl-3 t3 → ctrl-2 t4 → ctrl-1 t5): liveness intact, the fix did not over-suppress. The round-4 RDHUP logs show the GENUINE-loss path ("primary_ctrl_fd RDHUP → begin PreVote failover"), NOT the suppression message — correct, because the primary was truly dead (ha_heartbeat_is_fresh==0).

What FAILED — round-4 election stall (NEW residual liveness defect)

After 3 overlapping kills, round 4 kills ctrl-1 (tip 19, the only fully-up-to-date node). The two survivors are both restarted-and-lagging (ctrl-2 tip=15, ctrl-3 tip=11), and their PreVote rounds repeatedly time out before a decision (no grant quorum): each loops ha_start_prevote_async → ha_prevote_round_timeout_fn (LOSS) → RDHUP → ha_begin_failover → … indefinitely, prospective terms frozen (ctrl-2=5, ctrl-3=6, never bumped because no PreVote ever WINS). The soak wedged here until the 10-min cap.

This is NOT the spurious-failover flap (5.2 closed that) and NOT a job-safety bug. A SAFE candidate exists — ctrl-2 holds the full committed prefix [1..11] (Leader Completeness satisfied) and is the freshest survivor — yet the election never converges on it under sustained overlapping churn. Observed contributors (need a dedicated RED→bisect): freshness mutual-denial (PreVote DENIED … reason=freshness, floor lsn=11) between two lagging survivors, the in_catch_up-denies-all-votes gate (Req 2.5; obs 1240 "a catching-up node sends zero PreVote replies") with no live primary left to catch up FROM, and/or peer-to-peer vote connections not established between two nodes that each only held a control channel to the now-dead primary.

Verdict

7.2 = NO-GO. Job safety (R8.1/R8.2) holds; the acceptance liveness criterion (R8.3 — reconverge 3-of-3 + elect after EVERY kill round) fails at round 4. Per the design's chaotic-harness Contingency, this is a recorded new finding, not closure: added follow-up task 5.3 (overlapping-churn election stall — a lagging-but-safe survivor must be electable; the catching-up/freshness vote gates must not deadlock when the primary is gone). 7.2 stays blocked on 5.3. Re-run 7.2 after 5.3 lands.

---

Task 5.3 + 7.2 re-run — CORRECTED root cause: the soak stall is bug-924, not consensus

After landing task 5.3 (commit bf88681 — gate the in_catch_up PreVote deny on a LIVE primary), the chaotic soak was re-run on real-fs release binaries (/home/claude/tmp/ha-7p2b, controller rebuilt 02:44 with all Phase-2+5.2+5.3 fixes).

Result: round-4 stalls AGAIN — but the cause is the persist fdatasync wedge (bug-924), NOT election logic

• Rounds 1–3 elect a higher-term primary each round (term 4→5→6); job safety holds.
• Round 4 "NO new primary within window" — and ctrl-2's PreVote rounds get zero replies (timeout, not LOST), because the other survivor ctrl-3 is wedged: its persist flusher spins on flusher fallback fdatasync FAILED (errno=22 Invalid argument) from 02:45:49 onward and stops answering PreVotes. ctrl-3's process is alive but non-responsive — jammed in the durability layer, not the election layer.
• This is the bug-924 WAL-fdatasync-on-overlay artifact (memory obs 1476/1477): the WAL segment's fdatasync returns EINVAL in this sandbox. A file-level repro of the exact controller pattern (open O_RDWR|O_CREAT|O_TRUNC → posix_fallocate(64MB) → write → fdatasync) SUCCEEDS on /home/claude/tmp, /home/rshen/projects/xce (both ext4) and /dev/shm — so it is the controller's live overlay-backed WAL fd under chaotic restart, not a plain fs limitation. bug-924 is a Non-Goal of this spec (design §Non-Goals / §Out of Boundary: "the overlayfs fdatasync environment defect (bug-924) — excluded from the acceptance verdict, not fixed here"); durability sync is owned by ha-commit-durability.

Re-attribution of the round-4 stall (corrects the first §"Task 7.2" entry)

The first soak's round-4 stall was attributed to the in_catch_up mutual-deny. That IS a real latent defect — 5.3 reproduces it deterministically at unit level (a catching-up survivor with no live primary denied a fresh PreVote) and fixes it — but it is NOT what blocks the soak. The soak's actual round-4 blocker in BOTH runs is the bug-924 fdatasync wedge that makes a survivor non-responsive. 5.3 is kept as a valid correctness fix (unit RED→GREEN, test_ha_core_safety 20 Tests / 0 Failures); its 6-round-soak observable clause shares 7.2's environment block.

7.2 status: BLOCKED ON ENVIRONMENT (bug-924), not a consensus NO-GO

The consensus work this spec owns is unit-complete and validated: Phase-2 payload integrity (committed prefix byte-identical across members), 5.2 spurious-failover suppression, 5.3 catching-up-survivor electability. The live ≥6-round acceptance soak cannot reach a clean verdict in this sandbox because the controller's WAL fdatasync fails (bug-924) and wedges nodes mid-soak — the same env block that left ha-term-safety's live soak permanently BLOCKED. 7.2 (and 5.3's soak clause) require a real-fs host/CI where the controller WAL fdatasync succeeds, OR the soak harness must detect the fdatasync-wedge and exclude it from the liveness verdict (a Phase-1 harness improvement), OR bug-924 is resolved by ha-commit-durability. The consensus fixes do not need re-work for this; it is an environment/durability-layer blocker.

---

Task 7.2 on ~/tmp (non-overlay) — DEFINITIVE 6-round blocker: a controller restart SEGFAULT

Re-ran the chaotic soak on the user-provided non-overlay path ~/tmp (= /home/claude/tmp) with the release binaries (Phase-2 + 5.2 + 5.3) and a harness change to capture each controller's stdout+stderr (start_node previously sent it to /dev/null, hiding all crash output — the key diagnostic gap).

What is now PROVEN

• Elections work under chaotic churn. A 2-round chaotic soak COMPLETES (both rounds elect a higher-term primary). In 6-round runs, rounds 1–3 each elect (terms climb 4→5→6). 5.2 + 5.3 hold live. Job safety holds (zero committed-job loss every run).
• The 6-round blocker is a RELEASE-BUILD SEGFAULT on restart, captured at last in the per-node stderr: start.sh: line 14: <pid> Segmentation fault (core dumped) … xce_controller --myname ctrl-3 …. A controller segfaults a few seconds after being restarted into the churning cluster (during recovery→failover/PreVote). One node down → cluster at 1-of-3 → the lone survivor cannot reach quorum → "NO new primary within window." The harness's other stderr lines are the expected kill -9 ("Killed"); only the restarted node shows SEGV.
• This is NOT the in_catch_up eligibility deadlock (5.3 fixes that, unit-proven), NOT bug-924 fdatasync (that wedged a node in ONE earlier run but is absent here — zero fdatasync FAILED), and NOT the consensus logic this spec owns. It is a restart/rejoin stability crash.

Why it could not be root-caused in this sandbox

• The crash is a heisenbug: it does NOT reproduce under the ASAN controller build (all ASAN-run stderr shows only harness kill -9, no SEGV / no ASAN report) — memory-layout / timing dependent.
• No gdb and no usable core: core_pattern pipes to a non-running systemd-coredump, so the "core dumped" core is discarded; coredumpctl/gdb are absent from the env.
• Root-causing requires a debug-non-ASAN build with core dumps + gdb (or a host where the systemd-coredump pipeline is live), i.e. a tooling environment this sandbox lacks.

Harness accuracy note (separate, in-scope)

The soak counts committed_divergence via walcompare --report --committed $((1 + 2*njobs)) (run-ha-soak.sh:543/663). That boundary is the OPTIMISTIC job-count estimate, not the true majority-committed LSN, so a lagging follower's not-yet-replicated tail (uncommitted on that member) is mis-counted as committed divergence. The completed 2-round soak reported committed_divergence=3 committed_job_loss=0 — and a frozen-WAL walcompare at the true boundary shows the committed prefix byte-identical (the 3 are catch-up-lag MISSING tails, not corruption). ctrlstatus already reports committedLsn; the boundary should use it. (Even fixed, the 6-round verdict stays blocked by the segfault above.)

7.2 status (corrected): blocked by a restart SEGFAULT, not consensus, not bug-924

Consensus correctness (Phase-2 payload integrity, 5.2 spurious-failover suppression, 5.3 catching-up-survivor electability) is unit-complete and confirmed live for every round that completes. The clean ≥6-round acceptance verdict is blocked by the restart segfault — added follow-up task 5.4 (root-cause+fix the chaotic-restart crash; needs core+gdb / debug env) and bug-1281. The harness committedLsn-boundary fix is a separate Phase-1 accuracy improvement.

---

Task 7.2/5.4 — UNIFIED root cause via gdb: round-N stall is a restart-stability bug (I/O wedge / SIGSEGV), not a consensus-logic gap

Wrapped the release xce_controller under gdb -batch (handles RT signals; dumps all-thread bt on SIGSEGV) via XCE_HA_BIN=<gdb-wrapper> and re-ran the chaotic soak on ~/tmp.

Decisive findings

• Two round-4 faces, ONE root cause. Plain release: a restarted node SIGSEGVs (captured Segmentation fault (core dumped) … --myname ctrl-3). Under gdb the crash does NOT fire (ptrace perturbs the memory bug); instead the same restarted node's inbound-accept / iomgr path WEDGES — ctrl-3's last cluster.c:737 server sock accepted is 03:42:03, yet it keeps firing its own election timer (pre_vote_start) to 03:51:59. It campaigns but can neither accept peers' PreVote connections nor get answered; ctrl-2's outbound PreVote connects hang on the dead listener (zero connect-fail logs). The two restarted survivors can't exchange votes → permanent round-4 stall. Release SEGV and gdb accept-hang = the same restart-time corruption manifesting differently.
• Restart-STABILITY bug, not a consensus-logic gap. PreVote/eligibility/freshness (incl. the 5.3 fix) are correct; the node is simply I/O-dead after restart. Predates this session — baseline 21e26a1 failed round 3–4 (requirements.md symptom 3). 5.2/5.3 sit in the restart/failover path so 5.4 must rule them out during the backtrace, but they did not introduce the instability.
• Cannot backtrace in this sandbox. Heisenbug: invisible under ASAN (no SEGV) AND under gdb (becomes a hang, not a catchable signal); core_pattern is read-only (pipes to a non-running systemd-coredump) so the release core is discarded. Needs a non-ASAN DEBUG build (-O0 -g) run directly under gdb so the SIGSEGV is caught with symbols, or a host with a live coredump pipe.

Net for 7.2

Elections + job safety are sound for every round that COMPLETES (2-round soak PASSES; 6-round rounds 1–3 elect higher-term primaries; committed prefix byte-identical; zero job loss). The ≥6-round acceptance verdict is blocked by the restart-stability bug (bug-1281 / task 5.4), reachable at round 4 (kill the up-to-date node → two restarted survivors). Spec-owned consensus correctness (Phase-2/5.2/5.3) is unaffected.

Harness wins

start_node now captures per-node stdout+stderr (committed 75243fb) — this exposed the SEGV. The gdb-wrapper (XCE_HA_BIN=<wrapper> → gdb -batch … --args <release ctrl>) is the reusable technique to catch a restarted node's fault once a debug+core env is available.

---

Task 7.2/5.4 — ROOT CAUSE NAILED via gdb: ABBA lock-order deadlock in src/lib async dispatch (UPSTREAM raft-drive-async)

Wrapped the release controller under gdb -batch with a SIGQUIT handler that dumps all-thread backtraces; reproduced the round-4 hang; kill -QUIT'd the two wedged survivors. ctrl-2 and ctrl-3 showed the IDENTICAL two-thread deadlock.

The deadlock (both nodes)

• io_worker (netio.c:612): takes ep->lock (netio.c:672) and, still holding it, runs vtask_enq -> dataq_enq which blocks on q->lock (dataq.c:81). Order: ep->lock -> q->lock.
• vdispatcher_entry: dataq_deq (dataq.c:65) takes q->lock for the WHOLE match scan and runs the match callback vtask_eligible (vtaskmgr.c:255) -> endpoint_is_task_blocked (netio.c:1001) which blocks on ep->lock. Order: q->lock -> ep->lock.
• ABBA: io_worker holds ep wants q; vdispatcher holds q wants ep -> permanent deadlock. io_worker dies -> the node stops accepting / answering PreVotes -> two restarted survivors can't elect -> round-4 stall. In release the same ep/q race under churn corrupts -> SIGSEGV (the earlier crash).

Exact defect + minimal fix

dataq_deq invokes a match callback that acquires a SECOND lock (ep->lock via the endpoint_is_task_blocked eligibility gate) while holding q->lock — inverting the ep->lock->q->lock order io_worker uses. Minimal fix: make ep->busy atomic and read it locklessly in endpoint_is_task_blocked (remove the ep->lock acquire from under q->lock); or io_worker drops ep->lock before dataq_enq; or impose one global lock order.

Ownership: UPSTREAM (raft-drive-async / core libxce async-io), NOT this spec

Entirely in src/lib async-I/O dispatch (netio.c io_worker/endpoint, vtaskmgr.c eligibility, dataq.c) — the VTASK dispatch / endpoint-blocked eligibility layer raft-drive-async owns and this spec's Non-Goals exclude. Per upstream-ownership protocol: route to raft-drive-async (bug-1282), keep 7.2 blocked until the dispatch lock-order is repaired, then re-run the gate. This is the gate working as designed: 7.2 must be GREEN before raft-drive-async flips dispatch flags ON, and it just caught a real dispatch-layer deadlock.

Consensus correctness (this spec) — unaffected

Phase-2 payload integrity, 5.2 spurious-failover suppression, 5.3 catching-up electability are correct and unit-validated; elections + job safety hold for every round that completes. The round-4 blocker is the upstream dispatch deadlock, not consensus logic.

---

bug-1282 FIXED + bug-1283 isolated (the round-4 stall had TWO causes)

Implemented the bug-1282 fix (commit 25e741d): promoted endpoint_t.busy to _Atomic int and made endpoint_block_tasks/endpoint_unblock_tasks/endpoint_is_task_blocked lockless, removing the ep->lock acquire from the vtask-eligibility path that ran under the dataq q->lock.

Verified

• Deadlock gone: post-fix gdb soak shows ZERO __lll_lock_wait threads across all 3 nodes and endpoint_is_task_blocked no longer appears in any lock path. The ABBA inversion is eliminated.
• No regression: controller_unit_test all green (incl. test_ha_core_safety 20/0); libxce + controller + unit test build clean.

But the round-4 stall had TWO independent causes

A plain release soak WITH the fix STILL segfaults a restarted standby at round 4 (ctrl-3, "Segmentation fault (core dumped)"). Deadlocks hang; they do not SIGSEGV — so this crash is a SEPARATE memory bug (now bug-1283), not bug-1282. It is most likely an endpoint-lifetime UAF under chaotic restart (bug-1197 family): the dispatch eligibility path reads vdata->u.iodata.ep / endpoint state while io_worker tears down or mutates the same endpoint. It is a heisenbug — invisible under ASAN and under gdb (gdb perturbs it into a cold-start timeout / no crash) — and the sandbox has no usable core (read-only core_pattern → non-running systemd-coredump), so a backtrace needs a non-ASAN DEBUG (-O0 -g) build under gdb or a host with a live coredump pipeline.

Net for 7.2

The bug-1282 deadlock fix removes ONE of the two round-4 blockers (the iomgr wedge). The remaining blocker is bug-1283 (the restart segfault), likely UPSTREAM raft-drive-async / core libxce endpoint quiescence. 7.2 stays blocked on bug-1283; consensus correctness (Phase-2/5.2/5.3) is unaffected and elections + job safety hold for every round that completes.

---

bug-1286 (the round-4 segfault) ROOT-CAUSED — NULL ep->owner — via the new /xce-build debug profile

The /xce-build ... debug profile added this session (non-ASAN -O0 -g3, no sanitizer → build.dbg) reproduced the heisenbug AND gave a symbolized backtrace under gdb (ASAN and gdb-on-release both perturbed it away). The captured SIGSEGV:

Thread received SIGSEGV
#0 ha_dispatch_gate (ha=0x3ff0, opcode=37)  controller/ha.c:3194  (atomic_load &ha->role)
#1 myreq_handler                            controller/cluster.c:251
#2 vtask_entry                              controller/cluster.c:693
#3 run_task / vcpu_entry                    lib/vtaskmgr.c

Root cause: myreq_handler does cluster_t* c = (cluster_t*)ep->owner; then ha_dispatch_gate(&c->ha, opcode). On a chaotic restart an inbound endpoint dispatches a walCatchupEntry (op 37) vtask while ep->owner == NULL, so c == NULL and &c->ha == NULL + offsetof(cluster_t,ha) == 0x3ff0 — a garbage pointer ha_dispatch_gate dereferences → SIGSEGV. (DISTINCT from the bug-1282 deadlock, which is fixed; deadlocks hang, they do not crash.) Fix: NULL-guard c = ep->owner in myreq_handler (drop/reject the request if the endpoint's owner is not yet attached) instead of dereferencing; deeper, ensure ep->owner is set at accept time before the endpoint is monitored for input and never cleared while a vtask is in flight. Tracked as bug-1286 (an earlier auto-logged entry already held the id bug-1283).

7.2 now has ONE remaining blocker: bug-1286 (root-caused; small NULL-guard fix). bug-1282 (deadlock) is fixed. Consensus correctness (Phase-2/5.2/5.3) is unaffected.

---

Both fixes verified (bug-1282 + bug-1286) — round-4 now a CLEAN election-liveness cross-deny

6-round chaotic soak on real-fs release WITH bug-1282 (deadlock) + bug-1286 (NULL ep->owner crash) both fixed: rounds 1-3 elect; round 4 has NO segfault, NO fdatasync, NO iomgr deadlock (both fixes held, confirmed). The round-4 stall is now the GENUINE consensus-liveness edge the crash/deadlock were masking:

• ctrl-1 (primary term 6) killed round 4.
• ctrl-2 (recovered durable_floor=15, the FRESHER survivor) denies ctrl-3 on FRESHNESS (ctrl-3 advertises 12 < 15) - correct.
• ctrl-3 (floor 12) denies the fresher ctrl-2 on ELIGIBILITY (raft_cb_vote_eligible), repeatedly for 2+ minutes - because ctrl-3 still believes its primary (the dead ctrl-1) is alive (in_catch_up + heartbeat-not-yet-stale, primary_ctrl_fd still >=0). So the legitimate winner (ctrl-2) never gets ctrl-3's grant -> stall.

This is requirements.md symptom #3 (round-N election stall) in pure form. Contributors to chase: (a) ctrl-3's "primary alive" view not expiring within the round - the heartbeat-fresh/in_catch_up eligibility deny (gated on primary_ctrl_fd>=0 && ha_heartbeat_is_fresh by task 5.3) does not lift because last_activity is not going stale; SUSPECT task 5.2's ha_arm_heartbeat_timeout_vtask re-arm resetting last_activity=now on the RDHUP-suppress path keeps a node falsely-fresh after the primary dies - revisit whether the suppress path should reset last_activity. (b) a stale primary_ctrl_fd to the dead primary not cleared promptly. Next Phase-5 task: a node denying on heartbeat-freshness/ in_catch_up must detect a genuinely-dead primary within the failover window and release the deny so the freshest survivor is electable.

Session net

bug-1282 (ABBA iomgr deadlock) and bug-1286 (round-4 restart SIGSEGV, NULL ep->owner) are FIXED and VERIFIED (no crash / no deadlock / no fdatasync across round-4 churn). The remaining 7.2 blocker is this election-liveness cross-deny - the hard core of Phase 5, no longer masked.

---

Task 5.5 fix VERIFIED as progress; round-4 now reaches a DEEPER layer (dueling-candidate term escalation)

Landed the 5.5 fix (commit b8453b2: advertise persist_entry_term(p, log_len) as lastLogTerm instead of the stale c->persist.term) and re-ran the 6-round soak. Round-4 frozen state: ctrl-2 tip 15 @ term 5 (freshest), ctrl-3 tip 11 @ term 4 (staler).

5.5 is correct and made real progress

BEFORE the fix the freshest survivor advertised a too-low lastLogTerm and EVERY voter's logOk rejected it — it could not even win a PreVote. AFTER the fix ctrl-2 WINS PreVote (pre_vote_won) AND enters the real election ("PreVote round WON — entering real election"). The advertised-term asymmetry is fixed; the fresher candidate is now electable in principle.

The deeper layer it exposed (next task)

ctrl-2 wins PreVote -> real election (term N) -> RequestVote -> replies but NO became_leader -> retries at term N+1, N+2 ... prospective term climbing 21->25->28->29 unbounded. Mechanism: ctrl-2's primary (dead ctrl-1) keeps RDHUP'ing ~every 15s; 5.2 does NOT suppress (primary genuinely dead), so each RDHUP re-triggers ha_begin_failover -> fresh PreVote -> real election -> term++. ctrl-2 and ctrl-3 duel in lockstep (each denies the other's real vote on already-voted-this-term / stale-term), and the deterministic ~15s RDHUP cadence defeats the randomized-backoff tiebreak that normally resolves a split vote. Neither reaches majority; the term escalates without converging.

Classic Raft split-vote / dueling-candidates liveness problem, amplified by the RDHUP failover re-trigger. SEPARATE layer from 5.5. Next: break the duel (per-node randomized failover backoff / election jitter that desynchronizes the two survivors; and/or do not re-trigger a fresh failover via RDHUP while a real election the node just entered is still within its timeout window).

Session net (4 layers peeled on the round-4 stall)

1. bug-1282 ABBA iomgr deadlock — FIXED (25e741d).
2. bug-1286 NULL ep->owner restart SIGSEGV — FIXED (5db734a).
3. bug-851/5.5 stale advertised lastLogTerm — FIXED (b8453b2), verified: the fresher survivor now wins PreVote + enters the real election (could not before).
4. REMAINING: dueling-candidate split-vote + unbounded term escalation under the RDHUP failover cadence — the round-4 election round still does not complete. New follow-up. Consensus SAFETY intact throughout (no two leaders; job safety holds every completed round); this is purely election LIVENESS under sustained churn.

---

Task 5.6 fix VERIFIED (unit) + reduces churn; round-4 exposes a 5th layer (vote-reply term-lag escalation)

Landed 5.6 (commit 81cb09a: ha_on_primary_ctrl_fd_rdhup drops an RDHUP while role==CTRL_ELECTING or pre_vote_in_progress, instead of restarting failover). Unit RED→GREEN (test_ha_core_safety 22 Tests / 0 Failures); the soak's term churn is slower (peaked ~24-31 vs ~28-29+ before at the same elapsed time) — the fix does prevent RDHUP from preempting an in-flight election.

The 5th layer (next task 5.7)

Round 4 still stalls. The freshest survivor ctrl-2 (15@t5) repeatedly wins PreVote, enters the real election, AND ctrl-3 GRANTS it (vote_granted) — ctrl-3 is voting FOR ctrl-2, not leapfrogging (ctrl-3 wins only 1 prevote total). But ctrl-2 never tallies a majority: its log shows every grant as vote_reply_stale_drop — the reply's term is LOWER than ctrl-2's current election term. Mechanism: ctrl-2's term escalates via repeated election timeouts faster than ctrl-3 can process the request and reply, so ctrl-3's grant (stamped at ctrl-3's lower term) is always one term behind ctrl-2's current term → DropStaleResponse → not counted → election times out → term++ → the grant for the new term is again stale. Self-perpetuating term escalation; ctrl-2 never wins despite a willing grantor.

election_timeout_ms = 5000, jitter = 200ms (HA_DEFAULT_*). The 5s window should suffice for a warm RTT, so the lag points at: a just-restarted ctrl-2 with COLD peer connections (each real-vote opens a fresh endpoint_connect + SSL handshake; raft_cb_send_vote_request), plus parallel connect attempts to the dead ctrl-1 (DEFAULT_CONNECT_TIMEOUT_MS) — the grant from ctrl-3 lands after ctrl-2 has moved on. Fix directions (5.7): (a) don't bump the election term while a vote round-trip for the current term is still outstanding (give the grant time to land before re-escalating); (b) warm/reuse the peer vote connection so the grant RTT < election_timeout; (c) larger election-timeout jitter so the two survivors desynchronize and one's term stops racing the other's reply. Preserve Election Safety (DropStaleResponse itself is correct Raft — the bug is the escalation cadence, not the drop).

Session net (5 layers peeled on the round-4 stall; 4 fixed)

1. bug-1282 ABBA iomgr deadlock — FIXED (25e741d).
2. bug-1286 NULL ep->owner restart SIGSEGV — FIXED (5db734a).
3. bug-851/5.5 stale advertised lastLogTerm — FIXED (b8453b2), verified prevote-win.
4. 5.6 RDHUP preempts in-flight election — FIXED (81cb09a), unit-verified, reduces churn.
5. REMAINING (task 5.7): vote-reply term-lag / unbounded term escalation under cold-connection churn — the freshest survivor is granted but its term out-races the grant. New follow-up. Consensus SAFETY intact throughout (no two leaders; DropStaleResponse correct; zero job loss every completed round). Purely election LIVENESS convergence under sustained overlapping-kill churn.

---

Task 5.7 fix VERIFIED + round-4 blocker re-rooted (2026-06-30)

5.7 (election-timeout resonance) — FIXED, commit 74784b2.

The 5.6 note guessed the 5.7 cause as "term re-escalates faster than the grant can land." Reading the code pinned the mechanism precisely: the per-CAMPAIGN election watchdog (raft_start_election → on_schedule_timeout → raft_cb_schedule_timeout, ha.c) was armed with a FIXED election_timeout_ms — no randomization. Only the startup-discovery path (ha_wait_for_heartbeat_or_timeout) jittered; the recurring campaign watchdog did not. raft.c:157 passes the bare s->election_timeout_ms straight through.

A fixed deadline + a roughly-constant cold-connection grant RTT RESONATE: if ctrl-3's grant consistently lands a hair after a constant deadline, every campaign loses by the same margin forever → the campaign times out, the term re-escalates, and the now-stale grant is dropped by DropStaleResponse (raft.c:453, which is correct Raft). Terms climb 25→30+. The bug was never "RTT genuinely > 5 s"; it was that the deterministic cadence locks the grant permanently just outside every window.

Fix (Ongaro thesis §3.4 — randomized election timeout, re-drawn each campaign): new ha_jittered_election_delay(ha, base) returns base + rand[0, base] (the canonical [T, 2T] interval), wired into raft_cb_schedule_timeout so the watchdog is re-randomized per campaign. Done in the HA adapter so the pure, sim-tested raft library stays deterministic. Independent of election_jitter_max_ms (which tunes only the sub-second startup/stagger jitter) so it spreads even at jmax=200; never below the floor so genuine-loss failover is not sped up past election_timeout_ms.

Unit: test_5_7_campaign_timeout_is_randomized (CORE) + test_5_7_floor_preserved_and_zero_safe (PIN). RED verified by stubbing the jitter to 0 → the randomization test fails with the resonance message while the PIN stays green. test_ha_core_safety 24 Tests / 0 Failures. (Aside: the full aggregate controller_unit_test binary cannot reach the ha_core_safety suite because an EARLIER suite, test_ha_vep_teardown, SIGSEGVs in iomgr_fini→SSL_CTX_free under openssl 3.6 — unrelated to this change; run the suite in isolation with -t test_ha_core_safety.)

Soak confirmation: a fresh ASAN 6-round chaotic soak now elects with BOUNDED terms — rounds 1→3 elect ctrl-3/ctrl-2/ctrl-1 at terms 4→5→6 (not 25→30+). The unbounded escalation is gone; zero job loss on every completed round (3/6/9/12 tracked jobs all present on each new primary). The election-liveness layer is DONE.

Round-4 blocker RE-ROOTS to a non-election crash (task 5.8 / bug-1298). Round 4 still stalls, but no longer for any election reason. ctrl-3 ABORTs on an ASAN heap-use-after-free:

UAF READ size 4  endpoint_is_task_blocked  netio.c:1008      (vtask dispatcher T3)
                 vtask_eligible vtaskmgr.c:255 ← dataq_deq_first_match ← vdispatcher_entry
FREED BY (T2)    endpoint_killer  cluster.c:813               (io_worker)
                 handle_endpoint_error cluster.c:108 ← handle_error ← io_worker
ALLOC            endpoint_get netio.c:748 ← handle_connection cluster.c:741

io_worker (T2) tears down a connection on error and free()s the endpoint via endpoint_killer while the vtask dispatcher (T3) concurrently reads ep->busy on that just-freed endpoint during its dataq eligibility scan. The endpoint is freed while still referenced by a pending vtask; nothing drains its queued vtasks before the free and no refcount keeps it alive across the dispatcher read. ctrl-3 dies → the cluster drops to 1/3 (the round-4 survivors are the just-restarted ctrl-2 and a now-dead ctrl-3) → no quorum → "NO new primary within window." ctrl-2's logs show only PreVote-round timeouts (prospective term=6, never wins) — the textbook symptom of a lost quorum peer, not an election defect.

Same netio surface and routing precedent as task 5.4/bug-1282: an async-io dispatch concurrency bug, UPSTREAM-owned by raft-drive-async / core libxce. Likely a latent UAF that the bug-1282 lockless ep->busy read unmasked (the dispatcher now reaches the read without contending on ep->lock). Logged as bug-1298; tracked as task 5.8. Consensus safety (Phase-2/5.2/5.3/5.5/5.6/5.7) is sound; elections and job safety hold for every round that completes.

---

Cross-Spec Gap Analysis (2026-06-30)

Generated by /kiro-validate-gap ha-term-safety raft-drive-async raft-ha-core-safety ha-consistency-invariants.

Summary

4-spec HA safety family: ha-term-safety COMPLETE, raft-drive-async DORMANT (all tasks done, production flags OFF), raft-ha-core-safety IN PROGRESS (12/26 tasks, Phase 5 critical path — task 5.7), ha-consistency-invariants DESIGN ONLY. One blocker to production: raft-ha-core-safety 7.2 chaotic soak PASS.

Dependency Graph

ha-consistency-invariants (umbrella — defines invariants)
  ├── ha-term-safety (term recording + reconcile) ← COMPLETE
  ├── raft-ha-core-safety (payload safety + election liveness + soak gate) ← IN PROGRESS
  └── raft-drive-async (non-blocking drive threads) ← DORMANT
        gates on: raft-ha-core-safety 7.2 PASS

Implementation State

Spec	Tasks Done	Tasks Remaining	Status
ha-term-safety	20/20	0	COMPLETE (build green, 175 tests, clean failover verified)
raft-ha-core-safety	12/26	14 (5.7 critical)	IN PROGRESS (Phases 1-2+4+5.1-5.4 done; Phase 3 not started)
raft-drive-async	17/17	0 (dormant flags)	DORMANT (production flags OFF; gates on raft-ha-core-safety 7.2)
ha-consistency-invariants	0/0	design only	DESIGN ONLY (no tasks generated)

Critical Path

raft-ha-core-safety 5.7 (vote-reply term-lag) → 7.2 (soak PASS) → raft-drive-async flags ON

Remaining Gaps

1. Task 5.7 (NOT STARTED): vote-reply term-lag — candidate's term escalates faster than grant RTT, all grants arrive stale. Fix: don't bump election term while vote round-trip outstanding, or warm peer connections.
2. Task 7.2 (BLOCKED on 5.7): acceptance chaotic soak gate — ≥6 rounds, zero divergence, zero job loss. Currently: rounds 1-6 pass, round 7 stalls on PreVote cross-denial (bug-1296). Election liveness is the LAST remaining data-path gap; payload safety (Phase 2) and reconvergence (Phase 4) are proven.
3. Phase 3 (snapshot-boundary reconcile): lowest priority — snapshot install is rare (16-byte sentinel today).
4. raft-drive-async flags: peerctrl/snapshot/configpush all default OFF. Flag-flip gate = raft-ha-core-safety 7.2 PASS + borrowed-ep refcount for snapshot/configpush lane.
5. ha-consistency-invariants: umbrella spec with approved design, no tasks. Close by verifying all invariants are satisfied by the three implementation specs.

Risk

Risk	Likelihood	Impact
Task 5.7 reveals further election liveness layers (5.5→5.6→5.7 chain)	HIGH	Schedule slip; budget 2-3 more iterations
Chaotic soak still fails after 5.7	MEDIUM	7.2 blocks raft-drive-async; consider sequential-only fallback
Raft-drive-async flag-flip reveals new races	LOW	Gradual rollout: peerctrl first

Recommendations

1. Immediate: Fix task 5.7 (vote-reply term-lag) — approach (a): don't bump term while vote outstanding.
2. Short-term: Complete raft-ha-core-safety 5.7→7.2→7.3, then flip raft-drive-async flags.
3. Medium: Generate tasks from ha-consistency-invariants design; close the umbrella loop.

---

Task 5.8 fix VALIDATED + round-4..6 reveals pre-existing commit-loss (2026-06-30)

5.8 (endpoint teardown-vs-dispatch UAF, bug-1298) — FIXED, commit 6e05dc1.

Chosen fix (user picked the principled option over the targeted deferred-teardown patch): an endpoint REFERENCE COUNT consolidating the prior ad-hoc lifetime patches. endpoint_t.refcnt (_Atomic) + heap_alloc; endpoint_get → refcnt=1 (owner); vtask_enq takes a per-vtask ref BEFORE publishing the task to the dispatcher, run_task / taskq_item_killer release it; every heap-ep teardown drops its owner ref via endpoint_unref (last unref fini+frees). Embedded eps (client ctx->ep, heap_alloc=0) are never freed by unref. So the dispatcher reads ep->busy on a queued vtask without the io_worker freeing it underneath — the UAF is structurally impossible. Also fixed several latent ep-struct leaks (endpoint_fini-only sites).

Validation: controller_unit_test 182 tests / 0 failures (test_ha_vep_teardown OFF-path test updated — it asserted the now-fixed struct leak). 6-round ASAN chaotic soak: ZERO AddressSanitizer reports, ctrl-3 survives round 4, ALL 6 rounds elect at bounded terms (5→6→7→8). The round-4 crash-stall is gone.

The soak now reaches rounds 4-6 for the FIRST TIME — and reveals a pre-existing committed-job-loss (task 5.9, NOT a regression).

Verdict: committed_job_loss=4, committed_divergence=7 → FAIL. But this is NOT a regression of 5.8 nor of any previously-fixed bug. Proof:

1. ONE specific committed job (0-0-1782829181-1) goes missing at round 4 and stays missing through rounds 5 and 6, across THREE different elected primaries (ctrl-3, ctrl-2, ctrl-3). A permanent committed-prefix gap that propagates as catch-up streams FROM the gapped node — the exact bug-795/828/851 signature.
2. Rounds 1-3 are CLEAN (all jobs present, zero divergence). Loss begins precisely at round 4 — the first round whose overlapping kills elect a JUST-RESTARTED node (ctrl-2 restarted, ctrl-1 killed, ctrl-3 — itself recently restarted — elected). A protocol/catch-up condition, not a memory condition.
3. This session changed ZERO WAL/commit/catch-up/replication code (persist.c, config_repl.c, commit.c, repl*). The files touched are cluster.c, ha.c/.h, netio.c, vtaskmgr.c (ep refcount + the 5.7 election timeout) + 2 test files. The commit-durability path is byte-identical to the session start — it CANNOT have regressed because it did not change.
4. ASAN is fully clean — no refcount-induced UAF/double-free/leak. A refcount bug would crash (caught by ASAN) or corrupt broadly; it cannot selectively delete one committed job's payload from a WAL.
5. No chaotic soak EVER reached rounds 4-6 before (every prior run crashed/stalled at round 3-4: bug-851 cross-deny → bug-1282 deadlock → bug-1286 segv → 5.5/5.6/ 5.7 election stalls → bug-1298 UAF). There is no prior green baseline for rounds 4-6 to regress FROM — this is a first observation.

Conclusion: bug-795/828/851 FAILOVER CATCH-UP GAP — a restarted node rejoins and is elected primary without backfilling the full committed prefix (empty-rejoin catch-up streams [watermark..committed], no snapshot backfill → permanent gap). EXTERNAL spec ha-commit-durability (+ raft term/log-safety). Tracked as task 5.9; route there.

Phase 1 closeout — 10-round chaotic soak VALIDATES the election layer (2026-06-30)

Re-ran the chaotic soak at the full 10-round budget (run-ha-soak.sh --chaotic 10, deploy /home/claude/tmp/ha-1296) to confirm bug-1296 (the round-7 RDHUP→PreVote→timeout liveloc) is gone after tasks 5.5/5.6/5.7.

Election liveness: PASS (10/10). Every kill round elected a new primary in a single term increment — terms walked 4→5→6→7→8→9→10→11→12→13, one step per round, ZERO escalation. Round 7 (the historical bug-1296 liveloc point) and rounds 8–10 all elected cleanly. The per-campaign jittered watchdog (5.7) + don't-preempt-in-flight-election (5.6) fully resolve the resonance. bug-1296 marked FIXED.

Consistency: FAIL (expected, task 5.9 / EXTERNAL). Final verdict committed_divergence=11 committed_job_loss=5 result=FAIL. Job 0-0-1782854622-0 went missing at round 8 and stayed absent on the reachable primary through rounds 8/9/10 and the settle — a permanent committed-prefix gap. This is the documented bug-795/828/851/1306 failover catch-up gap (a just-restarted node elected primary without backfilling the full committed prefix), NOT an election fault and NOT a regression: rounds 1–7 are job-loss-free, the loss onset tracks kill timing, and the election machinery is perfect across all 10 rounds. The per-round compared=3 present=2 divergence before round 8 is the just-restarted node lagging its backfill (no convergence wait), which the catch-up backfill (Phase 2) closes.

Phase 1 (election liveness) is DONE. The remaining soak FAIL is wholly Phase 2 (task 5.9 / ha-commit-durability catch-up backfill).