# Implementation Plan: raft-ha-core-safety > **TDD (project steering, mandatory):** every implementation sub-task is preceded by a testsuite sub-task that lands failing (RED) tests first, then implements to GREEN. The deterministic harness (Phase 1) is the RED infrastructure that pins every later fix to a `walcompare`/invariant violation. > **No `(P)` markers:** Phases 2–6 modify shared files (`ha.c`, `persist.c`, `config_repl.c`) and the RED cases share `test_ha_core_safety.c`, so the plan is sequential — concurrent edits would contend (same constraint as `ha-convergence-recovery`). Foundation Phase 1 establishes shared prerequisites and is also sequential. > **Correctness oracle:** verify every change against `raft/raft.tla` and the invariant checklist in `src/controller/CLAUDE.md` (Election Safety, Leader Completeness, State Machine Safety, Commit Ordering/Durability). > **Verify anchors before editing** (files drift): the design and `research.md` carry 2026-06-27 file:line anchors; re-grep before each edit. > **Root cause (task 1.3, DONE + verified):** the term-blind hypothesis is **refuted**. The real defect is **cross-term catch-up payload loss** — `ha_stream_wal_to_standby` silently ships `PLEN=0` on a short `pread`/`malloc`-fail, and the `(term,lsn)`-only persist guard is payload-blind. Phase 2 targets the streamer + payload-aware guard (`research.md` §"Task 1.3"). A green synthetic unit test over a still-red soak does not satisfy 1.1/8.1 — closure requires the real-fs chaotic soak (7.2) clean. - [ ] 1. Foundation: deterministic chaotic harness and live-divergence reproduction - [x] 1.1 Extend the divergence detector to a machine-checkable per-member report - Extend `walcompare` to emit a parseable per-member `(lsn -> term, payload-crc, present/MISSING)` report over the committed prefix, and classify each divergence as committed vs uncommitted - Observable completion: `walcompare` prints a parseable per-member report and a committed-vs-uncommitted classification; the existing soak still detects the round-3 LSN divergence through it - _Requirements: 7.1, 7.5_ - _Boundary: Deterministic Chaotic Harness_ - [x] 1.2 Add a chaotic soak mode with an isolated verdict - Add an overlapping-kill mode to the soak driver (kills landing inside election windows, not only sequential kill→wait→rejoin); produce a verdict that counts `walcompare` divergences and committed-job loss while excluding client-discovery (bug-854) and overlayfs `fdatasync` (bug-924) failures from the divergence/job-loss result - Observable completion: the soak runs chaotic mode against real-fs release binaries and emits a verdict whose divergence/job-loss counts are unaffected by bug-854/bug-924 - _Requirements: 7.5, 8.4_ - _Boundary: Acceptance Soak Gate, Deterministic Chaotic Harness_ - [x] 1.3 Reproduce and classify the live divergence, then bisect to a seam - Run the chaotic soak to capture the actual `walcompare` committed-prefix divergence (RED), classify it via `(term, lsn, pcrc)` (which member, which term, committed vs uncommitted), and bisect it to the concrete ingest path; confirm or refute the term-blind-bypass hypothesis - Observable completion: a recorded RED divergence with member/term/lsn classification and the identified seam, written to `research.md`; the hypothesis is confirmed or a new trigger is named before any fix - _Requirements: 7.1, 1.1_ - _Boundary: Deterministic Chaotic Harness_ - [x] 1.4 Stand up the unit/integration harness infrastructure - Create `test_ha_core_safety.c` with the deterministic gate/probe seams (no sleeps; teardown drains pending delaymgr tasks to avoid the bug-1224 teardown-race UAF), the `(term, lsn)`/apply-gap/kill-injection fixtures, and register it in the controller unit-test build; include one smoke-RED case proving the harness can observe a divergence. The per-defect RED bodies are owned by their phases (2.3 ingest, 4.1 reconvergence, 5.1 election) — this task provides the shared seams they build on, not those cases - Observable completion: `test_ha_core_safety` builds and runs, the injection fixtures and probe seams are usable by later phases, and the smoke-RED case FAILs on the baseline with no ASAN teardown errors - _Requirements: 7.1, 7.4_ - _Boundary: Deterministic Chaotic Harness_ - [x] 1.5 Make soak primary-detection robust to the release-build log buffer - Task 1.3 found the release build block-buffers its `--logfile` and flushes only on signal, so the soak's log-grep primary/convergence detection cannot see an elected primary and false-aborts. Switch the soak's primary + convergence detection to a client probe (`ctrlstatus`/`xce_test`) instead of log-grep, so the chaotic soak runs unattended on release binaries. (Only if a client probe is insufficient, fall back to a controller release log-flush change that does not alter production logging semantics.) - Observable completion: `run-ha-soak.sh --chaotic` on the `~/tmp` real-fs path with the `src/build.release` binaries detects the elected primary each round via the client probe and no longer false-aborts on "no primary within Ns"; a short (2-round) run reaches its verdict line - _Requirements: 7.5, 8.3_ - _Boundary: Acceptance Soak Gate, Deterministic Chaotic Harness_ - _Depends: 1.2_ - [ ] 2. Committed-prefix divergence fix: cross-term catch-up payload integrity + payload-aware guard - _Root cause (task 1.3, verified): the cross-term catch-up streamer silently ships `PLEN=0` on a short `pread`/`malloc`-fail, and the `(term,lsn)`-only persist guard is payload-blind. Term and ingest-path are correct — the payload is lost._ - [x] 2.1 RED: payload-aware persistence guard tests - Add cases: an empty-payload (`PLEN=0`) entry written at an existing LSN N followed by the correct full-payload entry must end with the FULL payload (today the `(term,lsn)`-only idempotency guard silently no-ops the corrective write); the reverse (full then empty at the same `(term,lsn)`) must NOT degrade the stored entry to empty; an identical `(term, payload-crc)` re-write is a safe idempotent no-op - Observable completion: the new persistence cases FAIL on the baseline because the guard compares only `(term, lsn)` and cannot see the payload difference - _Requirements: 1.1, 1.2, 2.1, 2.2, 3.1, 3.4_ - _Boundary: Persist Conflict Guard_ - [x] 2.2 Make the persistence guard `(term, lsn, payload-crc)`-aware - Extend the LSN-only idempotency check (persist.c:822): identical `(term, payload-crc)` at an LSN is an idempotent no-op; a differing payload-crc (or term) at an existing LSN is surfaced for repair, never a silent `return 0`; never accept a `PLEN=0` write as the durable form of a known-non-empty entry; preserve persist-before-ACK, the term index, and `WAL_LEGACY_TERM` - Observable completion: the 2.1 cases pass; an empty-over-real write at an existing committed LSN no longer silently survives; identical re-writes stay idempotent - _Requirements: 1.1, 1.2, 2.1, 2.2, 2.4, 3.1_ - _Boundary: Persist Conflict Guard_ - [x] 2.3 RED: cross-term catch-up streamer payload-integrity tests - Add a case reproducing the live signature: a cross-term catch-up (stored term < leader current term, bug-1067 path) whose entry-body `pread` returns short / `malloc` fails must NOT result in a persisted `PLEN=0` entry at a committed LSN — instead the stream must error and retry. Also add the defensive case that a `term==0`-shaped catch-up entry routes through the reconcile seam (no direct-persist bypass), and the unrecoverable-fence case (a node that cannot obtain the majority-agreed committed prefix refuses promotion instead of serving divergent state). Instrument `payload_len` at send and at `on_persist_entry` to disambiguate send-side-empty vs transitive-empty propagation - Observable completion: the streamer case FAILs on the baseline because the short-read/malloc-fail path silently ships `PLEN=0`; the seam-routing and fence cases FAIL because those paths/fence do not exist yet - _Requirements: 1.1, 1.2, 2.1, 3.1, 3.2, 3.3, 7.1_ - _Boundary: Catch-up Payload Integrity, Unified Entry Ingest_ - _Depends: 2.2_ - [x] 2.4 Fix the streamer payload integrity + defensive seam routing + fence - Core: in `ha_stream_wal_to_standby` (config_repl.c ~2154-2163), fail loud on a short `pread`/`malloc`-fail with a bounded retry instead of freeing the buffer and shipping `PLEN=0`; a non-empty entry is never transmitted empty. Disambiguate and handle send-side-empty vs transitive-empty propagation per the 2.3 instrumentation. Defensive: retire the term-blind `ha_apply_single_entry` direct-persist branch so all ingest routes through the seam; surface genuinely-unrecoverable divergence as a loud non-promotable fence (never self-terminate); defer file I/O to delaymgr (never the callback/iomgr thread) - Observable completion: 2.3 passes (streamer never ships empty; seam-routing and fence cases green); on the 1.3 reproduction path (real-fs release chaotic soak) `walcompare` reports zero committed divergence at the previously-divergent LSNs - _Requirements: 1.1, 1.2, 1.3, 2.1, 2.3, 3.1, 3.2, 3.3, 3.4_ - _Boundary: Catch-up Payload Integrity, Unified Entry Ingest_ - _Depends: 2.2, 2.3_ - [ ] 3. Snapshot-boundary reconciliation - [ ] 3.1 RED: snapshot post-install reconcile test - Add a case where a divergent pre-snapshot entry must be reconciled against the installed checkpoint's `(term, lsn)` boundary (today snapshot install sets applied-lsn without a `(term, lsn)` reconcile) - Observable completion: the case FAILS on the baseline because no `(term, lsn)` reconcile occurs at the snapshot boundary - _Requirements: 3.1, 5.1_ - _Boundary: Snapshot Boundary Reconcile_ - [ ] 3.2 Establish the snapshot `(term, lsn)` boundary - Make snapshot install establish a verifiable `(term, lsn)` at the checkpoint so post-install AE intake reconciles correctly and no committed entry below the checkpoint can diverge - Observable completion: 3.1 passes; a member that installed a snapshot matches the leader's `(term, payload-crc)` at and below the checkpoint boundary - _Requirements: 3.1, 5.1_ - _Boundary: Snapshot Boundary Reconcile_ - _Depends: 2.4_ - [ ] 4. Standby reconvergence completeness - _Root cause (Phase-4/5 investigation, `research.md`): a recovered standby holding a divergent uncommitted tail above the new leader's committed point (former-primary shape, e.g. ctrl-2 with 12-15 at an old term while leader committed 11) never reaches `ae_already_have` — it loops on `ae_reject_logmismatch`, its control channel flaps, and it never rejoins. A short-suffix standby (ctrl-3) DOES reconverge, so the backtrack works for small divergence but not the former-primary tail._ - [x] 4.1 RED: divergent-tail reconvergence + apply-gap gate tests - Primary case: construct a recovered standby with a MULTI-entry divergent uncommitted tail above the leader's committed point and assert it reconverges to follower — truncates the conflicting suffix and reaches `ae_already_have` (no permanent `ae_reject_logmismatch` loop). Secondary case: a rejoining node with an injected catch-up apply gap must not be reported converged/healthy until `applied_lsn` and `log_len` equal the leader's committed boundary - Observable completion: both cases FAIL on the baseline (the divergent-tail case loops on log-mismatch / never truncates; the apply-gap case reports healthy at match-index alone) - _Requirements: 5.1, 5.2, 5.3, 5.4, 7.2_ - _Boundary: Standby Reconvergence_ - [x] 4.2 Verify divergent-tail reconvergence + applied-agreement gate (no production fix needed) - **Discriminator outcome (4.1, adversarially confirmed):** the follower-side AE log-matching backtrack + follower-AE-conflict-truncate path ALREADY reconverges a multi-entry former-primary divergent tail (truncates 12-15 old-term, accepts the leader's term-6 entries, preserves committed prefix 1..11) when driven deterministically WITHOUT a channel flap; and the applied-agreement healthy gate (`ha_process_standby_state`) already blocks "converged" until `applied_lsn == committed_lsn`. Neutering `ha_consume_append_entries` makes the test fail (`reached=0`), proving the real path does the work. **No production change required in Phase 4** — the live stall is NOT follower-side reconvergence. The fix moves to Phase 5 (channel-flap suppression + `in_catch_up`-blocks-PreVote-reply). - Observable completion: 4.1's divergent-tail + apply-gap tests pass as regression pins; no production diff in Phase 4 (verified by the discriminator) - _Requirements: 5.1, 5.2, 5.3, 5.4_ - _Boundary: Standby Reconvergence_ - _Depends: 2.4_ - [ ] 5. Election liveness and safety under churn - _Root cause (Phase-4/5 investigation, `research.md`): the live stall is NOT a missing election-restart (retry works) — it is a recovered standby treating a TRANSIENT control-channel RDHUP as "primary died" and launching an UNWINNABLE PreVote-failover storm while a current-term primary is alive (peers deny/ignore, round times out, reconnect, repeat ~15s). The node retries an election it can never win instead of rejoining the live primary._ - [x] 5.1 RED: spurious-failover suppression + election liveness + Election Safety tests - Primary case: a recovered standby whose primary control channel flaps (transient RDHUP) while a current-term primary is alive and quorum-reachable must NOT enter an unbounded `ha_begin_failover`/`PreVote timed out` loop — it reconnects and rejoins as follower. Liveness case: on GENUINE primary loss, repeated rapid kill/restart rounds yield a higher-term primary each round; a scripted failed round is followed by another election. Safety case: across overlapping kills, at most one member ever reaches committing-primary per term (Election Safety, mapped to `raft/raft.tla`) - Observable completion: the spurious-failover case FAILS on the baseline (unbounded prevote-timeout loop while a live primary exists); the liveness cases pass/fail per the stall; the Election Safety assertion runs and reports per-term primary counts - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3, 7.3_ - _Boundary: Election Liveness_ - [x] 5.2 Suppress unwinnable failover on transient RDHUP; keep genuine-loss liveness and single-leader safety - On a control-channel RDHUP, attempt reconnect + catch-up first, and/or back off failover when a node's own PreVotes repeatedly fail to reach quorum because a current-term leader exists — only escalate to a real election when the primary is genuinely unreachable to the quorum. Preserve: genuine primary loss still elects a higher-term primary (no permanent stall while quorum reachable); step-down-on-higher-term paired with election restart; at-most-one-committing-primary-per-term - Observable completion: 5.1 spurious-failover case passes (a recovered standby rejoins a live primary instead of prevote-spinning); genuine-loss liveness still completes each round; the Election Safety assertion still holds - _Requirements: 4.1, 4.2, 4.3, 6.1, 6.2, 6.3_ - _Boundary: Election Liveness_ - _Depends: 4.2_ - [x] 5.3 Lagging-but-safe survivor must be electable under overlapping churn (NEW — surfaced by the 7.2 soak) - _Root cause (task 7.2 first soak, `research.md` §"Task 7.2"): distinct from the 5.2 flap. After ≥3 overlapping kills the only fully-up-to-date node is killed and the two survivors are both restarted-and-lagging (e.g. tips 15 and 11 after the leader at 19 dies). Their PreVote rounds repeatedly time out with no grant quorum — prospective terms frozen, never bumped — so no primary is elected, even though the fresher survivor holds the FULL committed prefix (Leader Completeness satisfied) and is a SAFE candidate. Observed contributors: freshness mutual-denial between two lagging survivors, the in_catch_up-denies-all-votes gate (Req 2.5; obs 1240) with no live primary left to catch up FROM, and/or peer-to-peer vote connections never established between two nodes that each only held a control channel to the now-dead primary._ - RED: deterministically construct the round-4 shape (kill the up-to-date leader; two survivors each holding the full committed prefix but at different lagging tips, both flagged in_catch_up, no live primary) and assert the freshest survivor reaches CTRL_PRIMARY at a higher term within the failover window (no permanent PreVote-timeout loop). Reuse the 1.4 kill/election + apply-gap seams + the 5.1 PreVote dispatch seam. - Fix: the catching-up/freshness vote gates must not deadlock when the primary is gone — a survivor that holds the majority-agreed committed prefix must be able to grant/win a PreVote once no live primary remains (drop in_catch_up for vote-eligibility when the primary is unreachable AND the candidate is ≥ the voter's committed floor; ensure the peer vote connection is established during failover). Preserve Election Safety (≤1 primary/term), Leader Completeness (never elect a candidate below the committed floor), and the 5.2 spurious-failover suppression. - Observable completion: the RED passes; a 6-round chaotic soak elects a new primary after EVERY kill round (no round-N stall) with zero committed-job loss; Election Safety still holds - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_ - _Boundary: Election Liveness_ - _Depends: 5.2_ - _Note: UNIT clause DONE (RED→GREEN, commit bf88681; `test_ha_core_safety` 20 Tests / 0 Failures). The 6-round-soak clause shares 7.2's environment block — the soak re-run still stalls at round 4, but the cause is the bug-924 WAL-fdatasync wedge (a survivor jams its persist flusher on `fdatasync`→EINVAL and stops answering PreVotes), NOT the eligibility deadlock this task fixes. See `research.md` §"Task 5.3 + 7.2 re-run". The eligibility fix is a real latent-defect fix proven at unit level; its live confirmation rides on 7.2's env resolution._ - [x] 5.4 Root-cause the chaotic-restart round-N blocker (DONE: it is an UPSTREAM dispatch deadlock — route to raft-drive-async) - **ROOT-CAUSED via gdb (bug-1282):** the round-4 hang is an **ABBA lock-order deadlock in `src/lib` async dispatch** — `io_worker` (netio.c:672) holds `ep->lock` then blocks on `q->lock` (`dataq_enq`), while `vdispatcher`'s `dataq_deq` (dataq.c:65) holds `q->lock` then blocks on `ep->lock` via the `vtask_eligible`→`endpoint_is_task_blocked` (netio.c:1001) match callback. io_worker dies → the node stops accepting/answering PreVotes → two restarted survivors can't elect → stall. The release-build SIGSEGV is the same `ep`/`q` race corrupting under churn. Confirmed by identical all-thread backtraces on two independently-wedged nodes (gdb SIGQUIT-bt; `research.md` §"ROOT CAUSE NAILED"). **NOT a consensus-logic gap** (PreVote/eligibility/freshness incl. 5.3 are correct). - **Ownership = UPSTREAM raft-drive-async / core libxce async-io dispatch** (vtaskmgr/dataq/netio), which this spec's Non-Goals exclude. Per the upstream-ownership protocol, the FIX is routed there (bug-1282; minimal: make `ep->busy` atomic + lockless in `endpoint_is_task_blocked`, or io_worker drops `ep->lock` before `dataq_enq`, or one global lock order). 7.2 stays blocked until the upstream lock-order is repaired, then re-run the gate. The 5.4 obligation (root-cause to a concrete site + assign ownership) is met. - _Requirements: 5.1, 6.2, 7.4, 8.3_ - _Boundary: Standby Reconvergence, Election Liveness (verification only — fix is upstream)_ - _Depends: 5.3_ - [x] 5.5 Release the heartbeat-fresh/in_catch_up vote deny on genuine primary death within the failover window (NEW — the unmasked round-4 stall) - _CLOSED (commit b8453b2; advertised-lastLogTerm fix live-verified). Checkbox flipped: the election-liveness layer is validated end-to-end by the 5.7 6-round soak (bounded-term election every completed round). Kept open historically only because its observable was the green soak, which 5.7 delivered._ - _Root cause (7.2 soak with bug-1282+bug-1286 fixed, `research.md` §"Both fixes verified"): with the crash + deadlock gone, round 4 stalls on a pure election-liveness cross-deny. The freshest survivor (ctrl-2, floor 15) is the legitimate winner, but the other survivor (ctrl-3, floor 12) denies it on ELIGIBILITY (`raft_cb_vote_eligible`) for minutes because ctrl-3 still believes its primary (the dead ctrl-1) is alive (in_catch_up + heartbeat-not-yet-stale, primary_ctrl_fd≥0). The "primary alive" view does not expire within the round → the freshest survivor never gets a grant._ - Investigate/fix: a node denying a PreVote on heartbeat-freshness or in_catch_up must detect a genuinely-dead primary within the failover window and release the deny so the freshest survivor is electable. Prime suspect: task 5.2's `ha_arm_heartbeat_timeout_vtask` re-arm resets `last_activity=now` on the RDHUP-suppress path, keeping a node falsely-fresh after the primary dies — revisit whether the suppress path should reset last_activity (or should only re-arm the watchdog without bumping the freshness clock). Also ensure a stale `primary_ctrl_fd` to a dead primary is cleared promptly. RED: two divergent-freshness survivors, primary killed; assert the freshest reaches CTRL_PRIMARY within the failover window (no multi-round eligibility cross-deny). Preserve Election Safety + the 5.2 spurious-failover suppression (do not re-introduce the flap). - Observable completion: the RED passes; a 6-round chaotic soak elects after every kill round (no round-N cross-deny stall) with bug-1282/bug-1286 fixes in place - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_ - _Boundary: Election Liveness_ - _Depends: 5.2, 5.3_ - _ACTUAL root cause (NOT the last_activity hypothesis) + PARTIAL FIX (commit b8453b2): the cross-deny was a stale **advertised lastLogTerm** — the candidate advertised `c->persist.term` (the persist running term, stale/0 after restart) instead of `persist_entry_term(p, log_len)` (the term of the last WAL entry, the same source voters use via on_get_entry_term). The freshest survivor advertised a too-low term → every voter's logOk rejected it. bug-851 family. FIX VERIFIED AS PROGRESS: the fresher survivor now WINS PreVote + enters the real election (could not before). But the round still does not COMPLETE → a deeper layer remains (task 5.6). Detail in `research.md` §"Task 5.5 fix VERIFIED"._ - [x] 5.6 Break the dueling-candidate split-vote / unbounded term escalation under the RDHUP failover cadence (NEW — the layer under 5.5) - _CLOSED (commit 81cb09a, don't-preempt-in-flight-election; reduced term churn). Checkbox flipped: 5.7's per-campaign jittered watchdog is the final fix that bounds the term escalation this task targeted; 6-round soak elects with bounded terms. Layer DONE._ - _Root cause (`research.md` §"Task 5.5 fix VERIFIED"): with 5.5's advertised-term fix in, round-4 ctrl-2 (freshest) wins PreVote and enters the real election, but never reaches majority — it retries at ever-higher terms (21→25→28→29…). Mechanism: the dead primary keeps RDHUP'ing ~every 15s; 5.2 correctly does not suppress (primary genuinely dead), so each RDHUP re-triggers `ha_begin_failover` → fresh PreVote → real election → term++. The two survivors duel in lockstep (each denies the other's real vote on already-voted-this-term / stale-term), and the deterministic ~15s RDHUP cadence defeats the randomized-backoff tiebreak that normally resolves a split vote._ - Fix direction: introduce/strengthen per-node randomized failover backoff (election jitter) so two surviving candidates DESYNCHRONIZE and one wins; and/or do not re-trigger a fresh failover via RDHUP while a real election the node just entered is still within its election-timeout window (let the in-progress round complete or time out before re-arming). Preserve genuine-loss liveness (R6), spurious-failover suppression (5.2), and Election Safety. RED: two divergent-freshness survivors + a periodic RDHUP cadence; assert a primary is elected within a bounded number of rounds (term does not escalate unbounded). - Observable completion: the RED passes; a 6-round chaotic soak elects after every kill round (no unbounded term escalation / split-vote stall) with all prior fixes in place — the green 7.2 gate - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_ - _Boundary: Election Liveness_ - _Depends: 5.5_ - _PARTIAL FIX (commit 81cb09a): the RDHUP-entry now drops an RDHUP while role==CTRL_ELECTING or pre_vote_in_progress (don't preempt an in-flight election). Unit RED→GREEN (22 Tests/0 Failures); reduces the term churn. But the soak still stalls at round 4 → a deeper 5th layer (task 5.7): the freshest survivor is GRANTED but its term out-races the grant (vote_reply_stale_drop). Detail in `research.md` §"Task 5.6 fix VERIFIED"._ - [x] 5.7 Stop the vote-reply term-lag / unbounded term escalation under cold-connection churn (NEW — the 5th layer under 5.6) - _FIXED + VERIFIED (commit 74784b2, `research.md` §"Task 5.7 fix VERIFIED"). Actual root cause was narrower than the 5.6 note guessed: the per-CAMPAIGN election watchdog was armed with a FIXED `election_timeout_ms` (`raft_start_election`→`on_schedule_timeout`→`raft_cb_schedule_timeout`); only the startup-discovery path jittered. A fixed deadline + a roughly-constant cold-connection grant RTT RESONATE — the single needed grant lands a hair past every deadline forever, the campaign times out, the term re-escalates, and the now-stale grant is dropped (DropStaleResponse correctly). Fix (Ongaro §3.4): re-randomize the watchdog per campaign to the canonical [T, 2T] interval, injected in the HA adapter so the pure (sim-tested) raft library stays deterministic; independent of `election_jitter_max_ms`; never below the floor. New `ha_jittered_election_delay()` + 2 unit tests (RED verified — stubbing jitter to 0 fails `test_5_7_campaign_timeout_is_randomized` with the resonance message; PIN guards floor + modulo-by-zero). `test_ha_core_safety` 24 Tests / 0 Failures. SOAK-CONFIRMED: a fresh 6-round chaotic soak now elects with BOUNDED terms (rounds 1→3 at terms 4→5→6, NOT 25→30+); zero job loss across completed rounds. The election-liveness layer is DONE._ - _Root cause (`research.md` §"Task 5.6 fix VERIFIED"): with 5.6 in, round-4 ctrl-2 (freshest, 15@t5) wins PreVote, enters the real election, and ctrl-3 GRANTS it (vote_granted) — ctrl-3 votes FOR ctrl-2, not leapfrogging. But ctrl-2 never tallies a majority: every grant arrives as vote_reply_stale_drop (reply term < ctrl-2's current term). ctrl-2's term escalates via repeated election timeouts faster than ctrl-3 can reply, so each grant is one term stale → dropped → timeout → term++ → self-perpetuating. election_timeout_ms=5000, jitter=200ms._ - Observable completion: the RED passes; a 6-round chaotic soak elects after every kill round (bounded term, no escalation stall) with all prior fixes in place — the green 7.2 gate ✅ (election layer met; 7.2 now blocked on a SEPARATE non-election crash, task 5.8) - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_ - _Boundary: Election Liveness_ - _Depends: 5.6_ - [x] 5.8 Endpoint teardown-vs-dispatch heap-use-after-free crash (NEW — the round-4 7.2 blocker after 5.7) - _FIXED + VALIDATED (commit 6e05dc1, `research.md` §"Task 5.8 fix VALIDATED"). With 5.7's election fix in, the round-4 stall RE-ROOTED off elections entirely: ctrl-3 ABORTed on an ASAN heap-use-after-free — the vtask dispatcher read `ep->busy` (endpoint_is_task_blocked, netio.c) on a queued vtask while the io_worker freed that ep via `endpoint_killer` (cluster.c, from handle_endpoint_error). ctrl-3 died → cluster 1/3 → no quorum → "no new primary."_ - _FIX (chosen by user over the targeted deferred-teardown patch): a principled endpoint REFERENCE COUNT, consolidating the prior ad-hoc lifetime patches (bug-1282 lockless busy, bug-1197 dispatch_inflight). endpoint_t.refcnt (_Atomic) + heap_alloc; endpoint_get starts refcnt=1 (owner); vtask_enq takes a per-vtask ref BEFORE publishing the task to the dispatcher, run_task/taskq_item_killer release it; every heap-ep teardown site drops its owner ref via endpoint_unref (last unref fini+frees). Embedded eps (client ctx->ep, heap_alloc=0) are never freed by unref. So the dispatcher reads ep->busy on a queued vtask without the io_worker freeing it underneath — UAF eliminated. Also fixed several latent ep-struct leaks (endpoint_fini-only sites)._ - _VALIDATION: controller_unit_test 182 tests / 0 failures (test_ha_vep_teardown updated — its OFF-path test asserted the now-fixed struct leak). 6-round ASAN chaotic soak: ZERO AddressSanitizer reports, ctrl-3 survives round 4, ALL 6 rounds elect at bounded terms (5→6→7→8). The round-4 crash-stall is gone._ - Observable completion: the 6-round chaotic soak runs all rounds with NO controller crash ✅ (met). 7.2 still NOT green — see below. - _Requirements: 6.1, 6.2, 6.3_ - _Boundary: Async-IO dispatch (endpoint lifetime)_ - _Depends: 5.7_ - [x] 5.9 Committed JOB LOSS under chaotic failover — FIXED (the durability half; divergence tracked separately in 5.10) - _ROOT CAUSE (sharpened 2026-06-30, waldump+log bisect — NOT the catch-up prefix-gap the earlier note guessed): `xce_run_hooks` (src/lib/hook.c:134-157) returns only the LAST-registered hook's rc. The durable-majority gate worked (a term-9 leader's job at LSN 30 timed out commit → `commit_submit` -ETIMEDOUT → `persist_job_hook_fun` returned -ETIMEDOUT), but a later JOB AFTER_EVENT hook (sched/apiserver) returned 0 and CLOBBERED it → `handle_jobsubmit` replied `jobsubmit ok` for an uncommitted entry → a term-10 leader legally overwrote LSN 30 → committed-job-loss. Violated the Commit Durability invariant (reply-success ⇒ majority-durable). All 3 WALs were gapless/CRC-intact/full-payload (rules out prefix-trim + bug-847); lost job only on ctrl-1@LSN30 term9 vs term-10 NOOP on the majority._ - _FIX (raft-apply-decoupling item #1 down payment, commit pending): extracted the WAL append + majority-commit barrier out of `persist_job_hook_fun`'s WAIT branch into an explicit `persist_submit_wal_and_commit(p, jlist)` (persist.c/.h); `handle_jobsubmit` (job.c) calls it directly and reads the commit rc DIRECTLY — rejects on <0 BEFORE firing the now notification-only AFTER hook (canonical commit→notify→reply). The commit verdict is no longer a clobber-prone hook return. Regression pin `test_job_hook_new_is_notify_only` + 4 relocated hook tests; persist_hooks 19/0, e2e_hooks 3/0, commit_barrier_reply 22/0, commit_durability 38/0._ - _PROVEN: 10-round chaotic soak `committed_job_loss` 5 → **0** (same election code, A/B isolates the durability fix)._ - _Boundary: Commit durability (raft-apply-decoupling item #1). Full canonical inversion (async keystone + apply_committed_entry seam + facts + restart validation) = the follow-on ha spec._ - _Depends: 5.8_ - [ ] 5.10 Residual soak FAILs after 5.9 (NOT job loss) — divergence lag [5.10a, OPEN] + election-escalation tail [5.10b, FIX LANDED] - _5.10b DONE (bug-1310, commit pending): RDHUP failover DEBOUNCE. Root cause = a dead primary's fd RDHUPs faster than an election settles; the 5.6 point-in-time guard misses RDHUPs that land after a lost election (role briefly back to STANDBY) → term++ per RDHUP (48 begin_failovers, terms 6→24). Fix: ha_state_t.last_failover_ms stamped at ha_begin_failover; ha_on_primary_ctrl_fd_rdhup drops an RDHUP within 2×election_timeout of the last trigger. Unit-proven (test_rdhup_within_cooldown_is_debounced); soak 10/10 elect, terms 4→14 bounded, begin_failover 14/14/12 (was 48), no regression — but the stochastic storm did not recur this run (debounce fired 0×), so soak-reproduction of the storm is still pending. 5.10a (divergence catch-up lag) remains OPEN below._ - _(a) committed_divergence>0 from the just-restarted node's catch-up LAG (walcompare `present=2`: entry on the 2 live nodes, not yet backfilled on the restarted one). Transient, settles; NOT lost data. Closed by entity-bearing snapshots / the full raft-apply-decoupling apply seam (EXTERNAL — ha-commit-durability / ha-convergence-recovery). (b) Election-escalation TAIL: under an adversarial overlap (kill inside a split-vote window) a round can still stall + ladder terms (observed round-4 term 6→25 before electing) despite 5.7's jitter — liveness held (elected next round) but degraded. Harden: cap/dampen term escalation or widen the jitter under repeated same-round failovers._ - Observable completion: a 10-round chaotic soak verdict is committed_divergence=0 with no per-round election stall — the green 7.2 gate - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_ - _Boundary: catch-up backfill (EXTERNAL) + election-liveness tail (5.7 family)_ - _Depends: 5.9_ - [ ] 6. Integration: single-seam invariant end-to-end - [ ] 6.1 Verify leader→standby replication carries (term, payload) faithfully end-to-end - Integration check that the live leader replication AND the cross-term catch-up streaming path transmit each entry's full `(term, payload)` to the receiver: no `term==0` on the wire from a live leader (defensive), and — the real concern — no non-empty entry ever arrives with `PLEN=0`; the walAck catch-up continuation feeds only the seam ingest path - Observable completion: an end-to-end test (or instrumented soak round) shows leader and standby hold identical `(term, payload-crc)` for every replicated and cross-term-caught-up LSN; `walcompare --report` clean across members - _Requirements: 2.1, 2.3, 3.2_ - _Boundary: Catch-up Payload Integrity, Unified Entry Ingest_ - _Depends: 2.4_ - [ ] 6.2 Verify the single-seam invariant and restart identity - Assert no ingest path (live replication, catch-up batch, snapshot post-install, rejoin) writes the WAL outside the reconcile seam, and that each entry's `(term, lsn)` identity survives a restart/recovery. **Carry-forward from 2.4:** the term-blind `ha_apply_single_entry` direct-persist branch was NOT structurally retired in 2.4 (its empty-over-full safety is enforced by the 2.2 payload guard and pinned by `test_2_3_termblind_route_payload_guarded`/`test_2_3_unrepairable_catchup_raises_nonpromotable_fence`). This task must either structurally retire that bypass (route term-0 catch-up through the seam, with its own RED first) OR formally document it as safe-by-persist-guard and adjust the invariant wording accordingly - Observable completion: an integration test confirms every ingest path reconciles through the seam and that entry terms are preserved across restart; the term-blind bypass is either removed (no direct-persist path remains) or documented safe-by-guard with the pin tests cited - _Requirements: 1.1, 1.2, 2.4, 3.1_ - _Depends: 2.4, 3.2, 4.2_ - [ ] 7. Validation: regression and acceptance soak gate - [ ] 7.1 Full controller suite under ASAN with the new harness green - Build and run the full controller unit suite under ASAN (suite-by-suite if the aggregate hits the known 300s cap); fix any regression from the ingest/persist/election changes - Observable completion: the full suite is green under ASAN and every `test_ha_core_safety` case passes with no ASAN errors - _Requirements: 7.4_ - _Depends: 6.2, 5.2_ - [ ] 7.2 Acceptance chaotic-failover soak gate - Run the chaotic soak (≥6 rounds, overlapping kills) against real-fs release binaries and assert zero `walcompare` divergence, zero committed-job loss, and 3-of-3 reconvergence plus a new primary after every kill round, with bug-854/bug-924 isolated from the verdict - Observable completion: the soak exits PASS over ≥6 chaotic rounds with zero divergence and zero job loss — the green result that gates the `raft-drive-async` production flag-flip - _Requirements: 1.4, 7.4, 8.1, 8.2, 8.3, 8.4_ - _Depends: 1.5, 6.2, 5.2, 5.3, 5.4_ - _Blocked on bug-1283 (the round-4 stall had TWO independent causes). (1) bug-1282 — an ABBA lock-order deadlock in `src/lib` async dispatch (`io_worker` ep->lock→q->lock vs `vdispatcher` `dataq_deq` q->lock→ep->lock via `endpoint_is_task_blocked`) — is **FIXED** (commit 25e741d: `endpoint_t.busy` → `_Atomic int`, lockless; verified zero `__lll_lock_wait` post-fix; unit suite green). (2) bug-1286 — a SEPARATE release-only SIGSEGV on a restarted standby at round 4 (deadlocks hang, they don't crash) — is now **ROOT-CAUSED** (via the new `/xce-build debug` non-ASAN `-O0 -g3` profile + gdb): `myreq_handler` (cluster.c:251) does `c = ep->owner` then `ha_dispatch_gate(&c->ha, …)`; on a chaotic restart a `walCatchupEntry` (op 37) vtask runs with `ep->owner == NULL`, so `&c->ha == 0x3ff0` → deref crash at ha.c:3194. Fix = NULL-guard `c = ep->owner` in `myreq_handler` (+ ensure owner is set at accept time / not cleared mid-flight). is **FIXED** (commit 5db734a: NULL-guard `c = ep->owner` in myreq_handler). With BOTH bug-1282 and bug-1286 fixed, the round-4 soak shows NO crash / NO deadlock / NO fdatasync — but still stalls on a THIRD, now-unmasked cause: **task 5.5** — a pure election-liveness cross-deny (the freshest survivor is denied on eligibility because the other survivor still believes its dead primary is alive). 5.5 (advertised-term) is FIXED (b8453b2, verified: fresher survivor now wins PreVote + enters real election); 5.6 (don't preempt an in-flight election) is FIXED (81cb09a, unit-verified, reduces churn); the remaining 7.2 blocker is now **task 5.7** — vote-reply term-lag / unbounded term escalation under cold-connection churn (the freshest survivor is granted but its term out-races the grant). Consensus safety (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs primary `committedLsn`). Detail in `research.md` §"Both fixes verified". Re-run 7.2 after 5.5 lands._ - [ ] 7.3 Hypothesis-validation closeout - Confirm the live soak divergence is eliminated by the implemented fixes (not by synthetic unit tests alone); if any residual divergence remains, classify it via `walcompare (term, lsn, pcrc)`, record the new finding in `research.md`, and add a follow-up fix task rather than declaring closure - Observable completion: the soak divergence count is zero with closure justified by the live soak, OR a recorded new finding plus a follow-up task; `raft/raft.tla` invariants (Election Safety, Leader Completeness, State Machine Safety, Commit Ordering) reviewed against the final behavior - _Requirements: 1.1, 1.4, 7.1, 8.1_ - _Depends: 7.2_ ## Implementation Notes - **Task 2.2 — payload repair must be TIP-ONLY (consensus-safety, caught in review round 1).** The `(term,lsn,payload-crc)`-aware guard repairs an empty-over-real entry by truncate-to-`lsn-1`+re-append, but this is safe ONLY when `lsn == current_lsn-1` (the conflict is at the tip with nothing committed above). For a BELOW-tip conflict (`lsn < current_lsn-1` — the actual live LSN-9 case with 10/11/12 above), truncating the tail would drop committed entries and roll `committed_lsn` backward (State Machine Safety violation). Below-tip is handled as a loud non-promotable fence (reuses `recovery_unrecoverable` sticky flag), never mutating any boundary; the real reconvergence comes from the 2.4 streamer fix + re-stream. The tip path snapshots/restores `committed_lsn` under `p->lock` so the boundary is never observably lowered. Any future in-WAL repair must preserve: never drop a committed entry above the conflict, never lower `committed_lsn`. - **Task 5.1 — RED lands (3 cases in `test_ha_core_safety.c`).** CORE RED `test_5_1_spurious_failover_suppressed_while_primary_alive` FAILs on baseline (`Expected 0 Was 5`): it drives the REAL `ha_on_primary_ctrl_fd_rdhup` (unit-test `c==NULL` path) 5× with a FRESH heartbeat (`ha_heartbeat_is_fresh()==1`) and counts PreVote launches — baseline launches one unwinnable round per flap. **Root-cause asymmetry the fix targets:** the DEFERRED PreVote path (`ha_failover_stagger_fn`, ha.c ~6664) ALREADY guards with `ha_heartbeat_is_fresh` and skips, but the RDHUP-INITIATION path (`ha_on_primary_ctrl_fd_rdhup`→`ha_begin_failover`→`ha_start_prevote_async` synchronous/`c==NULL` branch) has NO such guard. **Task 5.2 GREEN = add the `ha_heartbeat_is_fresh` guard at the RDHUP initiation path** (reconnect+rejoin instead of campaigning) WITHOUT regressing the two PASSing pins: `test_5_1_genuine_loss_elects_and_advances_term` (genuine loss — stale heartbeat — must STILL launch; failed round re-arms; rapid rounds advance term) guards against over-suppression, and `test_5_1_election_safety_one_primary_per_term` (vote-once via the real `handle_request_vote`, A=3/B=0 → ≤1 primary/term) is the falsifiable Election Safety check mapped to raft.tla. Note: `handle_request_vote` already denies same-term challengers when `primary_ctrl_fd>=0 && ha_heartbeat_is_fresh` — the VOTE path is safe; only failover-INITIATION is not. - **Task 5.2 — GREEN (the one-guard fix, `ha.c:ha_on_primary_ctrl_fd_rdhup`).** Added the spurious-failover-suppression guard between the fd-close and the failover INFO: `if (rdhup_role == CTRL_STANDBY && ha_heartbeat_is_fresh(ha)) { log; if (c) ha_arm_heartbeat_timeout_vtask(ha,c); return; }`. A transient RDHUP while a current-term primary is alive (fresh heartbeat) now stays CTRL_STANDBY and awaits reconnect instead of launching an unwinnable PreVote — the same guard `ha_failover_stagger_fn` already applies on the deferred path. Genuine loss (stale heartbeat) falls through unchanged; no term bump / no role change, so Election Safety is untouched. The watchdog re-arm (production-only, `c!=NULL`) resets `last_activity` to now for a fresh grace window, so a primary that truly never heartbeats again still fires a real failover after `heartbeat_timeout_ms`. **Why one guard suffices over the `in_catch_up`-denies-PreVote facet (obs 1240):** that deny is correct Raft (Req 2.5) and only deadlocks if catch-up never finishes — suppressing the flap keeps the standby attached so catch-up completes, `in_catch_up` clears, and votes flow. Validation: `test_ha_core_safety` 17 Tests / 0 Failures (spurious `launches=0/5` GREEN; genuine-loss `final_term=8`; Election Safety A=3/B=0→1 primary); zero failing suites binary-wide; LSan unchanged (174379 B / 1553 allocs = pre-fix baseline — suppression path allocates a liveness ctx only under real `c`, untouched by the c==NULL unit tests). Live ≥6-round chaotic soak proof is task 7.2. - **Task 2.4 — root-cause source fix + a deferral.** `ha_stream_wal_to_standby` now reads the entry body via `stream_read_entry_body` (bounded retry, max 4 no-progress attempts, partial-read-accumulating, no sleep); on persistent short `pread`/`malloc`-fail it emits a loud ERROR and ABORTS the stream (flushes the valid contiguous prefix, drops the unreadable entry, returns `-EIO`) instead of shipping `PLEN=0`. The incomplete stream is re-driven by the ACK-driven catch-up continuation. Combined with the 2.2 self-healing guard, this closes the live cross-term payload-loss divergence. **Deferred:** the structural retirement of the term-blind `ha_apply_single_entry` bypass — its empty-over-full safety is enforced by the 2.2 guard + pinned by two passing tests, so it cannot corrupt committed state; the structural removal is tracked to task 6.2. Live ≥6-round soak proof is task 7.2.