myxce/.kiro/specs/raft-ha-core-safety/research.md

# Gap Analysis — raft-ha-core-safety

_Generated by `/kiro-validate-gap`. Information over decisions: options and findings, not final choices._

## Headline Finding (reframes the spec)

The `(term, lsn)` commit-identity + reconcile + commit-safety machinery the requirements describe **already exists and is wired at the baseline**. Confirmed:

- **Library half is 100% complete and validated**: `raft-log-replication` (24/24, G1/G2/G3/G5) and `raft-commit-safety` (18/18, G4 Figure-8 + G6 vote freshness). All five callbacks present in `src/lib/raft/raft.{c,h}`.
- **Controller adapter is 16/20 wired** by `ha-term-safety`: all five callbacks implemented in `s_real_cbs` (`ha.c:1480/1539/1656/1711/1739`), primary append routed through `raft_log_append` (`ha.c:3408+`), heartbeat clean (no `on_persist_entry` on `n_entries==0`), per-LSN term index (`persist.c:414/476/487/752`), config markers, and `walcompare` (`src/controller/testsuite/walcompare.c`).
- **Baseline `21e26a1` already contains all of it.** `git`: `21e26a1` is itself a `raft-drive-async` task-2.3 commit, ancestor of HEAD, and contains `s_real_cbs` + `walcompare`. So the machinery is NOT absent at baseline.

**Yet the live chaotic-failover soak still fails on this branch AND on baseline `21e26a1`**: `walcompare FAIL: committed-prefix divergence at LSN 9` (round 3) and standby `ctrl-2` rejoin failure (round 4), with intermittent round-N election stall.

**Therefore raft-ha-core-safety is NOT a build-the-machinery spec. It is a residual-defect + convergence/liveness + live-verification spec.** The work is: root-cause why divergence survives the existing reconcile path under overlapping chaotic failover; close the convergence and election-liveness gaps no completed spec actually closed; and finish the live chaotic soak that `ha-term-safety` left permanently BLOCKED on env. **Do not rebuild the `(term,lsn)`/callback machinery — extend and correct it.**

## Requirement-to-Asset Map

| Req | Capability | Existing asset | Gap tag |
|-----|-----------|----------------|---------|
| R1 Committed-prefix safety | quorum commit + Figure-8 guard | `raft_advance_commit` raft.c:769-821; `on_commit_advanced` ha.c:1739 | **Constraint** — machinery present but live soak still diverges at LSN 9 → residual defect |
| R2 Per-entry commit identity | term stamped from leader; per-LSN term index; legacy-0→1 | raft.c:571-580; `persist_set_term` persist.c:476; `record_entry_term` persist.c:414/752; `WAL_LEGACY_TERM` persist.c:456 | **Done (verify)** — confirm no path still emits term-0 under chaos |
| R3 Divergence detect + repair | logOk + conflict-truncate-then-persist on AE intake | raft.c:701-765; `on_truncate_from` ha.c:1711; `walcompare` detector | **Constraint** — repair fires on AE-intake path only; rejoin/catch-up path may bypass it (see Research Needed #1) |
| R4 Single leader per term | term++ on election, vote-once, step-down, pre-vote, vote freshness | raft.c:105-249; `raft_cb_vote_eligible` ha.c:342 | **Constraint** — two-leaders-per-term window under churn not yet proven closed live |
| R5 Standby rejoin → 3-of-3 | catch-up apply, applied-lsn floor, snapshot backfill, contiguity guard | `ha_apply_wal_catchup_entry` ha.c:2233; `ha_advance_applied_lsn` ha.c:2078; `ha_handle_snapshot_chunk` ha.c:2492; contiguity persist.c:839 | **Missing (effective)** — bug-1042 standby-never-converges has NO dedicated owning task; `ha-convergence-recovery` claims it but soak still fails round 4 |
| R6 Election liveness under churn | election restart, candidate advertise floor | `raft_start_election` raft.c:105; `ha_candidate_advertise_lsn` ha.c:3921 | **Missing (effective)** — round-N stall not closed; no deterministic churn test |
| R7 RED→GREEN harness | `xce-test-ha` soak, `walcompare`, `controller_unit_test`, `test_ha_dispatch_integration` | `.claude/skills/xce-test-ha/`; `walcompare.c`; `src/xce.build/controller/controller_unit_test` | **Missing** — no deterministic harness that *forces* chaotic divergence/rejoin/stall as a RED unit/integration gate (current soak is probabilistic) |
| R8 Acceptance soak gate | real-fs release soak; BIN/XT overridable | `run-ha-soak.sh` (BIN/XT env override added this branch); non-ASAN release build | **Unknown** — never passed at 6+ chaotic rounds; was BLOCKED on overlayfs `fdatasync` (bug-924) |

## Existing-Code Constraints (must respect)

- **Reconcile is single-seam**: divergence repair happens only inside `raft_on_append_entries_request` (logOk → `on_truncate_from` → re-persist, raft.c:730-765). Any code path that brings a node's log forward *without* routing through this function (e.g. a snapshot/catch-up stream that writes WAL directly) will not detect or repair a divergent committed prefix.
- **Truncate must never cut the majority-committed prefix** (raft.c monotonic commit; promotion truncate-clamp `max(safe_committed_lsn, durable_tip)` ha.c:999-1000, bug-1060 fix). R3.4 depends on this clamp holding under chaos.
- **Persist-before-ACK durability is owned by `ha-commit-durability`, not this spec.** R1/R2 must not weaken it. `on_commit_advanced` is a barrier trigger, not the durability mechanism.
- **`raft-drive-async` is out of scope and dormant.** Do not change SlowLane / peer-control dispatch / endpoint-quiescence behavior (Boundary Context).
- **Client-side bug-854** (libxce primary discovery) can fail the soak harness at EXIT-1 even when server data safety passes. Must be isolated so it does not mask/forge R8 verdicts.

## Implementation Approach Options

### Option A — Extend & correct existing machinery (recommended)
Keep all library + adapter machinery. Root-cause the residual divergence, route the rejoin/catch-up path through the reconcile seam, close the convergence + liveness gaps with targeted fixes, add a deterministic chaotic harness, finish the live soak.
- ✅ No rebuild of consensus-critical code that already passed library + clean-failover validation
- ✅ Smallest blast radius; aligns with TLA+ already-proven invariants
- ❌ Requires precise root-cause of a hard-to-repro chaotic defect before fixing
- ❌ Risk of "fix the symptom not the invariant violation" if harness stays probabilistic

### Option B — New convergence/divergence-repair subsystem
Build a standalone post-failover reconcile/repair component separate from the AE-intake seam.
- ✅ Clean place to own rejoin reconcile + unrecoverable-divergence fencing
- ❌ Duplicates logic that already lives in `raft_on_append_entries_request`; two reconcile paths is exactly the class of bug being fixed
- ❌ High risk of new two-leaders/two-reconcile divergence

### Option C — Hybrid (Option A + new deterministic harness component)
Extend/correct in place (A), but build the **deterministic chaotic harness** (R7) as genuinely new test components (forced overlapping kills, injected divergent prefix, scripted round-N stall) rather than relying on the probabilistic soak.
- ✅ Keeps consensus code as A; isolates the new surface to test infrastructure where new code is safe
- ✅ Gives a RED gate that pins each fix to an invariant violation
- ❌ Harness design is itself non-trivial (deterministic chaos, no sleeps, gate/probe seams like `test_ha_dispatch_integration.c`)

## Effort & Risk

| Item | Effort | Risk | Justification |
|------|--------|------|---------------|
| Root-cause residual LSN-9 divergence | M | High | Chaotic, intermittent; survives existing reconcile — invariant violation is subtle |
| Route rejoin/catch-up through reconcile seam (R3/R5) | M | High | Touches recovery + commit prefix; must not cut majority-committed |
| Election liveness under churn (R6) | M | Medium | Election machinery exists; needs churn-stall root cause + restart guarantee |
| Deterministic chaotic harness (R7) | L | Medium | New, but test-only; pattern exists (`test_ha_dispatch_integration.c`) |
| Live 6+round real-fs soak (R8) | M | High | Was permanently blocked on overlayfs `fdatasync` (bug-924); needs real-fs env + bug-854 isolation |
| **Total** | **L–XL** | **High** | Consensus correctness under chaos + env-gated live verification |

## Research Needed (carry to design phase)

1. **Why does divergence survive the reconcile seam?** Trace the rejoin/catch-up path (`ha_apply_wal_catchup_entry` ha.c:2233, snapshot `ha.c:2492`): does it route through `raft_on_append_entries_request` so logOk/`on_truncate_from` fire, or does it write WAL directly and bypass conflict repair? Hypothesis: catch-up streams `[watermark..committed]` directly (per memory bug-795/828 family) → no `(term,lsn)` conflict check on rejoin.
2. **Is LSN-9 divergence in the committed or uncommitted prefix?** If committed, the violation is upstream (two-leaders-per-term or premature commit-advance), not a repair gap. Use `walcompare` `(lsn→term,pcrc)` output to classify.
3. **Two-leaders-per-term under overlapping kills** — does pre-vote + vote-once fully close the window when kills overlap election timeouts? Map against `raft/raft.tla` Election Safety.
4. **bug-1042 ownership** — `ha-convergence-recovery` claims the rejoin path but soak still fails round 4. Determine whether its tasks are actually landed on this branch or stranded, to avoid duplicate work.
5. **R8 env** — confirm real-fs (ext4) soak runs now via `run-ha-soak.sh` BIN/XT override + non-ASAN release build; quantify how bug-924 (overlayfs) and bug-854 (client discovery) are excluded from the R8 verdict.

## Recommendation for Design Phase

- **Preferred: Option C** (extend/correct in place + new deterministic chaotic harness). Treat the spec as residual-defect + convergence + liveness + live-verification, explicitly **not** a machinery rebuild.
- **Key design decisions to make**: (a) route rejoin/catch-up through the single reconcile seam vs. add a guarded repair at promotion; (b) classify LSN-9 as committed vs uncommitted before choosing repair vs upstream fix; (c) harness determinism mechanism (forced-kill gates/probes) for R7; (d) how R8 isolates bug-854/bug-924 so the gate measures only server-side committed-job safety.
- **Carry forward**: Research Needed #1–#5; verify every change against `raft/raft.tla` (Election Safety, Leader Completeness, State Machine Safety, Commit Ordering/Durability) per `src/controller/CLAUDE.md`.

---

# Design-Phase Discovery (kiro-spec-design)

## ROOT CAUSE (term-blind catch-up bypass) — ⚠️ SUPERSEDED / REFUTED by task 1.3

> **This hypothesis was tested live (task 1.3) and REFUTED.** The real defect is cross-term catch-up *payload* loss (see §"Task 1.3 — Live divergence reproduction & bisect" below). The divergent entry carries the correct term and arrives through the reconcile seam; the term-blind path is a latent hazard but not the observed cause. Retained below for the diagnostic record.

Deep-trace of the standby catch-up/rejoin paths found a **split** where one branch bypasses the single reconcile seam:

| Path | Entry | Routes through `raft_on_append_entries_request`? | logOk | conflict-truncate | term stamped |
|------|-------|---|---|---|---|
| 1 Term-carrying | `entry->term != 0` → `ha_apply_term_entry_via_library` (ha.c:2173) → `ha_consume_append_entries` (ha.c:2383) → raft.c:646 | YES | YES (raft.c:701) | YES (raft.c:735) | leader term via `persist_set_term` |
| 2 **Term-blind (defect)** | `entry->term == 0` → `ha_apply_single_entry` (ha.c:2097) → `persist_wal_write_replicated` (persist.c:802) DIRECT | **NO** | **NO** | **NO** | **standby's own `p->term`** (persist.c:862) |
| 3 Snapshot | `ha_handle_snapshot_chunk` (ha.c:2492) → `persist_load_snapshot` sets applied_lsn=checkpoint | NO | N/A | N/A | N/A (canonical state) |
| 4 Stream+ACK | `ha_stream_wal_to_standby` (config_repl.c:2032) feeds Path 1 or 2 by `entry->term` | depends | depends | depends | depends |
| 5 `ha_rejoin_reconcile` | ha_convergence.c:254 — **now a no-op stub** (committed-watermark truncate retired; relies on library) | NO | N/A | retired | N/A |

**Divergence mechanism (LSN 9):**
1. An entry reaches a standby via Path 2 (term-blind). `persist_wal_write_replicated` stamps `hdr.term = standby p->term` (wrong) and records it (`record_entry_term`, persist.c:902). No logOk, no conflict check.
2. The correct term-carrying entry for the same LSN later arrives via Path 1. The persist **idempotency guard** (`if lsn+1 <= p->current_lsn return 0`, persist.c:822) **silently no-ops** — the divergent entry is never overwritten/repaired.
3. `walcompare` (testsuite/walcompare.c:135) compares `(term, payload_crc)` per committed LSN; the member with the term-blind write shows a different `term` → `WALCMP_DIVERGENT`. `match_term` maps raw 0 → `WAL_LEGACY_TERM`, so legacy-0 vs genuine-term at the same LSN reads as divergence.

**Why the existing `(term,lsn)` machinery does not catch it:** all conflict detection lives *inside* `raft_on_append_entries_request`; Path 2 never enters it, and the idempotency guard treats "LSN already present" as "done" regardless of term.

## Synthesis

> ⚠️ **D1/D3 partially superseded by task 1.3.** The "eliminate every bypass of the one seam / collapse Path 2 into Path 1" framing assumed the term-blind hypothesis, now refuted. The single-seam invariant (D2) and harness-first discipline (D4) still hold; but the actual fix is the cross-term catch-up streamer payload integrity + a `(term,lsn,payload-crc)`-aware guard (see §"Task 1.3"), not seam-bypass elimination. Term-blind retirement remains as defensive hardening only.

- **Generalization (D1):** R1/R2/R3/R4 are one underlying invariant — *every* entry that enters a member's log must pass the single `(term,lsn)` reconcile seam. The fix is not four mechanisms but **eliminating every bypass of the one seam** + making persistence `(term,lsn)`-conflict-aware rather than LSN-only-idempotent.
- **Build vs adopt (D2):** Adopt the existing library seam (`raft_on_append_entries_request` + `on_truncate_from` + `persist_set_term`) — already validated (raft-log-replication/raft-commit-safety). Build nothing new in the consensus core; only route into it and harden the persist guard. Reuse `ha_rejoin_reconcile` intake hook (currently stub) rather than a second reconcile path.
- **Simplification (D3):** Do not add a separate "divergence repair subsystem" (gap-analysis Option B rejected) — that would create the exact second-reconcile-path class of bug being fixed. Collapse Path 2 into Path 1.
- **Harness-first (D4):** The live bug is intermittent/chaotic. Make it deterministic: a RED unit test injects a term-blind/divergent entry at an LSN and asserts the seam repairs it; a chaotic soak mode forces overlapping kills. RED before any fix (R7), pins each fix to a `walcompare`/invariant violation.

## Open risks carried to tasks
- Confirm Path 2 is the *only* divergence source live (harness RED test will confirm/refute; if soak still diverges after Path-2 closure, classify next via `walcompare (term,lsn,pcrc)`).
- Snapshot boundary (Path 3): post-install AE intake must reconcile against the snapshot's `(term,lsn)`; verify no committed entry below the checkpoint can diverge.
- Standby reconvergence (R5) and election-liveness (R6) are separable seams from divergence-repair; phaseable but kept in one spec per the unified acceptance gate.
- bug-854 (client discovery) and bug-924 (overlayfs fdatasync) must be excluded from the R8 verdict so the gate measures server-side committed-job safety only.

---

# Design Review Resolution (kiro-validate-design)

Three critical issues raised and folded into design.md (GO):

1. **Repair-vs-clamp conflation (correctness).** Split truncate into two contracts: *promotion-truncate* keeps the bug-1060 `max(safe_committed_lsn, durable_tip)` clamp; *follower-AE-conflict-truncate* obeys Raft Log Matching and truncates the conflicting suffix regardless of local `durable_tip` (a conflicting entry was never majority-committed → safe to cut). Applying the promotion clamp on the follower path would have converted repairable divergence into a permanent fence and failed R8. Updated: §Existing Architecture, §Unified Entry Ingest, §Persist Conflict Guard, §Error Handling, Traceability 3.x, Testing.
2. **R4 two-leaders-per-term unverified.** Added a falsifiable Election Safety assertion (≤1 committing-primary per term under overlapping kills, mapped to raft.tla); failure promotes a concrete adapter-gate guard from verification to required work. Updated: §Election Liveness, Testing Integration.
3. **Hypothesis-validation risk.** RED gate (7.1) must reproduce the *live* `walcompare` divergence and bisect to a seam before synthetic `(term,lsn)` injection counts; baseline 21e26a1 already retired live term-0 (ha-term-safety 3.4), so the term-blind path may not be the live trigger. Contingency: if soak still diverges after Path-2 closure, re-classify via `walcompare (term,lsn,pcrc)` and add a fix task — do not mark R1/R8 done on green unit tests alone. Updated: §Deterministic Chaotic Harness, Testing E2E + RED-first.

Verdict: **GO** (unconditional after edits). Architecture unchanged; refinements were bounded contract clarifications.

---

# Task 1.3 — Live divergence reproduction & bisect

_Generated by the task-1.3 investigation subagent (REPRODUCE + CLASSIFY + BISECT). Evidence-backed; no production code changed. Real-fs (ext4) `~/tmp`, non-ASAN release binaries `src/build.release/controller/{xce_controller,xce_test}` (built 08:48 this day)._

## TL;DR / VERDICT

**The design's primary "term-blind catch-up bypass" hypothesis is REFUTED for the reproduced live committed divergence.** The divergent committed entry has the **CORRECT term**, was ingested **through** the single library reconcile seam (`ae_append` → `on_persist_entry`), and the LSN-only idempotency no-op was **not** the mechanism. The real defect is **payload loss on cross-term catch-up**: a committed `JOB_SUBMIT` entry is persisted on the catching-up member with `payload_len=0` (empty body) while its `(term, lsn)` are correct. It is a sibling of the bug-795/828/1060 committed-prefix-loss family, located in the **catch-up replication payload path**, not in the `(term,lsn)`/idempotency-guard seam Phase 2 currently targets.

## Commands run & reproduction status

- Two `--chaotic` soak attempts via `run-ha-soak.sh` (rounds=6, `XCE_HA_BIN/XT` → release binaries, dir `~/tmp/ha-1p3`) **both aborted at cold-start** with `FAIL: no primary elected within 45s`. A `--sequential` attempt failed identically.
- **This cold-start "election stall" is a HARNESS/LOG-BUFFER ARTIFACT, not a consensus stall.** Evidence: in every aborted run a primary WAS elected on time (e.g. `ctrl-1 → CTRL_PRIMARY (term=2)` logged ~15s after start, `quorum_reached`), yet `wait_for_primary_log` never tripped. All three nodes' last log line is the identical cleanup-kill timestamp (= start+45s): the 08:48 release build **block-buffers its `--logfile` and only flushes on signal**, so the harness's log-grep primary detection (`latest_primary_term`) cannot observe progress live. The OLD binaries used by the prior passing sequential soaks (`~/tmp/xce-soak-{rel,bl}.log`, 02:56/03:01) did not have this buffering behaviour. The client (`ctrlstatus`) path is unaffected.
- **Reproduced cleanly** by driving the cluster with a client-based (log-buffer-immune) driver (`scratchpad/drive.sh`, fresh deploy `~/tmp/ha19`, sequential kill→restart→re-elect, 6 rounds). Committed-prefix divergence appeared by round 1 and persisted: round 1 LSN 8 (present=2), round 2 LSN 8 (present=2), **round 3 LSN 9 (present=3 — all members hold it but disagree)**. Driver wedged at round 4 on a transient quorum loss; the frozen WAL dirs were classified directly.

## walcompare --report classification (frozen `~/tmp/ha19`, `--committed 25`)

Decisive divergence — **committed LSN 9, class=committed, compared=3, present=3** (every member holds the entry; they disagree on payload):

```
WALCMP_DIVERGENCE lsn=9 class=committed compared=3 present=3
  member 0 (ctrl-1): term=3 pcrc=00000000   <-- MINORITY / divergent
  member 1 (ctrl-2): term=3 pcrc=512dc5b5   <-- majority
  member 2 (ctrl-3): term=3 pcrc=512dc5b5   <-- majority
```

Same pattern at committed LSN 10 (ctrl-1 pcrc=0 vs majority 06e88dac) and LSN 11 (ctrl-1 pcrc=0 vs majority fea75587). **Term is identical (3) across all three members; only the payload-CRC diverges.** (ctrl-3 additionally shows `status=MISSING` for LSN 12-19 — a separate catch-up-lag/convergence gap from the round-4 quorum loss, bug-1042 family, not the payload defect.)

Raw `waldump` confirms the payload loss, not a term mismatch:

```
ctrl-1 (divergent):   9 | 3 | JOB_SUBMIT | PLEN=0    | 00000000 | jobid=?
ctrl-2/ctrl-3 (maj):  9 | 3 | JOB_SUBMIT | PLEN=3560 | 512dc5b5 | jobid=0-0-1782550813-0 ...
```

The origin is intact — both majority members hold the full 3560-byte body with a decodable jobid; **ctrl-1 lost the payload bytes during catch-up while keeping the correct type (JOB_SUBMIT) and term (3).**

## Bisect — the concrete ingest path

ctrl-1's `--logfile` (fully flushed post-kill) traces exactly how it acquired the divergent entries:

1. ctrl-1 was the term-1 cold-start primary (wrote LSN 1-7), killed round 1, **restarted round 2** (`persist_recover done — recovered durable_floor=7 current_lsn=8`). It then needed LSN 8-onward.
2. `09:00:13` — term-3 leader ctrl-3 streamed catch-up `op 37 walCatchupEntry`, but starting at `prevIdx=8` (entry 9) while ctrl-1 only held up to 7 → library `ae_reject_logmismatch`, `ha_consume_append_entries … success=0`. Nothing written. (ctrl-3 then died — round 2 failover.)
3. `09:00:26` — new term-4 leader ctrl-2 streamed `op 37 walCatchupEntry` from `prevIdx=7`. The **library accepted and APPENDED**: `RAFT[p0] term=4 event=ae_append`, `ha_consume_append_entries … prevIdx=7 n=1 → success=1 match=8`, then match=9, 10, 11, 12. These were **fresh appends through the reconcile seam (`on_persist_entry`)**, not idempotent no-ops (current_lsn was 8 immediately prior). LSN 9-11 landed with `payload_len=0`.
4. In the SAME catch-up, LSN 13-15 (stored term=4, the streaming leader's OWN term) landed with FULL payloads. **Signature: same-term catch-up preserves payload; cross-term catch-up of prior-term entries (stored term 3 streamed under leader term 4) drops it.**

Code-path mapping (read at the research anchors):
- Receiver `ha_consume_append_entries` (ha.c:2410-2440) copies the body **unconditionally** (`if (payload && plen>0) memcpy(...)`, ha.c:2438) from `e->raw_payload`/`e->raw_payload_len`. So the empty body **arrived empty on the wire** → the loss is SEND-side / source-side, not in the seam's persist.
- Both documented send paths nominally read the payload: batch streamer `ha_stream_wal_to_standby` (config_repl.c:2154-2161, `pread` when `hdr.payload_len>0`, and sets `entry->term = persist_entry_term(...)` + `entry->leader_current_term = leader_term`, config_repl.c:2130-2151) and the single-entry `on_send_append_entries` → `ha_read_wal_entry_at` (ha.c:1611-1640). Neither *statically* drops a cross-term payload — so the loss is either (a) the streaming leader (ctrl-2) **itself held an empty copy at stream time** and the full payload was reconciled into ctrl-2's WAL only later (a transitive empty-propagation across the failover chain), or (b) the wire encode/decode of the body for these messages. Distinguishing (a) vs (b) requires one instrumented run capturing per-entry `payload_len` at the leader's send and at the receiver's `on_persist_entry` for a cross-term LSN — see "Implied next work".

## Why the term-blind hypothesis is refuted here

- Predicted by design.md ROOT CAUSE: `entry->term==0` → `ha_apply_single_entry` (ha.c:2097) → `persist_wal_write_replicated` DIRECT, **wrong term** (standby's `p->term`), idempotency no-op of the later correct entry.
- Observed: the entry's **term is correct (3)**, it was ingested via `ha_apply_term_entry_via_library` → `ha_consume_append_entries` → `on_persist_entry` (the term-carrying LIBRARY seam, `event=ae_append`), and the catch-up was a **fresh append** (no idempotency-guard no-op involved). The streamer stamps a non-zero term (config_repl.c:2141), so the `term==0` → `ha_apply_single_entry` branch (ha.c:2253/2291 gate `sub->term != 0`) is not taken for these entries.

## VERDICT

- **Term-blind catch-up bypass (wrong-term + LSN-only idempotency no-op): REFUTED** as the trigger of the reproduced live committed divergence.
- **Actual trigger: payload-loss on cross-term catch-up** — committed `JOB_SUBMIT` entries persisted with `payload_len=0` (term/type correct) on a member that catches up prior-term entries from a higher-term leader; the divergence is a **payload-CRC mismatch at an identical `(term,lsn)`**, in the committed prefix. A member carrying this can (and in round 3 did) become primary, so it is a silent committed-state corruption in the bug-795/828/1060 family.

## Implied next work (steers Phase 2)

1. **Re-scope the Phase-2 fix away from the term-stamp/idempotency seam.** Collapsing Path-2 into Path-1 (design D3/D1) will NOT fix this divergence — the entry already goes through Path-1 with the correct term. The fix must target the **catch-up payload source-of-truth**: add an instrumented capture of `payload_len` at `ha_stream_wal_to_standby`/`on_send_append_entries` (send) and `on_persist_entry` (receive) for cross-term LSNs to prove send-side empty vs transitive-empty, then fix whichever drops/propagates the empty body.
2. **Harden the persist conflict guard to be `(term,lsn,payload)`-aware**, not term-only: a committed `(term,lsn)` whose payload-CRC disagrees with the quorum must be detected/repaired (or the member fenced), since term-only reconcile is blind to this class.
3. **Fix the divergence detector / RED gate to assert payload-CRC at committed LSN even when terms match** (it already does via `pcrc`; keep `walcompare --report (term,lsn,pcrc)` as the canonical RED classifier — it caught this).
4. **R6/Harness defect (secondary):** the release build's log buffering defeats `run-ha-soak.sh`'s log-based `wait_for_primary_log`/`primary_from_log` primary detection (false "election stall"). The chaotic acceptance gate (R8) needs client-based (`ctrlstatus`) primary detection, or the controller must line-buffer/flush its INFO log, before the soak can run unattended on release binaries.

## Evidence artifacts (under ~/tmp, ephemeral)

- `~/tmp/ha19/` — frozen 3-member deploy with the reproduced LSN 9/10/11 payload divergence (`controller/ctrl-{1,2,3}/data`, `logs/ctrl-1.log`).
- `~/tmp/ha19-drive.log` — driver round-by-round output (divergence first seen round 1).
- `~/tmp/ha-1p3-run{1,2}.log`, `~/tmp/ha-1p3-seq.log` — the cold-start log-buffer "stall" runs.

---

# Phase-2 interim live validation (post-2.4, real-fs release rebuild)

Rebuilt the release controller at HEAD 63b9c5a (2.2 payload-aware guard + 2.4 streamer fail-loud) and ran a chaotic soak on real ext4 (`~/tmp/ha-verify-2p4`). The 4-round run **wedged (timed out ~9m40s) on an election stall** before reaching its own verdict — the un-fixed Phase-5 liveness defect. Froze the WALs and measured directly with `walcompare --report`:

- **Universally-held committed prefix (LSN 1..11) is content-identical across all 3 members** (`walcompare: OK — committed prefixes identical`).
- **No present-but-empty committed divergence** — the bug-1.3 signature (one member `pcrc=0/PLEN=0` PRESENT where the majority holds a full payload) is **absent**. The `pcrc=00000000` entries at LSN 1/8/12/16 are present+identical on ALL members (legitimate empty-payload marker/NOOP entries, same term+crc everywhere).
- Remaining apparent divergences are all `status=MISSING` tail entries (ctrl-1 tip=19, ctrl-2=15, ctrl-3=11) — **catch-up lag** on lagging/killed members, not committed-content corruption. Classifying these as "committed divergence" only happens when an inflated `--committed` boundary over-claims the committed region; at the true majority-held boundary they are uncommitted tail / in-flight catch-up.

**Conclusion:** the Phase-2 fix empirically eliminates the cross-term payload-loss divergence on the live repro. A *clean end-to-end* `committed_divergence=0` soak verdict is now gated on:
1. **Phase 5 (election liveness under churn)** — the chaotic soak wedges on an election stall before completing, so it never reaches its verdict line. This is the top blocker for the acceptance gate (7.2).
2. **Phase 4 (standby reconvergence)** — lagging members must finish catching up so the tail `MISSING` resolves once the cluster stabilizes.

**Steering implication:** Phase 3 (snapshot-boundary reconcile) is defense-in-depth for a path NOT implicated in the live repro; Phases 5 and 4 are on the critical path to a clean soak verdict. Consider doing 5 (and 4) before 3.

---

# Phase 4/5 — Post-Phase-2 stall root cause (investigation, salvaged from a hung run)

Reproduced the chaotic-soak wedge on real-fs release binaries (HEAD with Phase-2 fixes). The committed/payload prefix is clean (Phase-2 holds). The remaining wedge is a **reconvergence-triggered spurious-failover loop**, NOT a simple missing-election-restart.

## Evidence (frozen `~/tmp/ha-estall` logs)
- **ctrl-1 = stable primary at term 6**: `became_leader`, repeated `repl_broadcast` / `term=6 event=log_append`. The cluster HAS a leader.
- **ctrl-3 reconverges (slow but works)**: `term=6 ae_recv → ae_reject_logmismatch` ×4, then `ae_already_have`. The AE log-matching backtrack converges for a short divergent suffix.
- **ctrl-2 = the stall**. It recovered as a former primary with an uncommitted higher tail (`durable_floor=15 current_lsn=16`, entries 12-15 at an old term) vs the leader's committed 11. Its log shows a ~15s flap cycle, repeating indefinitely:
  1. `handle_ctrlhello: primary control channel established (fd=6)` → receives heartbeats → re-attaches CTRL_STANDBY.
  2. ~12-15s later: `ha_on_primary_ctrl_fd_rdhup: primary_ctrl_fd RDHUP → begin PreVote failover (Req 1.1)`.
  3. `ha_start_prevote_async (prospective term=6)` → `ha_prevote_round_timeout_fn: PreVote round timed out before a decision — treating as LOSS, staying CTRL_STANDBY` (NO `pre_vote_reply_in` — peers see the primary alive and don't grant).
  4. Primary reconnects → loop repeats. (Last heartbeat and last prevote-timeout share the same timestamp — the channel is flapping, not cleanly down.)

## Root cause
The primary↔ctrl-2 control channel keeps **RDHUP-flapping** (~15s period). Because "control-channel RDHUP = primary died → failover" is the designed trigger (CLAUDE.md), each flap spawns a spurious PreVote failover that **cannot win** (the primary is genuinely alive, so the other voters don't grant), times out, and re-arms. The likely driver of the flap: ctrl-2 cannot reconverge its **divergent uncommitted tail (12-15)** against the leader — the catch-up/AE path stalls or the leader drops the channel — so ctrl-2 never settles as a follower and the channel cycles. Net: the recovered node never rejoins → no stable 3-of-3 → soak `wait_for_full_convergence` times out (the observed wedge).

## What the fix must address (Phase 4 + Phase 5, intertwined)
1. **Reconvergence (Phase 4 / R5):** a standby that recovers with a divergent *uncommitted* tail must truncate it (follower-AE-conflict-truncate contract) and catch up to the leader — the same way ctrl-3 does for a short suffix — so the control channel stays up. Determine why ctrl-2's longer/former-primary divergent tail does not reconcile while ctrl-3's short one does (candidate: the backtrack doesn't reach far enough before the channel drops, or the former-primary recovery advertises a tip that the leader's nextIndex backtracking never crosses).
2. **Spurious-failover suppression (Phase 5 / R6/R4):** a transient/flapping control-channel RDHUP must not spawn an unwinnable PreVote storm while the primary is demonstrably alive (peers still see it). A standby should prefer to **reconnect + catch up** on RDHUP before (or instead of) launching failover, or back off failover when its own PreVotes repeatedly get no quorum because a current-term leader exists. (The retry itself works — R6 retry is present — but it retries an election that can never win instead of rejoining.)

## RED test direction (5.1 / 4.1)
- Force a recovered standby holding a divergent uncommitted tail (former-primary shape, tip above the new leader's committed) to rejoin a cluster with a live higher-term primary, and assert it **reconverges to follower (truncates tail, catches up) without entering a PreVote-failover loop** — i.e. no repeated `ha_begin_failover`/`PreVote timed out` while a current-term primary is reachable. This is deterministically constructible with the 1.4 kill/election + apply-gap seams plus a divergent-tail fixture.

## Process note
The dispatched investigation subagent backgrounded the soak and hung (~no findings returned); this section was reconstructed by the parent directly from the frozen `~/tmp/ha-estall` controller logs.

---

# Task 7.2 — Acceptance chaotic soak (first run with Phase-2+5 fixes): NO-GO + new finding

Ran `run-ha-soak.sh --chaotic 6` on real-fs (ext4 `/home/claude/tmp/ha-7p2`) with the
**release binaries rebuilt at HEAD `01419e1`** (Phase-2 payload fix + Phase-5 5.2
spurious-failover suppression). Frozen at `/home/claude/tmp/ha-7p2-frozen`.

## What PASSED (job-safety + the landed fixes)
- **Zero committed-JOB loss.** Every round the job-verify confirmed all tracked jobs
  present on the new primary (round1 6/6, round2 9/9, round3 12/12).
- **Zero TRUE committed divergence.** `walcompare --committed 11` over the frozen
  WALs: **"OK — committed prefixes identical"** across all 3 members. The Phase-2
  payload-loss signature (present=3, one member `pcrc=0` vs majority real) is ABSENT.
  `lsn=8` is `pcrc=0` *identically on all three* — a legit empty NOOP marker, not loss.
- **The counted `walcompare` divergences are catch-up LAG, not corruption.** The only
  boundary-19 divergence is `lsn=12 present=2`: ctrl-3 (tip=11, restarted round 3) is
  MISSING it; ctrl-1/ctrl-2 hold it with matching content. All "divergences" are
  `present=2` MISSING tails of restarted-but-uncaught followers (tips ctrl-1=19,
  ctrl-2=15, ctrl-3=11), which an inflated `--committed` boundary over-claims as
  committed — they are uncommitted/in-flight catch-up tail, exactly the research.md
  Phase-2-interim note.
- **5.2 spurious-failover suppression confirmed working.** Rounds 1–3 each elected a
  strictly higher-term primary (ctrl-3 t3 → ctrl-2 t4 → ctrl-1 t5): liveness intact,
  the fix did not over-suppress. The round-4 RDHUP logs show the GENUINE-loss path
  ("primary_ctrl_fd RDHUP → begin PreVote failover"), NOT the suppression message —
  correct, because the primary was truly dead (`ha_heartbeat_is_fresh==0`).

## What FAILED — round-4 election stall (NEW residual liveness defect)
After 3 overlapping kills, round 4 kills ctrl-1 (tip 19, the only fully-up-to-date
node). The two survivors are both restarted-and-lagging (ctrl-2 tip=15, ctrl-3 tip=11),
and their PreVote rounds **repeatedly time out before a decision** (no grant quorum):
each loops `ha_start_prevote_async → ha_prevote_round_timeout_fn (LOSS) → RDHUP →
ha_begin_failover → …` indefinitely, prospective terms frozen (ctrl-2=5, ctrl-3=6,
never bumped because no PreVote ever WINS). The soak wedged here until the 10-min cap.

**This is NOT the spurious-failover flap (5.2 closed that) and NOT a job-safety bug.**
A SAFE candidate exists — ctrl-2 holds the full committed prefix [1..11] (Leader
Completeness satisfied) and is the freshest survivor — yet the election never converges
on it under sustained overlapping churn. Observed contributors (need a dedicated
RED→bisect): freshness mutual-denial (`PreVote DENIED … reason=freshness, floor lsn=11`)
between two lagging survivors, the in_catch_up-denies-all-votes gate (Req 2.5; obs 1240
"a catching-up node sends zero PreVote replies") with no live primary left to catch up
FROM, and/or peer-to-peer vote connections not established between two nodes that each
only held a control channel to the now-dead primary.

## Verdict
**7.2 = NO-GO.** Job safety (R8.1/R8.2) holds; the acceptance liveness criterion
(R8.3 — reconverge 3-of-3 + elect after EVERY kill round) fails at round 4. Per the
design's chaotic-harness Contingency, this is a recorded new finding, not closure:
added follow-up **task 5.3** (overlapping-churn election stall — a lagging-but-safe
survivor must be electable; the catching-up/freshness vote gates must not deadlock when
the primary is gone). 7.2 stays blocked on 5.3. Re-run 7.2 after 5.3 lands.

---

# Task 5.3 + 7.2 re-run — CORRECTED root cause: the soak stall is bug-924, not consensus

After landing task 5.3 (commit bf88681 — gate the `in_catch_up` PreVote deny on a
LIVE primary), the chaotic soak was re-run on real-fs release binaries
(`/home/claude/tmp/ha-7p2b`, controller rebuilt 02:44 with all Phase-2+5.2+5.3 fixes).

## Result: round-4 stalls AGAIN — but the cause is the persist fdatasync wedge (bug-924), NOT election logic
- Rounds 1–3 elect a higher-term primary each round (term 4→5→6); job safety holds.
- Round 4 "NO new primary within window" — and ctrl-2's PreVote rounds get **zero
  replies** (timeout, not LOST), because the *other* survivor **ctrl-3 is wedged**:
  its persist flusher spins on `flusher fallback fdatasync FAILED (errno=22 Invalid
  argument)` from 02:45:49 onward and stops answering PreVotes. ctrl-3's process is
  alive but non-responsive — jammed in the durability layer, not the election layer.
- This is the **bug-924** WAL-fdatasync-on-overlay artifact (memory obs 1476/1477):
  the WAL segment's `fdatasync` returns EINVAL in this sandbox. A file-level repro of
  the exact controller pattern (`open O_RDWR|O_CREAT|O_TRUNC` → `posix_fallocate(64MB)`
  → write → `fdatasync`) SUCCEEDS on `/home/claude/tmp`, `/home/rshen/projects/xce`
  (both ext4) and `/dev/shm` — so it is the controller's live overlay-backed WAL fd
  under chaotic restart, not a plain fs limitation. bug-924 is a **Non-Goal of this
  spec** (design §Non-Goals / §Out of Boundary: "the overlayfs `fdatasync`
  environment defect (bug-924) — excluded from the acceptance verdict, not fixed
  here"); durability sync is owned by `ha-commit-durability`.

## Re-attribution of the round-4 stall (corrects the first §"Task 7.2" entry)
The first soak's round-4 stall was attributed to the `in_catch_up` mutual-deny. That
IS a real latent defect — 5.3 reproduces it deterministically at unit level (a
catching-up survivor with no live primary denied a fresh PreVote) and fixes it — but
it is NOT what blocks the soak. The soak's actual round-4 blocker in BOTH runs is the
bug-924 fdatasync wedge that makes a survivor non-responsive. 5.3 is kept as a valid
correctness fix (unit RED→GREEN, `test_ha_core_safety` 20 Tests / 0 Failures); its
6-round-soak observable clause shares 7.2's environment block.

## 7.2 status: BLOCKED ON ENVIRONMENT (bug-924), not a consensus NO-GO
The consensus work this spec owns is unit-complete and validated: Phase-2 payload
integrity (committed prefix byte-identical across members), 5.2 spurious-failover
suppression, 5.3 catching-up-survivor electability. The live ≥6-round acceptance soak
cannot reach a clean verdict in this sandbox because the controller's WAL `fdatasync`
fails (bug-924) and wedges nodes mid-soak — the same env block that left
`ha-term-safety`'s live soak permanently BLOCKED. 7.2 (and 5.3's soak clause) require a
real-fs host/CI where the controller WAL `fdatasync` succeeds, OR the soak harness must
detect the fdatasync-wedge and exclude it from the liveness verdict (a Phase-1 harness
improvement), OR bug-924 is resolved by `ha-commit-durability`. The consensus fixes do
not need re-work for this; it is an environment/durability-layer blocker.

---

# Task 7.2 on ~/tmp (non-overlay) — DEFINITIVE 6-round blocker: a controller restart SEGFAULT

Re-ran the chaotic soak on the user-provided non-overlay path `~/tmp` (= `/home/claude/tmp`)
with the release binaries (Phase-2 + 5.2 + 5.3) and a harness change to **capture each
controller's stdout+stderr** (`start_node` previously sent it to `/dev/null`, hiding all
crash output — the key diagnostic gap).

## What is now PROVEN
- **Elections work under chaotic churn.** A 2-round chaotic soak COMPLETES (both rounds
  elect a higher-term primary). In 6-round runs, rounds 1–3 each elect (terms climb
  4→5→6). 5.2 + 5.3 hold live. Job safety holds (zero committed-job loss every run).
- **The 6-round blocker is a RELEASE-BUILD SEGFAULT on restart**, captured at last in the
  per-node stderr: `start.sh: line 14: <pid> Segmentation fault (core dumped) … xce_controller
  --myname ctrl-3 …`. A controller **segfaults a few seconds after being restarted into the
  churning cluster** (during recovery→failover/PreVote). One node down → cluster at 1-of-3 →
  the lone survivor cannot reach quorum → "NO new primary within window." The harness's other
  stderr lines are the expected `kill -9` ("Killed"); only the restarted node shows SEGV.
- **This is NOT** the in_catch_up eligibility deadlock (5.3 fixes that, unit-proven), **NOT**
  bug-924 fdatasync (that wedged a node in ONE earlier run but is absent here — zero
  `fdatasync FAILED`), and **NOT** the consensus logic this spec owns. It is a restart/rejoin
  stability crash.

## Why it could not be root-caused in this sandbox
- The crash is a **heisenbug**: it does NOT reproduce under the ASAN controller build (all
  ASAN-run stderr shows only harness `kill -9`, no SEGV / no ASAN report) — memory-layout /
  timing dependent.
- No `gdb` and no usable core: `core_pattern` pipes to a non-running `systemd-coredump`, so
  the "core dumped" core is discarded; `coredumpctl`/`gdb` are absent from the env.
- Root-causing requires a debug-non-ASAN build with core dumps + gdb (or a host where the
  systemd-coredump pipeline is live), i.e. a tooling environment this sandbox lacks.

## Harness accuracy note (separate, in-scope)
The soak counts `committed_divergence` via `walcompare --report --committed $((1 + 2*njobs))`
(run-ha-soak.sh:543/663). That boundary is the OPTIMISTIC job-count estimate, not the true
majority-committed LSN, so a lagging follower's not-yet-replicated tail (uncommitted on that
member) is mis-counted as committed divergence. The completed 2-round soak reported
`committed_divergence=3 committed_job_loss=0` — and a frozen-WAL `walcompare` at the true
boundary shows the committed prefix byte-identical (the 3 are catch-up-lag MISSING tails, not
corruption). `ctrlstatus` already reports `committedLsn`; the boundary should use it. (Even
fixed, the 6-round verdict stays blocked by the segfault above.)

## 7.2 status (corrected): blocked by a restart SEGFAULT, not consensus, not bug-924
Consensus correctness (Phase-2 payload integrity, 5.2 spurious-failover suppression, 5.3
catching-up-survivor electability) is unit-complete and confirmed live for every round that
completes. The clean ≥6-round acceptance verdict is blocked by the restart segfault — added
follow-up **task 5.4** (root-cause+fix the chaotic-restart crash; needs core+gdb / debug env)
and **bug-1281**. The harness `committedLsn`-boundary fix is a separate Phase-1 accuracy
improvement.

---

# Task 7.2/5.4 — UNIFIED root cause via gdb: round-N stall is a restart-stability bug (I/O wedge / SIGSEGV), not a consensus-logic gap

Wrapped the release `xce_controller` under `gdb -batch` (handles RT signals; dumps all-thread
bt on SIGSEGV) via `XCE_HA_BIN=<gdb-wrapper>` and re-ran the chaotic soak on `~/tmp`.

## Decisive findings
- **Two round-4 faces, ONE root cause.** Plain release: a restarted node **SIGSEGVs**
  (captured `Segmentation fault (core dumped) … --myname ctrl-3`). Under gdb the crash does NOT
  fire (ptrace perturbs the memory bug); instead the same restarted node's **inbound-accept /
  iomgr path WEDGES** — ctrl-3's last `cluster.c:737 server sock accepted` is 03:42:03, yet it
  keeps firing its own election timer (`pre_vote_start`) to 03:51:59. It campaigns but can
  neither accept peers' PreVote connections nor get answered; ctrl-2's outbound PreVote connects
  hang on the dead listener (zero connect-fail logs). The two restarted survivors can't exchange
  votes → permanent round-4 stall. Release SEGV and gdb accept-hang = the same restart-time
  corruption manifesting differently.
- **Restart-STABILITY bug, not a consensus-logic gap.** PreVote/eligibility/freshness (incl. the
  5.3 fix) are correct; the node is simply I/O-dead after restart. Predates this session —
  baseline `21e26a1` failed round 3–4 (requirements.md symptom 3). 5.2/5.3 sit in the
  restart/failover path so 5.4 must rule them out during the backtrace, but they did not
  introduce the instability.
- **Cannot backtrace in this sandbox.** Heisenbug: invisible under ASAN (no SEGV) AND under gdb
  (becomes a hang, not a catchable signal); `core_pattern` is read-only (pipes to a non-running
  systemd-coredump) so the release core is discarded. Needs a non-ASAN DEBUG build (-O0 -g) run
  directly under gdb so the SIGSEGV is caught with symbols, or a host with a live coredump pipe.

## Net for 7.2
Elections + job safety are sound for every round that COMPLETES (2-round soak PASSES; 6-round
rounds 1–3 elect higher-term primaries; committed prefix byte-identical; zero job loss). The
≥6-round acceptance verdict is blocked by the restart-stability bug (bug-1281 / task 5.4),
reachable at round 4 (kill the up-to-date node → two restarted survivors). Spec-owned consensus
correctness (Phase-2/5.2/5.3) is unaffected.

## Harness wins
`start_node` now captures per-node stdout+stderr (committed 75243fb) — this exposed the SEGV.
The gdb-wrapper (`XCE_HA_BIN=<wrapper>` → `gdb -batch … --args <release ctrl>`) is the reusable
technique to catch a restarted node's fault once a debug+core env is available.

---

# Task 7.2/5.4 — ROOT CAUSE NAILED via gdb: ABBA lock-order deadlock in src/lib async dispatch (UPSTREAM raft-drive-async)

Wrapped the release controller under gdb -batch with a SIGQUIT handler that dumps all-thread
backtraces; reproduced the round-4 hang; kill -QUIT'd the two wedged survivors. ctrl-2 and ctrl-3
showed the IDENTICAL two-thread deadlock.

## The deadlock (both nodes)
- io_worker (netio.c:612): takes ep->lock (netio.c:672) and, still holding it, runs
  vtask_enq -> dataq_enq which blocks on q->lock (dataq.c:81). Order: ep->lock -> q->lock.
- vdispatcher_entry: dataq_deq (dataq.c:65) takes q->lock for the WHOLE match scan and runs the
  match callback vtask_eligible (vtaskmgr.c:255) -> endpoint_is_task_blocked (netio.c:1001) which
  blocks on ep->lock. Order: q->lock -> ep->lock.
- ABBA: io_worker holds ep wants q; vdispatcher holds q wants ep -> permanent deadlock. io_worker
  dies -> the node stops accepting / answering PreVotes -> two restarted survivors can't elect ->
  round-4 stall. In release the same ep/q race under churn corrupts -> SIGSEGV (the earlier crash).

## Exact defect + minimal fix
dataq_deq invokes a match callback that acquires a SECOND lock (ep->lock via the
endpoint_is_task_blocked eligibility gate) while holding q->lock — inverting the ep->lock->q->lock
order io_worker uses. Minimal fix: make ep->busy atomic and read it locklessly in
endpoint_is_task_blocked (remove the ep->lock acquire from under q->lock); or io_worker drops
ep->lock before dataq_enq; or impose one global lock order.

## Ownership: UPSTREAM (raft-drive-async / core libxce async-io), NOT this spec
Entirely in src/lib async-I/O dispatch (netio.c io_worker/endpoint, vtaskmgr.c eligibility,
dataq.c) — the VTASK dispatch / endpoint-blocked eligibility layer raft-drive-async owns and this
spec's Non-Goals exclude. Per upstream-ownership protocol: route to raft-drive-async (bug-1282),
keep 7.2 blocked until the dispatch lock-order is repaired, then re-run the gate. This is the gate
working as designed: 7.2 must be GREEN before raft-drive-async flips dispatch flags ON, and it just
caught a real dispatch-layer deadlock.

## Consensus correctness (this spec) — unaffected
Phase-2 payload integrity, 5.2 spurious-failover suppression, 5.3 catching-up electability are
correct and unit-validated; elections + job safety hold for every round that completes. The
round-4 blocker is the upstream dispatch deadlock, not consensus logic.

---

# bug-1282 FIXED + bug-1283 isolated (the round-4 stall had TWO causes)

Implemented the bug-1282 fix (commit 25e741d): promoted `endpoint_t.busy` to `_Atomic int` and
made `endpoint_block_tasks`/`endpoint_unblock_tasks`/`endpoint_is_task_blocked` lockless, removing
the `ep->lock` acquire from the vtask-eligibility path that ran under the dataq `q->lock`.

## Verified
- **Deadlock gone:** post-fix gdb soak shows ZERO `__lll_lock_wait` threads across all 3 nodes and
  `endpoint_is_task_blocked` no longer appears in any lock path. The ABBA inversion is eliminated.
- **No regression:** `controller_unit_test` all green (incl. `test_ha_core_safety` 20/0); libxce +
  controller + unit test build clean.

## But the round-4 stall had TWO independent causes
A plain release soak WITH the fix STILL segfaults a restarted standby at round 4 (ctrl-3,
"Segmentation fault (core dumped)"). **Deadlocks hang; they do not SIGSEGV** — so this crash is a
SEPARATE memory bug (now bug-1283), not bug-1282. It is most likely an endpoint-lifetime UAF under
chaotic restart (bug-1197 family): the dispatch eligibility path reads `vdata->u.iodata.ep` /
endpoint state while io_worker tears down or mutates the same endpoint. It is a heisenbug —
invisible under ASAN and under gdb (gdb perturbs it into a cold-start timeout / no crash) — and the
sandbox has no usable core (read-only `core_pattern` → non-running systemd-coredump), so a
backtrace needs a non-ASAN DEBUG (-O0 -g) build under gdb or a host with a live coredump pipeline.

## Net for 7.2
The bug-1282 deadlock fix removes ONE of the two round-4 blockers (the iomgr wedge). The remaining
blocker is bug-1283 (the restart segfault), likely UPSTREAM raft-drive-async / core libxce endpoint
quiescence. 7.2 stays blocked on bug-1283; consensus correctness (Phase-2/5.2/5.3) is unaffected and
elections + job safety hold for every round that completes.

---

# bug-1286 (the round-4 segfault) ROOT-CAUSED — NULL ep->owner — via the new /xce-build debug profile

The `/xce-build ... debug` profile added this session (non-ASAN `-O0 -g3`, no sanitizer → `build.dbg`)
reproduced the heisenbug AND gave a symbolized backtrace under gdb (ASAN and gdb-on-release both
perturbed it away). The captured SIGSEGV:

```
Thread received SIGSEGV
#0 ha_dispatch_gate (ha=0x3ff0, opcode=37)  controller/ha.c:3194  (atomic_load &ha->role)
#1 myreq_handler                            controller/cluster.c:251
#2 vtask_entry                              controller/cluster.c:693
#3 run_task / vcpu_entry                    lib/vtaskmgr.c
```

Root cause: `myreq_handler` does `cluster_t* c = (cluster_t*)ep->owner;` then
`ha_dispatch_gate(&c->ha, opcode)`. On a chaotic restart an inbound endpoint dispatches a
`walCatchupEntry` (op 37) vtask while `ep->owner == NULL`, so `c == NULL` and
`&c->ha == NULL + offsetof(cluster_t,ha) == 0x3ff0` — a garbage pointer `ha_dispatch_gate`
dereferences → SIGSEGV. (DISTINCT from the bug-1282 deadlock, which is fixed; deadlocks hang,
they do not crash.) Fix: NULL-guard `c = ep->owner` in `myreq_handler` (drop/reject the request if
the endpoint's owner is not yet attached) instead of dereferencing; deeper, ensure `ep->owner` is
set at accept time before the endpoint is monitored for input and never cleared while a vtask is in
flight. Tracked as bug-1286 (an earlier auto-logged entry already held the id bug-1283).

7.2 now has ONE remaining blocker: bug-1286 (root-caused; small NULL-guard fix). bug-1282 (deadlock)
is fixed. Consensus correctness (Phase-2/5.2/5.3) is unaffected.

---

# Both fixes verified (bug-1282 + bug-1286) — round-4 now a CLEAN election-liveness cross-deny

6-round chaotic soak on real-fs release WITH bug-1282 (deadlock) + bug-1286 (NULL ep->owner crash)
both fixed: rounds 1-3 elect; round 4 has NO segfault, NO fdatasync, NO iomgr deadlock (both fixes
held, confirmed). The round-4 stall is now the GENUINE consensus-liveness edge the crash/deadlock
were masking:

- ctrl-1 (primary term 6) killed round 4.
- ctrl-2 (recovered durable_floor=15, the FRESHER survivor) denies ctrl-3 on FRESHNESS (ctrl-3
  advertises 12 < 15) - correct.
- ctrl-3 (floor 12) denies the fresher ctrl-2 on ELIGIBILITY (raft_cb_vote_eligible), repeatedly
  for 2+ minutes - because ctrl-3 still believes its primary (the dead ctrl-1) is alive
  (in_catch_up + heartbeat-not-yet-stale, primary_ctrl_fd still >=0). So the legitimate winner
  (ctrl-2) never gets ctrl-3's grant -> stall.

This is requirements.md symptom #3 (round-N election stall) in pure form. Contributors to chase:
(a) ctrl-3's "primary alive" view not expiring within the round - the heartbeat-fresh/in_catch_up
eligibility deny (gated on primary_ctrl_fd>=0 && ha_heartbeat_is_fresh by task 5.3) does not lift
because last_activity is not going stale; SUSPECT task 5.2's ha_arm_heartbeat_timeout_vtask re-arm
resetting last_activity=now on the RDHUP-suppress path keeps a node falsely-fresh after the primary
dies - revisit whether the suppress path should reset last_activity. (b) a stale primary_ctrl_fd to
the dead primary not cleared promptly. Next Phase-5 task: a node denying on heartbeat-freshness/
in_catch_up must detect a genuinely-dead primary within the failover window and release the deny so
the freshest survivor is electable.

## Session net
bug-1282 (ABBA iomgr deadlock) and bug-1286 (round-4 restart SIGSEGV, NULL ep->owner) are FIXED and
VERIFIED (no crash / no deadlock / no fdatasync across round-4 churn). The remaining 7.2 blocker is
this election-liveness cross-deny - the hard core of Phase 5, no longer masked.

---

# Task 5.5 fix VERIFIED as progress; round-4 now reaches a DEEPER layer (dueling-candidate term escalation)

Landed the 5.5 fix (commit b8453b2: advertise persist_entry_term(p, log_len) as lastLogTerm instead
of the stale c->persist.term) and re-ran the 6-round soak. Round-4 frozen state: ctrl-2 tip 15 @
term 5 (freshest), ctrl-3 tip 11 @ term 4 (staler).

## 5.5 is correct and made real progress
BEFORE the fix the freshest survivor advertised a too-low lastLogTerm and EVERY voter's logOk
rejected it — it could not even win a PreVote. AFTER the fix ctrl-2 WINS PreVote (pre_vote_won) AND
enters the real election ("PreVote round WON — entering real election"). The advertised-term
asymmetry is fixed; the fresher candidate is now electable in principle.

## The deeper layer it exposed (next task)
ctrl-2 wins PreVote -> real election (term N) -> RequestVote -> replies but NO became_leader ->
retries at term N+1, N+2 ... prospective term climbing 21->25->28->29 unbounded. Mechanism: ctrl-2's
primary (dead ctrl-1) keeps RDHUP'ing ~every 15s; 5.2 does NOT suppress (primary genuinely dead),
so each RDHUP re-triggers ha_begin_failover -> fresh PreVote -> real election -> term++. ctrl-2 and
ctrl-3 duel in lockstep (each denies the other's real vote on already-voted-this-term / stale-term),
and the deterministic ~15s RDHUP cadence defeats the randomized-backoff tiebreak that normally
resolves a split vote. Neither reaches majority; the term escalates without converging.

Classic Raft split-vote / dueling-candidates liveness problem, amplified by the RDHUP failover
re-trigger. SEPARATE layer from 5.5. Next: break the duel (per-node randomized failover backoff /
election jitter that desynchronizes the two survivors; and/or do not re-trigger a fresh failover
via RDHUP while a real election the node just entered is still within its timeout window).

## Session net (4 layers peeled on the round-4 stall)
1. bug-1282 ABBA iomgr deadlock — FIXED (25e741d).
2. bug-1286 NULL ep->owner restart SIGSEGV — FIXED (5db734a).
3. bug-851/5.5 stale advertised lastLogTerm — FIXED (b8453b2), verified: the fresher survivor now
   wins PreVote + enters the real election (could not before).
4. REMAINING: dueling-candidate split-vote + unbounded term escalation under the RDHUP failover
   cadence — the round-4 election round still does not complete. New follow-up.
Consensus SAFETY intact throughout (no two leaders; job safety holds every completed round); this is
purely election LIVENESS under sustained churn.

---

# Task 5.6 fix VERIFIED (unit) + reduces churn; round-4 exposes a 5th layer (vote-reply term-lag escalation)

Landed 5.6 (commit 81cb09a: ha_on_primary_ctrl_fd_rdhup drops an RDHUP while role==CTRL_ELECTING
or pre_vote_in_progress, instead of restarting failover). Unit RED→GREEN (test_ha_core_safety 22
Tests / 0 Failures); the soak's term churn is slower (peaked ~24-31 vs ~28-29+ before at the same
elapsed time) — the fix does prevent RDHUP from preempting an in-flight election.

## The 5th layer (next task 5.7)
Round 4 still stalls. The freshest survivor ctrl-2 (15@t5) repeatedly wins PreVote, enters the real
election, AND ctrl-3 GRANTS it (vote_granted) — ctrl-3 is voting FOR ctrl-2, not leapfrogging
(ctrl-3 wins only 1 prevote total). But ctrl-2 never tallies a majority: its log shows every grant
as **vote_reply_stale_drop** — the reply's term is LOWER than ctrl-2's current election term.
Mechanism: ctrl-2's term escalates via repeated election timeouts faster than ctrl-3 can process
the request and reply, so ctrl-3's grant (stamped at ctrl-3's lower term) is always one term behind
ctrl-2's current term → DropStaleResponse → not counted → election times out → term++ → the grant
for the new term is again stale. Self-perpetuating term escalation; ctrl-2 never wins despite a
willing grantor.

election_timeout_ms = 5000, jitter = 200ms (HA_DEFAULT_*). The 5s window should suffice for a warm
RTT, so the lag points at: a just-restarted ctrl-2 with COLD peer connections (each real-vote opens
a fresh endpoint_connect + SSL handshake; raft_cb_send_vote_request), plus parallel connect attempts
to the dead ctrl-1 (DEFAULT_CONNECT_TIMEOUT_MS) — the grant from ctrl-3 lands after ctrl-2 has moved
on. Fix directions (5.7): (a) don't bump the election term while a vote round-trip for the current
term is still outstanding (give the grant time to land before re-escalating); (b) warm/reuse the
peer vote connection so the grant RTT < election_timeout; (c) larger election-timeout jitter so the
two survivors desynchronize and one's term stops racing the other's reply. Preserve Election Safety
(DropStaleResponse itself is correct Raft — the bug is the escalation cadence, not the drop).

## Session net (5 layers peeled on the round-4 stall; 4 fixed)
1. bug-1282 ABBA iomgr deadlock — FIXED (25e741d).
2. bug-1286 NULL ep->owner restart SIGSEGV — FIXED (5db734a).
3. bug-851/5.5 stale advertised lastLogTerm — FIXED (b8453b2), verified prevote-win.
4. 5.6 RDHUP preempts in-flight election — FIXED (81cb09a), unit-verified, reduces churn.
5. REMAINING (task 5.7): vote-reply term-lag / unbounded term escalation under cold-connection churn
   — the freshest survivor is granted but its term out-races the grant. New follow-up.
Consensus SAFETY intact throughout (no two leaders; DropStaleResponse correct; zero job loss every
completed round). Purely election LIVENESS convergence under sustained overlapping-kill churn.

---

## Task 5.7 fix VERIFIED + round-4 blocker re-rooted (2026-06-30)

**5.7 (election-timeout resonance) — FIXED, commit 74784b2.**

The 5.6 note guessed the 5.7 cause as "term re-escalates faster than the grant
can land." Reading the code pinned the *mechanism* precisely: the per-CAMPAIGN
election watchdog (`raft_start_election` → `on_schedule_timeout` →
`raft_cb_schedule_timeout`, ha.c) was armed with a **FIXED** `election_timeout_ms`
— **no randomization**. Only the *startup-discovery* path (`ha_wait_for_heartbeat_or_timeout`)
jittered; the recurring campaign watchdog did not. raft.c:157 passes the bare
`s->election_timeout_ms` straight through.

A fixed deadline + a roughly-constant cold-connection grant RTT **RESONATE**: if
ctrl-3's grant consistently lands a hair *after* a constant deadline, every
campaign loses by the *same* margin forever → the campaign times out, the term
re-escalates, and the now-stale grant is dropped by `DropStaleResponse`
(raft.c:453, which is *correct* Raft). Terms climb 25→30+. The bug was never
"RTT genuinely > 5 s"; it was that the deterministic cadence locks the grant
permanently just outside every window.

**Fix (Ongaro thesis §3.4 — randomized election timeout, re-drawn each campaign):**
new `ha_jittered_election_delay(ha, base)` returns `base + rand[0, base]` (the
canonical [T, 2T] interval), wired into `raft_cb_schedule_timeout` so the watchdog
is re-randomized per campaign. Done in the HA adapter so the pure, sim-tested raft
library stays deterministic. Independent of `election_jitter_max_ms` (which tunes
only the sub-second startup/stagger jitter) so it spreads even at jmax=200; never
below the floor so genuine-loss failover is not sped up past `election_timeout_ms`.

Unit: `test_5_7_campaign_timeout_is_randomized` (CORE) + `test_5_7_floor_preserved_and_zero_safe`
(PIN). RED verified by stubbing the jitter to 0 → the randomization test fails with
the resonance message while the PIN stays green. `test_ha_core_safety` 24 Tests / 0
Failures. (Aside: the full aggregate `controller_unit_test` binary cannot reach the
ha_core_safety suite because an EARLIER suite, `test_ha_vep_teardown`, SIGSEGVs in
`iomgr_fini`→`SSL_CTX_free` under openssl 3.6 — unrelated to this change; run the
suite in isolation with `-t test_ha_core_safety`.)

**Soak confirmation:** a fresh ASAN 6-round chaotic soak now elects with **BOUNDED
terms** — rounds 1→3 elect ctrl-3/ctrl-2/ctrl-1 at terms **4→5→6** (not 25→30+).
The unbounded escalation is gone; zero job loss on every completed round (3/6/9/12
tracked jobs all present on each new primary). The election-liveness layer is DONE.

**Round-4 blocker RE-ROOTS to a non-election crash (task 5.8 / bug-1298).**
Round 4 still stalls, but no longer for any election reason. ctrl-3 **ABORTs** on
an ASAN heap-use-after-free:

```
UAF READ size 4  endpoint_is_task_blocked  netio.c:1008      (vtask dispatcher T3)
                 vtask_eligible vtaskmgr.c:255 ← dataq_deq_first_match ← vdispatcher_entry
FREED BY (T2)    endpoint_killer  cluster.c:813               (io_worker)
                 handle_endpoint_error cluster.c:108 ← handle_error ← io_worker
ALLOC            endpoint_get netio.c:748 ← handle_connection cluster.c:741
```

io_worker (T2) tears down a connection on error and `free()`s the endpoint via
`endpoint_killer` while the vtask dispatcher (T3) concurrently reads `ep->busy` on
that just-freed endpoint during its dataq eligibility scan. The endpoint is freed
while still referenced by a pending vtask; nothing drains its queued vtasks before
the free and no refcount keeps it alive across the dispatcher read. ctrl-3 dies →
the cluster drops to 1/3 (the round-4 survivors are the just-restarted ctrl-2 and a
now-dead ctrl-3) → no quorum → "NO new primary within window." ctrl-2's logs show
only PreVote-round timeouts (`prospective term=6`, never wins) — the textbook
symptom of a lost quorum peer, not an election defect.

Same netio surface and routing precedent as task 5.4/bug-1282: an async-io dispatch
concurrency bug, **UPSTREAM-owned** by raft-drive-async / core libxce. Likely a
latent UAF that the bug-1282 lockless `ep->busy` read unmasked (the dispatcher now
reaches the read without contending on `ep->lock`). Logged as **bug-1298**; tracked
as task 5.8. Consensus safety (Phase-2/5.2/5.3/5.5/5.6/5.7) is sound; elections and
job safety hold for every round that completes.

---

# Cross-Spec Gap Analysis (2026-06-30)

Generated by `/kiro-validate-gap ha-term-safety raft-drive-async raft-ha-core-safety ha-consistency-invariants`.

## Summary

4-spec HA safety family: **ha-term-safety COMPLETE**, **raft-drive-async DORMANT** (all tasks done, production flags OFF), **raft-ha-core-safety IN PROGRESS** (12/26 tasks, Phase 5 critical path — task 5.7), **ha-consistency-invariants DESIGN ONLY**. One blocker to production: raft-ha-core-safety 7.2 chaotic soak PASS.

## Dependency Graph

```
ha-consistency-invariants (umbrella — defines invariants)
  ├── ha-term-safety (term recording + reconcile) ← COMPLETE
  ├── raft-ha-core-safety (payload safety + election liveness + soak gate) ← IN PROGRESS
  └── raft-drive-async (non-blocking drive threads) ← DORMANT
        gates on: raft-ha-core-safety 7.2 PASS
```

## Implementation State

| Spec | Tasks Done | Tasks Remaining | Status |
|------|-----------|----------------|--------|
| ha-term-safety | 20/20 | 0 | COMPLETE (build green, 175 tests, clean failover verified) |
| raft-ha-core-safety | 12/26 | 14 (5.7 critical) | IN PROGRESS (Phases 1-2+4+5.1-5.4 done; Phase 3 not started) |
| raft-drive-async | 17/17 | 0 (dormant flags) | DORMANT (production flags OFF; gates on raft-ha-core-safety 7.2) |
| ha-consistency-invariants | 0/0 | design only | DESIGN ONLY (no tasks generated) |

## Critical Path

```
raft-ha-core-safety 5.7 (vote-reply term-lag) → 7.2 (soak PASS) → raft-drive-async flags ON
```

## Remaining Gaps

1. **Task 5.7** (NOT STARTED): vote-reply term-lag — candidate's term escalates faster than grant RTT, all grants arrive stale. Fix: don't bump election term while vote round-trip outstanding, or warm peer connections.
2. **Task 7.2** (BLOCKED on 5.7): acceptance chaotic soak gate — ≥6 rounds, zero divergence, zero job loss. Currently: rounds 1-6 pass, round 7 stalls on PreVote cross-denial (bug-1296). Election liveness is the LAST remaining data-path gap; payload safety (Phase 2) and reconvergence (Phase 4) are proven.
3. **Phase 3** (snapshot-boundary reconcile): lowest priority — snapshot install is rare (16-byte sentinel today).
4. **raft-drive-async flags**: peerctrl/snapshot/configpush all default OFF. Flag-flip gate = raft-ha-core-safety 7.2 PASS + borrowed-ep refcount for snapshot/configpush lane.
5. **ha-consistency-invariants**: umbrella spec with approved design, no tasks. Close by verifying all invariants are satisfied by the three implementation specs.

## Risk

| Risk | Likelihood | Impact |
|------|-----------|--------|
| Task 5.7 reveals further election liveness layers (5.5→5.6→5.7 chain) | HIGH | Schedule slip; budget 2-3 more iterations |
| Chaotic soak still fails after 5.7 | MEDIUM | 7.2 blocks raft-drive-async; consider sequential-only fallback |
| Raft-drive-async flag-flip reveals new races | LOW | Gradual rollout: peerctrl first |

## Recommendations

1. **Immediate**: Fix task 5.7 (vote-reply term-lag) — approach (a): don't bump term while vote outstanding.
2. **Short-term**: Complete raft-ha-core-safety 5.7→7.2→7.3, then flip raft-drive-async flags.
3. **Medium**: Generate tasks from ha-consistency-invariants design; close the umbrella loop.

---

## Task 5.8 fix VALIDATED + round-4..6 reveals pre-existing commit-loss (2026-06-30)

**5.8 (endpoint teardown-vs-dispatch UAF, bug-1298) — FIXED, commit 6e05dc1.**

Chosen fix (user picked the principled option over the targeted deferred-teardown
patch): an endpoint REFERENCE COUNT consolidating the prior ad-hoc lifetime
patches. endpoint_t.refcnt (_Atomic) + heap_alloc; endpoint_get → refcnt=1 (owner);
vtask_enq takes a per-vtask ref BEFORE publishing the task to the dispatcher,
run_task / taskq_item_killer release it; every heap-ep teardown drops its owner ref
via endpoint_unref (last unref fini+frees). Embedded eps (client ctx->ep,
heap_alloc=0) are never freed by unref. So the dispatcher reads ep->busy on a queued
vtask without the io_worker freeing it underneath — the UAF is structurally
impossible. Also fixed several latent ep-struct leaks (endpoint_fini-only sites).

Validation: controller_unit_test 182 tests / 0 failures (test_ha_vep_teardown
OFF-path test updated — it asserted the now-fixed struct leak). 6-round ASAN chaotic
soak: ZERO AddressSanitizer reports, ctrl-3 survives round 4, ALL 6 rounds elect at
bounded terms (5→6→7→8). The round-4 crash-stall is gone.

**The soak now reaches rounds 4-6 for the FIRST TIME — and reveals a pre-existing
committed-job-loss (task 5.9, NOT a regression).**

Verdict: committed_job_loss=4, committed_divergence=7 → FAIL. But this is NOT a
regression of 5.8 nor of any previously-fixed bug. Proof:

1. ONE specific committed job (0-0-1782829181-1) goes missing at round 4 and stays
   missing through rounds 5 and 6, across THREE different elected primaries
   (ctrl-3, ctrl-2, ctrl-3). A permanent committed-prefix gap that propagates as
   catch-up streams FROM the gapped node — the exact bug-795/828/851 signature.
2. Rounds 1-3 are CLEAN (all jobs present, zero divergence). Loss begins precisely
   at round 4 — the first round whose overlapping kills elect a JUST-RESTARTED node
   (ctrl-2 restarted, ctrl-1 killed, ctrl-3 — itself recently restarted — elected).
   A protocol/catch-up condition, not a memory condition.
3. This session changed ZERO WAL/commit/catch-up/replication code (persist.c,
   config_repl.c, commit.c, repl*). The files touched are cluster.c, ha.c/.h,
   netio.c, vtaskmgr.c (ep refcount + the 5.7 election timeout) + 2 test files. The
   commit-durability path is byte-identical to the session start — it CANNOT have
   regressed because it did not change.
4. ASAN is fully clean — no refcount-induced UAF/double-free/leak. A refcount bug
   would crash (caught by ASAN) or corrupt broadly; it cannot selectively delete
   one committed job's payload from a WAL.
5. No chaotic soak EVER reached rounds 4-6 before (every prior run crashed/stalled
   at round 3-4: bug-851 cross-deny → bug-1282 deadlock → bug-1286 segv → 5.5/5.6/
   5.7 election stalls → bug-1298 UAF). There is no prior green baseline for rounds
   4-6 to regress FROM — this is a first observation.

Conclusion: bug-795/828/851 FAILOVER CATCH-UP GAP — a restarted node rejoins and is
elected primary without backfilling the full committed prefix (empty-rejoin catch-up
streams [watermark..committed], no snapshot backfill → permanent gap). EXTERNAL spec
ha-commit-durability (+ raft term/log-safety). Tracked as task 5.9; route there.

## Phase 1 closeout — 10-round chaotic soak VALIDATES the election layer (2026-06-30)

Re-ran the chaotic soak at the full 10-round budget (`run-ha-soak.sh --chaotic 10`,
deploy `/home/claude/tmp/ha-1296`) to confirm bug-1296 (the round-7 RDHUP→PreVote→timeout
liveloc) is gone after tasks 5.5/5.6/5.7.

**Election liveness: PASS (10/10).** Every kill round elected a new primary in a single
term increment — terms walked 4→5→6→7→8→9→10→11→12→13, one step per round, ZERO escalation.
Round 7 (the historical bug-1296 liveloc point) and rounds 8–10 all elected cleanly. The
per-campaign jittered watchdog (5.7) + don't-preempt-in-flight-election (5.6) fully resolve
the resonance. bug-1296 marked FIXED.

**Consistency: FAIL (expected, task 5.9 / EXTERNAL).** Final verdict
`committed_divergence=11 committed_job_loss=5 result=FAIL`. Job `0-0-1782854622-0` went
missing at round 8 and stayed absent on the reachable primary through rounds 8/9/10 and the
settle — a permanent committed-prefix gap. This is the documented bug-795/828/851/1306
failover catch-up gap (a just-restarted node elected primary without backfilling the full
committed prefix), NOT an election fault and NOT a regression: rounds 1–7 are job-loss-free,
the loss onset tracks kill timing, and the election machinery is perfect across all 10 rounds.
The per-round `compared=3 present=2` divergence before round 8 is the just-restarted node
lagging its backfill (no convergence wait), which the catch-up backfill (Phase 2) closes.

Phase 1 (election liveness) is DONE. The remaining soak FAIL is wholly Phase 2 (task 5.9 /
ha-commit-durability catch-up backfill).