myxce/.kiro/specs/raft-ha-core-safety/design.md

Design Document — raft-ha-core-safety

Overview

This feature makes repeated and chaotic primary failover on an XCE controller HA cluster provably safe and live. It is a correction-and-verification spec, not a build-from-scratch one: the (term, lsn) commit-identity, conflict-reconcile, Figure-8 commit, and vote-freshness machinery already exist and are validated (raft-log-replication, raft-commit-safety, ha-term-safety), yet the live chaotic soak still produces committed-prefix divergence (LSN 9), standby-rejoin failure, and intermittent election stalls — on this branch and on baseline 21e26a1.

Discovery (task 1.3, live real-fs release soak, independently verified — see research.md) root-caused the residual divergence to cross-term catch-up payload loss, not the term-blind-bypass hypothesis the early analysis proposed (that hypothesis is refuted: the divergent entry carries the correct term and arrives through the reconcile seam). At committed LSN 9 all members agree on term, but the minority member holds an empty payload (PLEN=0, pcrc=0) where the majority holds the real entry (PLEN=3560). The bisect: when a restarted standby catches up an entry whose stored term is below the current leader's term (the bug-1067 cross-term path), the leader-side streamer ha_stream_wal_to_standby (config_repl.c ~2154-2163) silently ships PLEN=0 on a short pread or malloc failure instead of failing loud; the receiver faithfully persists the empty body; and the persistence layer's (term,lsn)-only idempotency guard is payload-blind, so it never detects that an existing-LSN entry now has a divergent payload. The result is a permanently divergent committed prefix that carries the right term but the wrong (empty) data.

The design therefore: (1) collapses every entry-ingest path into the one reconcile seam and makes persistence (term, lsn)-conflict-aware instead of LSN-only-idempotent; (2) closes the snapshot-boundary and standby-reconvergence gaps; (3) guarantees election liveness under churn; and (4) delivers a deterministic chaotic harness that forces each defect RED before its fix and a green 6+ round real-fs soak as the acceptance gate.

Goals

• Committed log prefix is identical across all members under repeated/chaotic failover (zero walcompare divergence).
• Every entry entering a member's log passes the single (term, lsn) reconcile seam; no path stamps a non-leader term or silently skips conflict repair.
• A restarted node deterministically reconverges to 3-of-3 (committed + applied), and elections always complete under churn.
• Each defect is pinned by a deterministic RED→GREEN test; the spec closes only on a green 6+ round chaotic real-fs release soak with zero divergence and zero committed-job loss.

Non-Goals

• Rebuilding or re-deriving the Raft library consensus core (log replication, commit advance, vote freshness) — adopted as-is.
• Persist-before-ACK durability and the sync barrier — owned by ha-commit-durability.
• raft-drive-async async-drive/offload behavior (SlowLane, peer-control dispatch, snapshot/config-push offload, endpoint-quiescence) — out of scope and must not change.
• Client-side primary discovery (libxce bug-854) and the overlayfs fdatasync environment defect (bug-924) — excluded from the acceptance verdict, not fixed here.
• Cluster membership resize, multi-region, non-Raft replication.

Boundary Commitments

This Spec Owns

• The single-seam ingest invariant: routing every replicated/catch-up entry through raft_on_append_entries_request so logOk + (term,lsn) conflict-truncate + leader-term stamping always run.
• Making the WAL persistence idempotency/contiguity guard (term,lsn)-conflict-aware (detect same-LSN/different-term as a conflict to repair, not a no-op to skip).
• Snapshot-boundary (term,lsn) consistency so post-install AE intake reconciles correctly and no committed entry below the checkpoint can diverge.
• Standby reconvergence completeness: post-recovery log_len/applied sync and catch-up apply completeness before a node reports healthy/3-of-3.
• Election liveness under churn: a member that loses its primary always proceeds to a completed election; no permanent stall.
• The deterministic chaotic harness (unit/integration + xce-test-ha soak extension + walcompare use) and the acceptance soak gate.

Out of Boundary

• Raft library internals (src/lib/raft/*) beyond consuming the existing callbacks/seam — no new consensus primitives.
• Persist-before-ACK durability mechanism and the commit sync barrier (ha-commit-durability).
• raft-drive-async dispatch behavior.
• libxce client discovery (bug-854); overlayfs env (bug-924).
• Sequential-failover rejoin reconcile already delivered by ha-convergence-recovery (reused, not re-implemented).

Allowed Dependencies

• Raft library reconcile seam + five callbacks (raft_on_append_entries_request, on_persist_entry, on_get_entry_term, on_truncate_from, on_commit_advanced).
• persist_set_term / per-LSN term index / WAL_LEGACY_TERM mapping (ha-term-safety).
• ha_rejoin_reconcile intake hook and ha_self_restart convergence primitive (ha-convergence-recovery).
• walcompare (term, lsn, pcrc) detector; xce-test-ha soak driver; run-ha-soak.sh BIN/XT override + non-ASAN release build.
• The controller TLA+ spec raft/raft.tla and the invariant checklist in src/controller/CLAUDE.md as the correctness oracle.

Revalidation Triggers

• Any change to the Raft callback signatures or the reconcile-seam contract → revalidate ha-term-safety, raft-log-replication, raft-commit-safety.
• Any change to WAL header (term, lsn) layout or the term index → revalidate walcompare and ha-commit-durability.
• Any change to the snapshot install boundary contract → revalidate ha-convergence-recovery rejoin path.
• Closing this gate (green soak) is itself the revalidation trigger that unblocks the raft-drive-async production flag-flip.

Retroactive Boundary Crossings (2026-07-09 validate-impl finding)

Two tasks materially crossed the "Out of Boundary" lines above. Both are self-labeled in tasks.md, non-conflicting with the specs that inherited them, and confirmed by a later cross-task validation pass — documented here so a future validate-impl doesn't re-flag them as new findings:

• Task 5.9 shipped a commit-durability fix (persist_submit_wal_and_commit extraction, hook-rc-clobber job-loss) — territory line 19/36 assigns to ha-commit-durability. Landed here as a "down payment" (tasks.md's own words) because raft-apply-decoupling had no tasks.md yet and this spec's Req 8 soak needed zero job-loss to close. raft-apply-decoupling's own task 3.1/3.2 later built the canonical version on the same code path — confirmed non-duplicating successor, not a competing fix.
• Task 5.8 shipped an endpoint-refcount/quiescence mechanism — territory line 20 explicitly excludes ("raft-drive-async... endpoint-quiescence — out of scope and must not change"). In practice it became the shared foundation raft-drive-async's own task 3.2/3.4a cite as the reason their UAF hazards are "closed for real" — both specs' authors acknowledged the dependency in their own tasks.md, making this real, coherent, bidirectional coupling rather than silent scope creep.

Architecture

Existing Architecture Analysis

• The real defect (task 1.3, verified): the cross-term catch-up payload is lost at its source. ha_stream_wal_to_standby (config_repl.c ~2154-2163) reads the entry body with pread into a malloc'd buffer; on a short read or allocation failure it frees the buffer and ships the entry with PLEN=0 silently (no error, no retry). The receiver persists the empty body, and the (term,lsn)-only persistence guard cannot tell that an existing-LSN entry now carries a divergent (empty) payload. Term and ingest-path are correct; the data is gone.
• Single reconcile seam (preserve, do not duplicate): all (term,lsn) conflict detection/repair lives inside raft_on_append_entries_request (raft.c:646–765). This seam is sound and is preserved as an invariant — but it is not where this bug lives (the divergent entry already passes through it with the correct term). The fix is upstream (streamer payload integrity) and underneath (payload-aware persistence guard).
• Layering (must hold): Raft library (pure, callback-driven) ← controller HA adapter (ha.c, ha_convergence.c, config_repl.c) ← persistence (persist.c, WAL). The adapter calls into the library and implements callbacks; the library never calls the adapter except through callbacks. The fix routes adapter ingest into the library, never around it.
• Two distinct truncate contracts (must not be conflated):
  • Promotion-truncate (a node becoming leader): clamps at max(safe_committed_lsn, durable_tip) (ha.c:999–1000, bug-1060) — a fresh leader must not discard its own durable tip.
  • Follower-AE-conflict-truncate (a follower/standby receiving an entry that conflicts at (term, lsn)): obeys Raft Log Matching — truncates the conflicting suffix regardless of local durable_tip. The bug-1060 promotion clamp does not apply here. Rationale: an entry that conflicts with the leader's (term, lsn) was, by Raft's Leader Completeness, never majority-committed, so it is safe — and required — to remove even if the local node had marked it durable/committed. Applying the promotion clamp on this path would block legitimate repair and convert repairable divergence into a permanent fence.
  • The only entries the follower-conflict path must never cut are those that match the leader (no conflict) — guaranteed by logOk before any on_truncate_from.
• Technical debt addressed: the silent empty-payload ship in the cross-term catch-up streamer and the payload-blind persistence idempotency guard are the primary debt this spec removes. The term-blind ingest branch is retired as defensive hardening (it is a latent hazard, not the cause of the observed divergence).

Architecture Pattern & Boundary Map

graph TB
    subgraph Library [Raft Library pure]
        AE[raft_on_append_entries_request]
        LOGOK[logOk term lsn check]
        TRUNC[on_truncate_from]
        PERSISTCB[on_persist_entry]
        AE --> LOGOK --> TRUNC
        LOGOK --> PERSISTCB
    end
    subgraph Adapter [Controller HA Adapter]
        STREAM[Catchup Streamer payload integrity]
        INGEST[Unified Entry Ingest]
        SNAP[Snapshot Boundary Reconcile]
        CONV[Standby Reconvergence]
        ELECT[Election Liveness]
    end
    subgraph Persist [Persistence WAL]
        GUARD[term lsn payload conflict guard]
        SETTERM[persist_set_term leader term]
        TERMIDX[per lsn term index]
    end
    subgraph Verify [Verification]
        HARNESS[Deterministic Chaotic Harness]
        WALCMP[walcompare term lsn pcrc]
        SOAK[xce-test-ha chaotic soak]
    end
    STREAM --> INGEST
    INGEST --> AE
    SNAP --> AE
    CONV --> INGEST
    PERSISTCB --> SETTERM --> GUARD --> TERMIDX
    TRUNC --> GUARD
    ELECT -.guarantees.-> AE
    HARNESS --> WALCMP
    HARNESS --> INGEST
    SOAK --> WALCMP

Architecture Integration

• Selected pattern: payload-integrity-at-source + payload-aware-persistence, with the single reconcile seam preserved as an invariant. The cross-term catch-up streamer must never ship a non-empty entry as empty (fail-loud on short pread/malloc-fail); persistence enforces (term, lsn, payload-crc) conflict semantics so a divergent-payload entry at an existing LSN is detected, not idempotent-skipped.
• Boundaries: catch-up payload integrity (Adapter streamer), persistence-conflict (Persist), reconvergence + liveness (Adapter), verification (test-only) are separable seams under one acceptance gate.
• Patterns preserved: single reconcile seam, callback-driven library, truncate-clamp, persist-before-ACK.
• New components rationale: the harness is genuinely new code; the streamer fix hardens a silent-failure path; the persistence change extends an existing guard from (term,lsn) to (term,lsn,payload-crc).
• Steering compliance: async-I/O (no blocking in vtask handlers; file I/O + restart on delaymgr, never callback thread), fail-loud over silent skip, controller-never-self-terminates-on-operational-condition.

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Consensus core	src/lib/raft (in-repo)	reconcile seam + callbacks (consumed)	no library changes beyond callback use
Controller HA adapter	C (ha.c, ha_convergence.c, config_repl.c)	ingest routing, snapshot/rejoin, election	primary change surface
Persistence	C (persist.c, WAL segments)	(term,lsn,payload-crc)-aware guard, term index	harden guard, no format change
Verification	C unit (controller_unit_test), walcompare, xce-test-ha bash	RED→GREEN + soak gate	new test_ha_core_safety.c; chaotic soak mode
Runtime	real filesystem (ext4) release build	acceptance soak	overlayfs excluded (bug-924)

File Structure Plan

Modified Files

• src/controller/config_repl.c — Core fix. Leader-side catch-up streamer ha_stream_wal_to_standby (~2154-2163): on a short pread or malloc failure when reading an entry body, fail loud (error + retry/abort the stream) instead of silently freeing the buffer and shipping PLEN=0. A non-empty entry must never be transmitted with an empty payload. Disambiguate and handle both send-side-empty (short read/alloc fail here) and any transitive-empty-propagation source uncovered during implementation.
• src/controller/persist.c — Extend the replicated-write guard (persist_wal_write_replicated persist.c:802, idempotency guard at :822/:839) from (term,lsn)-only to (term, lsn, payload-crc)-aware: an entry arriving for an existing LSN whose payload-crc differs (e.g. a real entry over a previously-stored empty one, or vice versa) is a conflict to surface for repair, not an idempotent return 0. Never accept an empty payload as the durable form of a known-non-empty entry. Preserve persist-before-ACK.
• src/controller/ha.c — Receiver-side handling so a recovered/divergent-payload entry is repaired through the reconcile seam (ha_consume_append_entries ha.c:2410-2440 body copy; ha_apply_wal_catchup_entry ha.c:2233). Snapshot post-install (ha_handle_snapshot_chunk ha.c:2492) establishes the (term,lsn) boundary. Standby reconvergence completion (applied/log_len sync, healthy gating). Election-liveness guarantee on the failover/vote path. Defensive: retire the term-blind ha_apply_single_entry (ha.c:2097) direct-persist branch so no future path can bypass the seam.
• src/controller/ha_convergence.c — Reuse the ha_rejoin_reconcile intake hook for chaotic-rejoin reconciliation through the seam; retain the ha_self_restart convergence primitive and once-guard.
• src/controller/testsuite/meson.build — Register test_ha_core_safety.c.
• src/controller/testsuite/walcompare.c — Extend to emit a machine-checkable (lsn → term, pcrc, present) per-member report consumable by the harness verdict (committed-vs-uncommitted classification).
• .claude/skills/xce-test-ha/run-ha-soak.sh (and SKILL.md) — Chaotic mode + isolated verdict (done, tasks 1.1/1.2). Plus (task 1.3 follow-up): make primary-detection robust to the release build's block-buffered --logfile — switch the soak's primary/convergence detection to a client probe (ctrlstatus) rather than log-grep, so the chaotic soak runs unattended on release binaries.
• src/controller/ log init (only if the client-probe approach is insufficient) — ensure the release build flushes (line-buffers) its --logfile so operator-observable progress is not lost until signal. Prefer the soak-side client-probe fix first; treat a controller log-flush change as a fallback that must not alter production logging semantics.

New Files

• src/controller/testsuite/test_ha_core_safety.c — Deterministic RED→GREEN harness: forces the cross-term catch-up payload-loss divergence (a non-empty entry streamed/persisted as empty at an existing committed LSN), snapshot-boundary divergence, standby reconvergence-failure, and election-stall paths using existing gate/probe seams (pattern: test_ha_dispatch_integration.c); no sleeps.

Dependency direction (enforced): persist.c (term/lsn storage) ← ha.c/config_repl.c/ha_convergence.c (adapter ingest/election) → Raft library seam. Tests depend on all; production never depends on tests.

System Flows

Cross-term catch-up payload integrity (the fix)

sequenceDiagram
    participant L as Leader streamer
    participant S as Standby ingest
    participant Seam as raft_on_append_entries_request
    participant P as persist term lsn payload guard
    L->>L: read entry body pread into malloc buffer
    alt short read or malloc fail
        L->>L: FAIL LOUD error retry or abort stream
        Note over L: never ship a non empty entry as PLEN 0
    else full body
        L->>S: stream entry at LSN N with full payload and term
    end
    S->>Seam: route through reconcile seam
    Seam->>P: on_persist_entry N term payload
    alt existing LSN payload crc differs
        P->>P: conflict surface for repair not idempotent skip
    else identical term and payload crc
        P->>P: idempotent no op safe
    end
    P-->>S: WAL holds full payload at N
    Note over S,P: walcompare term and pcrc identical across members

Key decisions: the streamer fails loud on a short pread/malloc failure instead of silently shipping PLEN=0; the persistence guard compares (term, lsn, payload-crc) so an empty-over-real (or real-over-empty) write at an existing LSN is a conflict to repair, not an idempotent skip. The single reconcile seam is preserved; truncation (when a genuine (term,lsn) conflict exists) never cuts below max(safe_committed_lsn, durable_tip) on the promotion path and follows the follower-conflict contract on the receive path.

Standby reconvergence gating

stateDiagram-v2
    [*] --> Rejoining
    Rejoining --> CatchingUp: route entries through seam
    CatchingUp --> AppliedSync: log tip reached at match level
    AppliedSync --> Healthy: applied_lsn and log_len equal leader committed
    AppliedSync --> CatchingUp: apply gap remains
    Healthy --> [*]: report 3 of 3

A node is reported healthy/3-of-3 only when committed and applied state match the leader — not at match-index alone (closes the bug-1042 reconvergence gap).

Requirements Traceability

Requirement	Summary	Components	Key Contracts	Flows
1.1–1.4	Committed prefix identical; no divergent entry at a position; divergent treated non-authoritative; TLA+ State-Machine/Commit safety	Catch-up Payload Integrity; Persist Conflict Guard	streamer fail-loud; (term,lsn,payload-crc) guard	Payload integrity
2.1–2.4	Entry carries faithful identity+payload; never empty-over-real at a committed LSN; identity distinguishes conflicts; survives restart	Persist Conflict Guard; Catch-up Payload Integrity	(term,lsn,payload-crc) guard; persist_set_term; term index	Payload integrity
3.1–3.4	Detect conflict; repair to majority prefix; surface unrecoverable; never cut majority-committed	Unified Entry Ingest; Snapshot Boundary	on_truncate_from; follower-conflict truncate contract (logOk-gated)	Divergence repair
4.1–4.3	At most one committing primary/term; step-down on higher term; Election-Safety/Leader-Completeness	Election Liveness (step-down)	vote-once; pre-vote; vote freshness	—
5.1–5.4	Rejoin → 3-of-3 in bound; applied+log_len sync; catch-up completeness; not converged until applied matches	Standby Reconvergence	healthy-gating contract	Reconvergence gating
6.1–6.3	Elect higher-term primary in bound; keep electing under churn; retry on failed round	Election Liveness	election-restart guarantee	—
7.1–7.5	Force divergence/reconverge/stall RED on baseline; GREEN after; extend soak + unit/integration	Deterministic Chaotic Harness	RED→GREEN gate	all
8.1–8.4	6+ round chaotic real-fs soak: zero divergence, zero job loss, reconverge+elect each round; gate for flag-flip	Acceptance Soak Gate	soak verdict (bug-854/924 isolated)	all

Components and Interfaces

Component	Layer	Intent	Req Coverage	Key Dependencies	Contracts
Catch-up Payload Integrity	Adapter	Stream a non-empty entry never as empty; fail-loud on short read/alloc-fail	1,2,3	catch-up streamer (P0); Persist guard (P0)	Service, State
Persist Conflict Guard	Persist	(term,lsn,payload-crc)-conflict-aware write; never accept empty-over-real	1,2,3	term index (P0); persist_set_term (P0)	State
Unified Entry Ingest	Adapter	Route every entry through the one reconcile seam; retire term-blind bypass (defensive)	1,2,3	Raft seam (P0); Persist guard (P0)	Service, State
Snapshot Boundary Reconcile	Adapter	Establish (term,lsn) at checkpoint; post-install reconcile	3,5	snapshot install (P0); seam (P1)	State
Standby Reconvergence	Adapter	Gate healthy/3-of-3 on committed+applied match	5	catch-up apply (P0); ha_rejoin_reconcile (P1)	Service, State
Election Liveness	Adapter	Always complete an election under churn; step-down on higher term	4,6	vote/election path (P0)	Service
Deterministic Chaotic Harness	Verify	Force each defect RED→GREEN; no sleeps	7	gate/probe seams (P0); walcompare (P0)	Batch
Acceptance Soak Gate	Verify	6+ round chaotic real-fs soak verdict	8	xce-test-ha; release build	Batch

Adapter

Catch-up Payload Integrity

Field	Detail
Intent	The cross-term catch-up streamer must transmit every non-empty entry with its full payload, or fail loud — never silently ship PLEN=0.
Requirements	1.1, 1.2, 2.1, 2.3, 3.1

Responsibilities & Constraints

• In ha_stream_wal_to_standby (config_repl.c ~2154-2163): when reading an entry body (pread into a malloc'd buffer), a short read or allocation failure must be handled as an error (log + retry/abort the stream), not by freeing the buffer and sending the entry with an empty payload.
• Disambiguate the two possible sources the reviewer flagged: (a) send-side empty — the short-read/alloc-fail here; (b) transitive-empty propagation — the streamed source WAL already holds an empty body (itself produced by an earlier instance of this defect). The fix must close (a) directly and, if (b) is observed, trace it to its own streamer instance rather than masking it.
• Cross-term entries (stored term < leader current term, the bug-1067 path) are the trigger seen live — ensure they are read and shipped identically to same-term entries.

Dependencies: Inbound: walAck catch-up continuation (P0). Outbound: Persist Conflict Guard (P0) on the receiver; the reconcile seam (P1).

Contracts: Service [x] / State [x]

• Precondition: the source WAL holds a non-empty body for the entry being streamed.
• Postcondition: the receiver persists the same (term, lsn, payload-crc) the leader holds; a transmit failure is surfaced, never silently degraded to an empty entry.
• Invariant: walcompare reports identical (term, pcrc) for the streamed LSN across leader and standby.

Implementation Notes

• Validation: the RED test reproduces the live signature — a cross-term catch-up of an entry whose body read short/fails must NOT result in a persisted PLEN=0 entry at a committed LSN; instead the stream errors and retries. Pin send-side vs transitive via an instrumented payload_len capture at send and at on_persist_entry.
• Risks: failing loud on the stream must not wedge catch-up permanently — pair the error with a bounded retry so a transient short read recovers, while a persistent failure is surfaced (fail-loud over silent skip; controller never self-terminates).

Unified Entry Ingest

Field	Detail
Intent	Every replicated/catch-up entry passes raft_on_append_entries_request (logOk + (term,lsn) conflict + leader-term stamp). The term-blind direct-persist branch is closed by reachability, not by removal — task 6.2 traced every production sender and found it unreachable with real data; documented SAFE-BY-GUARD rather than structurally retired (see task 6.2's Implementation Note below).
Requirements	1.2, 2.3, 3.1, 3.2, 3.4

Responsibilities & Constraints

• Defensive: the entry->term == 0 direct-persist branch (ha_apply_single_entry) is NOT structurally removed (task 6.2 decision). Every real sender (repl_broadcast for live replication, ha_stream_wal_to_standby for catch-up) always supplies n_entries>=1 with a non-zero term, so this branch is unreachable with production data; the sole function that could ever drive it (repl_send_append_entries) is invoked only by the raft library's raft_send_append_entries, which has zero production callers today. It remains documented and guarded in ha.c, not retired. (Not the cause of the observed divergence, but a latent bypass hazard — now shown to be dormant, not live.)
• Conflict repair on this (follower/standby) path uses the follower-AE-conflict-truncate contract (truncate the conflicting suffix per Raft Log Matching) — not the promotion clamp. logOk gates the truncate, so only genuinely-conflicting (never-majority-committed) entries are cut. The promotion clamp applies only to the leader-promotion path, which this component does not touch.
• File I/O / truncation / restart never on a callback or iomgr thread — defer to delaymgr (existing pattern).

Dependencies: Inbound: catch-up/replication/rejoin call sites (P0). Outbound: Raft seam (P0); Persist Conflict Guard (P0).

Contracts: State [x] / Service [x]

State Management

• State model: per-member WAL log indexed by LSN, each entry carrying (term, payload); library log_len/commit_index mirror the WAL.
• Consistency: post-ingest, on_get_entry_term(N) equals the leader's term for N across all members; walcompare reports identical (term, pcrc) on the committed prefix.
• Concurrency: ingest may run on iomgr; deferred truncate/persist follow the existing delaymgr one-shot pattern.

Implementation Notes

• Integration: reuse ha_apply_term_entry_via_library/ha_consume_append_entries for every reachable ingest path.
• Validation (task 6.2, closing task 2.4's deferral): rather than retiring ha_apply_single_entry's direct-persist path, task 6.2 traced every production sender of a wal_catchup_entry_t message and found the term-blind branches (the batch-mode sub->term==0 fallthrough and the unconditional entry->n_entries==0 single-entry mode in ha_apply_wal_catchup_entry) unreachable with real data: repl_broadcast always sends n_entries==1 with the real stamped term; ha_stream_wal_to_standby always batches n_entries>=1 and sources each sub-entry's term via persist_entry_term, whose legacy-zero rule maps a stored term of 0 to WAL_LEGACY_TERM for any present LSN (never 0). The only function that could emit the term-blind shape, repl_send_append_entries, is driven solely by the raft library's raft_send_append_entries, which has zero production call sites (only the pure library simulator and an XCE_TESTING test helper call it). Decision: document safe-by-guard, do not structurally retire — retiring would mean building a term-0-safe seam variant for a path with no live trigger, which is speculative work the evidence does not warrant. The disposition and reachability trace are recorded as comments at ha_apply_single_entry and both term-blind branches in ha.c, and pinned by test_2_3_termblind_route_payload_guarded, test_2_3_unrepairable_catchup_raises_nonpromotable_fence, and test_6_2_real_senders_never_produce_termblind_shape (test_ha_core_safety.c).
• Risks: the safety boundary on this path is logOk, not a durable-tip clamp — only entries that fail the (term, lsn) match are truncated, and a failed match proves the entry was never majority-committed (Leader Completeness). A unit test must assert that an entry which matches the leader is never truncated, and that a conflicting entry below local durable_tip is truncated.

Standby Reconvergence

Field	Detail
Intent	A recovered standby — including a former primary holding a divergent uncommitted tail above the new leader's committed point — reconverges to follower (truncates the divergent tail, catches up) and is reported healthy/3-of-3 only after committed and applied state match the leader.
Requirements	5.1, 5.2, 5.3, 5.4

Responsibilities & Constraints:

• Divergent-tail reconvergence (the live-stall fix, task 1.3/Phase-4 investigation): when a standby recovers with an uncommitted tail whose entries conflict with the leader's log (the former-primary current_lsn > leader_committed shape — e.g. ctrl-2 holding 12-15 at an old term while the leader committed 11), the AE log-matching backtrack must reach the divergence point and the follower must truncate the conflicting suffix (follower-AE-conflict-truncate contract) and accept the leader's entries — the same way a short-suffix standby already reconverges (ae_reject_logmismatch ×N → ae_already_have). Determine and remove whatever prevents a longer/former-primary divergent tail from reaching ae_already_have (candidate: the leader's next_index backtrack does not cross the divergence before the control channel drops; or the recovered tip is advertised in a way that stalls the backtrack).
• Post-recovery log_len/applied-lsn sync; complete outstanding catch-up apply before "healthy"; bounded recovery window.

Dependencies: Inbound: rejoin path (P0). Outbound: Unified Entry Ingest (P0); ha_rejoin_reconcile hook (P1); the leader-side AE streamer next_index backtrack (P0).

Contracts: State [x] / Service [x]

• Postcondition: a recovered standby with a divergent uncommitted tail truncates it and reaches applied_lsn == log_len == leader_committed_lsn; reported converged only then, otherwise CatchingUp.

Implementation Notes

• Integration: reuse the ha-convergence-recovery rejoin hook; ensure the AE backtrack + follower-conflict-truncate path actually fires for a former-primary divergent tail, not only a 1-2 entry suffix.
• Validation: RED integration test — construct a recovered standby with a multi-entry divergent uncommitted tail above the leader's committed point and assert it reconverges to ae_already_have (no permanent ae_reject_logmismatch loop) and is not reported converged until applied state matches.

Election Liveness

Field	Detail
Intent	A member that loses its primary always proceeds to a completed election; at most one committing primary per term.
Requirements	4.1, 4.2, 4.3, 6.1, 6.2, 6.3

Responsibilities & Constraints:

• On genuine primary loss with quorum reachable, elect a higher-term primary within the failover window; restart a failed election round; step down on observing a higher/equal-term leader. No permanent stall under repeated rapid rounds.
• Spurious-failover suppression (the live-stall fix, task 1.3/Phase-5 investigation): a transient primary control-channel RDHUP must not launch an unwinnable PreVote-failover storm while a current-term primary is actually alive and reachable by the quorum. The observed stall: a recovered standby whose channel to the live primary flaps (~15s) treats each RDHUP as "primary died" → ha_begin_failover → PreVote that times out (peers see the primary alive, deny/ignore) → reconnect → repeat forever, so the node never rejoins. The fix: on a control-channel RDHUP, attempt reconnect + catch-up first, and/or back off failover when a node's own PreVotes repeatedly fail to reach a quorum because a current-term leader exists — only escalate to a real failover when the primary is genuinely unreachable to the quorum. The election retry already works (R6); the defect is retrying an unwinnable election instead of rejoining the live primary.

Dependencies: Inbound: failover detection / ha_on_primary_ctrl_fd_rdhup (P0). Outbound: Raft vote/election + raft_cb_vote_eligible (P0); peer reconnect path (P0).

Contracts: Service [x]

• Invariant: never two committing primaries in one term (Election Safety); a node without a reachable primary keeps initiating elections (liveness); a node WITH a reachable current-term primary rejoins it rather than launching repeated unwinnable failovers.

Implementation Notes

• Validation (liveness, R6): RED integration test forces repeated rapid kill/restart rounds and asserts a higher-term CTRL_PRIMARY appears each round; a scripted single-round failure asserts a follow-on election starts.
• Validation (spurious-failover, the live stall): RED test/soak asserts a recovered standby whose control channel flaps while a current-term primary is alive does NOT enter an unbounded ha_begin_failover/PreVote timed out loop — it reconnects and rejoins as follower.
• Validation (safety, R4 — two-leaders-per-term): an explicit Election Safety assertion runs in the chaotic harness and soak — for every observed term, at most one member ever reaches committing-primary state. This is a falsifiable verification, not an assertion of correctness: it maps to raft/raft.tla Election Safety and must hold under overlapping kills. If it fails, the design's "library already enforces single-leader" assumption is disproven and a concrete guard (e.g. tighten the pre-vote/vote-once window at the adapter promotion gate) becomes required work rather than verification.
• Risks: over-eager step-down could itself cause a stall — step-down must be paired with election restart. The harness must distinguish a liveness failure (no primary) from a safety failure (two primaries) so a fix for one does not regress the other.

Persist

Persist Conflict Guard

Field	Detail
Intent	(term, lsn, payload-crc)-aware write: identical (term, payload-crc) at an LSN is an idempotent no-op; a differing payload-crc (or term) at an existing LSN is a conflict surfaced for repair — never a silent return 0. An empty payload is never accepted as the durable form of a known-non-empty entry.
Requirements	1.1, 1.2, 2.1, 2.2, 2.4, 3.1, 3.4

Responsibilities & Constraints: extend the LSN-only idempotency check (persist.c:822) to compare (term, payload-crc); preserve persist-before-ACK; preserve the term index and WAL_LEGACY_TERM mapping.

Dependencies: Inbound: on_persist_entry/on_truncate_from (P0). Outbound: term index record_entry_term/persist_entry_term (P0).

Contracts: State [x]

• Precondition: caller has passed the entry through the seam (leader term known via persist_set_term; logOk already decided match vs conflict).
• Postcondition: WAL at LSN N holds exactly the leader's (term, payload); no silent retention of a stale-term or empty-payload entry.
• Invariant: never write term == 0 for a live-replicated entry, and never let a PLEN=0 write overwrite or shadow a known-non-empty entry at the same LSN. On the seam-driven conflict path the guard honors on_truncate_from (follower-AE-conflict contract) — it does not re-apply the promotion durable-tip clamp, which belongs to the leader-promotion path only. The guard never initiates a truncate on its own; it only executes the seam's decision.

Implementation Notes

• Integration: the guard distinguishes "already have identical entry (same term + payload-crc)" from "have a different-payload entry at this LSN"; the latter is surfaced for repair instead of return 0.
• Validation: unit test writes an empty-payload entry at LSN N, then the correct full-payload entry, and asserts the WAL ends with the full payload (no silent idempotent skip); and the reverse (full then empty) never degrades a stored entry.
• Risks: mis-tagging an identical (term, payload-crc) entry as a conflict would cause needless work — the guard compares both term and payload-crc and treats a true match as idempotent.

Verify

Deterministic Chaotic Harness

Field	Detail
Intent	Force each defect deterministically; RED on baseline, GREEN after fix; no sleeps.
Requirements	7.1, 7.2, 7.3, 7.4, 7.5

Contracts: Batch [x]

• Trigger: unit/integration run (controller_unit_test) + xce-test-ha chaotic soak.
• Input: injected divergent (term,lsn) entry; injected apply gap; scripted overlapping kills inside election windows.
• Output: pass/fail per path; walcompare (term,lsn,pcrc) report.
• Idempotency & recovery: deterministic gate/probe seams (pattern test_ha_dispatch_integration.c); teardown drains pending tasks (avoid the known teardown-race UAF class, bug-1224).

Hypothesis-validation discipline (mandatory):

• The live-repro gate (task 1.3) has already run and refuted the term-blind hypothesis (verified): the real defect is cross-term catch-up payload loss (research.md §"Task 1.3"). The RED unit/integration cases must therefore reproduce the observed signature — a non-empty entry persisted as PLEN=0 at a committed LSN under cross-term catch-up — not the synthetic term-conflict the earlier draft assumed.
• A green synthetic unit test over a still-red soak is not a pass: R1/R8 close only when the chaotic real-fs soak (task 7.2) reports zero walcompare divergence with the streamer + guard fixes in place.
• Contingency: if the soak still diverges after the streamer payload-integrity fix and the payload-aware guard land, treat it as a new finding — classify via walcompare (term,lsn,pcrc), record in research.md, and add the next fix task; do not mark R1/R8 done on the strength of green unit tests alone. (The reviewer flagged one still-open sub-question: send-side-empty vs transitive-empty propagation — disambiguate during the streamer fix.)

Acceptance Soak Gate

Field	Detail
Intent	6+ round chaotic failover on real fs (release build): zero divergence, zero job loss, reconverge+elect each round.
Requirements	8.1, 8.2, 8.3, 8.4

Contracts: Batch [x]

• Trigger: run-ha-soak.sh chaotic mode, real-fs (ext4) release binaries.
• Output: verdict that isolates bug-854 (client discovery) and bug-924 (overlayfs) from the divergence/job-loss result.
• Idempotency & recovery: verdict counts walcompare divergences and committed-job loss; gate passes only at zero across ≥6 rounds. This green result is the trigger that unblocks the raft-drive-async production flag-flip.

Error Handling

Error Strategy

• Conflict at an LSN (same LSN, different term) on the follower/standby path: logOk fails ⇒ seam truncates the conflicting suffix (follower-AE-conflict contract, no promotion clamp) and repersists the leader's (term, payload); never a silent no-op (fail-loud over silent skip). This is the normal, recoverable case — a conflicting entry was never majority-committed, so cutting it loses no committed data.
• Genuinely unrecoverable divergence: only when a node cannot obtain the majority-agreed prefix at all (e.g. the leader's log no longer covers the needed range and no snapshot can backfill it) — surface a loud, non-promotable fence and keep retrying; the controller does not self-terminate on an operational condition (steering invariant). Operator-visible WARN/ERROR with both LSNs/terms. Note: a local entry conflicting below the node's own durable_tip is not in this category — it is repairable via truncate, because the conflict proves it was never committed cluster-wide.
• Reconvergence apply gap: remain CatchingUp, keep streaming via the seam; never report healthy with an open gap.
• Election round failure: start a further election; never park without a primary while quorum is reachable.

Monitoring

• walcompare divergence count (soak verdict), per-edge quorum transition WARN (existing), election round/term progression logs, reconvergence applied/log_len deltas. Each defect path emits an operator-observable line so the soak verdict is explainable.

Testing Strategy

Unit Tests (test_ha_core_safety.c, controller_unit_test)

• Streamer payload integrity: a cross-term catch-up whose entry-body read returns short / malloc fails must NOT persist a PLEN=0 entry at a committed LSN — the stream errors and retries instead (1.1, 2.1, 3.1).
• Payload-aware guard: an empty-payload write at existing LSN N followed by the correct full-payload entry ⇒ WAL ends with the full payload (guard did not silently idempotent-skip); the reverse (full then empty) never degrades the stored entry (1.1, 1.2, 2.1, 2.2).
• Defensive term-blind retirement: a term==0-shaped catch-up entry routes through the seam and is reconciled (no direct-persist bypass) (1.2, 3.2).
• Follower-conflict truncate contract: a conflicting entry below local durable_tip is truncated and repaired (the promotion clamp does not block it); a matching entry is never truncated (3.2, 3.4).
• Reconvergence gate: injected apply gap ⇒ node not reported converged until closed (5.2, 5.3, 5.4).

Integration Tests

• Snapshot post-install reconcile: a divergent pre-snapshot entry is reconciled against the checkpoint (term,lsn) (3.1, 5.1).
• Election liveness under churn: repeated rapid kill/restart ⇒ higher-term CTRL_PRIMARY each round; scripted failed round ⇒ follow-on election (6.1, 6.2, 6.3).
• Election Safety (two-leaders-per-term): across overlapping kills, assert at most one member ever reaches committing-primary per term — falsifiable check mapped to raft/raft.tla Election Safety; failure promotes a concrete adapter-gate guard to required work (4.1, 4.3).

E2E / Soak (xce-test-ha chaotic mode, real fs, release)

• Live-repro first: the chaotic soak must first reproduce the actual walcompare divergence (RED) and bisect it to a seam before synthetic unit tests are accepted as representative (7.1).
• ≥6 rounds chaotic failover: zero walcompare divergence (8.1); zero committed-job loss (8.2); 3-of-3 reconverge + new primary each round (8.3); verdict isolates bug-854/bug-924 (8.4).

RED-first ordering

• Every implementation task is preceded by its failing test on the unfixed baseline (project TDD steering). The live-divergence repro (7.1) and harness/RED tests land before the ingest/persist/election fixes so each fix is pinned to a walcompare/invariant violation (7.1–7.4). A green synthetic unit test over a still-red soak does not satisfy R1/R8.

Security Considerations

• No new external surface; all changes are intra-cluster consensus/persistence. The single-seam invariant strengthens integrity (no unvalidated entry enters a member's committed log). Fenced unrecoverable-divergence prevents serving divergent committed state. No secrets, auth, or input-boundary changes.