myxce/.kiro/specs/ha-commit-durability/research.md

Gap Analysis — ha-commit-durability

Brownfield analysis against the XCE controller (src/controller/). Informs /kiro-spec-design. Line numbers from the working tree at analysis time (2026-06-18); re-confirm before editing.

Current-State Summary

The HA stack is mature and well-factored: per-LSN majority commit engine (commit.c), fan-out WAL replication with ACK/catch-up (repl.c, config_repl.c), a commit barrier that gates client replies (persist_commit_barrier), demotion truncate + rejoin reconciliation (ha_convergence.c), single-leader own-term gate, PreVote, and a vote-freshness gate. The durability gap is beneath all of it: "committed" (majority-ACKed) does not imply "durable on a majority", and recovery/promotion trust in-memory position rather than the durable log.

The submit reply path is already commit-gated and runs off the iomgr thread (job.c:568-569 — "the AFTER_EVENT hook writes WAL and waits for majority replication"), so durability work can be added to that path without stalling heartbeats.

Requirement-to-Asset Map

Requirement 1 — Durable Commit (acknowledgment ⇒ majority persistence)

Need	Existing asset	Status
Reply gated on commit	persist_commit_barrier → commit_submit(COMMIT_MODE_WAIT) (persist.c); callers job.c:1024, queue.c:144, worker.c:713, join.c:130	✅ Exists — reuse; strengthen meaning of "committed"
Standby persists before ACK	persist_wal_write_replicated fdatasync before return (persist.c:515)	✅ Exists — pin with regression test (Req 1.3)
Primary self-counts only when durable	commit.c:majority_candidate_locked sets cap = highest_appended_lsn(ce->p) (appended, not durable)	❌ Missing — core defect (bug-661)
Primary persists before self-count	persist_wal_async (persist.c:340) appends via pwrite, broadcasts, returns without fsync; durability deferred to fsync_worker timer (persist.c:173)	❌ Missing
Track highest durable LSN	none — persist_t has committed_lsn/current_lsn but no durable_lsn	❌ Missing (new field)
Liveness during durability work	commit-wait path already off iomgr (job.c:568-569)	✅ Constraint satisfied by placement

Minimal change shape: add durable_lsn to persist_t, advanced when the entry's bytes are fdatasync'd; persist_wal_async fsyncs (inline, synchronous design — see brief decision) and sets durable_lsn before returning; majority_candidate_locked uses min(durable_lsn, …) for the self contribution instead of highest_appended_lsn. Reply gate then enforces the invariant unchanged.

Requirement 2 — Recovery Integrity + Leadership Fence

Need	Existing asset	Status
No silent truncate-to-zero on corrupt recovery	replay_wal breaks on CRC chain break / CRC mismatch / short read (persist.c:1021, 1050, 1042) and returns the replayed count as success; only -ENOMEM propagates (persist.c:1107, 1152)	❌ Missing — silent truncate-accept is the amplifier
Durable floor re-derived on restart	persist_recover never sets committed_lsn — it stays 0 after restart; floor is reconstructed only at promotion (persist.c:1114)	❌ Missing — restart advertises in-memory floor unbacked by WAL
Pre-promotion integrity / caught-up gate	raft_cb_become_leader (ha.c:551) computes safe_lsn from votes (ha.c:572) then persist_truncate_wal_above(safe_lsn) (ha.c:577) — no check that local WAL is intact or ≥ safe_lsn	❌ Missing — hollow node can promote, truncate collapses tip without backfill
Freshness floor from durable WAL	ha_vote_self_freshness = max(applied_lsn, last_known_committed_lsn) — in-memory atomics, not re-derived from durable WAL (ha.c:2127; used cluster.c:380-396 vote, cluster.c:467-474 PreVote)	❌ Missing — truncated candidate advertises a floor it can't back
Fence "ineligible until caught up"	in_catch_up atomic exists but only a standby uses it to deny its own outgoing vote (ha.c:325, cluster.c:529); no fence for a freshly-recovered/promoting node	⚠️ Partial — extend concept
Re-eligibility on catch-up	standby catch-up apply ha_apply_wal_catchup_entry (ha.c:1192); re-inclusion exists	✅ Exists — hook the fence release here

Requirement 3 — Deterministic Durability Verification

Need	Existing asset	Status
Configurable durability cadence / disable background flush	wal_sync_ms = DEFAULT_WAL_SYNC_MS (config.h:86, value 10); persist_config_defaults (persist.h:30); no env/CLI/config override — callers myapp.c:164, cluster.c:1267 pass defaults unmutated	❌ Missing — must add an override
Test-mode seam convention	#ifdef XCE_TESTING hook shims (ha.c:2270-2335), test-only handle_request_vote (ha.c:2161); cluster.c not linked under XCE_TESTING (cluster.c:67)	✅ Pattern exists — follow it

Doc drift noted: persistence-and-ha-design.md:4178 says wal_sync_ms=50; code is 10 (config.h:86). Reconcile in design.

Implementation Approach Options

Option A — Extend existing components (recommended). All change sites are inside existing, well-scoped functions (commit.c, persist.c, ha.c, cluster.c). R1 = durable_lsn field + 2 edits (persist_wal_async fsync, majority_candidate_locked self cap). R2 = make replay_wal return an integrity verdict + a pre-promotion gate in raft_cb_become_leader + floor-from-durable-WAL in ha_vote_self_freshness. R3 = config override + an XCE_TESTING seam.

• ✅ Leverages the commit-barrier and in_catch_up machinery already present; smallest surface; matches TLA+ mapping in CLAUDE.md.
• ❌ Touches correctness-critical election/commit code — demands rigorous TDD and TLA+ re-verification.

Option B — New durability module. Extract a durability/durable_lsn tracker as a new component.

• ✅ Isolated unit testing. ❌ Over-engineered — durable_lsn is one atomic on persist_t; a new module adds indirection without separation benefit. Not recommended.

Option C — Hybrid (recommended for R2 only). Extend in place for R1/R3; add a small new recovery-integrity + promotion-eligibility helper (e.g. persist_recovered_intact() / ha_promotion_durable_ok()) for R2 so the integrity verdict and the fence predicate are unit-testable and don't bloat raft_cb_become_leader.

• ✅ Keeps the election hot path readable; testable fence predicate. Recommended: A for R1+R3, C for R2.

Effort & Risk

Req	Effort	Risk	Justification
R1 Durable commit	M (3–7d)	Medium-High	Localized (durable_lsn + 2 edits) but correctness-critical; needs TLA+ re-check (commit rule, leader completeness) and durability TDD. Liveness already safe (off-iomgr path).
R2 Recovery + fence	L (1–2wk)	High	Touches recovery, promotion, and the vote-freshness floor; must compose with ha-convergence-recovery (truncate/rejoin) and ha-single-leader-role (own-term gate) without double-handling, and honor CLAUDE.md's "never truncate from stale ctrlhello" rule. Election-safety sensitive.
R3 Verification seam	S (1–3d)	Low	Config override + XCE_TESTING hook following an established pattern.

Recommendations for Design Phase

Preferred approach: Option A (extend) for R1 + R3; Option C (extend + small fence helper) for R2.

Key design decisions to resolve:

1. Durability mechanism — already decided (brief): synchronous persist-before-self-count, eliminate fsync_worker as the durability mechanism; group-commit batching deferred to roadmap. Design must confirm fsync_worker's fate: delete entirely vs retain a best-effort flush for any non-client-acked / local-only write (audit wal_classify for writes that gate no reply; checkpoint markers already fsync inline at persist.c:573).
2. durable_lsn semantics — single atomic on persist_t; advanced under p->lock after fdatasync; consumed by majority_candidate_locked. Confirm interaction with persist_truncate_wal_above (must not let durable_lsn exceed a post-truncation tip).
3. Recovery integrity verdict — replay_wal/persist_recover must distinguish "clean EOF" from "stopped at corruption". Decide: FATAL on corruption (fail-loud, simplest, honors the user's fail-loud preference) vs flag-and-fence (recover what's intact, fence from leadership until caught up). The fail-loud preference ([[feedback_fail_loud_over_timeout]]) leans FATAL unless the node can provably catch up.
4. Freshness floor source — derive the candidate's advertised floor from the durable WAL tip, not in-memory applied_lsn/last_known_committed_lsn alone, so a truncated node cannot advertise an unbacked floor. Reconcile with CLAUDE.md: never truncate using a stale ctrlhello last_known_committed_lsn; the safe-commit point still comes from voter replies at promotion.
5. Compose with convergence-recovery — R2's pre-promotion fence must not duplicate ha_convergence.c's demotion-truncate/rejoin logic; define the seam (this spec fences promotion; convergence handles demotion and rejoin reconciliation of an intact log).

Research items to carry forward:

• Measure fdatasync latency on the target storage vs the existing replication round-trip, to confirm the synchronous-design latency add is acceptable before group-commit is needed.
• Confirm no client-acked operation class bypasses the commit barrier (wal_classify audit) — any that do would violate the invariant.
• Decide the wal_sync_ms override surface (env var vs CLI flag vs config field) consistent with existing --sync-timeout-ms precedent (main.c OPT_SYNC_TIMEOUT).
• Re-verify against raft/raft.tla and the CLAUDE.md Protocol Verification Checklist after the self-count change (commit rule, leader completeness).

---

Design Synthesis (2026-06-18)

Decisions resolved into design.md:

• Durability mechanism = synchronous. Add persist_t.durable_lsn (atomic, advanced after fdatasync); persist_wal_async persists before return; commit.c:majority_candidate_locked self-contribution uses durable_lsn not highest_appended_lsn. fsync_worker removed as the durability mechanism (checkpoint markers keep their inline fsync). Group-commit batching deferred to roadmap as a strict, contract-preserving optimization.
• No new module (Option A) for R1/R3; small fence helper (Option C) for R2. durable_lsn is one atomic — a module would be over-abstraction (build-vs-adopt: extend).
• Recovery: fail-loud + fence. replay_wal returns a verdict (RECOVER_CLEAN vs RECOVER_TRUNCATED); a truncated member is fenced from promotion and catches up from a healthy peer; FATAL only when it cannot reach a consistent committed state and is required for quorum. Balances [[feedback_fail_loud_over_timeout]] with availability.
• Freshness floor from durable WAL, not volatile in-memory last_known_committed_lsn; safe-commit point at promotion still from voter replies (honors CLAUDE.md).
• Compose with convergence-recovery: this spec fences promotion + verifies recovery integrity; convergence-recovery owns demotion truncate and rejoin of an intact log. No shared ownership of the commit index (kept volatile per Raft).
• Liveness: the added fdatasync lands on the already-off-iomgr commit path (job.c:568-569), so heartbeats are unaffected — Option A's only real risk is retired.

Generalizations/simplifications: reuse the existing in_catch_up fence concept (extend from streaming-standby to recovery-truncated promoter) rather than inventing a new fence; reuse persist_commit_barrier reply gate unchanged; follow --sync-timeout-ms and XCE_TESTING hook conventions for R3.

---

Gap Re-Audit (2026-06-18)

Post-implementation gap audit. All 17 tasks (1.1–5.3) + the never-self-exit rework are marked [x] (landed 2026-06-18). This pass re-verifies each EARS acceptance criterion against the on-disk code (codegraph + grep), tags each MET / Missing / Unknown / Constraint, and records the known residual test-thoroughness items called out in the Implementation Notes. Line numbers from the working tree at audit time; re-confirm before editing.

Requirement-to-Asset Map (verified against code)

Requirement 1 — Durable Commit (acknowledgment ⇒ majority persistence)

Crit	Asset (file:line)	Tag
1.1 ack ⇒ durable on majority	Reply gated on commit barrier (unchanged) + self durable-capped (commit.c:93) + standby persist-before-ACK (persist.c:761); end-to-end test test_durable_majority_commit_end_to_end	MET
1.2 self counts only when durable	majority_candidate_locked self-cap cap = persist_durable_lsn(ce->p) (commit.c:93, was highest_appended_lsn); debug invariant assert(cand <= persist_durable_lsn) (commit.c:162)	MET
1.3 standby persists before ACK	persist_wal_write_replicated inline fdatasync rc-gated, return -EIO on failure so the standby does NOT ACK (persist.c:761-773); caller ha_apply_single_entry ACKs only on rc==0 (no-apply/no-ACK on negative). Pinned by test_replicated_persist_syncs_before_return / test_replicated_sync_failure_no_ack (counter test_replicated_fdatasync_count)	MET
1.4 not reported until durable-majority	committed_lsn cannot advance over a non-durable entry (1.2 cap) → existing WAIT-mode reply gate enforces it unchanged	MET
1.5 minority loss survives	Property test test_sole_durable_contributor_loss_survives (committed ⇒ ≥⌊N/2⌋+1 durable copies, drop one); live failover proof via soak (5.2)	MET (live N=2 fault is decision/property-level — see residual G1)
1.6 liveness during durability work	Group-commit flusher (fsync_worker) syncs OFF the iomgr/caller thread; fdatasync under p->lock only (never with fsync_mutex); test_durability_work_off_iomgr_preserves_liveness asserts flush on a non-caller thread + heartbeat reads <50ms	MET

Mechanism note: persist_wal_async no longer fsyncs inline (persist.c:643-652 signals the flusher); durability watermark persist_t.durable_lsn (_Atomic uint64_t) advanced ONLY after a successful fdatasync (flusher persist.c:281-311, persist_flush persist.c:801-806). fsync_worker repurposed from fixed-interval timer to demand-driven group-commit flusher; checkpoint markers retain their inline fdatasync (persist.c:162). Legacy sync_append ACK-commit store now durable-gated (sync_append_legacy_commit_store, store only when written_lsn <= durable_lsn).

Requirement 2 — Recovery Integrity + Leadership Fence

Crit	Asset (file:line)	Tag
2.1 no silent truncate-accept	replay_wal(p, from_lsn, recover_verdict_t*) classifies CRC chain break / CRC mismatch / short header / short payload → RECOVER_TRUNCATED; clean EOF / pre-alloc zero tail / empty WAL → RECOVER_CLEAN (persist.c:1411-1605); verdict stored on persist_t.recovery_verdict, accessors persist_recovery_verdict/persist_recovery_truncated (production, not XCE_TESTING-gated)	MET
2.2 ineligible until caught up	Pre-promotion gate in raft_cb_become_leader (ha.c:952-1010): REFUSE if persist_recovery_truncated OR durable_floor < promotion_safe_lsn (read AFTER truncate-clamp), set in_catch_up, return WITHOUT becoming primary	MET
2.3 below-HWM never authoritative	Durable-backed freshness floor: candidate advertises bare ha_candidate_advertise_lsn = persist_durable_floor (ha.c:2587), voter floor max(durable_floor, learned); promotion gate compares clamped durable_floor vs voter-reply promotion_safe_lsn; floor re-derived from recovered WAL tip in persist_recover (assign, not max)	MET
2.4 loud alert, hold non-promotable, STAY ALIVE, never self-exit	persist_recover_loud_fence_if_unrecoverable (persist.c:1787): loud ERROR with truncation boundary + sticky _Atomic bool recovery_unrecoverable, NO abort/exit/tlog_exit; RECOVER_DECISION_FENCE_UNRECOVERABLE (non-terminating). grep confirms zero abort/exit on the recovery path; persist_fatal_hook removed from production (only a test comment remains)	MET
2.5 preserve durable-commit at degraded/minimal quorum	Hollow node refused regardless of quorum size (gate is unconditional on truncated/behind) + 4.2 freshness floor; test_promotion_gate_hollow_refused_at_minimal_quorum (decision-level, Req 2.5)	MET (decision-level; no dedicated live N=2 fault-injection — residual G1)
2.6 re-eligibility on catch-up	Fence/flag clear automatically when !truncated && durable_floor >= HWM: ha_evaluate_promotion_fence ELIGIBLE branch clears in_catch_up (ha.c:592-597); promotion_fence_timeout_fn releases + persist_clear_recovery_unrecoverable on catch-up (ha.c:755-763); gate ELIGIBLE branch clears + cancels timer (ha.c:986-993)	MET

Bounded-window catch-up (task 4.4): raft_cb_become_leader NEVER FATALs/aborts synchronously — on FENCE/would-be-unrecoverable it arms a one-shot promotion_fence_timeout_fn (DEFAULT_RECOVERY_CATCHUP_WINDOW_MS=30000) that re-reads live state, releases on catch-up, re-emits the loud alert + re-arms if still hollow with no peer — never exits (ha.c:721-807). UAF-safe cancel; cancelled in ha_fini.

Requirement 3 — Deterministic Durability Verification

Crit	Asset (file:line)	Tag
3.1 configurable durability cadence	--wal-sync-ms flag (main.c:67,118) → myconf.wal_sync_ms → persist_config_set_wal_sync_ms (persist.h:130) applied at BOTH persist_init sites (myapp.c:169 production, cluster.c:1284 phase2); 0-sentinel = keep DEFAULT_WAL_SYNC_MS. Test seam persist_test_durable_lsn_on_disk (read-only on-disk CRC-validated durable LSN) + persist_test_set_force_fsync_fail fault injection, all XCE_TESTING-gated (persist.h:445-506)	MET
3.2 ack ⇒ recoverable after ungraceful kill in verify-mode	Deterministic E2E (test-commit-durability.sh --deterministic): with --wal-sync-ms 30000, in-window primary fdatasync >= 1 (=2) precedes ack ("durable-before-ack"), flipped RED→GREEN stable 3/3 / 9-9 probes (5.2)	MET
3.3 same observable ack semantics with/without verify-mode	Documented as a property in the test header (test_commit_durability.c:33-34): verify-mode "changes nothing observable about the acknowledgment contract" — but there is NO dedicated A/B assertion running the same op with mode on vs off and asserting identical observable acknowledgment	PARTIAL (residual G2)

Constraint compliance: safe_committed_lsn arithmetic UNTOUCHED (voter-reply-sourced, honors CLAUDE.md "never truncate from stale ctrlhello"). raft.tla election/freshness logic maps faithfully; raft.tla does NOT model durability (restores log intact on restart, raft.tla:165-174) — the commit-durability invariant is verified by tests 2.x/5.1/5.2 + the manual Protocol Verification Checklist, NOT by raft.tla. (Tagged Constraint — by-design deferral, see residual G3.)

Residual Gaps

ID	Gap	Severity	Notes
G1	Req 2.5 has no dedicated minimal-quorum live fault-injection test (N=2, lose the sole durable contributor across a real failover). Current coverage = decision/property-level (test_promotion_gate_hollow_refused_at_minimal_quorum, test_sole_durable_contributor_loss_survives) + the rolling-SIGKILL soak.	LOW	Invariant is enforced by the unconditional gate + freshness floor and the soak observed zero loss; a targeted N=2 case would harden the assertion against a future regression but is not a correctness gap.
G2	Req 3.3 has no verify-mode A/B observable-equivalence assertion (same op, mode on vs off, identical externally observable acknowledgment). Property is asserted only by header comment + the structural fact that the seam is read-only / production-excluded.	LOW	The seam (persist_test_*, --wal-sync-ms) demonstrably changes only timing, not the reply path (reply gate unchanged). An explicit A/B test would close the letter of 3.3.
G3	raft.tla does not model durable-vs-volatile log; the formal model cannot prove the commit-durability invariant.	LOW (by design)	Explicitly deferred (task 5.3 note, requirements out-of-scope). Durability is test-backed. A durable-log TLA+ extension is out of scope unless explicitly opted in.

No CRITICAL/HIGH residuals. No criterion is Missing or Unknown: 16/16 verifiable EARS criteria are MET (1.1–1.6, 2.1–2.6, 3.1–3.2) with 3.3 PARTIAL (property held but not A/B-asserted). The never-self-exit invariant (Req 2.4) is upheld — grep/nm confirm no abort/exit/tlog_exit on the recovery/promotion path.

Integration Risks

• Group-commit flusher latency on a degrading disk (Implementation Note 2.1 follow-up): the flusher tight-spins on SUSTAINED fdatasync failure (releases p->lock each iter so liveness holds, but CPU/log burn). A HANGING (not erroring) fdatasync blocks appends since it runs under p->lock — pre-existing, not introduced here. Non-blocking; carry to roadmap with group-commit batching.
• Test-fixture durable priming (bug-704 class): any commit-engine / election fixture that primes current_lsn/applied_lsn/last_known_committed_lsn but NOT durable_lsn will see floor/advertise=0 and committed-stall, because both the self-cap (2.2) and the freshness floor (4.2) now read durable state. Future tests MUST prime durable_lsn to the modeled tip. (4 fixtures already corrected; bug-760 caught one adjacent real-flusher test.)
• fsync-able filesystem for validation: /tmp overlay in some sandboxes EIOs on fdatasync; because committed_lsn is now durable-gated, an EIO correctly HOLDS committed_lsn and surfaces as adjacent-test failures. Final validation MUST run on a real fsync-able fs. Gate on ANSI-stripped :FAIL: count (meson Fail:0 masks Unity sub-group failures).

Options for the Open Test-Thoroughness Items (G1, G2)

Option A — Close both with targeted unit/integration tests (recommended).

• G1: add an N=2 minimal-quorum integration test reusing the test_sole_durable_contributor_loss_survives two-persist-dir harness, but drive it through a real failover-style commit + drop-the-durable-contributor and assert the acked op is present on the survivor.
• G2: add an A/B test that submits the same op with verify-mode (--wal-sync-ms slow / forced) and without, asserting byte-identical externally observable acknowledgment (same rc, same reply gate release semantics).
• Effort: S (1–3d). Risk: Low — test-only, no production change; reuses existing harnesses/seams.

Option B — Cover via the soak/E2E harness only.

• Extend test-commit-durability.sh with an N=2 deterministic round (G1) and a mode-on/mode-off ack-equivalence probe (G2).
• Effort: S (1–2d). Risk: Low-Medium — E2E is less deterministic than unit; the strace-gap probe is the established deterministic surface, so prefer it over joblist-loss for G2.

Option C — Accept as documented residuals (status quo).

• Record G1/G2 as LOW-severity known limitations; rely on the gate/floor invariants + the zero-loss soak already in place.
• Effort: none. Risk: Low — acceptable given no CRITICAL/HIGH gap, but leaves the letter of Req 3.3 (A/B) and a dedicated Req 2.5 live case unasserted, so a future regression in those exact paths could go uncaught until soak.

Recommendation: Option A — both items are S-effort, test-only, and turn two LOW residuals into fully-asserted criteria; G3 stays a by-design deferral.

---

REOPENED 2026-06-19 — cold-restart-all (total-cluster-outage) committed-job loss (bug-795)

Trigger: the xce-test-e2e-primitives spec converted test-commit-durability.sh to exit-code primitives and ADDED a new run_cold_restart_recovery phase (kill ALL 3 controllers simultaneously, then restart ALL from disk, then assert every acknowledged job recovers). Running it live on a 3-node cluster surfaced a NON-DETERMINISTIC committed-job loss that this spec's original validation never exercised.

What was observed (3 live soak runs, sandbox 3-node cluster):

• Rolling single-node SIGKILL + restart: ZERO loss every round (consistent with this spec's 5.2 result — single-host loss is non-deterministic and standbys fdatasync-before-WalAck, so a committed job is durable off the primary).
• Cold-restart-all (kill ALL 3, restart ALL): a job that returned jobsubmit ok AND survived every rolling round VANISHED after the full restart (expect-jobs expected N, actual N-2; expect-job <id> not found). PASSED in one run, FAILED in another → a race. Recovery logs show WAL truncate on the rejoining nodes.

Why this spec did not catch it: 5.2's soak was rolling single-node kills only; the durability guarantee was validated (and CLAUDE.md flipped to "GUARANTEED") against that scenario. The total simultaneous outage path — where ALL members lose their volatile/page-cache tail at once and recovery must reconstruct committed state from on-disk WAL alone — was never tested. When every node dies together, the "durable off the primary" assumption that makes single-host loss non-deterministic no longer applies: only what was fdatasync'd on a majority before the ack can survive.

Open question — REAL durability gap vs OVERLAYFS artifact (must resolve first):

• This sandbox's /tmp is overlay (NOT a real fsync-able fs). This spec's own notes repeatedly flag that overlay /tmp can EIO on fdatasync and that "final validation must run on a real fsync-able fs."
• If fdatasync on overlay is weak (returns rc=0 without true durability) OR if recovery's truncation (replay_wal / persist_truncate_wal_above / durable-floor re-derivation, tasks 3.1/3.4) is too aggressive after a total outage, a committed entry's tail could be lost or truncated on cold restart.
• The durable-gating (2.2/2.4) should make committed_lsn HOLD if fdatasync failed — so an un-fdatasync'd job should NOT have returned jobsubmit ok. Yet these jobs DID get jobsubmit ok. So either fdatasync returned success on overlay without real durability, or a non-durable-gated ack path exists under the total-outage timing, or recovery truncates committed entries.

Disposition: the xce-test-e2e-primitives soak is the reproduction surface and is working correctly (it detects the loss via exit codes). The fix (if real) belongs HERE. See Phase 6.

UPDATE 2026-06-19b — trustworthy soak sharpens the finding (rolling-failover loss + liveness)

After the xce-test-e2e-primitives soak was HARDENED (commit d82cc78: it no longer conflates liveness with durability — durability is asserted ONLY against a confirmed-serving cluster, with a catch-up-lag confirm re-check; a non-serving cluster is INCONCLUSIVE, never "DATA LOSS"), three 6–10-round live runs give a cleaner signal:

1. The committed-job loss reproduces in ROLLING single-node failover too — not only cold-restart-all. One run lost a committed job at round 6 with the cluster CONFIRMED SERVING (DATA LOSS — persistently absent from a SERVING cluster); two other runs had zero loss. So the durability race is NON-DETERMINISTIC (~1 in 3 runs here) and broader than the cold-restart-all framing in the original reopen note — it is a general acked-but-not-durable-on-majority race that a kill can expose during normal rolling failover. This is exactly the bug-661 invariant this spec claimed CLOSED for rolling failover (task 5.2) — it is not fully closed. (Caveat still applies: sandbox /tmp is overlayfs; 6.1 must confirm on a real fsync-able fs.)
2. Separate LIVENESS concern: every run shows 2–3 rounds where, after killing the current primary (consistently ctrl-1), NO new primary served within 60s (the cluster always eventually recovered). A >60s failover for a single-node kill in a 3-node cluster is poor availability and may share a root cause with the durability race (a behind/just-rejoined node interacting with the promotion fence / freshness floor). Worth a dedicated liveness check in 6.2.

Phase 6 scope note: 6.1's reproduction MUST include BOTH the cold-restart-all step AND the rolling-failover rounds (the hardened soak runs both and reports trustworthy per-round verdicts), and should capture the >60s-election cases for the liveness angle.

UPDATE 2026-06-19c — 6.1 ANSWERED: the loss is REAL, not (only) an overlay artifact

The cold-restart-all / rolling durability loss was initially suspected to be an overlay-/tmp fdatasync-EIO artifact of the dev sandbox. That is now ruled out as the primary cause:

• User runs the soak OUTSIDE the container (real host, real fsync-able fs, NO overlay fdatasync EIO) and still observes the DATA LOSS + the liveness failures. So they are genuine, not sandbox fdatasync noise.
• In-sandbox confirmation on a real ext2/3 deploy dir (fdatasync succeeds, ZERO EIO): a 3-round run PASSED clean, but an 8-round run reproduced BOTH a liveness cascade (after killing ctrl-1 in round 1 the cluster never returned to 3-healthy) AND a cold-restart expect-jobs shortfall with a PERSISTENTLY-absent committed job — with zero fdatasync EIO. (Caveat: the 8-round run's cold-restart followed a degraded run, so the cleanest real-fs evidence is the user's outside-container host.)

Verdict for 6.1: the durability loss and the liveness stall reproduce off overlay → REAL controller findings, not an fdatasync-EIO artifact. The overlay EIO remains a SEPARATE, sandbox-only confound (it makes the dev-box controller_unit_test flaky — see bug-760 — and inflates loss on overlay deploys), but it is not the root cause of the soak loss.

Phase 6 is now at 6.2 (root-cause). Two real, non-deterministic findings to root-cause:

• Durability: an acked (jobsubmit ok) job is PERSISTENTLY absent from a SERVING cluster after failover / cold restart — i.e. bug-661 is NOT fully closed (Phase 5 only exercised rolling single-node kills with the racy soak). Buckets: acked-without-durable-majority (commit-gate timing) / recovery truncates committed entries / behind-then-killed node re-leads with a truncated log.
• Liveness: after killing the current primary (consistently ctrl-1 in these runs), the cluster sometimes takes >60s — or fails entirely — to elect a serving primary, and a restarted node sometimes never returns to a healthy 3-node serving set. Possibly coupled to the promotion-fence / freshness-floor interaction with a just-rejoined node.

The hardened xce-test-e2e-primitives soak (trustworthy: liveness vs durability separated, confirm-settle, quiet wait-primary) is the reproduction surface for both.