myxce/.kiro/specs/ha-commit-durability/design.md

Design Document — ha-commit-durability

Overview

Purpose: Make acknowledgment imply durability in the XCE HA controller. Today the controller advances its commit position on a majority of replication acknowledgments, but the primary counts its own contribution from a page-cache write (commit.c:majority_candidate_locked uses highest_appended_lsn), while background fdatasync runs on a timer (fsync_worker). A "committed" operation can therefore be durable on as few as one member — bug-661, observed as permanent job loss across failover.

Users: Cluster operators and every client whose acknowledged job/queue/worker operations must survive controller failover.

Impact: Strengthens the meaning of "committed" from "majority-ACKed (volatile)" to "persisted to stable storage on a fault-tolerant majority." The client reply path is already gated on the commit barrier, so once committed_lsn implies a durable majority, the invariant holds with no change to the reply path. Durability becomes group-commit: fsync_worker is repurposed from a fire-and-forget timer into a demand-driven flusher that, on append, batches fdatasync across all pending entries, advances durable_lsn, and wakes the commit evaluation. The commit majority counts durable_lsn (not appended). This amortizes one fdatasync across many entries — important because the highest-volume WAL traffic is WAL_MODE_RETAIN facts (job start/finish/fail/rusage), which are not client-latency-bound but must be durable before committed_lsn passes them. A second concern — a member with a corrupt or truncated log becoming authoritative — is closed by a recovery-integrity verdict and a promotion fence.

Goals

• Durable commit: when the controller replies ok, the operation's log entry is fdatasync'd on the acknowledging member plus ⌊N/2⌋ others (Req 1).
• Recovery integrity & promotion fence: a member that cannot reconstruct its committed state, or that lags the cluster's committed high-water mark, is barred from leadership until caught up — fail-loud when it cannot recover safely (Req 2).
• Deterministic verification: durability timing is controllable in a test mode so the guarantee is provable without racing the clock (Req 3).
• Preserve cluster liveness: durability work never stalls heartbeat/election processing.

Non-Goals

• Single-record job-array persistence (job-array-single-wal-record, roadmap) — complements this spec by bounding per-array fsync count; independent of the group-commit flusher introduced here.
• Per-entry synchronous fsync as the steady-state mechanism — superseded by group-commit (decision: the high-volume WAL_MODE_RETAIN path needs batching; see Components/DurableCommit).
• Replication/ACK wire format (ha-wal-replication), per-entry majority arithmetic (ha-commit-engine), demotion truncate / rejoin reconciliation of an intact log (ha-convergence-recovery), promotion no-op / own-term gate (ha-single-leader-role), raw vote-grant arithmetic (raft-library).

Boundary Commitments

This Spec Owns

• The durability precondition on the commit majority: a member contributes to the commit majority for an LSN only once that LSN is fdatasync'd locally. Introduces and is authoritative for persist_t.durable_lsn.
• Group-commit durability: fsync_worker repurposed from a timer into a demand-driven flusher that batches fdatasync, advances durable_lsn, and wakes commit evaluation; persist_wal_async signals the flusher instead of deferring to a timer. Benefits all members on the high-volume WAL_MODE_RETAIN path.
• Recovery integrity verdict: replay_wal/persist_recover distinguish a clean log end from a corruption-truncated end, and re-derive the member's durable floor from the recovered WAL.
• Promotion eligibility fence: a recovery-truncated member, or one whose durable position is below the cluster committed high-water mark, is ineligible to become primary until caught up; re-eligible on catch-up.
• Durable-backed freshness advertisement: a candidate's advertised log position and a voter's freshness floor derive from durable WAL state, not from volatile in-memory positions that survive a truncating restart.
• Durability verification seam: a configurable durability cadence / background-flush control plus an XCE_TESTING hook.

Out of Boundary

• Counting the majority or advancing committed_lsn arithmetic (ha-commit-engine — consumed; this spec changes only the durability precondition of its inputs).
• The reply ordering / classification sweep (ha-commit-barrier-rollout — reused unchanged).
• Demotion truncate + self-restart and rejoin reconciliation of an intact log (ha-convergence-recovery). This spec fences promotion; convergence-recovery handles demotion and rejoin.
• Vote-grant arithmetic and term handling (raft-library); this spec adds only the durability/high-water precondition to the existing freshness gate.
• WAL group-commit batching; job-array single-record persistence.

Allowed Dependencies

• ha-commit-engine: commit_engine_t, commit_submit, majority_candidate_locked (modify self-contribution input only).
• ha-wal-replication: standby ACK path (persist_wal_write_replicated) and catch-up streaming (consumed; in_catch_up reused).
• ha-commit-barrier-rollout: persist_commit_barrier reply gate (consumed unchanged).
• persist_t internals (current_lsn, committed_lsn, last_crc, segment I/O), ha_state_t (applied_lsn, last_known_committed_lsn, in_catch_up, term).
• Constraint: must not write/persist committed_lsn to disk (Raft keeps the commit index volatile); durability is about the log, not the commit index.

Revalidation Triggers

• Change to the durable_lsn semantics or to majority_candidate_locked's self-contribution → re-check ha-commit-engine and ha-convergence-recovery (which assume durable committed state).
• Change to the promotion-eligibility gate or freshness-floor source → re-check ha-single-leader-role, ha-prevote, raft-library, and the CLAUDE.md TLA+ checklist.
• Removal of fsync_worker → re-check any caller that assumed deferred fsync (audit wal_classify).
• New wal_sync override surface → re-check main.c flag conventions and deployment scripts.

Architecture

Existing Architecture Analysis

• Commit gate already off the iomgr thread: the AFTER_EVENT hook "writes WAL and waits for majority replication" before the reply (job.c:568-569); persist_commit_barrier → commit_submit(COMMIT_MODE_WAIT) blocks on a condvar. Adding fdatasync to this path is liveness-safe.
• Standby already persists-before-ACK: persist_wal_write_replicated fdatasyncs before returning (persist.c:515). The primary is the asymmetric gap.
• Recovery is silent-truncating: replay_wal breaks on CRC chain break / mismatch / short read (persist.c:1021/1050/1042) and returns the replayed count as success; persist_recover never sets committed_lsn (stays 0).
• Promotion has no integrity gate: raft_cb_become_leader computes safe_lsn from votes then persist_truncate_wal_above(safe_lsn) (ha.c:572/577) with no check that the local log is intact or ≥ safe_lsn.
• Freshness floor is volatile: ha_vote_self_freshness = max(applied_lsn, last_known_committed_lsn) (ha.c:2127), in-memory atomics not re-derived from durable WAL.
• Test-seam convention: #ifdef XCE_TESTING hook shims (ha.c:2270-2335); wal_sync_ms is a hard constant (config.h:86) with no override.

Architecture Pattern & Boundary Map

graph TB
    subgraph RequestPath
        Hook[AFTER_EVENT hook off iomgr]
        Barrier[persist_commit_barrier]
    end
    subgraph Persistence
        Append[persist_wal_async sync fsync]
        DurLsn[durable_lsn]
        Recover[persist_recover integrity verdict]
        ReplApply[persist_wal_write_replicated standby fsync]
    end
    subgraph Commit
        Majority[majority_candidate_locked durable self]
        Committed[committed_lsn]
    end
    subgraph Election
        Fresh[freshness floor from durable WAL]
        Promote[raft_cb_become_leader promotion fence]
    end
    Hook --> Append
    Append --> DurLsn
    Append --> ReplApply
    DurLsn --> Majority
    ReplApply --> Majority
    Majority --> Committed
    Committed --> Barrier
    Barrier --> Reply[reply ok to client]
    Recover --> Fresh
    Recover --> Promote
    Fresh --> Promote
    DurLsn --> Promote

Architecture Integration

• Selected pattern: extend-in-place for the durability precondition (Req 1, 3) plus a small testable fence helper for promotion eligibility (Req 2). No new module — durable_lsn is one atomic on persist_t.
• Boundaries: persistence owns durable_lsn and the recovery verdict; the commit engine consumes durable_lsn; election consumes the recovery verdict + durable-backed floor. No shared ownership of the commit index.
• Patterns preserved: commit-barrier reply gate, in_catch_up fence concept, XCE_TESTING hooks, the standby persist-before-ACK already in place.
• Steering compliance: never block the iomgr thread (fsync lands on the off-iomgr commit path); re-verify against raft/raft.tla (commit rule, leader completeness) per CLAUDE.md.

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Backend / Services	C (gnu17), controller daemon	Durable commit, recovery integrity, promotion fence	Extends persist.c, commit.c, ha.c, cluster.c
Data / Storage	Local WAL segments + fdatasync	Synchronous durability; integrity-verified recovery	posix_fallocate'd segments; fdatasync (data + pre-allocated metadata)
Infrastructure / Runtime	--wal-sync-ms (new) + XCE_TESTING seam	Deterministic durability verification	Follows --sync-timeout-ms (OPT_SYNC_TIMEOUT) precedent

File Structure Plan

Modified Files

• src/controller/persist.c — persist_wal_async: append then signal the flusher (no inline per-entry fsync); convert fsync_worker from a timer wait into a demand-driven group-commit flusher (wake on append-signal, fdatasync, advance durable_lsn to the highest synced LSN, broadcast the commit-eval condvar); keep its thread/mutex/cond lifecycle in persist_start/persist_stop/persist_fini. replay_wal/persist_recover: return an integrity verdict; re-derive durable floor from the recovered WAL tip. persist_wal_sync_append (dormant, no production caller): delete or re-route through the durable-majority evaluation — it must not advance the commit position on majority ACK without the durable precondition (task 2.4).
• src/include/persist.h — add _Atomic uint64_t durable_lsn and a recovery-verdict type to persist_t/the recover signature; keep fsync_* fields (now driving the group-commit flusher); add an override field to persist_config_t (or its setter).
• src/controller/commit.c — majority_candidate_locked: use durable_lsn (not highest_appended_lsn) for the self contribution cap.
• src/controller/ha.c — raft_cb_become_leader: insert the pre-promotion eligibility gate; ha_vote_self_freshness: derive floor from durable WAL state; fence-release on catch-up completion. New XCE_TESTING durability-verification hook in the seam region (ha.c:2270-2335).
• src/controller/cluster.c — vote / PreVote freshness comparison (cluster.c:380-396, 467-474) consume the durable-backed floor; promotion path honors the fence.
• src/include/config.h — keep DEFAULT_WAL_SYNC_MS; reconcile the stale 50 in persistence-and-ha-design.md.
• src/controller/main.c — parse --wal-sync-ms (and optionally --wal-sync-disable) into persist_config_t.
• src/controller/myapp.c / cluster.c (persist_config_defaults callers) — apply the override before persist_init.

New Files

• src/controller/testsuite/test_commit_durability.c — unit/white-box tests for durable_lsn advancement, self-counts-durable, recovery verdict, promotion fence (Req 1, 2, 3).
• src/tools/test-commit-durability.sh — exists (soak E2E); extend to drive the new --wal-sync-* seam for a deterministic mode.

Each modified function keeps one responsibility: persistence owns durability + recovery verdict; commit owns majority math (consuming durable_lsn); election owns eligibility (consuming the verdict + floor).

System Flows

Durable commit (reply ok ⇒ persisted on a majority)

sequenceDiagram
    participant Client
    participant Primary
    participant Standby
    participant Flusher
    Client->>Primary: submit op
    Primary->>Primary: persist_wal_async pwrite then signal flusher
    Primary->>Standby: replicate entry
    Flusher->>Flusher: batched fdatasync then durable_lsn advance then wake commit
    Standby->>Standby: persist_wal_write_replicated fdatasync
    Standby-->>Primary: WalAck acked_lsn
    Primary->>Primary: majority_candidate_locked self equals durable_lsn plus standby acks
    Primary->>Primary: committed_lsn advance then commit barrier releases
    Primary-->>Client: reply ok

Key decisions: the reply is released only after committed_lsn covers the entry, and committed_lsn now requires a durable self plus a durable standby majority. The fdatasync runs on the dedicated flusher thread (the repurposed fsync_worker), batched across all pending appends; the AFTER_EVENT caller blocks on the commit barrier (off iomgr) until durable_lsn + standby acks cover its entry, so heartbeats are unaffected (Req 1.6) and one fdatasync amortizes across the high-volume RETAIN-fact stream.

Recovery integrity & promotion fence

graph TD
    Start[startup persist_recover] --> Replay[replay_wal]
    Replay --> Verdict{integrity verdict}
    Verdict -->|clean eof| Floor[set durable floor from WAL tip]
    Verdict -->|corruption truncated| Truncated[mark recovery_truncated]
    Floor --> Eligible[eligible to lead]
    Truncated --> CanCatchUp{healthy peer reachable}
    CanCatchUp -->|yes| Fence[fence from promotion until caught up]
    CanCatchUp -->|no and needed for quorum| Fatal[FATAL fail loud]
    Fence --> Caught{caught up past last committed boundary}
    Caught -->|yes| Eligible

Key decisions: corruption is never silently accepted (Req 2.1). A truncated member is fenced from promotion and catches up from a healthy primary (Req 2.2, 2.6); if it cannot reach a consistent committed state and the cluster needs it, the controller fails loudly (Req 2.4), honoring the project's fail-loud preference. A candidate never advertises or leads beyond its durable WAL tip (Req 2.3), so a hollow node cannot win even under degraded quorum (Req 2.5).

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1	reply ok ⇒ durable on majority	DurableCommit	commit barrier, durable_lsn	Durable commit
1.2	self counts only when durable	DurableCommit	majority_candidate_locked	Durable commit
1.3	standby persists before ACK	DurableCommit (pin)	persist_wal_write_replicated	Durable commit
1.4	no premature ok	DurableCommit	persist_commit_barrier	Durable commit
1.5	survives minority crash+restart	DurableCommit	durable-majority property	Durable commit / soak
1.6	durability work preserves liveness	DurableCommit	off-iomgr placement	Durable commit
2.1	no silent truncate-accept	RecoveryIntegrity	replay_wal verdict	Recovery fence
2.2	fence until caught up	PromotionFence	promotion gate, in_catch_up	Recovery fence
2.3	never lead below high-water	PromotionFence + FreshnessFloor	raft_cb_become_leader, ha_vote_self_freshness	Recovery fence
2.4	fail-loud when unrecoverable	RecoveryIntegrity	FATAL policy	Recovery fence
2.5	degraded quorum preserves invariant	DurableCommit + PromotionFence	durable-majority + fence	Both
2.6	re-eligibility on catch-up	PromotionFence	fence-release hook	Recovery fence
3.1	configurable durability cadence	DurabilityConfig	--wal-sync-ms	—
3.2	recoverable after ungraceful kill in verify mode	DurabilityTestSeam	XCE_TESTING hook	soak deterministic mode
3.3	identical external semantics in/out of mode	DurabilityTestSeam	seam affects timing only	—

Components and Interfaces

Component	Layer	Intent	Req Coverage	Key Dependencies (P0/P1)	Contracts
DurableCommit	Persistence + Commit	Synchronous persist + self-counts-durable	1.1–1.6, 2.5	persist_t (P0), commit_engine (P0), repl ACK (P1)	State
RecoveryIntegrity	Persistence	Verdict on recovery; durable floor; FATAL policy	2.1, 2.4	replay_wal (P0)	State
PromotionFence	Election	Eligibility gate + fence-release	2.2, 2.3, 2.6, 2.5	RecoveryIntegrity (P0), in_catch_up (P1)	State
FreshnessFloor	Election	Durable-backed vote floor & advertisement	2.3	durable_lsn (P0), ha_state (P1)	State
DurabilityConfig + TestSeam	Runtime / Test	Override cadence; verification hook	3.1–3.3	persist_config (P0), XCE_TESTING (P1)	State

Persistence + Commit

DurableCommit

Field	Detail
Intent	Make the primary persist before it contributes to the commit majority
Requirements	1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 2.5

• persist_wal_async appends and signals the flusher; it does not fsync inline per entry.
• The group-commit flusher (repurposed fsync_worker) wakes on the append-signal, performs one fdatasync, advances durable_lsn to the highest synced LSN, and broadcasts the commit-eval condvar — amortizing one sync across all entries pending at wake time (the high-volume RETAIN stream benefits most).
• majority_candidate_locked caps the self contribution at durable_lsn, not highest_appended_lsn.
• Standby persist-before-ACK is preserved and regression-pinned (persist_wal_write_replicated:515); the standby path may later adopt the same flusher (out of scope here).
• Checkpoint markers retain their existing inline fdatasync (persist.c:573).
• durable_lsn is monotonic, never exceeds the live segment tip; on persist_truncate_wal_above, clamp durable_lsn to the new tip.

Dependencies

• Inbound: AFTER_EVENT hooks (job.c, queue.c, worker.c, join.c), ha.c WAL_NOOP — purpose: append path (P0).
• Outbound: commit_engine_t — purpose: self contribution input (P0).

Contracts: State [x]

State Management

• State model: _Atomic uint64_t durable_lsn on persist_t; invariant durable_lsn <= current_lsn - 1, monotonic non-decreasing except clamp-on-truncate.
• Persistence & consistency: durable_lsn advances only after a successful fdatasync in the flusher; the commit engine reads it for self-coverage so committed_lsn ⇒ durable-majority.
• Concurrency strategy: the flusher takes p->lock to read the appended tip and fdatasync, advances durable_lsn, releases, then signals commit-eval; majority_candidate_locked reads durable_lsn via atomic load. Define the append-signal → flush → commit-wake ordering precisely so a WAIT caller cannot miss a wake (lost-wakeup guard).

Implementation Notes

• Integration: the WAIT-mode caller blocks on the commit barrier off the iomgr thread; only the flusher thread blocks on fdatasync. Heartbeats unaffected.
• Validation: assert committed_lsn <= durable_lsn on the local member after every commit advance (debug build); verify the flusher batches (one fsync advances durable_lsn past multiple LSNs under load).
• Risks: lost-wakeup / batching-window correctness between persist_wal_async signal and the flusher (an entry must never be considered durable before its fdatasync); design the signal as level-triggered on the appended-vs-synced gap, not edge-triggered. Bounded flush latency vs throughput is a tunable (--wal-sync-ms becomes the max batching delay, not a durability gap).

Persistence — RecoveryIntegrity

Field	Detail
Intent	Distinguish clean log end from corruption; re-derive durable floor; fail loud when unrecoverable
Requirements	2.1, 2.4

Responsibilities & Constraints

• replay_wal/persist_recover return a verdict: RECOVER_CLEAN (reached a clean entry boundary) vs RECOVER_TRUNCATED (stopped at CRC break / mismatch / short read).
• On RECOVER_TRUNCATED, the member is marked recovery-truncated and is NOT treated as a clean recovery; it must not advertise or lead beyond the last intact LSN.
• The recovered durable floor (highest intact replayed LSN) is published so the freshness advertisement is durable-backed.
• FATAL policy: if a recovery-truncated member cannot reach a healthy peer to catch up and the cluster requires it, the controller terminates loudly rather than serve divergent state.

Contracts: State [x]

State Management

• State model: verdict enum + recovery_truncated flag (on ha_state_t or persist_t); recovered durable floor = highest intact LSN.
• Persistence & consistency: does not persist committed_lsn; only re-derives the durable log tip from on-disk segments.
• Concurrency strategy: runs single-threaded during cluster_init_phase2 before the node participates.

Implementation Notes

• Integration: composes with ha-convergence-recovery — that spec truncates an intact log on demotion/rejoin; this spec refuses to accept a corrupt log silently.
• Validation: unit test injects a CRC break at LSN 1 and asserts verdict RECOVER_TRUNCATED + non-promotability.
• Risks: must not FATAL on a legitimately empty first-boot WAL (no entries ≠ corruption).

Boot Policy Decision Table (task 3.2)

Status: DECISION recorded (task 3.2). No production behavior change is made by 3.2 — this table is the contract that task 3.3 (the FATAL branch) and task 4.3 (the promotion fence + release) implement. The skeleton tests in test_commit_durability.c (one per row, TEST_IGNORE-skipped pending 3.3/4.3) encode each row's expected outcome.

Decision (Req 2.2, 2.4): A member that boots with a structurally clean log but below the cluster's committed high-water mark, or with a recovery-truncated log, is fenced from leadership at boot (never promoted while behind/truncated). It then attempts catch-up from a healthy peer. It becomes eligible only once catch-up brings its durable position to a committed-consistent state at/above the cluster boundary. It terminates loudly (FATAL) only on a bounded failure to reach that state — i.e. it cannot recover locally and cannot catch up within the bounded window (no healthy peer reachable / cannot reach a committed-consistent state).

Invariant (authoritative): the fence is never a silent indefinite wait. A truncated/behind member always resolves to exactly one of two terminal outcomes: it catches up → ELIGIBLE, or it FATALs (fail-loud) on a bounded failure. There is no third "fenced forever, still running, silently divergent" state. Snapshot-sentinel recovery is treated as CLEAN-but-behind (catch-up), never FATAL on a legitimate sentinel.

Inputs (evaluated at boot / promotion-eligibility):

• verdict — recovery integrity verdict from task 3.1: CLEAN vs TRUNCATED.
• durable-vs-HWM — local durable position relative to the cluster committed high-water mark: at/above vs below.
• peer — a healthy peer reachable for catch-up: yes vs no.
• catch-up — catch-up reaches a committed-consistent state within the bounded window: succeeds vs fails (n/a when no catch-up is attempted).

Outcomes:

• ELIGIBLE — may promote (no fence).
• FENCE+CATCH-UP → ELIGIBLE — ineligible to promote; catch up from a healthy peer; regain eligibility on success (Req 2.6).
• FATAL — fail loud; terminate with the truncation/boundary logged rather than serve divergent state (Req 2.4).

#	verdict	durable-vs-HWM	healthy peer	catch-up result	Outcome	Rationale / Req
1	CLEAN	at/above	n/a	n/a	ELIGIBLE	Log intact and not behind — normal promotable member. (Req 2.3)
2	CLEAN	below	yes	succeeds	FENCE+CATCH-UP → ELIGIBLE	Intact but behind the HWM; must not lead below the boundary; catch up then promote. (Req 2.2, 2.3, 2.6)
3	TRUNCATED	(below — a truncated log is by definition not provably at/above)	yes	succeeds	FENCE+CATCH-UP → ELIGIBLE	Structurally truncated but a healthy peer exists; fence, catch up, regain eligibility. Never FATAL when a peer is reachable. (Req 2.2, 2.6)
4	TRUNCATED	below	no	fails (bounded window elapses / no committed-consistent state reachable)	FATAL	Cannot recover locally AND cannot catch up — bounded failure ⇒ fail loud, never silent indefinite fence. (Req 2.4)
4b	CLEAN	below	no	fails (bounded window elapses, no peer to advance from)	FATAL	Behind the HWM and no healthy peer to catch up from within the bounded window — same bounded-failure terminal as row 4; never a silent indefinite fence. (Req 2.4, 2.2)
5	CLEAN (snapshot-sentinel: committed at sentinel LSN, no sub-sentinel entries)	below (no entries below the sentinel)	yes	succeeds	FENCE+CATCH-UP → ELIGIBLE	A legitimate sentinel snapshot is CLEAN-but-behind, NOT corruption; treat as must-catch-up, never FATAL on a legitimate sentinel. (Req 2.1, 2.2)
6	CLEAN	at/above (empty first-boot WAL, fresh cold start: durable=0, HWM=0)	n/a	n/a	ELIGIBLE	An empty first-boot WAL is not behind anything — cold-start members are promotable. (Req 2.1, 2.3)

Notes on the dimensions:

• A TRUNCATED verdict is never simultaneously "provably at/above the HWM": truncation means the log stops below its last intact boundary, so a truncated member is treated as behind for fence purposes regardless of the volatile in-memory position (this is why 4.2/3.4 derive the freshness floor from the durable WAL tip, not a volatile value).
• "bounded window" is the catch-up deadline pinned by task 3.3; the policy requires it to be finite so the fence cannot become a silent indefinite wait.
• The healthy peer = no + at/above combination does not need catch-up and is covered by row 1 (ELIGIBLE) — a member already at/above the HWM is promotable without a peer.

Election — PromotionFence + FreshnessFloor

Field	Detail
Intent	Bar a truncated/behind member from leadership until caught up; advertise/vote on durable WAL state
Requirements	2.2, 2.3, 2.5, 2.6

Responsibilities & Constraints

• Pre-promotion gate in raft_cb_become_leader: if recovery-truncated, or local durable position < cluster committed high-water mark, refuse promotion (set/extend in_catch_up-style fence) and instead catch up from a healthy primary.
• ha_vote_self_freshness: floor derives from the durable WAL tip (re-derived on recovery), so a truncated member cannot over-advertise a volatile last_known_committed_lsn. Honors CLAUDE.md: the post-election safe-commit point still comes from voter replies, never from a stale ctrlhello.
• Fence-release: when catch-up brings the member past its last-known committed boundary, restore eligibility (hook the existing catch-up completion / re-inclusion path).
• Degraded quorum: because a candidate cannot lead beyond its durable tip, a hollow node cannot win even at minimal quorum — preserving Req 1's invariant across leadership changes.

Contracts: State [x]

Implementation Notes

• Integration: reuses in_catch_up (ha.c:954, cluster.c:529) and the PreVote/vote freshness comparison (cluster.c:380-396, 467-474); extends the fence from "streaming standby" to "recovery-truncated / behind-high-water promoter."
• Validation: simulate a node restarting truncated; assert it is denied promotion until catch-up, then re-enabled (Req 2.2/2.6).
• Risks: interaction with ha-single-leader-role own-term gate and ha-prevote — re-verify the combined eligibility predicate against raft.tla.

Runtime / Test — DurabilityConfig + TestSeam

Field	Detail
Intent	Deterministic, configurable durability timing for verification
Requirements	3.1, 3.2, 3.3

Responsibilities & Constraints

• --wal-sync-ms plumbed main.c → persist_config_t before persist_init; default unchanged (DEFAULT_WAL_SYNC_MS). Under group-commit this bounds the maximum batching delay (a flush fires on the append-signal or when this elapses), never a durability gap — an acked op is always durable regardless of the value.
• The seam's verification role: an XCE_TESTING hook (and/or a large --wal-sync-ms to widen the batching window) lets a test assert an acknowledged op is recoverable after an immediate ungraceful termination — proving durability comes from the commit gate, not the flush cadence.
• External acknowledgment semantics are identical with or without the mode (timing controllability only).

Contracts: State [x]

Implementation Notes

• Integration: follows --sync-timeout-ms (OPT_SYNC_TIMEOUT) and the ha.c:2270-2335 hook convention.
• Validation: deterministic E2E mode in test-commit-durability.sh drives the seam so the soak becomes a deterministic RED on the unfixed build and GREEN after.
• Risks: ensure the seam is compiled out of production (#ifdef XCE_TESTING).

Error Handling

Error Strategy

• Durability failure (fdatasync error in the flusher): durable_lsn does not advance, so the entry is NOT counted durable, the commit does not advance, and the client does not receive ok (Req 1.4). The flusher surfaces the I/O error loudly (fail-loud); a persistent local sync failure is treated as a member fault, not silently retried into a false-durable state.
• Recovery corruption (RECOVER_TRUNCATED): never silently accepted (Req 2.1). Fence from promotion + catch up; FATAL when catch-up is impossible and the node is required (Req 2.4) — fail-loud.
• Promotion while behind: refused at the gate (Req 2.3); logged with the local durable tip vs the cluster high-water mark.

Monitoring

• Log every durable_lsn regression-clamp on truncate, every recovery verdict (with the truncation boundary), every promotion refusal (durable tip vs high-water), and every fence set/release — operator-visible, once per transition (consistent with the quorum-transition logging already in the HA stack).

Testing Strategy

Unit Tests

• durable_lsn advances only after fdatasync; majority_candidate_locked self-cap equals durable_lsn not highest_appended_lsn (1.2).
• replay_wal returns RECOVER_TRUNCATED on an injected CRC break at LSN 1 and RECOVER_CLEAN on an intact log (2.1).
• Promotion gate denies a recovery-truncated / behind-high-water member and re-enables it after catch-up (2.2, 2.6).
• ha_vote_self_freshness floor reflects the durable WAL tip, not a stale in-memory last_known_committed_lsn (2.3).
• Standby persist_wal_write_replicated fdatasyncs before ACK — regression pin (1.3).

Integration Tests

• Commit advances committed_lsn only when self is durable + a standby ACK is durable; reply released only after (1.1, 1.4).
• Kill the sole durable contributor mid-commit; acknowledged op survives on the majority (1.5).
• Durability work does not delay heartbeat handling beyond tolerance (1.6).

E2E / Failover

• src/tools/test-commit-durability.sh soak: rolling SIGKILL+restart under load asserts every acknowledged job survives (1.1, 1.5, 2.x). With the --wal-sync-* seam, run a deterministic variant: widen/disable background timing, kill a durable contributor in-window, assert survival — RED on the current build, GREEN after the fix (3.1, 3.2).

Performance

• Measure submit latency delta from synchronous fdatasync vs the existing replication round-trip; confirm it is within the round-trip already paid before deciding group-commit is needed (research item).

Performance & Scalability

• Group-commit amortizes one fdatasync across all entries pending at flush time, so the per-entry sync cost falls as load rises — critical for the high-volume WAL_MODE_RETAIN fact stream (job start/finish/fail/rusage) which is not client-latency-bound. --wal-sync-ms bounds the maximum batching delay (a latency/throughput tunable), never a durability gap. The job-array-single-wal-record roadmap item independently cuts per-array entry count. Target: zero acknowledged-job loss across failover (correctness) takes precedence over per-submit latency; group-commit keeps the throughput cost sub-linear in write rate.

Open Questions / Gating Items (resolve in /kiro-spec-tasks)

• Freshness-floor formula (gating, election-safety): pin the exact reconciliation of the durable-WAL-derived floor vs the existing max(applied_lsn, last_known_committed_lsn). Too low ⇒ a truncated node still wins (the bug); too high ⇒ election livelock. Make a raft/raft.tla Leader-Completeness and liveness re-check a blocking sub-task before the freshness change merges (Req 2.3).
• FATAL-vs-fence decision point (gating): a booting truncated member cannot know it is "required for quorum" at cluster_init. Define the concrete policy: fence at boot, attempt catch-up from a healthy primary, and FATAL only on a bounded failure to reach a committed-consistent state — never a silent indefinite fence (Req 2.2, 2.4). RESOLVED (task 3.2): see the Boot Policy Decision Table (task 3.2) above for the recorded input-state → outcome table; tasks 3.3 (FATAL branch) and 4.3 (fence + release) implement it.
• Flusher lost-wakeup proof: the append-signal → flush → commit-wake path must be shown free of lost wakeups (level-triggered on the appended-vs-synced gap) before relying on it for the durability invariant (Req 1.1, 1.2).
• Snapshot-recovery edge: current snapshots are 16-byte LSN sentinels (no entity data), so a member that recovers "from snapshot" has a committed position at the sentinel LSN but no log entries below it. The recovery verdict (3.1) must treat snapshot-sentinel + clean replay as CLEAN (not TRUNCATED), and the promotion fence (4.3) must treat such a member as behind-and-must-catch-up (never FATAL on a legitimate sentinel snapshot). Interacts with the roadmap entity-bearing snapshots item; pin the classification before implementing 3.1/4.3 (Req 2.1, 2.2).
• Dormant synchronous-append path: persist_wal_sync_append advances the commit position directly on majority ACK and assumes inline fsync — a premise group-commit removes (task 2.4). It has no production caller today but must be deleted or re-routed so it cannot reintroduce ACK-without-durability commit (Req 1.1, 1.2).