myxce/.kiro/steering/persistence.md

WAL & Persistence Durability

How the controller turns appended state into durable, recoverable, majority-backed state. This is the persistence-layer companion to ha-async-io.md (HA protocol / replication) and the src/controller/CLAUDE.md Protocol Verification Checklist (the authoritative safety gate). Owning code: src/controller/persist.c, src/include/persist.h, src/controller/commit.c. Landed by spec ha-commit-durability (2026-06-18, closes bug-661).

Core Invariant: Acknowledged ⇒ Durable on a Majority

When the controller replies ok to a client, the operation's WAL entry is fdatasync'd on a fault-tolerant majority (the acking member + ⌊N/2⌋ others). The reply path is unchanged — it is gated on the commit barrier; the durability guarantee comes from strengthening what "committed" means, not from a new gate.

The load-bearing rule: a member contributes to the commit majority for an LSN only once that LSN is durable locally. The primary counts its own copy at durable_lsn, never at the appended tip.

durable_lsn — The Prefix Watermark

persist_t.durable_lsn (_Atomic uint64_t) is the highest LSN whose bytes are fdatasync'd locally. It is a prefix watermark, never a per-entry set: durable_lsn ≥ L implies every entry ≤ L is durable (because the WAL is a contiguous single-writer log). Invariants:

• durable_lsn <= current_lsn - 1 (you cannot be durable past what you have appended).
• Monotonic non-decreasing except clamp-on-truncate (persist_truncate_wal_above clamps durable_lsn down to the new tip). Because the clamp lowers durable_lsn, a truncate cutoff that lags the true commit point physically destroys committed entries — only ever call persist_truncate_wal_above with the votes-derived post-election safe_committed_lsn or the library (term,lsn) conflict point, NEVER with the lagging last_known_committed_lsn (bug-828).
• Advances only after a successful fdatasync — fail-loud: on fdatasync error the watermark holds and an ERROR is logged. Never advance a watermark on a failed sync.

Read it via persist_durable_lsn(p). The commit engine's self-cap (majority_candidate_locked in commit.c) uses exactly this value.

Group-Commit Flusher (the fsync_worker pattern)

fsync_worker is demand-driven, not a timer. On append, persist_wal_async pwrites under p->lock, releases, then signals the flusher. The flusher does one fdatasync covering all pending appends, advances durable_lsn, then wakes commit evaluation. This amortizes one sync across the high-volume WAL_MODE_RETAIN fact stream (job start/finish/fail/rusage).

Two correctness rules when touching this path:

• Lost-wakeup guard is level-triggered on the appended-vs-synced gap (durable_lsn < current_lsn - 1), re-checked under p->lock before waiting and after each flush — never edge-triggered. An entry must never be considered durable before its fdatasync.
• Commit wake-up via callback, not a back-reference. The flusher calls on_durable_advance(ctx) (registered with persist_set_durable_cb, ctx = the commit_engine_t) only when durable_lsn advanced, and after releasing p->lock. Lock order is strictly p->lock → ce->lock (cycle-free; persistence never includes commit). On shutdown, join the flusher before commit_engine_fini or the callback can fire into freed state.

--wal-sync-ms (default DEFAULT_WAL_SYNC_MS = 10) is the maximum batching delay, a latency/throughput tunable — it is never a durability gap. An acked op is durable regardless of its value (a large value just widens the batch window; the commit barrier still waits for durable_lsn).

Standby Persist-Before-ACK

A standby fdatasyncs a replicated entry before it sends its WalAck. persist_wal_write_replicated rc-gates the inline sync and returns -EIO on failure; the caller only ACKs on rc == 0. A standby must never ACK a non-durable entry — that is half of the majority guarantee.

Recovery Integrity Verdict

replay_wal / persist_recover distinguish a clean log end from corruption and surface a recover_verdict_t:

• RECOVER_CLEAN — valid CRC chain to a clean entry boundary, OR an empty first-boot WAL, OR a pre-allocated-zero tail, OR a snapshot sentinel + clean replay.
• RECOVER_TRUNCATED — replay stopped on structural corruption (CRC chain break, CRC mismatch, short header/payload read).

The verdict is monotonic (init CLEAN; only ever set TRUNCATED; a later clean segment cannot un-set it) and is structural integrity only — "behind the cluster" is a separate concern handled by the freshness/promotion fence. Recovery never silently discards committed entries. Consume via persist_recovery_verdict(p) / persist_recovery_truncated(p) (these are NOT XCE_TESTING-gated — the promotion gate reads them in production).

Durable Floor & Durable-Backed Freshness

On recovery, the member publishes its durable floor = highest intact replayed LSN (the WAL tip) via persist_durable_floor(p). It is assigned, not max'd, so a truncating restart lands at the lower intact tip and never over-advertises. This floor is a durable last-log index, not the commit position.

Election freshness is then durable-backed (formula lives in ha.c/cluster.c, consumed from here):

• A candidate advertises bare persist_durable_floor for the LSN axis — it may only claim a position it can durably back.
• A voter floor = max(durable_floor, last_known_committed_lsn) — durable tip with the committed-high-water backstop retained (bug-339).

Result: a truncated/behind node cannot win an election or pass the promotion gate; the most-durable reachable node always can (no livelock).

What Persistence Does NOT Persist

• Never write committed_lsn to disk. Raft keeps the commit index volatile; durability is about the log, not the commit index. The post-election safe-commit point is re-derived from voter replies (ha_compute_safe_committed_lsn), never from a stale ctrlhello.
• LSN 0 is a reserved sentinel — never a real entry; the first append is LSN 1. The commit engine rejects lsn == 0.

Fail-Loud, Never Self-Exit

When a member cannot reach a committed-consistent state (recovery-truncated/behind, no healthy peer), it enters a loud permanent fence: log a loud operator-visible ERROR (truncation boundary vs cluster HWM), set the sticky recovery_unrecoverable flag, hold non-promotable, stay alive, and keep retrying — it self-heals when a peer returns. It MUST NEVER exit/abort as a designed response. The decision is the pure persist_recover_decision → RECOVER_DECISION_{ELIGIBLE,FENCE,FENCE_UNRECOVERABLE}; the action is persist_recover_loud_fence_if_unrecoverable (no terminal). Authoritative statement: error-handling.md + src/controller/CLAUDE.md "Process Lifecycle".

Testing Durability Deterministically

• Force durability synchronously with persist_flush(p) (does an fdatasync and advances durable_lsn) when a test needs durable_lsn ≥ L before exercising a commit. The demand-driven flusher is async, so any test that synchronously expects "covering ACK ⇒ committed advances" must first make the entry durable (mirror this in in-memory fixtures by priming durable_lsn to the appended tip).
• XCE_TESTING seams (e.g. persist_test_set_force_fsync_fail, persist_test_durable_lsn_on_disk) are compiled out of production — verify exclusion with nm.
• Gate test runs on the ANSI-stripped :FAIL: count from the test binary's stdout, not on meson's Fail:0 summary (meson aggregates and hides Unity sub-group failures).
• Run final durability validation on a real fsync-able filesystem — some sandbox /tmp mounts mis-handle fdatasync.

Quick Reference

Concern	Symbol / Site
Durable watermark	persist_durable_lsn(p) / persist_t.durable_lsn
Durable floor (WAL tip)	persist_durable_floor(p)
Commit self-cap	majority_candidate_locked (commit.c) caps self at durable_lsn
Group-commit wake	persist_set_durable_cb → on_durable_advance(ctx)
Synchronous flush (tests)	persist_flush(p)
Recovery verdict	persist_recovery_verdict(p) / persist_recovery_truncated(p)
Loud fence (no exit)	persist_recover_loud_fence_if_unrecoverable, recovery_unrecoverable
Flush cadence (max batch delay)	--wal-sync-ms, DEFAULT_WAL_SYNC_MS = 10