myxce/.kiro/steering/persistence.md

# WAL & Persistence Durability

How the controller turns appended state into durable, recoverable, majority-backed state. This is the persistence-layer companion to `ha-async-io.md` (HA protocol / replication) and the `src/controller/CLAUDE.md` Protocol Verification Checklist (the authoritative safety gate). Owning code: `src/controller/persist.c`, `src/include/persist.h`, `src/controller/commit.c`. Landed by spec `ha-commit-durability` (2026-06-18, closes bug-661).

## Core Invariant: Acknowledged ⇒ Durable on a Majority

When the controller replies *ok* to a client, the operation's WAL entry is `fdatasync`'d on a fault-tolerant majority (the acking member + ⌊N/2⌋ others). The reply path is unchanged — it is gated on the commit barrier; the durability guarantee comes from strengthening what "committed" *means*, not from a new gate.

The load-bearing rule: **a member contributes to the commit majority for an LSN only once that LSN is durable locally.** The primary counts its own copy at `durable_lsn`, never at the appended tip.

## `durable_lsn` — The Prefix Watermark

`persist_t.durable_lsn` (`_Atomic uint64_t`) is the highest LSN whose bytes are `fdatasync`'d locally. It is a **prefix watermark**, never a per-entry set: `durable_lsn ≥ L` implies every entry `≤ L` is durable (because the WAL is a contiguous single-writer log). Invariants:

- `durable_lsn <= current_lsn - 1` (you cannot be durable past what you have appended).
- Monotonic non-decreasing **except** clamp-on-truncate (`persist_truncate_wal_above` clamps `durable_lsn` down to the new tip). **Because the clamp lowers `durable_lsn`, a truncate cutoff that lags the true commit point physically destroys committed entries** — only ever call `persist_truncate_wal_above` with the votes-derived post-election `safe_committed_lsn` or the library `(term,lsn)` conflict point, NEVER with the lagging `last_known_committed_lsn` (bug-828).
- Advances **only after a successful `fdatasync`** — fail-loud: on `fdatasync` error the watermark holds and an ERROR is logged. Never advance a watermark on a failed sync.

Read it via `persist_durable_lsn(p)`. The commit engine's self-cap (`majority_candidate_locked` in `commit.c`) uses exactly this value.

## Group-Commit Flusher (the `fsync_worker` pattern)

`fsync_worker` is **demand-driven, not a timer.** On append, `persist_wal_async` pwrites under `p->lock`, releases, then signals the flusher. The flusher does **one** `fdatasync` covering all pending appends, advances `durable_lsn`, then wakes commit evaluation. This amortizes one sync across the high-volume `WAL_MODE_RETAIN` fact stream (job start/finish/fail/rusage).

Two correctness rules when touching this path:

- **Lost-wakeup guard is level-triggered** on the appended-vs-synced gap (`durable_lsn < current_lsn - 1`), re-checked under `p->lock` before waiting and after each flush — never edge-triggered. An entry must never be considered durable before its `fdatasync`.
- **A sustained `fdatasync` failure must still back off (bug-1563).** The level-triggered gap above can't distinguish "fresh work just landed" from "still stuck from the last failed attempt" — both read as pending. `fsync_worker` skips its wait ONLY when the just-completed flush made genuine progress (succeeded) or there was nothing pending to begin with; a flush that WAS pending but made no progress (failed) still waits out the normal `wal_sync_ms` cadence before retrying. Without this, a sustained failure (real disk error, or this sandbox's overlayfs quirk) spins `flusher_flush_once`/`fdatasync` as fast as the CPU/lock allow — a live 40-round chaotic soak hit 300k+ failed attempts on one node in under a minute (one core pinned, log flooded) before this fix.
- **Commit wake-up via callback, not a back-reference.** Both the flusher AND `persist_flush` (below) call `on_durable_advance(ctx)` (registered with `persist_set_durable_cb`, ctx = the `commit_engine_t`) **only when `durable_lsn` advanced, and after releasing `p->lock`**. Lock order is strictly `p->lock → ce->lock` (cycle-free; persistence never includes commit). On shutdown, **join the flusher before** `commit_engine_fini` or the callback can fire into freed state. **`persist_flush` firing this callback is itself a fix (bug-1564)** — it used to advance `durable_lsn` silently, so a WAIT-blocked commit caller was only ever woken by the async flusher's own next successful cycle, never by a synchronous flush. The gap was latent until bug-1563's backoff fix above removed an accidental cover: pre-1563, the flusher's un-throttled retry loop happened to notice any state change and re-fire the callback itself within microseconds, masking that `persist_flush`'s own advance was unobserved.

`--wal-sync-ms` (default `DEFAULT_WAL_SYNC_MS = 10`) is the **maximum batching delay**, a latency/throughput tunable — it is *never* a durability gap. An acked op is durable regardless of its value (a large value just widens the batch window; the commit barrier still waits for `durable_lsn`).

## Standby Persist-Before-ACK

A standby `fdatasync`s a replicated entry before it sends its WalAck. `persist_wal_write_replicated` rc-gates the inline sync and returns `-EIO` on failure; the caller only ACKs on `rc == 0`. A standby must never ACK a non-durable entry — that is half of the majority guarantee.

## Recovery Integrity Verdict

`replay_wal` / `persist_recover` distinguish a clean log end from corruption and surface a `recover_verdict_t`:

- `RECOVER_CLEAN` — valid CRC chain to a clean entry boundary, OR an empty first-boot WAL, OR a pre-allocated-zero tail, OR a snapshot sentinel + clean replay.
- `RECOVER_TRUNCATED` — replay stopped on structural corruption (CRC chain break, CRC mismatch, short header/payload read).

The verdict is **monotonic** (init CLEAN; only ever set TRUNCATED; a later clean segment cannot un-set it) and is **structural integrity only** — "behind the cluster" is a separate concern handled by the freshness/promotion fence. Recovery never silently discards committed entries. Consume via `persist_recovery_verdict(p)` / `persist_recovery_truncated(p)` (these are NOT `XCE_TESTING`-gated — the promotion gate reads them in production).

## Durable Floor & Durable-Backed Freshness

On recovery, the member publishes its **durable floor** = highest intact replayed LSN (the WAL tip) via `persist_durable_floor(p)`. It is **assigned, not max'd**, so a truncating restart lands at the lower intact tip and never over-advertises. This floor is a durable *last-log index*, **not** the commit position.

Election freshness is then durable-backed (formula lives in `ha.c`/`cluster.c`, consumed from here):

- A **candidate advertises bare `persist_durable_floor`** for the LSN axis — it may only claim a position it can durably back.
- A **voter floor = `max(durable_floor, last_known_committed_lsn)`** — durable tip with the committed-high-water backstop retained (bug-339).

Result: a truncated/behind node cannot win an election or pass the promotion gate; the most-durable reachable node always can (no livelock).

## What Persistence Does NOT Persist

- **Never write `committed_lsn` to disk.** Raft keeps the commit index volatile; durability is about the **log**, not the commit index. The post-election safe-commit point is re-derived from voter replies (`ha_compute_safe_committed_lsn`), never from a stale ctrlhello.
- **LSN 0 is a reserved sentinel** — never a real entry; the first append is LSN 1. The commit engine rejects `lsn == 0`.

## Apply Correctness: `APPLY_DIVERGENT` (materialized state vs. WAL, not durability)

A **separate** verdict from `recover_verdict_t` (above): `persist_apply_verdict(p)` (`APPLY_CLEAN`/`APPLY_DIVERGENT`) answers "does the live materialized state (job/queue tables) actually match what a clean replay of the durable WAL would produce" — durability says the *bytes* are safe; this says the *applied result* is trustworthy. Computed once, at boot, in `persist_recover()`: an entity-count check (WAL_JOB_SUBMIT entries scanned vs. live table growth) plus a position check against `p->last_applied_lsn` (the LOGICAL apply cursor — never the physical WAL-scan tip, which also advances on checkpoint-marker/unknown-type entries and so falsely triggers DIVERGENT on an ordinary crash-during-checkpoint, bug-1419).

**Unconditional, restart-only fence (deliberate — landed by `raft-apply-decoupling` task 5.1-5.3, closes bug-1418's promotion-gate liveness-deadlock family):** once `APPLY_DIVERGENT`, the promotion gate (`ha.c` `raft_cb_become_leader` and `promotion_fence_timeout_fn`) refuses `CTRL_PRIMARY` **unconditionally** — no peer/HWM/durable-floor condition may clear it live, since a divergent local materialization can't be trusted for a peer/catch-up comparison either. Only a fresh, clean `persist_recover()` replay clears it (`apply_verdict` has exactly two production write sites: `persist_init()`'s reset and `persist_recover()`'s boot-time computation — no live path resets it). **Do not "fix" an apply-divergent liveness stall by relaxing this gate** — the actual fix is eliminating the false-positive DIVERGENT verdicts at the source (a boot-time TOCTOU where a returning node's verify pass raced the live apply pump / live-replication apply path, bug-1416, closed via `persist_t.recovery_complete` + `persist_mark_recovery_pending` called before any network traffic is processed).

**`APPLY_CLEAN` is entity-count + cursor-position only — it does NOT prove per-field completeness (bug-1678).** `persist_apply_verdict()`'s two checks (WAL_JOB_SUBMIT-count-vs-table-growth, `last_applied_lsn` position) both passed while a job's `allocs` (the resource allocation the scheduler assigned) silently replayed as `NULL` after a full restart — `job.state` was correctly `RUN`, the row existed, the cursor was in the right place, and `APPLY_CLEAN` reported healthy the entire time. Root cause: `WAL_JOB_DISPATCH`'s payload (`wal_jobstate_t`) never carried the allocation at all, so no apply-side check could have caught the gap even in principle — the field was never in the log to compare against. **The real invariant `APPLY_CLEAN` should imply — but today only partially enforces — is: online-apply and replay-apply of the same committed log MUST produce byte-identical materialized state for every persisted field of every entity, not just matching row counts and an advancing cursor.** When adding a new WAL fact type or a new field to an existing one, verify explicitly that (a) the field is actually encoded into the WAL payload (not just the in-memory struct at dispatch time), and (b) both apply seams — `apply_committed_entry()` (primary boot replay) and `apply_wal_entry()` (standby catch-up / replicated-apply) — decode and adopt it. Tracked as unimplemented future work at the systemic level in `roadmap.md`'s raft-apply-decoupling item #4 / raft-sim-conformance follow-up (a generic per-field snapshot-diff regression harness); until that lands, treat this as a standing audit dimension (see `xce-audit-invariants` skill) rather than a one-time fix.

## Snapshot Save/Load Field-Completeness (Scheduler)

**The identical invariant applies to the scheduler's own, separate persistence mechanism** (`xce_sched_core_save_snapshot()`/`xce_sched_core_load_snapshot()`, `src/scheduler/src/snapshot{,_wire,_load}.c`) -- a completely different code path from this file's WAL-based durability (no shared code, no shared fixes), but the same failure shape: a snapshot-restored entity can report the *correct state* while silently missing a *field* that state depends on, with nothing in the load path able to detect it because the field was never written to the wire type in the first place. Confirmed instance: `job->allocs`/`job->accounts` (a running job's resource allocation) were never captured by `build_wire_job()` -- `snapshot_wire.c:167`'s own comment flags it -- so a snapshot-restored RUNNING job showed the right `state` but permanently zero resource usage, the scheduler-side mirror of this section's `job->allocs`/EXEC_HOST story above (bug-1678). Scoped as its own spec, `scheduler-snapshot-job-allocation`, rather than folded into WAL work, since the two persistence mechanisms don't share fixes. Unlike the WAL side, `snapshot.c`'s own "Scope decisions" header already documents SOME gaps explicitly (`job->priority`, queue `project_name`/`job_pick_plugin`) -- those are fine, the invariant is about UNDOCUMENTED gaps. Tracked as `xce-audit-invariants`' scheduler dimension 4 -- when adding a new field to a snapshot-round-tripped entity, verify explicitly that it's written in the matching `build_wire_*()` function AND read back in `xce_sched_core_load_snapshot()`, or add it to the documented-gaps list if omitting it is a deliberate choice.

## Fail-Loud, Never Self-Exit

When a member cannot reach a committed-consistent state (recovery-truncated/behind, no healthy peer), it enters a **loud permanent fence**: log a loud operator-visible ERROR (truncation boundary vs cluster HWM), set the sticky `recovery_unrecoverable` flag, hold non-promotable, **stay alive, and keep retrying** — it self-heals when a peer returns. It MUST NEVER `exit`/`abort` as a designed response. The decision is the pure `persist_recover_decision` → `RECOVER_DECISION_{ELIGIBLE,FENCE,FENCE_UNRECOVERABLE}`; the action is `persist_recover_loud_fence_if_unrecoverable` (no terminal). Authoritative statement: `error-handling.md` + `src/controller/CLAUDE.md` "Process Lifecycle".

## Testing Durability Deterministically

- **Force durability synchronously** with `persist_flush(p)` (does an `fdatasync`, advances `durable_lsn`, and fires `on_durable_advance` on genuine advance — bug-1564) when a test needs `durable_lsn ≥ L` before exercising a commit. The demand-driven flusher is async, so any test that synchronously expects "covering ACK ⇒ committed advances" must first make the entry durable (mirror this in in-memory fixtures by priming `durable_lsn` to the appended tip). A WAIT-blocked commit caller now wakes from either path — don't assume only the async flusher can release one.
- **`XCE_TESTING` seams** (e.g. `persist_test_set_force_fsync_fail`, `persist_test_durable_lsn_on_disk`) are compiled out of production — verify exclusion with `nm`.
- **Gate test runs on the ANSI-stripped `:FAIL:` count** from the test binary's stdout, not on meson's `Fail:0` summary (meson aggregates and hides Unity sub-group failures).
- Run final durability validation on a **real fsync-able filesystem** — some sandbox `/tmp` mounts mis-handle `fdatasync`.

## Quick Reference

| Concern | Symbol / Site |
|---------|---------------|
| Durable watermark | `persist_durable_lsn(p)` / `persist_t.durable_lsn` |
| Durable floor (WAL tip) | `persist_durable_floor(p)` |
| Commit self-cap | `majority_candidate_locked` (commit.c) caps self at `durable_lsn` |
| Group-commit wake (async + sync) | `persist_set_durable_cb` → `on_durable_advance(ctx)`, fired by both `fsync_worker` and `persist_flush` on genuine advance |
| Synchronous flush (tests) | `persist_flush(p)` — also fires `on_durable_advance` (bug-1564) |
| Recovery verdict | `persist_recovery_verdict(p)` / `persist_recovery_truncated(p)` |
| Loud fence (no exit) | `persist_recover_loud_fence_if_unrecoverable`, `recovery_unrecoverable` |
| Flush cadence (max batch delay) | `--wal-sync-ms`, `DEFAULT_WAL_SYNC_MS = 10` |