# Design Document — ha-commit-durability

## Overview

**Purpose:** Make acknowledgment imply durability in the XCE HA controller. Today the controller advances its commit position on a majority of replication **acknowledgments**, but the primary counts its own contribution from a page-cache write (`commit.c:majority_candidate_locked` uses `highest_appended_lsn`), while background `fdatasync` runs on a timer (`fsync_worker`). A "committed" operation can therefore be durable on as few as one member — bug-661, observed as permanent job loss across failover.

**Users:** Cluster operators and every client whose acknowledged job/queue/worker operations must survive controller failover.

**Impact:** Strengthens the meaning of "committed" from "majority-ACKed (volatile)" to "persisted to stable storage on a fault-tolerant majority." The client reply path is already gated on the commit barrier, so once `committed_lsn` implies a durable majority, the invariant holds with no change to the reply path. Durability becomes **group-commit**: `fsync_worker` is repurposed from a fire-and-forget timer into a demand-driven flusher that, on append, batches `fdatasync` across all pending entries, advances `durable_lsn`, and wakes the commit evaluation. The commit majority counts `durable_lsn` (not appended). This amortizes one `fdatasync` across many entries — important because the highest-volume WAL traffic is `WAL_MODE_RETAIN` facts (job start/finish/fail/rusage), which are not client-latency-bound but must be durable before `committed_lsn` passes them. A second concern — a member with a corrupt or truncated log becoming authoritative — is closed by a recovery-integrity verdict and a promotion fence.

### Goals
- **Durable commit:** when the controller replies *ok*, the operation's log entry is `fdatasync`'d on the acknowledging member plus ⌊N/2⌋ others (Req 1).
- **Recovery integrity & promotion fence:** a member that cannot reconstruct its committed state, or that lags the cluster's committed high-water mark, is barred from leadership until caught up — fail-loud when it cannot recover safely (Req 2).
- **Deterministic verification:** durability timing is controllable in a test mode so the guarantee is provable without racing the clock (Req 3).
- Preserve cluster liveness: durability work never stalls heartbeat/election processing.

### Non-Goals
- Single-record job-array persistence (`job-array-single-wal-record`, roadmap) — complements this spec by bounding per-array fsync count; independent of the group-commit flusher introduced here.
- Per-entry synchronous fsync as the steady-state mechanism — superseded by group-commit (decision: the high-volume `WAL_MODE_RETAIN` path needs batching; see Components/DurableCommit).
- Replication/ACK wire format (`ha-wal-replication`), per-entry majority arithmetic (`ha-commit-engine`), demotion truncate / rejoin reconciliation of an intact log (`ha-convergence-recovery`), promotion no-op / own-term gate (`ha-single-leader-role`), raw vote-grant arithmetic (`raft-library`).

## Boundary Commitments

### This Spec Owns
- The **durability precondition** on the commit majority: a member contributes to the commit majority for an LSN only once that LSN is `fdatasync`'d locally. Introduces and is authoritative for `persist_t.durable_lsn`.
- **Group-commit durability**: `fsync_worker` repurposed from a timer into a demand-driven flusher that batches `fdatasync`, advances `durable_lsn`, and wakes commit evaluation; `persist_wal_async` signals the flusher instead of deferring to a timer. Benefits all members on the high-volume `WAL_MODE_RETAIN` path.
- **Recovery integrity verdict**: `replay_wal`/`persist_recover` distinguish a clean log end from a corruption-truncated end, and re-derive the member's durable floor from the recovered WAL.
- **Promotion eligibility fence**: a recovery-truncated member, or one whose durable position is below the cluster committed high-water mark, is ineligible to become primary until caught up; re-eligible on catch-up.
- **Durable-backed freshness advertisement**: a candidate's advertised log position and a voter's freshness floor derive from durable WAL state, not from volatile in-memory positions that survive a truncating restart.
- **Durability verification seam**: a configurable durability cadence / background-flush control plus an `XCE_TESTING` hook.

### Out of Boundary
- Counting the majority or advancing `committed_lsn` arithmetic (`ha-commit-engine` — consumed; this spec changes only the *durability precondition* of its inputs).
- The reply ordering / classification sweep (`ha-commit-barrier-rollout` — reused unchanged).
- Demotion truncate + self-restart and rejoin reconciliation of an **intact** log (`ha-convergence-recovery`). This spec fences **promotion**; convergence-recovery handles **demotion** and **rejoin**.
- Vote-grant arithmetic and term handling (`raft-library`); this spec adds only the durability/high-water **precondition** to the existing freshness gate.
- WAL group-commit batching; job-array single-record persistence.

### Allowed Dependencies
- `ha-commit-engine`: `commit_engine_t`, `commit_submit`, `majority_candidate_locked` (modify self-contribution input only).
- `ha-wal-replication`: standby ACK path (`persist_wal_write_replicated`) and catch-up streaming (consumed; `in_catch_up` reused).
- `ha-commit-barrier-rollout`: `persist_commit_barrier` reply gate (consumed unchanged).
- `persist_t` internals (`current_lsn`, `committed_lsn`, `last_crc`, segment I/O), `ha_state_t` (`applied_lsn`, `last_known_committed_lsn`, `in_catch_up`, `term`).
- Constraint: must not write/persist `committed_lsn` to disk (Raft keeps the commit index volatile); durability is about the **log**, not the commit index.

### Revalidation Triggers
- Change to the `durable_lsn` semantics or to `majority_candidate_locked`'s self-contribution → re-check `ha-commit-engine` and `ha-convergence-recovery` (which assume durable committed state).
- Change to the promotion-eligibility gate or freshness-floor source → re-check `ha-single-leader-role`, `ha-prevote`, `raft-library`, and the `CLAUDE.md` TLA+ checklist.
- Removal of `fsync_worker` → re-check any caller that assumed deferred fsync (audit `wal_classify`).
- New `wal_sync` override surface → re-check `main.c` flag conventions and deployment scripts.

## Architecture

### Existing Architecture Analysis

- **Commit gate already off the iomgr thread:** the AFTER_EVENT hook "writes WAL and waits for majority replication" before the reply (`job.c:568-569`); `persist_commit_barrier → commit_submit(COMMIT_MODE_WAIT)` blocks on a condvar. Adding `fdatasync` to this path is liveness-safe.
- **Standby already persists-before-ACK:** `persist_wal_write_replicated` `fdatasync`s before returning (`persist.c:515`). The primary is the asymmetric gap.
- **Recovery is silent-truncating:** `replay_wal` `break`s on CRC chain break / mismatch / short read (`persist.c:1021/1050/1042`) and returns the replayed count as success; `persist_recover` never sets `committed_lsn` (stays 0).
- **Promotion has no integrity gate:** `raft_cb_become_leader` computes `safe_lsn` from votes then `persist_truncate_wal_above(safe_lsn)` (`ha.c:572/577`) with no check that the local log is intact or ≥ `safe_lsn`.
- **Freshness floor is volatile:** `ha_vote_self_freshness = max(applied_lsn, last_known_committed_lsn)` (`ha.c:2127`), in-memory atomics not re-derived from durable WAL.
- **Test-seam convention:** `#ifdef XCE_TESTING` hook shims (`ha.c:2270-2335`); `wal_sync_ms` is a hard constant (`config.h:86`) with no override.

### Architecture Pattern & Boundary Map

```mermaid
graph TB
    subgraph RequestPath
        Hook[AFTER_EVENT hook off iomgr]
        Barrier[persist_commit_barrier]
    end
    subgraph Persistence
        Append[persist_wal_async sync fsync]
        DurLsn[durable_lsn]
        Recover[persist_recover integrity verdict]
        ReplApply[persist_wal_write_replicated standby fsync]
    end
    subgraph Commit
        Majority[majority_candidate_locked durable self]
        Committed[committed_lsn]
    end
    subgraph Election
        Fresh[freshness floor from durable WAL]
        Promote[raft_cb_become_leader promotion fence]
    end
    Hook --> Append
    Append --> DurLsn
    Append --> ReplApply
    DurLsn --> Majority
    ReplApply --> Majority
    Majority --> Committed
    Committed --> Barrier
    Barrier --> Reply[reply ok to client]
    Recover --> Fresh
    Recover --> Promote
    Fresh --> Promote
    DurLsn --> Promote
```

**Architecture Integration**
- **Selected pattern:** extend-in-place for the durability precondition (Req 1, 3) plus a small testable fence helper for promotion eligibility (Req 2). No new module — `durable_lsn` is one atomic on `persist_t`.
- **Boundaries:** persistence owns `durable_lsn` and the recovery verdict; the commit engine consumes `durable_lsn`; election consumes the recovery verdict + durable-backed floor. No shared ownership of the commit index.
- **Patterns preserved:** commit-barrier reply gate, `in_catch_up` fence concept, `XCE_TESTING` hooks, the standby persist-before-ACK already in place.
- **Steering compliance:** never block the iomgr thread (fsync lands on the off-iomgr commit path); re-verify against `raft/raft.tla` (commit rule, leader completeness) per `CLAUDE.md`.

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Backend / Services | C (gnu17), controller daemon | Durable commit, recovery integrity, promotion fence | Extends `persist.c`, `commit.c`, `ha.c`, `cluster.c` |
| Data / Storage | Local WAL segments + `fdatasync` | Synchronous durability; integrity-verified recovery | `posix_fallocate`'d segments; `fdatasync` (data + pre-allocated metadata) |
| Infrastructure / Runtime | `--wal-sync-ms` (new) + `XCE_TESTING` seam | Deterministic durability verification | Follows `--sync-timeout-ms` (`OPT_SYNC_TIMEOUT`) precedent |

## File Structure Plan

### Modified Files
- `src/controller/persist.c` — `persist_wal_async`: append then **signal the flusher** (no inline per-entry fsync); convert `fsync_worker` from a timer wait into a **demand-driven group-commit flusher** (wake on append-signal, `fdatasync`, advance `durable_lsn` to the highest synced LSN, broadcast the commit-eval condvar); keep its thread/mutex/cond lifecycle in `persist_start`/`persist_stop`/`persist_fini`. `replay_wal`/`persist_recover`: return an integrity verdict; re-derive durable floor from the recovered WAL tip. `persist_wal_sync_append` (dormant, no production caller): delete or re-route through the durable-majority evaluation — it must not advance the commit position on majority ACK without the durable precondition (task 2.4).
- `src/include/persist.h` — add `_Atomic uint64_t durable_lsn` and a recovery-verdict type to `persist_t`/the recover signature; keep `fsync_*` fields (now driving the group-commit flusher); add an override field to `persist_config_t` (or its setter).
- `src/controller/commit.c` — `majority_candidate_locked`: use `durable_lsn` (not `highest_appended_lsn`) for the self contribution `cap`.
- `src/controller/ha.c` — `raft_cb_become_leader`: insert the pre-promotion eligibility gate; `ha_vote_self_freshness`: derive floor from durable WAL state; fence-release on catch-up completion. New `XCE_TESTING` durability-verification hook in the seam region (`ha.c:2270-2335`).
- `src/controller/cluster.c` — vote / PreVote freshness comparison (`cluster.c:380-396`, `467-474`) consume the durable-backed floor; promotion path honors the fence.
- `src/include/config.h` — keep `DEFAULT_WAL_SYNC_MS`; reconcile the stale `50` in `persistence-and-ha-design.md`.
- `src/controller/main.c` — parse `--wal-sync-ms` (and optionally `--wal-sync-disable`) into `persist_config_t`.
- `src/controller/myapp.c` / `cluster.c` (`persist_config_defaults` callers) — apply the override before `persist_init`.

### New Files
- `src/controller/testsuite/test_commit_durability.c` — unit/white-box tests for `durable_lsn` advancement, self-counts-durable, recovery verdict, promotion fence (Req 1, 2, 3).
- `src/tools/test-commit-durability.sh` — **exists** (soak E2E); extend to drive the new `--wal-sync-*` seam for a deterministic mode.

> Each modified function keeps one responsibility: persistence owns durability + recovery verdict; commit owns majority math (consuming `durable_lsn`); election owns eligibility (consuming the verdict + floor).

## System Flows

### Durable commit (reply ok ⇒ persisted on a majority)

```mermaid
sequenceDiagram
    participant Client
    participant Primary
    participant Standby
    participant Flusher
    Client->>Primary: submit op
    Primary->>Primary: persist_wal_async pwrite then signal flusher
    Primary->>Standby: replicate entry
    Flusher->>Flusher: batched fdatasync then durable_lsn advance then wake commit
    Standby->>Standby: persist_wal_write_replicated fdatasync
    Standby-->>Primary: WalAck acked_lsn
    Primary->>Primary: majority_candidate_locked self equals durable_lsn plus standby acks
    Primary->>Primary: committed_lsn advance then commit barrier releases
    Primary-->>Client: reply ok
```

Key decisions: the reply is released only after `committed_lsn` covers the entry, and `committed_lsn` now requires a **durable** self plus a **durable** standby majority. The `fdatasync` runs on the dedicated flusher thread (the repurposed `fsync_worker`), batched across all pending appends; the AFTER_EVENT caller blocks on the commit barrier (off iomgr) until `durable_lsn` + standby acks cover its entry, so heartbeats are unaffected (Req 1.6) and one `fdatasync` amortizes across the high-volume RETAIN-fact stream.

### Recovery integrity & promotion fence

```mermaid
graph TD
    Start[startup persist_recover] --> Replay[replay_wal]
    Replay --> Verdict{integrity verdict}
    Verdict -->|clean eof| Floor[set durable floor from WAL tip]
    Verdict -->|corruption truncated| Truncated[mark recovery_truncated]
    Floor --> Eligible[eligible to lead]
    Truncated --> CanCatchUp{healthy peer reachable}
    CanCatchUp -->|yes| Fence[fence from promotion until caught up]
    CanCatchUp -->|no and needed for quorum| Fatal[FATAL fail loud]
    Fence --> Caught{caught up past last committed boundary}
    Caught -->|yes| Eligible
```

Key decisions: corruption is never silently accepted (Req 2.1). A truncated member is fenced from promotion and catches up from a healthy primary (Req 2.2, 2.6); if it cannot reach a consistent committed state and the cluster needs it, the controller fails loudly (Req 2.4), honoring the project's fail-loud preference. A candidate never advertises or leads beyond its durable WAL tip (Req 2.3), so a hollow node cannot win even under degraded quorum (Req 2.5).

## Requirements Traceability

| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|------------|------------|-------|
| 1.1 | reply ok ⇒ durable on majority | DurableCommit | `commit barrier`, `durable_lsn` | Durable commit |
| 1.2 | self counts only when durable | DurableCommit | `majority_candidate_locked` | Durable commit |
| 1.3 | standby persists before ACK | DurableCommit (pin) | `persist_wal_write_replicated` | Durable commit |
| 1.4 | no premature ok | DurableCommit | `persist_commit_barrier` | Durable commit |
| 1.5 | survives minority crash+restart | DurableCommit | durable-majority property | Durable commit / soak |
| 1.6 | durability work preserves liveness | DurableCommit | off-iomgr placement | Durable commit |
| 2.1 | no silent truncate-accept | RecoveryIntegrity | `replay_wal` verdict | Recovery fence |
| 2.2 | fence until caught up | PromotionFence | promotion gate, `in_catch_up` | Recovery fence |
| 2.3 | never lead below high-water | PromotionFence + FreshnessFloor | `raft_cb_become_leader`, `ha_vote_self_freshness` | Recovery fence |
| 2.4 | fail-loud when unrecoverable | RecoveryIntegrity | FATAL policy | Recovery fence |
| 2.5 | degraded quorum preserves invariant | DurableCommit + PromotionFence | durable-majority + fence | Both |
| 2.6 | re-eligibility on catch-up | PromotionFence | fence-release hook | Recovery fence |
| 3.1 | configurable durability cadence | DurabilityConfig | `--wal-sync-ms` | — |
| 3.2 | recoverable after ungraceful kill in verify mode | DurabilityTestSeam | `XCE_TESTING` hook | soak deterministic mode |
| 3.3 | identical external semantics in/out of mode | DurabilityTestSeam | seam affects timing only | — |

## Components and Interfaces

| Component | Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|-------|--------|--------------|--------------------------|-----------|
| DurableCommit | Persistence + Commit | Synchronous persist + self-counts-durable | 1.1–1.6, 2.5 | persist_t (P0), commit_engine (P0), repl ACK (P1) | State |
| RecoveryIntegrity | Persistence | Verdict on recovery; durable floor; FATAL policy | 2.1, 2.4 | replay_wal (P0) | State |
| PromotionFence | Election | Eligibility gate + fence-release | 2.2, 2.3, 2.6, 2.5 | RecoveryIntegrity (P0), in_catch_up (P1) | State |
| FreshnessFloor | Election | Durable-backed vote floor & advertisement | 2.3 | durable_lsn (P0), ha_state (P1) | State |
| DurabilityConfig + TestSeam | Runtime / Test | Override cadence; verification hook | 3.1–3.3 | persist_config (P0), XCE_TESTING (P1) | State |

### Persistence + Commit

#### DurableCommit

| Field | Detail |
|-------|--------|
| Intent | Make the primary persist before it contributes to the commit majority |
| Requirements | 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 2.5 |

- `persist_wal_async` appends and signals the flusher; it does **not** fsync inline per entry.
- The **group-commit flusher** (repurposed `fsync_worker`) wakes on the append-signal, performs one `fdatasync`, advances `durable_lsn` to the highest synced LSN, and broadcasts the commit-eval condvar — amortizing one sync across all entries pending at wake time (the high-volume RETAIN stream benefits most).
- `majority_candidate_locked` caps the self contribution at `durable_lsn`, not `highest_appended_lsn`.
- Standby persist-before-ACK is preserved and regression-pinned (`persist_wal_write_replicated:515`); the standby path may later adopt the same flusher (out of scope here).
- Checkpoint markers retain their existing inline `fdatasync` (`persist.c:573`).
- `durable_lsn` is monotonic, never exceeds the live segment tip; on `persist_truncate_wal_above`, clamp `durable_lsn` to the new tip.

**Dependencies**
- Inbound: AFTER_EVENT hooks (`job.c`, `queue.c`, `worker.c`, `join.c`), `ha.c` WAL_NOOP — purpose: append path (P0).
- Outbound: `commit_engine_t` — purpose: self contribution input (P0).

**Contracts**: State [x]

##### State Management
- **State model:** `_Atomic uint64_t durable_lsn` on `persist_t`; invariant `durable_lsn <= current_lsn - 1`, monotonic non-decreasing except clamp-on-truncate.
- **Persistence & consistency:** `durable_lsn` advances only after a successful `fdatasync` in the flusher; the commit engine reads it for self-coverage so `committed_lsn` ⇒ durable-majority.
- **Concurrency strategy:** the flusher takes `p->lock` to read the appended tip and `fdatasync`, advances `durable_lsn`, releases, then signals commit-eval; `majority_candidate_locked` reads `durable_lsn` via atomic load. Define the append-signal → flush → commit-wake ordering precisely so a WAIT caller cannot miss a wake (lost-wakeup guard).

**Implementation Notes**
- Integration: the WAIT-mode caller blocks on the commit barrier off the iomgr thread; only the flusher thread blocks on `fdatasync`. Heartbeats unaffected.
- Validation: assert `committed_lsn <= durable_lsn` on the local member after every commit advance (debug build); verify the flusher batches (one fsync advances `durable_lsn` past multiple LSNs under load).
- Risks: **lost-wakeup / batching-window correctness** between `persist_wal_async` signal and the flusher (an entry must never be considered durable before its `fdatasync`); design the signal as level-triggered on the appended-vs-synced gap, not edge-triggered. Bounded flush latency vs throughput is a tunable (`--wal-sync-ms` becomes the max batching delay, not a durability gap).

### Persistence — RecoveryIntegrity

| Field | Detail |
|-------|--------|
| Intent | Distinguish clean log end from corruption; re-derive durable floor; fail loud when unrecoverable |
| Requirements | 2.1, 2.4 |

**Responsibilities & Constraints**
- `replay_wal`/`persist_recover` return a verdict: `RECOVER_CLEAN` (reached a clean entry boundary) vs `RECOVER_TRUNCATED` (stopped at CRC break / mismatch / short read).
- On `RECOVER_TRUNCATED`, the member is marked recovery-truncated and is NOT treated as a clean recovery; it must not advertise or lead beyond the last intact LSN.
- The recovered durable floor (highest intact replayed LSN) is published so the freshness advertisement is durable-backed.
- FATAL policy: if a recovery-truncated member cannot reach a healthy peer to catch up and the cluster requires it, the controller terminates loudly rather than serve divergent state.

**Contracts**: State [x]

##### State Management
- **State model:** verdict enum + `recovery_truncated` flag (on `ha_state_t` or `persist_t`); recovered durable floor = highest intact LSN.
- **Persistence & consistency:** does not persist `committed_lsn`; only re-derives the durable log tip from on-disk segments.
- **Concurrency strategy:** runs single-threaded during `cluster_init_phase2` before the node participates.

**Implementation Notes**
- Integration: composes with `ha-convergence-recovery` — that spec truncates an **intact** log on demotion/rejoin; this spec refuses to accept a **corrupt** log silently.
- Validation: unit test injects a CRC break at LSN 1 and asserts verdict `RECOVER_TRUNCATED` + non-promotability.
- Risks: must not FATAL on a legitimately empty first-boot WAL (no entries ≠ corruption).

### Boot Policy Decision Table (task 3.2)

> **Status:** DECISION recorded (task 3.2). No production behavior change is made by 3.2 — this table is the contract that task **3.3** (the FATAL branch) and task **4.3** (the promotion fence + release) implement. The skeleton tests in `test_commit_durability.c` (one per row, `TEST_IGNORE`-skipped pending 3.3/4.3) encode each row's expected outcome.

**Decision (Req 2.2, 2.4):** A member that boots with a structurally clean log but below the cluster's committed high-water mark, or with a recovery-**truncated** log, is **fenced from leadership** at boot (never promoted while behind/truncated). It then **attempts catch-up** from a healthy peer. It becomes eligible **only** once catch-up brings its durable position to a committed-consistent state at/above the cluster boundary. It terminates **loudly (FATAL)** only on a **bounded** failure to reach that state — i.e. it cannot recover locally **and** cannot catch up within the bounded window (no healthy peer reachable / cannot reach a committed-consistent state). 

**Invariant (authoritative):** the fence is **never a silent indefinite wait**. A truncated/behind member always resolves to exactly one of two terminal outcomes: it **catches up → ELIGIBLE**, or it **FATALs (fail-loud)** on a bounded failure. There is no third "fenced forever, still running, silently divergent" state. Snapshot-sentinel recovery is treated as CLEAN-but-behind (catch-up), never FATAL on a legitimate sentinel.

**Inputs (evaluated at boot / promotion-eligibility):**
- **verdict** — recovery integrity verdict from task 3.1: `CLEAN` vs `TRUNCATED`.
- **durable-vs-HWM** — local durable position relative to the cluster committed high-water mark: `at/above` vs `below`.
- **peer** — a healthy peer reachable for catch-up: `yes` vs `no`.
- **catch-up** — catch-up reaches a committed-consistent state within the bounded window: `succeeds` vs `fails` (n/a when no catch-up is attempted).

**Outcomes:**
- **ELIGIBLE** — may promote (no fence).
- **FENCE+CATCH-UP → ELIGIBLE** — ineligible to promote; catch up from a healthy peer; regain eligibility on success (Req 2.6).
- **FATAL** — fail loud; terminate with the truncation/boundary logged rather than serve divergent state (Req 2.4).

| # | verdict | durable-vs-HWM | healthy peer | catch-up result | Outcome | Rationale / Req |
|---|---------|----------------|--------------|-----------------|---------|-----------------|
| 1 | CLEAN | at/above | n/a | n/a | **ELIGIBLE** | Log intact and not behind — normal promotable member. (Req 2.3) |
| 2 | CLEAN | below | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | Intact but behind the HWM; must not lead below the boundary; catch up then promote. (Req 2.2, 2.3, 2.6) |
| 3 | TRUNCATED | (below — a truncated log is by definition not provably at/above) | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | Structurally truncated but a healthy peer exists; fence, catch up, regain eligibility. Never FATAL when a peer is reachable. (Req 2.2, 2.6) |
| 4 | TRUNCATED | below | no | fails (bounded window elapses / no committed-consistent state reachable) | **FATAL** | Cannot recover locally AND cannot catch up — bounded failure ⇒ fail loud, never silent indefinite fence. (Req 2.4) |
| 4b | CLEAN | below | no | fails (bounded window elapses, no peer to advance from) | **FATAL** | Behind the HWM and no healthy peer to catch up from within the bounded window — same bounded-failure terminal as row 4; never a silent indefinite fence. (Req 2.4, 2.2) |
| 5 | CLEAN (snapshot-sentinel: committed at sentinel LSN, no sub-sentinel entries) | below (no entries below the sentinel) | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | A legitimate sentinel snapshot is CLEAN-but-behind, NOT corruption; treat as must-catch-up, **never FATAL** on a legitimate sentinel. (Req 2.1, 2.2) |
| 6 | CLEAN | at/above (empty first-boot WAL, fresh cold start: durable=0, HWM=0) | n/a | n/a | **ELIGIBLE** | An empty first-boot WAL is not behind anything — cold-start members are promotable. (Req 2.1, 2.3) |

**Notes on the dimensions:**
- A `TRUNCATED` verdict is never simultaneously "provably at/above the HWM": truncation means the log stops below its last intact boundary, so a truncated member is treated as behind for fence purposes regardless of the volatile in-memory position (this is why 4.2/3.4 derive the freshness floor from the **durable** WAL tip, not a volatile value).
- "bounded window" is the catch-up deadline pinned by task 3.3; the policy requires it to be **finite** so the fence cannot become a silent indefinite wait.
- The `healthy peer = no` + `at/above` combination does not need catch-up and is covered by row 1 (ELIGIBLE) — a member already at/above the HWM is promotable without a peer.

### Election — PromotionFence + FreshnessFloor

| Field | Detail |
|-------|--------|
| Intent | Bar a truncated/behind member from leadership until caught up; advertise/vote on durable WAL state |
| Requirements | 2.2, 2.3, 2.5, 2.6 |

**Responsibilities & Constraints**
- Pre-promotion gate in `raft_cb_become_leader`: if recovery-truncated, or local durable position < cluster committed high-water mark, refuse promotion (set/extend `in_catch_up`-style fence) and instead catch up from a healthy primary.
- `ha_vote_self_freshness`: floor derives from the **durable** WAL tip (re-derived on recovery), so a truncated member cannot over-advertise a volatile `last_known_committed_lsn`. Honors `CLAUDE.md`: the post-election safe-commit point still comes from voter replies, never from a stale ctrlhello.
- Fence-release: when catch-up brings the member past its last-known committed boundary, restore eligibility (hook the existing catch-up completion / re-inclusion path).
- Degraded quorum: because a candidate cannot lead beyond its durable tip, a hollow node cannot win even at minimal quorum — preserving Req 1's invariant across leadership changes.

**Contracts**: State [x]

**Implementation Notes**
- Integration: reuses `in_catch_up` (ha.c:954, cluster.c:529) and the PreVote/vote freshness comparison (cluster.c:380-396, 467-474); extends the fence from "streaming standby" to "recovery-truncated / behind-high-water promoter."
- Validation: simulate a node restarting truncated; assert it is denied promotion until catch-up, then re-enabled (Req 2.2/2.6).
- Risks: interaction with `ha-single-leader-role` own-term gate and `ha-prevote` — re-verify the combined eligibility predicate against `raft.tla`.

### Runtime / Test — DurabilityConfig + TestSeam

| Field | Detail |
|-------|--------|
| Intent | Deterministic, configurable durability timing for verification |
| Requirements | 3.1, 3.2, 3.3 |

**Responsibilities & Constraints**
- `--wal-sync-ms` plumbed `main.c → persist_config_t` before `persist_init`; default unchanged (`DEFAULT_WAL_SYNC_MS`). Under group-commit this bounds the **maximum batching delay** (a flush fires on the append-signal or when this elapses), never a durability gap — an acked op is always durable regardless of the value.
- The seam's verification role: an `XCE_TESTING` hook (and/or a large `--wal-sync-ms` to widen the batching window) lets a test assert an acknowledged op is recoverable after an immediate ungraceful termination — proving durability comes from the commit gate, not the flush cadence.
- External acknowledgment semantics are identical with or without the mode (timing controllability only).

**Contracts**: State [x]

**Implementation Notes**
- Integration: follows `--sync-timeout-ms` (`OPT_SYNC_TIMEOUT`) and the `ha.c:2270-2335` hook convention.
- Validation: deterministic E2E mode in `test-commit-durability.sh` drives the seam so the soak becomes a deterministic RED on the unfixed build and GREEN after.
- Risks: ensure the seam is compiled out of production (`#ifdef XCE_TESTING`).

## Error Handling

### Error Strategy
- **Durability failure** (`fdatasync` error in the flusher): `durable_lsn` does not advance, so the entry is NOT counted durable, the commit does not advance, and the client does not receive *ok* (Req 1.4). The flusher surfaces the I/O error loudly (fail-loud); a persistent local sync failure is treated as a member fault, not silently retried into a false-durable state.
- **Recovery corruption** (`RECOVER_TRUNCATED`): never silently accepted (Req 2.1). Fence from promotion + catch up; FATAL when catch-up is impossible and the node is required (Req 2.4) — fail-loud.
- **Promotion while behind**: refused at the gate (Req 2.3); logged with the local durable tip vs the cluster high-water mark.

### Monitoring
- Log every `durable_lsn` regression-clamp on truncate, every recovery verdict (with the truncation boundary), every promotion refusal (durable tip vs high-water), and every fence set/release — operator-visible, once per transition (consistent with the quorum-transition logging already in the HA stack).

## Testing Strategy

### Unit Tests
- `durable_lsn` advances only after `fdatasync`; `majority_candidate_locked` self-cap equals `durable_lsn` not `highest_appended_lsn` (1.2).
- `replay_wal` returns `RECOVER_TRUNCATED` on an injected CRC break at LSN 1 and `RECOVER_CLEAN` on an intact log (2.1).
- Promotion gate denies a recovery-truncated / behind-high-water member and re-enables it after catch-up (2.2, 2.6).
- `ha_vote_self_freshness` floor reflects the durable WAL tip, not a stale in-memory `last_known_committed_lsn` (2.3).
- Standby `persist_wal_write_replicated` `fdatasync`s before ACK — regression pin (1.3).

### Integration Tests
- Commit advances `committed_lsn` only when self is durable + a standby ACK is durable; reply released only after (1.1, 1.4).
- Kill the sole durable contributor mid-commit; acknowledged op survives on the majority (1.5).
- Durability work does not delay heartbeat handling beyond tolerance (1.6).

### E2E / Failover
- `src/tools/test-commit-durability.sh` soak: rolling SIGKILL+restart under load asserts every acknowledged job survives (1.1, 1.5, 2.x). With the `--wal-sync-*` seam, run a **deterministic** variant: widen/disable background timing, kill a durable contributor in-window, assert survival — RED on the current build, GREEN after the fix (3.1, 3.2).

### Performance
- Measure submit latency delta from synchronous `fdatasync` vs the existing replication round-trip; confirm it is within the round-trip already paid before deciding group-commit is needed (research item).

## Performance & Scalability
- **Group-commit** amortizes one `fdatasync` across all entries pending at flush time, so the per-entry sync cost falls as load rises — critical for the high-volume `WAL_MODE_RETAIN` fact stream (job start/finish/fail/rusage) which is not client-latency-bound. `--wal-sync-ms` bounds the maximum batching delay (a latency/throughput tunable), never a durability gap. The `job-array-single-wal-record` roadmap item independently cuts per-array entry count. Target: zero acknowledged-job loss across failover (correctness) takes precedence over per-submit latency; group-commit keeps the throughput cost sub-linear in write rate.

## Open Questions / Gating Items (resolve in `/kiro-spec-tasks`)
- **Freshness-floor formula (gating, election-safety):** pin the exact reconciliation of the durable-WAL-derived floor vs the existing `max(applied_lsn, last_known_committed_lsn)`. Too low ⇒ a truncated node still wins (the bug); too high ⇒ election livelock. Make a `raft/raft.tla` Leader-Completeness **and liveness** re-check a blocking sub-task before the freshness change merges (Req 2.3).
- **FATAL-vs-fence decision point (gating):** a booting truncated member cannot know it is "required for quorum" at `cluster_init`. Define the concrete policy: fence at boot, attempt catch-up from a healthy primary, and FATAL only on a bounded failure to reach a committed-consistent state — never a silent indefinite fence (Req 2.2, 2.4). **RESOLVED (task 3.2):** see the **Boot Policy Decision Table (task 3.2)** above for the recorded input-state → outcome table; tasks 3.3 (FATAL branch) and 4.3 (fence + release) implement it.
- **Flusher lost-wakeup proof:** the append-signal → flush → commit-wake path must be shown free of lost wakeups (level-triggered on the appended-vs-synced gap) before relying on it for the durability invariant (Req 1.1, 1.2).
- **Snapshot-recovery edge:** current snapshots are 16-byte LSN sentinels (no entity data), so a member that recovers "from snapshot" has a committed position at the sentinel LSN but **no log entries below it**. The recovery verdict (3.1) must treat snapshot-sentinel + clean replay as CLEAN (not TRUNCATED), and the promotion fence (4.3) must treat such a member as behind-and-must-catch-up (never FATAL on a legitimate sentinel snapshot). Interacts with the roadmap `entity-bearing snapshots` item; pin the classification before implementing 3.1/4.3 (Req 2.1, 2.2).
- **Dormant synchronous-append path:** `persist_wal_sync_append` advances the commit position directly on majority ACK and assumes inline fsync — a premise group-commit removes (task 2.4). It has no production caller today but must be deleted or re-routed so it cannot reintroduce ACK-without-durability commit (Req 1.1, 1.2).