lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha-commit-durability)!: controller never self-terminates — loud non-promotable fence replaces recovery FATAL/abort

Browse source 
Per design invariant (a controller exits only on crash or admin kill), replace the
self-abort recovery terminal (tasks 3.3/4.3/4.4) with a loud, permanent, non-promotable
fence that stays alive and keeps retrying catch-up (self-heals on peer return; no manual
admin recovery). Records the never-self-terminate invariant as project-wide guidance in
src/controller/CLAUDE.md + .kiro/steering/ha-async-io.md.
This commit is contained in:
Rongsong Shen 2026-06-18 11:10:19 +00:00
parent addf53e3b4
commit 59d9df246c
10 changed files with 607 additions and 406 deletions
Download patch file Download diff file Expand all files Collapse all files

39
.kiro/specs/ha-commit-durability/design.md
View file

@ -162,12 +162,13 @@ graph TD
Floor --> Eligible[eligible to lead]
Truncated --> CanCatchUp{healthy peer reachable}
CanCatchUp -->|yes| Fence[fence from promotion until caught up]
CanCatchUp -->|no and needed for quorum| Fatal[FATAL fail loud]
CanCatchUp -->|no| LoudFence[LOUD PERMANENT FENCE: non-promotable, STAYS ALIVE, keeps retrying catch-up — NEVER self-exit]
LoudFence --> Caught
Fence --> Caught{caught up past last committed boundary}
Caught -->|yes| Eligible
```
Key decisions: corruption is never silently accepted (Req 2.1). A truncated member is fenced from promotion and catches up from a healthy primary (Req 2.2, 2.6); if it cannot reach a consistent committed state and the cluster needs it, the controller fails loudly (Req 2.4), honoring the project's fail-loud preference. A candidate never advertises or leads beyond its durable WAL tip (Req 2.3), so a hollow node cannot win even under degraded quorum (Req 2.5).
Key decisions: corruption is never silently accepted (Req 2.1). A truncated member is fenced from promotion and catches up from a healthy primary (Req 2.2, 2.6); if it cannot reach a consistent committed state, the controller logs a LOUD operator-visible alert and holds itself PERMANENTLY non-promotable while STAYING ALIVE and continuing to retry catch-up (Req 2.4) — it refuses to serve divergent state by refusing leadership, NOT by terminating. **A controller MUST NEVER self-exit** (`exit`/`abort`) as a designed response to an operational condition (user invariant, 2026-06-18): a self-exited controller needs manual admin recovery and degrades quorum meanwhile, whereas one that stays alive self-heals when a peer returns (Req 2.6). A candidate never advertises or leads beyond its durable WAL tip (Req 2.3), so a hollow node cannot win even under degraded quorum (Req 2.5).
## Requirements Traceability
@ -182,7 +183,7 @@ Key decisions: corruption is never silently accepted (Req 2.1). A truncated memb
| 2.1 | no silent truncate-accept | RecoveryIntegrity | `replay_wal` verdict | Recovery fence |
| 2.2 | fence until caught up | PromotionFence | promotion gate, `in_catch_up` | Recovery fence |
| 2.3 | never lead below high-water | PromotionFence + FreshnessFloor | `raft_cb_become_leader`, `ha_vote_self_freshness` | Recovery fence |
| 2.4 | fail-loud when unrecoverable | RecoveryIntegrity | FATAL policy | Recovery fence |
| 2.4 | loud permanent non-promotable fence when unrecoverable (stays alive, NEVER self-exit) | RecoveryIntegrity | loud-fence policy + sticky `recovery_unrecoverable` flag | Recovery fence |
| 2.5 | degraded quorum preserves invariant | DurableCommit + PromotionFence | durable-majority + fence | Both |
| 2.6 | re-eligibility on catch-up | PromotionFence | fence-release hook | Recovery fence |
| 3.1 | configurable durability cadence | DurabilityConfig | `--wal-sync-ms` | — |
@ -194,7 +195,7 @@ Key decisions: corruption is never silently accepted (Req 2.1). A truncated memb
| Component | Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|-------|--------|--------------|--------------------------|-----------|
| DurableCommit | Persistence + Commit | Synchronous persist + self-counts-durable | 1.1–1.6, 2.5 | persist_t (P0), commit_engine (P0), repl ACK (P1) | State |
| RecoveryIntegrity | Persistence | Verdict on recovery; durable floor; FATAL policy | 2.1, 2.4 | replay_wal (P0) | State |
| RecoveryIntegrity | Persistence | Verdict on recovery; durable floor; loud permanent non-promotable fence policy (stays alive, NEVER self-exit) | 2.1, 2.4 | replay_wal (P0) | State |
| PromotionFence | Election | Eligibility gate + fence-release | 2.2, 2.3, 2.6, 2.5 | RecoveryIntegrity (P0), in_catch_up (P1) | State |
| FreshnessFloor | Election | Durable-backed vote floor & advertisement | 2.3 | durable_lsn (P0), ha_state (P1) | State |
| DurabilityConfig + TestSeam | Runtime / Test | Override cadence; verification hook | 3.1–3.3 | persist_config (P0), XCE_TESTING (P1) | State |
@ -235,14 +236,14 @@ Key decisions: corruption is never silently accepted (Req 2.1). A truncated memb
| Field | Detail |
|-------|--------|
| Intent | Distinguish clean log end from corruption; re-derive durable floor; fail loud when unrecoverable |
| Intent | Distinguish clean log end from corruption; re-derive durable floor; loud permanent non-promotable fence when unrecoverable (stays alive, NEVER self-exit) |
| Requirements | 2.1, 2.4 |
**Responsibilities & Constraints**
- `replay_wal`/`persist_recover` return a verdict: `RECOVER_CLEAN` (reached a clean entry boundary) vs `RECOVER_TRUNCATED` (stopped at CRC break / mismatch / short read).
- On `RECOVER_TRUNCATED`, the member is marked recovery-truncated and is NOT treated as a clean recovery; it must not advertise or lead beyond the last intact LSN.
- The recovered durable floor (highest intact replayed LSN) is published so the freshness advertisement is durable-backed.
- FATAL policy: if a recovery-truncated member cannot reach a healthy peer to catch up and the cluster requires it, the controller terminates loudly rather than serve divergent state.
- Loud-fence policy (NEVER self-exit, user invariant 2026-06-18): if a recovery-truncated / behind member cannot reach a healthy peer to catch up, the controller logs a LOUD operator-visible alert (truncation boundary vs cluster HWM) via `persist_recover_loud_fence_if_unrecoverable`, sets a sticky `recovery_unrecoverable` flag, and holds itself PERMANENTLY non-promotable while STAYING ALIVE and continuing to retry catch-up. It refuses to serve divergent state by refusing leadership — it does NOT `exit`/`abort`. The flag clears (and the fence releases → eligible, Req 2.6) automatically when a peer returns and catch-up reaches a committed-consistent state.
**Contracts**: State [x]
@ -254,15 +255,15 @@ Key decisions: corruption is never silently accepted (Req 2.1). A truncated memb
**Implementation Notes**
- Integration: composes with `ha-convergence-recovery` — that spec truncates an **intact** log on demotion/rejoin; this spec refuses to accept a **corrupt** log silently.
- Validation: unit test injects a CRC break at LSN 1 and asserts verdict `RECOVER_TRUNCATED` + non-promotability.
- Risks: must not FATAL on a legitimately empty first-boot WAL (no entries ≠ corruption).
- Risks: must not loud-fence on a legitimately empty first-boot WAL (no entries ≠ corruption).
### Boot Policy Decision Table (task 3.2)
> **Status:** DECISION recorded (task 3.2). No production behavior change is made by 3.2 — this table is the contract that task **3.3** (the FATAL branch) and task **4.3** (the promotion fence + release) implement. The skeleton tests in `test_commit_durability.c` (one per row, `TEST_IGNORE`-skipped pending 3.3/4.3) encode each row's expected outcome.
> **Status:** DECISION recorded (task 3.2); **terminal action corrected by the never-self-exit rework (2026-06-18).** The original table's `FATAL` outcome (a self-`abort`) was REPLACED: the unrecoverable terminal is now a **LOUD PERMANENT FENCE** — non-promotable, **stays alive**, keeps retrying catch-up. This table is the contract that task **3.3** (the loud-fence terminal) and task **4.3** (the promotion fence + release) implement.
**Decision (Req 2.2, 2.4):** A member that boots with a structurally clean log but below the cluster's committed high-water mark, or with a recovery-**truncated** log, is **fenced from leadership** at boot (never promoted while behind/truncated). It then **attempts catch-up** from a healthy peer. It becomes eligible **only** once catch-up brings its durable position to a committed-consistent state at/above the cluster boundary. It terminates **loudly (FATAL)** only on a **bounded** failure to reach that state — i.e. it cannot recover locally **and** cannot catch up within the bounded window (no healthy peer reachable / cannot reach a committed-consistent state).
**Decision (Req 2.2, 2.4):** A member that boots with a structurally clean log but below the cluster's committed high-water mark, or with a recovery-**truncated** log, is **fenced from leadership** at boot (never promoted while behind/truncated). It then **attempts catch-up** from a healthy peer. It becomes eligible **only** once catch-up brings its durable position to a committed-consistent state at/above the cluster boundary. If it cannot reach that state (no healthy peer reachable / cannot reach a committed-consistent state), it enters a **LOUD PERMANENT FENCE**: it logs a loud, operator-visible alert (truncation boundary vs cluster HWM), holds itself **permanently non-promotable**, and **stays alive while continuing to retry catch-up** — it **NEVER self-terminates**. The fence releases (→ ELIGIBLE) automatically once catch-up reaches a committed-consistent state.
**Invariant (authoritative):** the fence is **never a silent indefinite wait**. A truncated/behind member always resolves to exactly one of two terminal outcomes: it **catches up → ELIGIBLE**, or it **FATALs (fail-loud)** on a bounded failure. There is no third "fenced forever, still running, silently divergent" state. Snapshot-sentinel recovery is treated as CLEAN-but-behind (catch-up), never FATAL on a legitimate sentinel.
**Invariant (authoritative):** A truncated/behind member always resolves to exactly one of two outcomes: it **catches up → ELIGIBLE**, or it enters a **LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying catch-up — NEVER self-exit)**. It is never silently divergent (it refuses to lead), and the process never self-`exit`s/`abort`s as a designed response to this operational condition (user invariant, 2026-06-18) — a self-exited controller needs manual admin recovery and degrades quorum, whereas one that stays alive self-heals when a peer returns (Req 2.6). Snapshot-sentinel recovery is treated as CLEAN-but-behind (catch-up), never the unrecoverable terminal on a legitimate sentinel.
**Inputs (evaluated at boot / promotion-eligibility):**
- **verdict** — recovery integrity verdict from task 3.1: `CLEAN` vs `TRUNCATED`.
@ -273,21 +274,21 @@ Key decisions: corruption is never silently accepted (Req 2.1). A truncated memb
**Outcomes:**
- **ELIGIBLE** — may promote (no fence).
- **FENCE+CATCH-UP → ELIGIBLE** — ineligible to promote; catch up from a healthy peer; regain eligibility on success (Req 2.6).
- **FATAL** — fail loud; terminate with the truncation/boundary logged rather than serve divergent state (Req 2.4).
- **LOUD PERMANENT FENCE** — log a loud operator-visible alert (truncation/boundary), set the sticky `recovery_unrecoverable` flag, and hold the member permanently non-promotable while it **stays alive** and keeps retrying catch-up — **NEVER self-exit**; releases → ELIGIBLE on catch-up (Req 2.4, 2.6).
| # | verdict | durable-vs-HWM | healthy peer | catch-up result | Outcome | Rationale / Req |
|---|---------|----------------|--------------|-----------------|---------|-----------------|
| 1 | CLEAN | at/above | n/a | n/a | **ELIGIBLE** | Log intact and not behind — normal promotable member. (Req 2.3) |
| 2 | CLEAN | below | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | Intact but behind the HWM; must not lead below the boundary; catch up then promote. (Req 2.2, 2.3, 2.6) |
| 3 | TRUNCATED | (below — a truncated log is by definition not provably at/above) | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | Structurally truncated but a healthy peer exists; fence, catch up, regain eligibility. Never FATAL when a peer is reachable. (Req 2.2, 2.6) |
| 4 | TRUNCATED | below | no | fails (bounded window elapses / no committed-consistent state reachable) | **FATAL** | Cannot recover locally AND cannot catch up — bounded failure ⇒ fail loud, never silent indefinite fence. (Req 2.4) |
| 4b | CLEAN | below | no | fails (bounded window elapses, no peer to advance from) | **FATAL** | Behind the HWM and no healthy peer to catch up from within the bounded window — same bounded-failure terminal as row 4; never a silent indefinite fence. (Req 2.4, 2.2) |
| 5 | CLEAN (snapshot-sentinel: committed at sentinel LSN, no sub-sentinel entries) | below (no entries below the sentinel) | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | A legitimate sentinel snapshot is CLEAN-but-behind, NOT corruption; treat as must-catch-up, **never FATAL** on a legitimate sentinel. (Req 2.1, 2.2) |
| 3 | TRUNCATED | (below — a truncated log is by definition not provably at/above) | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | Structurally truncated but a healthy peer exists; fence, catch up, regain eligibility. Never the unrecoverable terminal when a peer is reachable. (Req 2.2, 2.6) |
| 4 | TRUNCATED | below | no | fails (no committed-consistent state reachable) | **LOUD PERMANENT FENCE** (non-promotable, stays alive, keeps retrying catch-up — NEVER self-exit) | Cannot recover locally AND no peer to catch up from ⇒ log loudly + hold non-promotable + alive; releases → ELIGIBLE when a peer returns and it catches up. NEVER `exit`/`abort`. (Req 2.4, 2.6) |
| 4b | CLEAN | below | no | fails (no peer to advance from) | **LOUD PERMANENT FENCE** (non-promotable, stays alive, keeps retrying catch-up — NEVER self-exit) | Behind the HWM and no healthy peer to catch up from — same non-terminating loud-fence as row 4; NEVER self-exit. (Req 2.4, 2.2, 2.6) |
| 5 | CLEAN (snapshot-sentinel: committed at sentinel LSN, no sub-sentinel entries) | below (no entries below the sentinel) | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | A legitimate sentinel snapshot is CLEAN-but-behind, NOT corruption; treat as must-catch-up, **never the unrecoverable terminal** on a legitimate sentinel. (Req 2.1, 2.2) |
| 6 | CLEAN | at/above (empty first-boot WAL, fresh cold start: durable=0, HWM=0) | n/a | n/a | **ELIGIBLE** | An empty first-boot WAL is not behind anything — cold-start members are promotable. (Req 2.1, 2.3) |
**Notes on the dimensions:**
- A `TRUNCATED` verdict is never simultaneously "provably at/above the HWM": truncation means the log stops below its last intact boundary, so a truncated member is treated as behind for fence purposes regardless of the volatile in-memory position (this is why 4.2/3.4 derive the freshness floor from the **durable** WAL tip, not a volatile value).
- "bounded window" is the catch-up deadline pinned by task 3.3; the policy requires it to be **finite** so the fence cannot become a silent indefinite wait.
- the catch-up retry is **indefinite** (the member stays alive and keeps trying) and **loud** (a periodic operator-visible re-alert, controlled by `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS`); it is never silently divergent because the member refuses leadership the entire time. The previous "bounded window before FATAL" deadline was REMOVED by the never-self-exit rework — there is no deadline-driven termination.
- The `healthy peer = no` + `at/above` combination does not need catch-up and is covered by row 1 (ELIGIBLE) — a member already at/above the HWM is promotable without a peer.
### Election — PromotionFence + FreshnessFloor
@ -458,7 +459,7 @@ Walking the relevant items under the pinned formula:
### Error Strategy
- **Durability failure** (`fdatasync` error in the flusher): `durable_lsn` does not advance, so the entry is NOT counted durable, the commit does not advance, and the client does not receive *ok* (Req 1.4). The flusher surfaces the I/O error loudly (fail-loud); a persistent local sync failure is treated as a member fault, not silently retried into a false-durable state.
- **Recovery corruption** (`RECOVER_TRUNCATED`): never silently accepted (Req 2.1). Fence from promotion + catch up; FATAL when catch-up is impossible and the node is required (Req 2.4) — fail-loud.
- **Recovery corruption** (`RECOVER_TRUNCATED`): never silently accepted (Req 2.1). Fence from promotion + catch up; when catch-up is impossible (no peer) the node enters a LOUD PERMANENT FENCE — non-promotable, **stays alive**, keeps retrying catch-up, NEVER self-exits (Req 2.4, 2.6).
- **Promotion while behind**: refused at the gate (Req 2.3); logged with the local durable tip vs the cluster high-water mark.
### Monitoring
@ -489,7 +490,7 @@ Walking the relevant items under the pinned formula:
## Open Questions / Gating Items (resolve in `/kiro-spec-tasks`)
- **Freshness-floor formula (gating, election-safety):** pin the exact reconciliation of the durable-WAL-derived floor vs the existing `max(applied_lsn, last_known_committed_lsn)`. Too low ⇒ a truncated node still wins (the bug); too high ⇒ election livelock. Make a `raft/raft.tla` Leader-Completeness **and liveness** re-check a blocking sub-task before the freshness change merges (Req 2.3).
- **FATAL-vs-fence decision point (gating):** a booting truncated member cannot know it is "required for quorum" at `cluster_init`. Define the concrete policy: fence at boot, attempt catch-up from a healthy primary, and FATAL only on a bounded failure to reach a committed-consistent state — never a silent indefinite fence (Req 2.2, 2.4). **RESOLVED (task 3.2):** see the **Boot Policy Decision Table (task 3.2)** above for the recorded input-state → outcome table; tasks 3.3 (FATAL branch) and 4.3 (fence + release) implement it.
- **Unrecoverable-terminal decision point (gating):** a booting truncated member cannot know it is "required for quorum" at `cluster_init`. Concrete policy: fence at boot, attempt catch-up from a healthy primary, and on a failure to reach a committed-consistent state (no peer) enter a **LOUD PERMANENT FENCE** — non-promotable, stays alive, keeps retrying catch-up, **NEVER self-exit** (Req 2.2, 2.4, 2.6). **RESOLVED (task 3.2; terminal corrected by the never-self-exit rework 2026-06-18 — the original self-`abort` FATAL was replaced by the loud permanent fence):** see the **Boot Policy Decision Table (task 3.2)** above; tasks 3.3 (loud-fence terminal) and 4.3 (fence + release) implement it.
- **Flusher lost-wakeup proof:** the append-signal → flush → commit-wake path must be shown free of lost wakeups (level-triggered on the appended-vs-synced gap) before relying on it for the durability invariant (Req 1.1, 1.2).
- **Snapshot-recovery edge:** current snapshots are 16-byte LSN sentinels (no entity data), so a member that recovers "from snapshot" has a committed position at the sentinel LSN but **no log entries below it**. The recovery verdict (3.1) must treat snapshot-sentinel + clean replay as CLEAN (not TRUNCATED), and the promotion fence (4.3) must treat such a member as behind-and-must-catch-up (never FATAL on a legitimate sentinel snapshot). Interacts with the roadmap `entity-bearing snapshots` item; pin the classification before implementing 3.1/4.3 (Req 2.1, 2.2).
- **Snapshot-recovery edge:** current snapshots are 16-byte LSN sentinels (no entity data), so a member that recovers "from snapshot" has a committed position at the sentinel LSN but **no log entries below it**. The recovery verdict (3.1) must treat snapshot-sentinel + clean replay as CLEAN (not TRUNCATED), and the promotion fence (4.3) must treat such a member as behind-and-must-catch-up (never the unrecoverable terminal on a legitimate sentinel snapshot). Interacts with the roadmap `entity-bearing snapshots` item; pin the classification before implementing 3.1/4.3 (Req 2.1, 2.2).
- **Dormant synchronous-append path:** `persist_wal_sync_append` advances the commit position directly on majority ACK and assumes inline fsync — a premise group-commit removes (task 2.4). It has no production caller today but must be deleted or re-routed so it cannot reintroduce ACK-without-durability commit (Req 1.1, 1.2).

61
.kiro/specs/ha-commit-durability/requirements.md Normal file
View file

@ -0,0 +1,61 @@
# Requirements Document
## Project Description (Input)
**Who has the problem:** Cluster operators running the XCE HA controller cluster — and every client whose acknowledged job/queue/worker operations must survive controller failover.
**Current situation:** In a 3-node failover test, committed jobs were permanently lost (`joblist` dropped 10→3; the 7 earliest, already acknowledged, vanished). Root cause (bug-661): **acknowledgment does not imply durability.** The controller advances its commit position on a majority of replication **acknowledgments**, but the primary counts its own contribution as soon as the entry is written to volatile buffers, not after it is persisted to stable storage — background persistence happens on a timer. A standby already persists before acknowledging, but the primary does not. So a "committed" operation can be durable on as few as one member; a single ill-timed crash + restart, or a restarted member with a truncated log winning leadership, discards it. This violates the documented assumption that a committed operation is held durably on a majority.
**What should change (chosen scope — durable commit + promotion fence):**
1. **Durable commit:** when the controller acknowledges an operation to a client, that operation's log entry must already be persisted to stable storage on a fault-tolerant majority (the acknowledging member plus ⌊N/2⌋ others). The controller must count its own copy toward the commit majority only once it is durably persisted.
2. **Recovery integrity and leadership fence (fail-loud):** a member that cannot reconstruct its committed state from persisted storage must be barred from leadership until it has caught up, and must never become authoritative while behind the cluster's known committed high-water mark — surfacing the failure loudly rather than silently discarding committed operations.
The mechanism (synchronous persistence vs. later group-commit batching) is a design decision; group-commit batching is deferred to the roadmap as a contract-preserving throughput optimization. See `.kiro/specs/ha-commit-durability/brief.md` for the full discovery analysis, evidence trail, and design decision.
## Introduction
This spec closes the durability gap beneath the HA consistency machinery: it makes acknowledgment imply durability. After this spec, any operation the controller reports as successful is guaranteed to be persisted on a majority of members at the moment of acknowledgment, and therefore survives the loss of any minority of members and any subsequent leadership change. It also fences a member that cannot recover its committed state — or that lags the cluster's committed high-water mark — from becoming the authoritative leader, so a truncated or corrupted log can never overwrite acknowledged operations. The client reply path already waits on the commit barrier; this spec strengthens the meaning of "committed" to "durably persisted on a majority," so the existing reply gate enforces the invariant. A testability provision makes the guarantee deterministically verifiable without depending on race timing.
## Boundary Context
- **In scope**: the durable-commit invariant on the primary's commit path (self contributes to the commit majority only when durably persisted); pinning the standby's persist-before-acknowledge behavior; the no-premature-acknowledgment guarantee; survival of acknowledged operations across the crash+restart of any minority; preserving cluster liveness (heartbeat/election) while performing durability work; recovery-integrity and leadership fencing for members that cannot reconstruct committed state or lag the committed high-water mark; degraded-quorum preservation of the invariant; a deterministic durability-verification provision; the soak E2E (`src/tools/test-commit-durability.sh`) plus unit coverage.
- **Out of scope**: the replication/acknowledgment wire format (`ha-wal-replication`); per-entry majority arithmetic and commit-index advancement (`ha-commit-engine` — consumed, this spec changes only the durability *precondition* on its inputs); demotion truncate/self-restart and rejoin reconciliation of an intact log (`ha-convergence-recovery`); promotion no-op and own-term gate (`ha-single-leader-role`); raw vote-grant arithmetic (`raft-library` — this spec adds a durability/high-water *precondition*, not new vote logic); WAL group-commit batching (roadmap — a contract-preserving throughput optimization, deferred); single-record job-array persistence (`job-array-single-wal-record`, roadmap); entity-bearing snapshots (roadmap); client-side controller failover (`libxce-failover`).
- **Adjacent expectations**: relies on `ha-commit-engine`'s commit-position semantics (monotonic, per-entry majority) and on `ha-wal-replication`'s acknowledgment and catch-up machinery; the client reply is already gated on the commit barrier (`ha-commit-barrier-rollout`), so strengthening "committed" to "durable-majority" enforces the invariant without changing the reply path. Downstream, `ha-convergence-recovery`'s single-outcome convergence depends on this spec making "persisted committed state" genuinely durable; a returning member that this spec fences from leadership until caught up complements that spec's truncate-and-restart reconciliation.
## Requirements
### Requirement 1: Durable Commit — Acknowledgment Implies Majority Persistence
**Objective:** As a cluster operator, on behalf of every client, I want any operation the controller acknowledges as successful to already be durably persisted on a majority of controllers, so that no acknowledged work is ever lost to a controller failure or failover.
#### Acceptance Criteria
1. When the controller acknowledges a state-changing operation to a client as successful, the controller shall have durably persisted that operation's log entry to stable storage on a majority of cluster members — the acknowledging member plus at least ⌊N/2⌋ other members.
2. While evaluating whether an operation is committed, the controller shall count its own log entry toward the commit majority only after that entry is durably persisted to its local stable storage, not when it is merely written to volatile buffers.
3. When a standby acknowledges a replicated log entry, the standby shall durably persist that entry to its local stable storage before sending the acknowledgment.
4. While a state-changing operation's log entry is not yet durably persisted on a majority of members, the controller shall not report that operation as successful to the requester.
5. If any minority of cluster members that contributed to an operation's commit are simultaneously lost and restarted, the acknowledged operation shall remain present and recoverable on the surviving cluster.
6. While performing durability persistence for a commit, the controller shall continue to service heartbeat and election processing without stalling, so that durability work does not induce spurious failover.
### Requirement 2: Recovery Integrity and Leadership Fence
**Objective:** As a cluster operator, I want a controller that cannot reconstruct its committed state to be barred from leadership until it has caught up, so that a member with a truncated or corrupted log can never become authoritative and discard acknowledged operations.
#### Acceptance Criteria
1. When the controller recovers its persisted log at startup, if the log cannot be reconstructed contiguously up to its last known committed boundary, the controller shall not silently discard committed entries and continue as if its committed state were complete.
2. If a controller's recovered log does not reach its last known committed boundary, the controller shall be ineligible to become primary until it has caught up to the cluster's committed boundary from a healthy member.
3. While a controller's committed position is below the cluster's known committed high-water mark, the controller shall not assume leadership in a way that would discard already-committed operations.
4. If a controller cannot recover to a consistent committed state and cannot catch up from a healthy member, the controller shall surface the condition as a loud, operator-visible alert and hold itself permanently non-promotable while staying alive and continuing to retry catch-up — refusing to serve potentially divergent state by refusing leadership, NOT by terminating the process. A controller MUST NEVER self-terminate (`exit`/`abort`) as a designed response to an operational condition: a self-exited controller requires manual admin recovery and degrades quorum meanwhile, whereas one that stays alive and keeps retrying self-heals automatically when a peer returns (see Requirement 2.6). (A genuine programming-invariant `assert`, or an external admin/supervisor kill, is not such a designed response.)
5. While the cluster operates with reduced redundancy or minimal quorum, the controller shall preserve the durable-commit guarantee of Requirement 1 across leadership changes, so that no acknowledged operation is lost even when quorum is temporarily degraded.
6. When a fenced member finishes catching up to the cluster's committed boundary, the controller shall restore that member's eligibility to participate in leadership.
### Requirement 3: Deterministic Durability Verification
**Objective:** As a developer verifying durability, I want the durability behavior to be deterministically controllable in a verification mode, so that the durable-before-acknowledge guarantee can be confirmed without depending on race timing.
#### Acceptance Criteria
1. Where a durability-verification mode is enabled, the controller shall make the durability behavior deterministically controllable (for example, by configuring or disabling background persistence) so that an acknowledged operation's durability can be asserted without relying on timing.
2. Where durability-verification mode removes reliance on background persistence, an operation that the controller acknowledged as successful shall be recoverable after an immediate, ungraceful termination of a contributing member.
3. The controller shall preserve the same externally observable acknowledgment semantics whether or not durability-verification mode is enabled, so that the mode changes timing controllability only, not the durability contract.

1
.kiro/specs/ha-commit-durability/tasks.md
View file

@ -143,3 +143,4 @@
- 4.3: Pre-promotion eligibility gate in `raft_cb_become_leader` (ha.c, post-truncate / pre-CTRL_PRIMARY, returns on refuse). `ha_promotion_gate_decision` (pure, delegates to 3.3's `persist_recover_decision` — table is single source of truth) + `ha_evaluate_promotion_fence` (live: ELIGIBLE clears `in_catch_up`, FENCE sets it, FATAL → 3.3 `persist_recover_fatal_if_unrecoverable`). REFUSE if `persist_recovery_truncated` OR `persist_durable_floor < promotion_safe_lsn` (the existing voter-reply-derived `safe_committed_lsn`, read AFTER `persist_truncate_wal_above` so it reflects 2.2's clamp). `peer_reachable` = some granting voter has a live endpoint (`ectx->votes[i].vote_granted && .ep != NULL`) — so a multi-node election winner ALWAYS has a peer ⇒ FENCE (never premature FATAL, honoring rows 3/5); FATAL only fires no-peer (single-node corrupt = genuinely unrecoverable, row 4). Reuses `in_catch_up` (NOT a parallel fence); release on a later promotion attempt once caught up (durable_floor ≥ HWM, not truncated) → ELIGIBLE clears the fence (Req 2.6, re-evaluation model). `safe_committed_lsn` arithmetic UNTOUCHED. Turned 3.2 rows 1/2/3/5/6 GREEN (0 ignored boot-policy rows remain) + minimal-quorum test (Req 2.5). DEFERRED to 5.1: the live bounded-window catch-up TIMER (grant `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS` to find a peer before FATAL) — here `catch_up_succeeded=false` at the gate, so it FENCEs (peer) or FATALs (no peer) immediately; fail-loud-conservative, never silently leads hollow. NOTE: reviewed in MAIN CONTEXT (review subagent hit a spend limit) — suite green ×1 (Fail:0, all 9 boot-policy/gate tests PASS, test_fact_hook_append passed too since this shell's /tmp is fsync-able). Phase 4 COMPLETE.
- 5.1: INTEGRATION verification — the end-to-end durable-majority path was ALREADY wired by 2.1/2.2/2.4/4.x (no production gap found); 5.1 adds 3 integration tests + a tiny `#ifdef XCE_TESTING` liveness-observability seam (`persist_test_last_fdatasync_thread`, nm-excluded from prod). `test_durable_majority_commit_end_to_end`: holds durability off (forced fsync-fail), delivers BOTH standby ACKs, asserts committed_lsn/WAIT do NOT release (ACK-majority alone insufficient) — then flush makes self durable ⇒ committed advances + WAIT releases rc=0; asserts `committed_lsn <= durable_lsn`. RED proof: injecting the pre-2.2 `highest_appended_lsn` cap trips `assert(cand <= persist_durable_lsn)`. `test_sole_durable_contributor_loss_survives` (Req 1.5/2.5): property-asserted (committed ⇒ ≥⌊N/2⌋+1 durable copies, two persist dirs, drop one) — LIVE failover proof is 5.2. `test_durability_work_off_iomgr_preserves_liveness` (Req 1.6): seam records the flusher thread id; asserts fdatasync runs off the caller/iomgr thread + heartbeat reads stay <50ms during flushes. GOTCHA: a real persist with registered standbys needs the `XCE_TESTING repl_broadcast_send_hook` (drop) installed before append, else dummy endpoints trip an ASAN overflow in `dataq_enq`. CAVEATS (for 5.3/validation): (1) meson `Fail:0` masks Unity sub-group failures — gate on `grep -c :FAIL: testlog.txt`==0; (2) run on a fsync-able fs (the pre-existing test_fact_hook_append EIO failure is /tmp-overlay-only). 4.3 catch-up-timer-before-FATAL still a follow-up. Reviewed in MAIN CONTEXT (subagent spend limit).
- 4.4: Closed the 5.2-soak liveness gap. `raft_cb_become_leader` NO LONGER FATALs synchronously — on FENCE / would-be-FATAL (truncated or durable_floor < HWM) it calls `ha_arm_promotion_fence_timer` (sets `in_catch_up`, schedules a one-shot `delay_task_ms(&c->delaymgr, ha_promotion_fence_window_ms(), 1, promotion_fence_timeout_fn, ctx)`) and RETURNS without becoming primary. The timer callback (`promotion_fence_timeout_fn`, ha.c:702) re-reads LIVE state after the window: caught-up (`!truncated && durable_floor>=HWM`) → release fence, NO FATAL (Req 2.6); peer reachable (`repl_active_standby_count>0`) OR durable_floor advanced past the arm-time baseline → progress possible, leave fenced (existing catch-up machinery / next promotion-gate ELIGIBLE releases it), NO FATAL; ONLY still-truncated/behind + no-peer + no-progress → 3.3 `persist_recover_fatal_if_unrecoverable` (Req 2.4). UAF-safe cancel (`ha_cancel_promotion_fence_timer`: cancelled flag + atomic-exchange the stored ctx; handle cleared before the cancelled-check, mirrors `raft_election_timeout_fn`); cancelled in `ha_fini`; fail-loud if `delay_task_ms` itself fails (never fence indefinitely). Test window seam `ha_set_promotion_fence_window_ms` + driver `ha_arm_promotion_fence_timer_for_test`. 3 tests (defers-fatal-to-timer, released-on-catchup-no-fatal, cancel-prevents-fatal) with RED proof (re-injecting sync FATAL → 3 fails). Reviewed MAIN CONTEXT (subagent spend limit): suite :FAIL:==0, /tmp fsync-able this run. NOTE: this is the bounded-window timer 4.3/5.1 deferred; with it, 5.2's soak should keep the cluster available (truncated rejoiners catch up, not die).
- REWORK (2026-06-18, user invariant — a controller MUST NEVER self-terminate): tasks 3.3/4.3/4.4 had implemented Req 2.4 "fail loud" as a SELF-ABORT (`persist_recover_fatal_if_unrecoverable → tlog_exit(); abort()` + `persist_fatal_hook`). That violated the invariant (a self-exited controller needs MANUAL admin recovery + leaves quorum degraded; a live retrying node self-heals when a peer returns) and was the true root cause of the 5.2 soak quorum collapse. CORRECTED: removed the abort/exit terminal entirely. New: `persist_recover_loud_fence_if_unrecoverable(persist_t*)` (NO abort) logs a loud operator-visible ERROR (truncation boundary vs cluster HWM) + sets sticky `_Atomic bool recovery_unrecoverable` (accessors `persist_recovery_unrecoverable`/`persist_clear_recovery_unrecoverable`); the member STAYS ALIVE, permanently non-promotable/fenced (`in_catch_up` + the sticky flag both consulted by the promotion gate), and KEEPS RETRYING catch-up. Decision enum `RECOVER_DECISION_FATAL` → `RECOVER_DECISION_FENCE_UNRECOVERABLE` (non-terminating). 4.4 timer no longer aborts — it re-emits the loud operator alert + re-arms (operator visibility), never exits. Flag + fence CLEAR automatically on catch-up (not truncated AND durable_floor ≥ HWM) → re-eligible (Req 2.6), fully automatic. `persist_fatal_hook` removed. Verified: grep/nm show NO abort/exit/tlog_exit on the recovery path; ASAN suite :FAIL:==0; reworked tests (row4/4b → loud_fence, unrecoverable-stays-alive-loud, released-on-catchup, cancel-stops-realert) PASS. Spec updated: requirements.md Req 2.4 + design.md (decision table rows 4/4b, RecoveryIntegrity, traceability, FATAL-vs-fence open question) reworded "FATAL/terminate" → "LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — NEVER self-exit)". PROJECT-WIDE GUIDANCE added: `src/controller/CLAUDE.md` §"Process Lifecycle (DESIGN INVARIANT)" + `.kiro/steering/ha-async-io.md` — controllers never self-terminate. (`ha_self_restart` execv-restart in convergence/demotion is a SEPARATE primitive, untouched.)

2
.kiro/steering/ha-async-io.md
View file

@ -6,6 +6,8 @@ The HA layer combines Raft-style consensus election with a single-writer replica
**Single-writer constraint**: only the primary writes WAL entries. This avoids Raft log replication complexity while the election mechanism inherits Raft's safety guarantees (term-based voting, freshness gates, monotonic commits).
**Process-lifecycle invariant (applies to ALL controller code)**: a controller MUST NEVER self-terminate. It exits only on a genuine crash or an admin/supervisor kill — never via `exit()`/`abort()`/`tlog_exit()` as a designed response to an operational condition (unrecoverable/corrupt WAL, lost quorum, unreachable peer, failed catch-up). "Fail loud" means log a loud operator-visible ERROR + enter a permanent non-promotable/fenced state + **stay alive and keep retrying catch-up** (so the node self-heals when a peer returns, without manual admin recovery) — it does NOT mean exit. See `src/controller/CLAUDE.md` → "Process Lifecycle (DESIGN INVARIANT)" for the authoritative statement.
## Async I/O Model
Three components in `src/lib/`:

17
src/controller/CLAUDE.md
View file

@ -24,6 +24,15 @@ Links against `libxce` from `src/lib/`.
- Component-specific headers in `include/`
- Shared types defined in `src/include/xce/` (libxce)
## Process Lifecycle (DESIGN INVARIANT — applies to ALL controller tasks)
**A controller process MUST NEVER self-terminate.** It exits ONLY on (a) a genuine crash (segfault/OOM) or (b) an admin/supervisor kill. A controller must NOT call `exit()`/`_exit()`/`abort()`/`tlog_exit()` (or any self-termination) as a **designed response to an operational condition** — e.g. a recovery-truncated/corrupt WAL, an inability to reach quorum, a peer being unreachable, a failed catch-up, or any cluster-state condition.
- **Rationale:** a self-exited controller requires **manual admin recovery** to come back and leaves quorum degraded in the meantime. A controller that **stays alive and keeps retrying self-heals automatically** when a peer returns — no admin in the loop.
- **"Fail loud" = refuse to participate, not exit.** When a controller detects it cannot safely act (e.g. it cannot reconstruct committed state, or would serve divergent state), it must: log a **loud, operator-visible ERROR** (with the actionable boundary, e.g. durable_floor vs cluster high-water), enter a **permanent non-promotable / fenced** state (never become authoritative, never serve divergent state), **stay alive**, and **keep retrying catch-up** so it recovers automatically when conditions allow. Clear the fenced state once a committed-consistent state is reached.
- **The only acceptable `assert()`/abort** is for a genuine *programming-invariant* violation (a bug that means the process is in an impossible state) — that is a "crash," not an operational response. Never use `assert()`/abort to handle expected operational/failure conditions.
- **When reviewing or writing controller code:** any new `exit`/`abort`/`tlog_exit`/`FATAL`-style terminal on an operational path is a design violation — replace it with the loud-fence-and-stay-alive pattern above. (Precedent: `ha-commit-durability` recovery-unrecoverable handling — `persist_recover_loud_fence_if_unrecoverable` + the `recovery_unrecoverable` sticky fence, which replaced an earlier incorrect `abort()` terminal.)
## HA Protocol
See `.kiro/steering/ha-async-io.md` for the full HA protocol reference: async I/O model (iomgr/vtaskmgr/delaymgr), connection ownership, Raft election, heartbeat model, two-tier replication consistency, dispatch gate, and TLA+ variable mapping. The TLA+ safety invariants and protocol verification checklist below are the authoritative controller-specific verification gates — keep them here.
@ -67,6 +76,10 @@ The file `raft/raft.tla` contains Diego Ongaro's TLA+ specification of the Raft
**Commit Rule** (current-term entry required): Each WAL entry is committed individually via `repl_wait_majority_ack()`. Since entries are single-writer (only the current primary writes), every committed entry is implicitly from the current leader's term. Equivalent to Raft's batched commit rule.
**Commit Ordering** (contiguous, in log order — RULE): `committed_lsn` advances only as a contiguous prefix in LSN/append order; entries are never committed or reordered out of log order. The single-writer append lock (`p->lock`) is the serialization point. Durability flushing (incl. group-commit batching) makes a contiguous **prefix** durable only — `durable_lsn` is a prefix watermark, never a per-entry set, so a later entry can never be durable while an earlier one is not. Log order must preserve **causal dependencies** (append a cause before its effect — e.g. job submit before job start); independent concurrent events take an arbitrary but consistent total order, which is linearizable via the single serialization point plus commit-gated reply. Out-of-order / parallel commit is NOT supported by this single-log architecture (it would require per-entity logs with dependency tracking — explicitly out of scope).
**Commit Durability** (acknowledged ⇒ durable on a majority — TARGET RULE): when the controller replies success to a client, the operation's entry must be `fdatasync`'d on a fault-tolerant majority (the acking member + ⌊N/2⌋ others); a member counts toward the commit majority for an LSN only once that LSN is durable locally (self caps at `durable_lsn`, not the appended tip). **KNOWN GAP — bug-661, spec `ha-commit-durability` (in progress, 2026-06-18):** the commit majority currently counts the primary's *appended* (page-cache) tip via `majority_candidate_locked`, and primary fsync is async (`fsync_worker` timer), so "committed" does NOT yet imply durable-on-a-majority — an acknowledged entry can be durable on a single member and lost across failover. Until that spec lands, do not assume an acked operation survives a crash of the acking minority.
### Post-Election Safe Commit Point (Critical)
After winning election, the new primary determines `safe_committed_lsn` from voter state collected during the election. This eliminates a round-trip vs waiting for the first heartbeat reply:
@ -89,6 +102,10 @@ When modifying HA protocol code (election, replication, heartbeat), verify:
- [ ] `safe_committed_lsn` is derived from voter replies, not from stale ctrlhello
- [ ] WAL truncation on promotion only removes entries above the safe commit point
- [ ] Freshness gate in RequestVote: candidate must have `lastLogTerm >= voter` AND `lastLogIndex >= voter`
- [ ] `committed_lsn` advances only as a contiguous LSN-ordered prefix — no out-of-order or per-entry commit
- [ ] Durability flushing makes a contiguous prefix durable; `durable_lsn` is a prefix watermark (never a per-entry set)
- [ ] Causal dependencies are appended in order (cause before effect, e.g. submit before start)
- [ ] (post `ha-commit-durability`) a member counts toward the commit majority only at its durably-persisted LSN; acknowledged ⇒ durable on a majority
## HA Design Constraints

202
src/controller/ha.c
View file

@ -572,8 +572,10 @@ recover_decision_t ha_promotion_gate_decision(bool recovery_truncated,
* the committed boundary regains eligibility, Req 2.6).
* - FENCE → set the fence, return FENCE (refuse promotion; catch up from a
* healthy peer).
* - FATAL → run the 3.3 loud-terminate path (logs the truncation boundary,
* then terminates — or fires persist_fatal_hook in tests).
* - FENCE_UNRECOVERABLE → run the 3.3 loud-fence action (logs LOUDLY + sets
* the sticky non-promotable flag) and ALSO set the in_catch_up
* fence. The member STAYS ALIVE, non-promotable, retrying
* catch-up — it NEVER self-exits (Req 2.4).
*/
recover_decision_t ha_evaluate_promotion_fence(ha_state_t *ha,
bool recovery_truncated,
@ -603,10 +605,11 @@ recover_decision_t ha_evaluate_promotion_fence(ha_state_t *ha,
"ha-commit-durability Req 2.2)",
recovery_truncated, durable_floor, safe_hwm);
break;
case RECOVER_DECISION_FATAL: {
/* Bounded-unrecoverable: route through the 3.3 loud-terminate action,
* which logs the truncation boundary at ERROR then terminates (or fires
* persist_fatal_hook under test). Req 2.4. */
case RECOVER_DECISION_FENCE_UNRECOVERABLE: {
/* Unrecoverable: route through the 3.3 loud-fence action — logs LOUDLY
* and sets the sticky non-promotable flag. Also hold the in_catch_up
* fence so the member refuses promotion. The process STAYS ALIVE and
* keeps retrying catch-up; it NEVER self-exits (Req 2.4). */
recover_policy_inputs_t in = {
.verdict = recovery_truncated ? RECOVER_TRUNCATED
: RECOVER_CLEAN,
@ -616,25 +619,36 @@ recover_decision_t ha_evaluate_promotion_fence(ha_state_t *ha,
.durable_floor_lsn = durable_floor,
.cluster_hwm_lsn = safe_hwm,
};
persist_recover_fatal_if_unrecoverable(&in);
persist_t *p = (ha != NULL && ha->raft_c != NULL) ? &ha->raft_c->persist
: NULL;
persist_recover_loud_fence_if_unrecoverable(p, &in);
if (ha != NULL)
atomic_store_explicit(&ha->in_catch_up, 1, memory_order_release);
break;
}
}
return d;
}
/* ── ha-commit-durability task 4.4: bounded-window catch-up before FATAL ─────
/* ── ha-commit-durability task 4.4: bounded-window catch-up re-alert ─────────
*
* The 4.3 gate above can DECIDE FATAL synchronously, but a member that won an
* election with no peer reachable AT THE PROMOTION INSTANT may simply be seeing
* a transient no-peer snapshot during aggressive failover (task-5.2 soak: a
* recovery-truncated winner self-FATAL'd immediately and repeatedly until
* quorum was lost — an availability collapse, not durable loss). Per the 3.2
* decision table the FATAL terminal must be reached ONLY after a BOUNDED
* failure: the member is fenced (in_catch_up), given DEFAULT_RECOVERY_CATCHUP_-
* WINDOW_MS to find a peer / make catch-up progress, and FATAL'd from a one-shot
* timer only if it is STILL hollow with no peer and no progress when the window
* elapses.
* (never-self-exit rework 2026-06-18.) A member that wins an election while
* recovery-truncated / behind the HWM with no peer reachable AT THE PROMOTION
* INSTANT is FENCED (in_catch_up set, non-promotable) and STAYS ALIVE, retrying
* catch-up indefinitely. This one-shot timer exists ONLY for operator
* VISIBILITY: when it fires, if the member is STILL unrecoverable (still
* truncated/behind AND no peer AND no progress) it RE-EMITS the loud,
* operator-visible alert via persist_recover_loud_fence_if_unrecoverable (which
* also keeps the sticky non-promotable flag set) — it NEVER terminates the
* process. If the member has caught up (not truncated AND durable_floor >=
* HWM) the timer RELEASES the fence (Req 2.6). If progress is possible (peer
* back, or durable floor advanced) it leaves the member fenced under the
* existing catch-up machinery.
*
* Earlier this timer DEFERRED-then-ABORTED. The abort is GONE: a controller
* MUST NEVER self-terminate as a designed response to an operational condition
* (user invariant, 2026-06-18). The task-5.2 soak quorum collapse was caused
* by that self-abort; staying alive + retrying self-heals when a peer returns.
*
* The window is a settable seam so the timer test runs fast. Default =
* DEFAULT_RECOVERY_CATCHUP_WINDOW_MS (persist.h). */
@ -684,21 +698,26 @@ void ha_cancel_promotion_fence_timer(ha_state_t *ha)
atomic_store_explicit(&t->cancelled, 1, memory_order_release);
}
/* promotion_fence_timeout_fn: the bounded-window timer callback. Owns its ctx —
* frees it on exit. Re-evaluates the LIVE recovery/durability state and reaches
* the 3.3 FATAL ONLY on a real bounded failure (Req 2.4):
* - still recovery-truncated/behind AND no healthy peer reachable AND no
* catch-up progress (durable_floor has not advanced past the baseline)
* → persist_recover_fatal_if_unrecoverable (fail loud).
/* Forward decl: the timeout callback re-arms itself for periodic re-alert. */
static void ha_arm_promotion_fence_timer(ha_state_t *ha, cluster_t *c,
bool truncated, uint64_t durable_floor,
uint64_t target_hwm);
/* promotion_fence_timeout_fn: the bounded-window re-alert timer callback. Owns
* its ctx — frees it on exit. Re-evaluates the LIVE recovery/durability state.
* It NEVER terminates the process (never-self-exit invariant, 2026-06-18):
* - caught up (NOT truncated AND durable_floor >= target_hwm) → release the
* in_catch_up fence; do NOT FATAL (Req 2.6 — eligible again).
* - a healthy peer is reachable now, OR catch-up is making progress but is not
* yet complete → leave the member fenced under the existing catch-up
* machinery and do NOT FATAL while progress is possible. The single bounded
* window has elapsed, so we do not re-arm: a member with a live peer is no
* longer in the "no peer + no progress" bounded-failure state the FATAL
* guards, and the existing catch-up / re-election path will release the
* fence (ha_evaluate_promotion_fence → ELIGIBLE) when it completes. */
* in_catch_up fence + clear the sticky unrecoverable flag; eligible again
* (Req 2.6).
* - a healthy peer is reachable now, OR catch-up is making progress → leave
* the member fenced under the existing catch-up machinery (it will release
* via ha_evaluate_promotion_fence → ELIGIBLE when it completes); do not
* re-arm the re-alert while progress is happening.
* - STILL unrecoverable (truncated/behind AND no peer AND no progress) →
* RE-EMIT the loud operator alert via persist_recover_loud_fence_if_-
* unrecoverable (keeps the sticky flag + in_catch_up set) and RE-ARM the
* timer so the alert keeps repeating until a peer returns. The process
* STAYS ALIVE and keeps retrying — it NEVER aborts. */
static int promotion_fence_timeout_fn(void *arg)
{
promotion_fence_ctx_t *t = (promotion_fence_ctx_t *)arg;
@ -721,7 +740,7 @@ static int promotion_fence_timeout_fn(void *arg)
if (!c) { free(t); return 0; } /* defensive: no persist to evaluate */
/* Re-read the LIVE state after the bounded window. */
/* Re-read the LIVE state after the window. */
bool truncated = persist_recovery_truncated(&c->persist);
uint64_t durable_floor = persist_durable_floor(&c->persist);
bool below_hwm = durable_floor < t->target_hwm;
@ -731,9 +750,11 @@ static int promotion_fence_timeout_fn(void *arg)
: false;
/* Caught up: not truncated AND durable position reached the boundary →
* release the fence; the member is eligible on its next election (Req 2.6). */
* release the fence + clear the sticky flag; the member is eligible on its
* next election (Req 2.6). */
if (!truncated && !below_hwm) {
atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release);
persist_clear_recovery_unrecoverable(&c->persist);
INFO("HA: promotion fence RELEASED after catch-up — durable_floor=%"
PRIu64 " >= cluster HWM=%" PRIu64 " (ha-commit-durability Req 2.6)",
durable_floor, t->target_hwm);
@ -742,14 +763,13 @@ static int promotion_fence_timeout_fn(void *arg)
}
/* Still hollow, but a peer is reachable or catch-up advanced the durable
* floor: progress is possible — do NOT FATAL. Leave the member fenced
* (in_catch_up stays set) under the existing catch-up machinery; the bounded
* window has elapsed so we do not re-arm. A member with a live peer is no
* longer in the no-peer bounded-failure state the FATAL guards. */
* floor: progress is possible. Leave the member fenced (in_catch_up stays
* set) under the existing catch-up machinery; do not re-arm the re-alert
* while progress is happening. */
if (peer_reachable || made_progress) {
INFO("HA: promotion fence catch-up window elapsed but progress is "
"possible (peer_reachable=%d durable_floor=%" PRIu64 " baseline=%"
PRIu64 " HWM=%" PRIu64 ") — remaining fenced, NOT FATAL "
PRIu64 " HWM=%" PRIu64 ") — remaining fenced, catching up "
"(ha-commit-durability Req 2.2)",
peer_reachable, durable_floor, t->baseline_durable_floor,
t->target_hwm);
@ -757,9 +777,11 @@ static int promotion_fence_timeout_fn(void *arg)
return 0;
}
/* Bounded failure: STILL truncated/behind AND no healthy peer AND no
* catch-up progress after the window → fail loud (Req 2.4). This is the
* ONLY path that reaches the 3.3 FATAL — never synchronously at promotion. */
/* STILL unrecoverable: truncated/behind AND no healthy peer AND no catch-up
* progress. RE-EMIT the loud operator alert and keep the member fenced +
* non-promotable + ALIVE (Req 2.4) — NEVER terminate. Re-arm the re-alert
* timer so the operator keeps hearing about it until a peer returns. */
uint64_t target_hwm = t->target_hwm; /* capture before free (UAF guard) */
{
recover_policy_inputs_t in = {
.verdict = truncated ? RECOVER_TRUNCATED
@ -768,27 +790,27 @@ static int promotion_fence_timeout_fn(void *arg)
.healthy_peer_reachable = false,
.catch_up_succeeded = false,
.durable_floor_lsn = durable_floor,
.cluster_hwm_lsn = t->target_hwm,
.cluster_hwm_lsn = target_hwm,
};
ERROR("HA: promotion catch-up window (%u ms) elapsed with no peer and "
"no progress — recovery_truncated=%d durable_floor=%" PRIu64
" < cluster HWM=%" PRIu64 "; failing loud rather than serving "
"divergent state (ha-commit-durability Req 2.4)",
ha_promotion_fence_window_ms(), truncated, durable_floor,
t->target_hwm);
persist_recover_fatal_if_unrecoverable(&in);
/* Loud, operator-visible alert + sticky non-promotable flag. STAYS
* ALIVE: no abort/exit anywhere on this path. */
persist_recover_loud_fence_if_unrecoverable(&c->persist, &in);
}
free(t);
/* Re-arm so the loud alert repeats periodically while the member remains
* unrecoverable. The member keeps retrying catch-up; the next firing
* re-evaluates and releases the fence once a peer returns and it catches up. */
ha_arm_promotion_fence_timer(ha, c, truncated, durable_floor, target_hwm);
return 0;
}
/* ha_arm_promotion_fence_timer: fence the member (set in_catch_up) and schedule
* the one-shot bounded-window FATAL timer. Cancels any previously-armed timer
* first so a re-evaluation never leaks or double-fires. Returns 0 on success.
* On a failure to schedule the timer it FATALs immediately via the 3.3 action so
* the bounded-failure terminal is never silently lost (a member that could not
* arm its own catch-up deadline must not run on indefinitely). */
* the one-shot bounded-window RE-ALERT timer. Cancels any previously-armed
* timer first so a re-evaluation never leaks or double-fires. On a failure to
* allocate/schedule the timer it emits the loud unrecoverable fence ONCE (stays
* alive, non-promotable) — it NEVER aborts (never-self-exit invariant). */
static void ha_arm_promotion_fence_timer(ha_state_t *ha, cluster_t *c,
bool truncated, uint64_t durable_floor,
uint64_t target_hwm)
@ -801,7 +823,7 @@ static void ha_arm_promotion_fence_timer(ha_state_t *ha, cluster_t *c,
promotion_fence_ctx_t *t = calloc(1, sizeof(*t));
if (!t)
goto fail_loud;
goto loud_fence_no_timer;
atomic_init(&t->cancelled, 0);
t->ha = ha;
@ -816,20 +838,22 @@ static void ha_arm_promotion_fence_timer(ha_state_t *ha, cluster_t *c,
if (rc < 0) {
WARN("ha_arm_promotion_fence_timer: delay_task_ms failed: %d", rc);
free(t);
goto fail_loud;
goto loud_fence_no_timer;
}
atomic_store_explicit(&ha->promotion_fence_ctx, t, memory_order_release);
INFO("HA: promotion FENCED + bounded catch-up window armed (%u ms) — "
INFO("HA: promotion FENCED + catch-up re-alert window armed (%u ms) — "
"recovery_truncated=%d durable_floor=%" PRIu64 " < cluster HWM=%" PRIu64
"; FATAL deferred to the timer (ha-commit-durability Req 2.2/2.4)",
"; staying ALIVE + non-promotable, retrying catch-up "
"(ha-commit-durability Req 2.2/2.4)",
window_ms, truncated, durable_floor, target_hwm);
return;
fail_loud:
loud_fence_no_timer:
{
/* Could not arm the bounded deadline — reach the FATAL terminal now
* rather than fence indefinitely. */
/* Could not arm the re-alert timer — still emit the loud unrecoverable
* fence once so the operator sees it; the member stays fenced
* (in_catch_up set above) + non-promotable + ALIVE. NO abort. */
recover_policy_inputs_t in = {
.verdict = truncated ? RECOVER_TRUNCATED
: RECOVER_CLEAN,
@ -839,7 +863,7 @@ fail_loud:
.durable_floor_lsn = durable_floor,
.cluster_hwm_lsn = target_hwm,
};
persist_recover_fatal_if_unrecoverable(&in);
persist_recover_loud_fence_if_unrecoverable(&c->persist, &in);
}
}
@ -919,10 +943,16 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
* the in_catch_up fence and return WITHOUT becoming primary; the member
* catches up from a healthy peer and a later promotion attempt passes the
* gate (Req 2.2/2.6). Only the production path (real persist) is gated; the
* c==NULL unit shim has no persist and skips it. */
* c==NULL unit shim has no persist and skips it.
*
* Belt-and-suspenders (never-self-exit rework 2026-06-18): a member that has
* been marked sticky-unrecoverable (persist_recovery_unrecoverable) must
* NEVER become primary while that flag is set — it stays alive, fenced, and
* keeps retrying catch-up. The flag clears automatically on catch-up. */
if (c != NULL) {
bool truncated = persist_recovery_truncated(&c->persist);
uint64_t durable_floor = persist_durable_floor(&c->persist);
bool unrecoverable = persist_recovery_unrecoverable(&c->persist);
/* A healthy peer is reachable for catch-up iff some voter granted with a
* live endpoint survives this election (these become the standby
@ -943,36 +973,36 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
/* catch_up_succeeded is false at the promotion-eligibility evaluation:
* a hollow member never silently proceeds to leadership. We evaluate
* the PURE 3.2/3.3 decision here and DEFER its terminal — task 4.4 does
* NOT reach the 3.3 FATAL synchronously at the promotion instant. A
* FENCE or a would-be-FATAL (truncated/behind + no peer right now) both
* fence the member (in_catch_up) and arm the bounded catch-up window;
* the FATAL is reached ONLY from that timer after a real bounded failure
* (still hollow + no peer + no progress). This avoids the task-5.2 soak
* collapse where a transient no-peer snapshot during aggressive failover
* self-FATAL'd a recovering winner repeatedly until quorum was lost. */
* the PURE 3.2/3.3 decision here. A FENCE or an UNRECOVERABLE
* (truncated/behind + no peer right now) both fence the member
* (in_catch_up) and arm the catch-up re-alert timer; the member STAYS
* ALIVE and keeps retrying — it NEVER self-exits (never-self-exit
* invariant, 2026-06-18; this is what fixes the task-5.2 soak collapse
* where a self-aborting recovering winner repeatedly dropped quorum). */
recover_decision_t gate = ha_promotion_gate_decision(
truncated, durable_floor, promotion_safe_lsn,
peer_reachable, /*catch_up_succeeded=*/false);
if (gate == RECOVER_DECISION_ELIGIBLE) {
/* ELIGIBLE: release the fence (a caught-up member regains
* eligibility, Req 2.6) and cancel any pending bounded-window FATAL
* timer — a member that is now promotable must never be FATAL'd by a
* stale timer. Proceed to authoritative promotion below. */
if (gate == RECOVER_DECISION_ELIGIBLE && !unrecoverable) {
/* ELIGIBLE and not sticky-unrecoverable: release the fence (a
* caught-up member regains eligibility, Req 2.6), clear the sticky
* flag if set, and cancel any pending re-alert timer. Proceed to
* authoritative promotion below. */
atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release);
persist_clear_recovery_unrecoverable(&c->persist);
ha_cancel_promotion_fence_timer(ha);
} else {
/* FENCE or would-be-FATAL: refuse promotion — do NOT become
* CTRL_PRIMARY. Fence the member and arm the bounded catch-up
* window; the 3.3 FATAL is deferred to the timer (Req 2.2/2.4).
* Never FATAL synchronously, never fence indefinitely. */
WARN("HA: promotion REFUSED (decision=%d) — recovery_truncated=%d "
"durable_floor=%" PRIu64 " cluster_hwm=%" PRIu64
" peer_reachable=%d; fencing + arming bounded catch-up window "
/* FENCE, UNRECOVERABLE, or sticky-unrecoverable: refuse promotion —
* do NOT become CTRL_PRIMARY. Fence the member and arm the catch-up
* re-alert timer. The member STAYS ALIVE, non-promotable, retrying
* catch-up (Req 2.2/2.4). NEVER self-exit, never fence silently. */
WARN("HA: promotion REFUSED (decision=%d unrecoverable=%d) — "
"recovery_truncated=%d durable_floor=%" PRIu64
" cluster_hwm=%" PRIu64 " peer_reachable=%d; fencing + arming "
"catch-up re-alert window, staying ALIVE "
"(ha-commit-durability Req 2.2/2.4)",
(int)gate, truncated, durable_floor, promotion_safe_lsn,
peer_reachable);
(int)gate, unrecoverable, truncated, durable_floor,
promotion_safe_lsn, peer_reachable);
ha_arm_promotion_fence_timer(ha, c, truncated, durable_floor,
promotion_safe_lsn);
return;

55
src/controller/ha.h
View file

@ -520,7 +520,10 @@ void ha_compute_safe_committed_lsn(ha_elect_ctx_t *ctx,
* 2.5, 2.6). Maps the promoting member's recovery/durability state to the Boot
* Policy Decision Table outcome by building recover_policy_inputs_t and
* delegating to persist_recover_decision (DRY — the 3.2/3.3 decision table is
* the single source of truth; this gate does not re-derive it):
* the single source of truth; this gate does not re-derive it). Note the
* FENCE_UNRECOVERABLE terminal is a LOUD PERMANENT NON-promotable fence that
* STAYS ALIVE and keeps retrying catch-up — it is NEVER process exit
* (never-self-exit invariant, 2026-06-18):
* - recovery_truncated → the task-3.1 structural-corruption flag.
* - durable_floor → the local CLAMPED durable position (read AFTER the
* promotion-path persist_truncate_wal_above so the
@ -547,9 +550,12 @@ recover_decision_t ha_promotion_gate_decision(bool recovery_truncated,
* boundary regains eligibility — Req 2.6).
* - FENCE → sets the in_catch_up fence (refuse promotion; catch up from a
* healthy peer) and returns FENCE.
* - FATAL → routes through persist_recover_fatal_if_unrecoverable (logs the
* truncation boundary at ERROR then terminates; the test seam
* persist_fatal_hook captures it) and returns FATAL.
* - FENCE_UNRECOVERABLE → routes through
* persist_recover_loud_fence_if_unrecoverable (logs LOUDLY at
* ERROR + sets the sticky persist recovery_unrecoverable flag),
* also sets the in_catch_up fence, and returns
* FENCE_UNRECOVERABLE. The process STAYS ALIVE, non-promotable,
* retrying catch-up — it NEVER self-exits (Req 2.4).
* Reuses the existing in_catch_up fence — does NOT invent a parallel fence. */
recover_decision_t ha_evaluate_promotion_fence(ha_state_t *ha,
bool recovery_truncated,
@ -558,34 +564,37 @@ recover_decision_t ha_evaluate_promotion_fence(ha_state_t *ha,
bool peer_reachable,
bool catch_up_succeeded);
/* ── ha-commit-durability task 4.4: bounded-window catch-up before FATAL ─────
/* ── ha-commit-durability task 4.4: catch-up re-alert window ─────────────────
* (never-self-exit rework 2026-06-18.)
*
* The promotion gate (4.3) no longer reaches the 3.3 FATAL synchronously at the
* promotion instant. When a recovery-truncated / behind member wins an election
* with no reachable peer AT THAT INSTANT (a transient no-peer snapshot during
* aggressive failover), it FENCES (sets in_catch_up, does NOT become primary)
* and arms a one-shot delaymgr timer for DEFAULT_RECOVERY_CATCHUP_WINDOW_MS.
* The 3.3 FATAL is reached ONLY from that timer, and ONLY if the member is STILL
* recovery-truncated/behind AND no healthy peer is reachable AND no catch-up
* progress was made when it fires (Req 2.4). Catch-up that brings durable_floor
* >= HWM releases the fence and cancels the timer (Req 2.6); shutdown cancels it
* too — never a synchronous FATAL, never an indefinite fence.
* When a recovery-truncated / behind member wins an election with no reachable
* peer AT THAT INSTANT (a transient no-peer snapshot during aggressive
* failover), it FENCES (sets in_catch_up, does NOT become primary) and arms a
* delaymgr re-alert timer for DEFAULT_RECOVERY_CATCHUP_WINDOW_MS. When the
* timer fires, if the member is STILL recovery-truncated/behind AND no healthy
* peer is reachable AND no catch-up progress was made, it RE-EMITS the loud
* operator alert (persist_recover_loud_fence_if_unrecoverable) and RE-ARMS —
* the process STAYS ALIVE and keeps retrying catch-up INDEFINITELY. It NEVER
* terminates: a controller must never self-exit as a designed response to an
* operational condition. Catch-up that brings durable_floor >= HWM releases
* the fence + clears the sticky flag and stops the re-alert (Req 2.6); shutdown
* cancels it too.
*
* ha_promotion_fence_window_ms: the bounded catch-up window in milliseconds.
* ha_promotion_fence_window_ms: the re-alert interval in milliseconds.
* Defaults to DEFAULT_RECOVERY_CATCHUP_WINDOW_MS; tests override it (via
* ha_set_promotion_fence_window_ms) to a small value so the timer fires fast. */
uint32_t ha_promotion_fence_window_ms(void);
/* ha_cancel_promotion_fence_timer: cancel a pending bounded-window FATAL timer
* (idempotent; no-op when none is armed). Called when the member catches up
* (fence released) or on shutdown, so neither a caught-up nor a stopped member
* ever reaches the deferred FATAL. */
/* ha_cancel_promotion_fence_timer: cancel a pending re-alert timer (idempotent;
* no-op when none is armed). Called when the member catches up (fence released)
* or on shutdown, so a stopped member stops re-alerting. It never gated a
* process exit — the timer never terminates the process. */
void ha_cancel_promotion_fence_timer(ha_state_t *ha);
#ifdef XCE_TESTING
/* Test seam: override the bounded catch-up window so the deferred-FATAL timer
* fires deterministically fast. 0 restores the DEFAULT_RECOVERY_CATCHUP_WINDOW_MS
* default. XCE_TESTING only — compiled out of production. */
/* Test seam: override the re-alert window so the timer fires deterministically
* fast. 0 restores the DEFAULT_RECOVERY_CATCHUP_WINDOW_MS default. XCE_TESTING
* only — compiled out of production. */
void ha_set_promotion_fence_window_ms(uint32_t ms);
#endif

121
src/controller/persist.c
View file

@ -449,6 +449,11 @@ int persist_init(persist_t* p, persist_config_t* cfg, cluster_t* c)
* replay outcome (RECOVER_TRUNCATED only on a structural log corruption). */
p->recovery_verdict = RECOVER_CLEAN;
/* Sticky "unrecoverable, loudly fenced" flag starts clear (Req 2.4): a fresh
* store is promotable. Set by persist_recover_loud_fence_if_unrecoverable
* on an unrecoverable boot; cleared on catch-up (never self-exits). */
atomic_store(&p->recovery_unrecoverable, false);
/* Durable-advance wake hook starts unregistered (ha-commit-durability 2.2):
* the commit engine registers it in commit_engine_init. Until then every
* flusher durable advance is a no-op wake. */
@ -1700,21 +1705,25 @@ bool persist_recovery_truncated(persist_t* p)
}
/* ------------------------------------------------------------------ *
* Boot-policy FATAL decision (ha-commit-durability task 3.3, Req 2.4).
* Boot-policy loud-fence decision (ha-commit-durability task 3.3, Req 2.4;
* never-self-exit rework 2026-06-18).
*
* Implements the FATAL terminal of the Boot Policy Decision Table
* (design.md). The decision is a PURE function of its inputs; the FATAL
* action (loud log + terminate) is separated into
* persist_recover_fatal_if_unrecoverable so the decision can be unit-tested
* without aborting. A reachable healthy peer NEVER FATALs (rows 3, 5);
* FATAL is reserved for a member that cannot recover locally AND cannot catch
* up within the bounded window (rows 4, 4b) — never a silent indefinite fence.
* Implements the FENCE_UNRECOVERABLE terminal of the Boot Policy Decision Table
* (design.md). The decision is a PURE function of its inputs; the action (loud
* operator-visible log + set the sticky non-promotable flag) is separated into
* persist_recover_loud_fence_if_unrecoverable so the decision can be unit-tested
* in isolation. A reachable healthy peer NEVER yields FENCE_UNRECOVERABLE
* (rows 3, 5); it is reserved for a member that cannot recover locally AND
* cannot catch up (no healthy peer, rows 4, 4b).
*
* CRITICAL: this terminal is a LOUD, PERMANENT, NON-promotable fence — the
* process STAYS ALIVE and keeps retrying catch-up. There is NO abort()/exit():
* a controller MUST NEVER self-terminate as a designed response to an
* operational condition (user invariant, 2026-06-18). A self-exited controller
* needs manual admin recovery and degrades quorum meanwhile; one that stays
* alive and keeps retrying self-heals when a peer returns (Req 2.6).
* ------------------------------------------------------------------ */
/* FATAL seam: NULL in production (real abort); a test may install a hook to
* capture the FATAL path without dying. Mirrors ha_fatal_hook. */
void (*persist_fatal_hook)(const char* msg) = NULL;
recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in)
{
if (in == NULL)
@ -1723,7 +1732,7 @@ recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in)
bool truncated = (in->verdict == RECOVER_TRUNCATED);
/* Rows 1 & 6: a structurally CLEAN log that is NOT behind the cluster
* high-water mark is a normal promotable member — no fence, no FATAL. */
* high-water mark is a normal promotable member — no fence. */
if (!truncated && !in->durable_below_hwm)
return RECOVER_DECISION_ELIGIBLE;
@ -1732,63 +1741,83 @@ recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in)
* (design.md note), so it is treated as behind for fence purposes. */
/* Rows 2, 3, 5: a healthy peer is reachable. Catch up from it and regain
* eligibility (task 4.3). A reachable peer NEVER yields FATAL — even for a
* truncated log or a legitimate snapshot-sentinel. If catch-up has already
* succeeded the member is eligible; otherwise it stays fenced (4.3 drives
* the catch-up + release; this task only refuses FATAL while a peer
* exists). */
* eligibility (task 4.3). A reachable peer NEVER yields the unrecoverable
* fence — even for a truncated log or a legitimate snapshot-sentinel. If
* catch-up has already succeeded the member is eligible; otherwise it stays
* fenced (4.3 drives the catch-up + release). */
if (in->healthy_peer_reachable) {
if (in->catch_up_succeeded)
return RECOVER_DECISION_ELIGIBLE;
return RECOVER_DECISION_FENCE;
}
/* No healthy peer. The bounded catch-up window is the member's only path to
* a committed-consistent state; if it did not reach one, the member can
* neither recover locally nor catch up — rows 4 & 4b: FATAL (fail loud),
* never a silent indefinite fence (Req 2.4). */
/* No healthy peer. If catch-up did not reach a committed-consistent state,
* the member can neither recover locally nor catch up right now — rows 4 &
* 4b: a LOUD, PERMANENT, NON-promotable fence (NOT process exit). It stays
* alive and keeps retrying; the fence releases when a peer returns
* (Req 2.4/2.6). */
if (!in->catch_up_succeeded)
return RECOVER_DECISION_FATAL;
return RECOVER_DECISION_FENCE_UNRECOVERABLE;
/* No peer but catch-up is reported succeeded (e.g. already at/above after a
* local catch-up): eligible. Defensive — the boot path reaches the
* no-peer branch with catch_up_succeeded=false in the bounded-failure case
* this task owns. */
* local catch-up): eligible. */
return RECOVER_DECISION_ELIGIBLE;
}
bool persist_recovery_unrecoverable(persist_t* p)
{
return p && atomic_load_explicit(&p->recovery_unrecoverable,
memory_order_acquire);
}
void persist_clear_recovery_unrecoverable(persist_t* p)
{
if (!p)
return;
/* Clear only on a genuine transition so the loud "fence released" log fires
* once, not on every catch-up tick. */
if (atomic_exchange_explicit(&p->recovery_unrecoverable, false,
memory_order_acq_rel)) {
INFO("persist_recover: unrecoverable fence RELEASED — member reached a "
"committed-consistent state via catch-up; promotable again "
"(ha-commit-durability Req 2.6)");
}
}
recover_decision_t
persist_recover_fatal_if_unrecoverable(const recover_policy_inputs_t* in)
persist_recover_loud_fence_if_unrecoverable(persist_t* p,
const recover_policy_inputs_t* in)
{
recover_decision_t decision = persist_recover_decision(in);
if (decision != RECOVER_DECISION_FATAL)
if (decision != RECOVER_DECISION_FENCE_UNRECOVERABLE)
return decision;
/* Fail loud: log the truncation boundary (durable_floor_lsn) and the
* cluster high-water mark so an operator sees exactly where the local log
* stops short of the cluster's committed state (Req 2.4). */
/* Loud, operator-visible alert: log the truncation boundary
* (durable_floor_lsn) and the cluster high-water mark so an operator sees
* exactly where the local log stops short of the cluster's committed state.
* The member STAYS ALIVE, stays non-promotable, and keeps retrying catch-up
* (Req 2.4) — it NEVER self-exits. */
const char* why = (in->verdict == RECOVER_TRUNCATED)
? "recovery-truncated log (structural corruption)"
: "log behind the cluster committed high-water mark";
ERROR("persist_recover: FATAL — %s and no healthy peer reachable to catch "
"up within the bounded window (%u ms); truncation boundary "
"durable_floor_lsn=%" PRIu64 " < cluster committed high-water "
"hwm=%" PRIu64 " — terminating rather than serving divergent state "
ERROR("persist_recover: UNRECOVERABLE — %s and no healthy peer reachable to "
"catch up; truncation boundary durable_floor_lsn=%" PRIu64
" < cluster committed high-water hwm=%" PRIu64 ". Staying ALIVE and "
"PERMANENTLY non-promotable (refusing to serve divergent state) and "
"RETRYING catch-up indefinitely — will self-heal when a healthy peer "
"returns; operator attention may be needed "
"(ha-commit-durability Req 2.4)",
why, (unsigned)DEFAULT_RECOVERY_CATCHUP_WINDOW_MS,
in->durable_floor_lsn, in->cluster_hwm_lsn);
why, in->durable_floor_lsn, in->cluster_hwm_lsn);
if (persist_fatal_hook != NULL) {
/* Test seam: capture the FATAL decision + boundary log without dying.
* Production leaves the hook NULL and aborts below. */
persist_fatal_hook("persist_recover: bounded-unrecoverable (FATAL)");
return RECOVER_DECISION_FATAL;
}
/* Set the sticky non-promotable flag so the promotion gate refuses
* leadership as belt-and-suspenders until catch-up clears it (Req 2.4/2.6).
* p may be NULL for pure-classification callers that only want the loud log. */
if (p != NULL)
atomic_store_explicit(&p->recovery_unrecoverable, true,
memory_order_release);
tlog_exit(); /* flush the async log buffer before dying */
abort(); /* NEVER returns in production */
return RECOVER_DECISION_FATAL; /* unreachable */
return RECOVER_DECISION_FENCE_UNRECOVERABLE;
}
/* Advance committed_lsn to `written_lsn` on a majority ACK — but ONLY if that

390
src/controller/testsuite/test_commit_durability.c
View file

@ -1587,19 +1587,36 @@ static void test_durable_floor_snapshot_sentinel_tip(void)
* failure. (Req 2.2, 2.4) *
* ------------------------------------------------------------------ */
/* task 3.3 — FATAL-path capture seam. The production FATAL terminates via
* abort(); a test installs persist_fatal_hook so the FATAL decision + boundary
* log can be asserted without the process dying. */
static int g_fatal_hook_calls = 0;
static char g_fatal_hook_msg[256];
static void dur_fatal_hook(const char* msg)
/* task 3.3 — unrecoverable loud-fence verification (never-self-exit rework
* 2026-06-18). The OLD behavior installed persist_fatal_hook to capture an
* abort()/exit() without dying. That self-exit terminal is GONE: an
* unrecoverable member STAYS ALIVE and is held permanently non-promotable.
* There is no hook to install — the tests assert the OBSERVABLE
* post-conditions instead:
* - the pure decision returns RECOVER_DECISION_FENCE_UNRECOVERABLE,
* - persist_recover_loud_fence_if_unrecoverable RETURNS (no abort — the test
* simply continuing past the call proves the process stayed alive),
* - the sticky persist_recovery_unrecoverable flag is SET (member is
* permanently non-promotable until catch-up),
* - and the flag CLEARS on catch-up (Req 2.6).
* dur_persist_for_flag builds a real persist_t the loud-fence helper can set
* the sticky flag on. */
static persist_t* dur_flag_persist(const char* tmpdir)
{
g_fatal_hook_calls++;
if (msg != NULL) {
strncpy(g_fatal_hook_msg, msg, sizeof(g_fatal_hook_msg) - 1);
g_fatal_hook_msg[sizeof(g_fatal_hook_msg) - 1] = '\0';
}
persist_t* p = calloc(1, sizeof(*p));
if (!p) return NULL;
persist_config_t cfg = persist_config_defaults();
cfg.wal_sync_ms = 5000;
persist_set_workdir(p, tmpdir, "ctrl-flag");
if (persist_init(p, &cfg, NULL) != 0) { free(p); return NULL; }
return p;
}
static void dur_free_flag_persist(persist_t* p)
{
if (!p) return;
persist_fini(p);
free(p);
}
/* task 3.3 — peer-reachable NEVER FATALs (the never-FATAL aspect of rows
@ -1613,10 +1630,13 @@ static void dur_fatal_hook(const char* msg)
* 2.4, design.md "never FATAL when a peer is reachable".) */
static void test_boot_policy_peer_reachable_never_fatal(void)
{
g_fatal_hook_calls = 0;
persist_fatal_hook = dur_fatal_hook;
char tmp[256];
strcpy(tmp, "/tmp/xce_dur_peerfence_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
persist_t* p = dur_flag_persist(tmp);
TEST_ASSERT_NOT_NULL(p);
/* Row 3: TRUNCATED + healthy peer (catch-up not yet done) -> FENCE, no FATAL. */
/* Row 3: TRUNCATED + healthy peer (catch-up not yet done) -> FENCE. */
recover_policy_inputs_t truncated_peer = {
.verdict = RECOVER_TRUNCATED,
.durable_below_hwm = true,
@ -1628,13 +1648,13 @@ static void test_boot_policy_peer_reachable_never_fatal(void)
TEST_ASSERT_EQUAL_INT_MESSAGE(
RECOVER_DECISION_FENCE, persist_recover_decision(&truncated_peer),
"row 3: a truncated member with a reachable healthy peer must FENCE "
"(catch up), NEVER FATAL");
"(catch up), NEVER the unrecoverable terminal");
TEST_ASSERT_EQUAL_INT(
RECOVER_DECISION_FENCE,
persist_recover_fatal_if_unrecoverable(&truncated_peer));
persist_recover_loud_fence_if_unrecoverable(p, &truncated_peer));
/* Row 5: snapshot-sentinel CLEAN-but-behind + healthy peer -> FENCE, no
* FATAL. A legitimate sentinel is CLEAN-but-behind, not corruption. */
/* Row 5: snapshot-sentinel CLEAN-but-behind + healthy peer -> FENCE.
* A legitimate sentinel is CLEAN-but-behind, not corruption. */
recover_policy_inputs_t sentinel_peer = {
.verdict = RECOVER_CLEAN,
.durable_below_hwm = true,
@ -1646,18 +1666,19 @@ static void test_boot_policy_peer_reachable_never_fatal(void)
TEST_ASSERT_EQUAL_INT_MESSAGE(
RECOVER_DECISION_FENCE, persist_recover_decision(&sentinel_peer),
"row 5: a snapshot-sentinel CLEAN-but-behind member with a reachable "
"peer must FENCE, NEVER FATAL");
"peer must FENCE, NEVER the unrecoverable terminal");
TEST_ASSERT_EQUAL_INT(
RECOVER_DECISION_FENCE,
persist_recover_fatal_if_unrecoverable(&sentinel_peer));
persist_recover_loud_fence_if_unrecoverable(p, &sentinel_peer));
/* The FATAL terminate path must NOT have fired for either peer-reachable
* case — a reachable healthy peer never triggers FATAL. */
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, g_fatal_hook_calls,
"peer-reachable rows must NEVER fire the FATAL terminate path");
/* The unrecoverable sticky flag must NOT have been set for either
* peer-reachable case — a reachable healthy peer never reaches the
* unrecoverable terminal. */
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(p),
"peer-reachable rows must NEVER set the unrecoverable sticky flag");
persist_fatal_hook = NULL;
dur_free_flag_persist(p);
printf("test_boot_policy_peer_reachable_never_fatal: PASS\n");
}
@ -1763,9 +1784,6 @@ static void test_boot_policy_row3_truncated_peer_catchup_eligible(void)
ha_config_t cfg = dur_ha_cfg();
TEST_ASSERT_EQUAL_INT(0, ha_init(&ha, &cfg, 0));
g_fatal_hook_calls = 0;
persist_fatal_hook = dur_fatal_hook;
/* Truncated boundary at 3, cluster HWM 10, peer reachable, not caught up. */
recover_decision_t d1 = ha_evaluate_promotion_fence(
&ha, /*truncated=*/true, /*durable_floor=*/3, /*safe_hwm=*/10,
@ -1773,13 +1791,10 @@ static void test_boot_policy_row3_truncated_peer_catchup_eligible(void)
TEST_ASSERT_EQUAL_INT_MESSAGE(
RECOVER_DECISION_FENCE, d1,
"row 3: a recovery-truncated member with a healthy peer must FENCE "
"(denied promotion), never FATAL");
"(denied promotion), never the unrecoverable terminal");
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, atomic_load_explicit(&ha.in_catch_up, memory_order_acquire),
"row 3: a truncated member must be fenced (in_catch_up set)");
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, g_fatal_hook_calls,
"row 3: FATAL must NOT fire while a healthy peer is reachable");
/* After catch-up to/above the committed boundary -> ELIGIBLE, fence released. */
recover_decision_t d2 = ha_evaluate_promotion_fence(
@ -1791,28 +1806,29 @@ static void test_boot_policy_row3_truncated_peer_catchup_eligible(void)
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, atomic_load_explicit(&ha.in_catch_up, memory_order_acquire),
"row 3: regained eligibility releases the fence (Req 2.6)");
TEST_ASSERT_EQUAL_INT(0, g_fatal_hook_calls);
persist_fatal_hook = NULL;
ha_fini(&ha);
printf("test_boot_policy_row3_truncated_peer_catchup_eligible: PASS\n");
}
/* Row 4: TRUNCATED + below HWM + NO healthy peer + catch-up fails
* (bounded window elapses) -> FATAL.
* Cannot recover locally AND cannot catch up within the bounded window:
* fail loud rather than serve divergent state. WILL ASSERT (3.3): the
* member terminates loudly (FATAL) with the truncation boundary logged
* when the bounded catch-up window elapses without a committed-consistent
* state. (Req 2.4) */
static void test_boot_policy_row4_truncated_no_peer_bounded_fatal(void)
* -> LOUD PERMANENT FENCE (non-promotable, STAYS ALIVE, keeps retrying), and
* RELEASES on catch-up (Req 2.4, 2.6).
*
* never-self-exit rework (2026-06-18): the member cannot recover locally AND
* cannot catch up right now. It does NOT self-exit; it logs LOUDLY, sets the
* sticky non-promotable flag, and stays alive. The flag clears once a peer
* returns and it catches up. RED→GREEN: pre-rework
* persist_recover_fatal_if_unrecoverable aborted (or fired the abort-capture
* hook); post-rework persist_recover_loud_fence_if_unrecoverable RETURNS — the
* test continuing past the call IS the proof the process stayed alive. */
static void test_boot_policy_row4_truncated_no_peer_loud_fence(void)
{
/* Row 4: TRUNCATED + below HWM + NO healthy peer + bounded catch-up fails
* -> FATAL. Cannot recover locally AND cannot catch up within the bounded
* window: fail loud with the truncation boundary logged (Req 2.4). */
g_fatal_hook_calls = 0;
g_fatal_hook_msg[0] = '\0';
persist_fatal_hook = dur_fatal_hook;
char tmp[256];
strcpy(tmp, "/tmp/xce_dur_row4_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
persist_t* p = dur_flag_persist(tmp);
TEST_ASSERT_NOT_NULL(p);
recover_policy_inputs_t in = {
.verdict = RECOVER_TRUNCATED,
@ -1823,39 +1839,49 @@ static void test_boot_policy_row4_truncated_no_peer_bounded_fatal(void)
.cluster_hwm_lsn = 12, /* cluster committed high-water */
};
/* Pure decision returns FATAL. */
/* Pure decision returns the non-terminating unrecoverable fence. */
TEST_ASSERT_EQUAL_INT_MESSAGE(
RECOVER_DECISION_FATAL, persist_recover_decision(&in),
"row 4: truncated + below + no peer + bounded catch-up fails must "
"decide FATAL");
RECOVER_DECISION_FENCE_UNRECOVERABLE, persist_recover_decision(&in),
"row 4: truncated + below + no peer + no catch-up must decide the LOUD "
"PERMANENT FENCE — NOT process exit");
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(p),
"row 4: the sticky flag is set by the ACTION helper, not the pure "
"decision");
/* The action helper routes FATAL through the hook (no abort in test) and
* still returns FATAL — proving the loud-terminate path was taken. */
recover_decision_t d = persist_recover_fatal_if_unrecoverable(&in);
TEST_ASSERT_EQUAL_INT(RECOVER_DECISION_FATAL, d);
/* The action helper logs LOUDLY, sets the sticky flag, and RETURNS (no
* abort/exit — the process stays alive; the test reaching the next line is
* the proof). */
recover_decision_t d = persist_recover_loud_fence_if_unrecoverable(p, &in);
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, g_fatal_hook_calls,
"row 4: the FATAL terminate path must fire exactly once (the "
"truncation boundary is logged at ERROR just before it)");
RECOVER_DECISION_FENCE_UNRECOVERABLE, d,
"row 4: the loud-fence action returns FENCE_UNRECOVERABLE (stays alive)");
TEST_ASSERT_TRUE_MESSAGE(
persist_recovery_unrecoverable(p),
"row 4: the member is held PERMANENTLY non-promotable (sticky flag set)");
persist_fatal_hook = NULL;
printf("test_boot_policy_row4_truncated_no_peer_bounded_fatal: PASS\n");
/* Req 2.6: once catch-up reaches a committed-consistent state the fence
* releases automatically — the flag clears, the member is promotable. */
persist_clear_recovery_unrecoverable(p);
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(p),
"row 4: catch-up CLEARS the unrecoverable fence (Req 2.6)");
dur_free_flag_persist(p);
printf("test_boot_policy_row4_truncated_no_peer_loud_fence: PASS\n");
}
/* Row 4b: CLEAN + below HWM + NO healthy peer + catch-up fails (bounded
* window elapses) -> FATAL.
* Behind the HWM with no peer to advance from within the bounded window:
* same bounded-failure terminal as row 4 — never a silent indefinite
* fence. WILL ASSERT (3.3): a behind member with no reachable peer FATALs
* once the bounded window elapses. (Req 2.4, 2.2) */
static void test_boot_policy_row4b_clean_below_no_peer_bounded_fatal(void)
/* Row 4b: CLEAN + below HWM + NO healthy peer + catch-up fails
* -> LOUD PERMANENT FENCE (non-promotable, STAYS ALIVE, keeps retrying).
* Behind the HWM with no peer to advance from — same non-terminating loud-fence
* terminal as row 4. NEVER self-exits. (Req 2.4, 2.2, 2.6) */
static void test_boot_policy_row4b_clean_below_no_peer_loud_fence(void)
{
/* Row 4b: CLEAN + below HWM + NO healthy peer + bounded catch-up fails
* -> FATAL. Behind the HWM with no peer to advance from within the bounded
* window — same bounded-failure terminal as row 4 (Req 2.4, 2.2). */
g_fatal_hook_calls = 0;
g_fatal_hook_msg[0] = '\0';
persist_fatal_hook = dur_fatal_hook;
char tmp[256];
strcpy(tmp, "/tmp/xce_dur_row4b_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
persist_t* p = dur_flag_persist(tmp);
TEST_ASSERT_NOT_NULL(p);
recover_policy_inputs_t in = {
.verdict = RECOVER_CLEAN, /* structurally intact... */
@ -1867,19 +1893,24 @@ static void test_boot_policy_row4b_clean_below_no_peer_bounded_fatal(void)
};
TEST_ASSERT_EQUAL_INT_MESSAGE(
RECOVER_DECISION_FATAL, persist_recover_decision(&in),
"row 4b: clean-but-behind + no peer + bounded catch-up fails must "
"decide FATAL — never a silent indefinite fence");
RECOVER_DECISION_FENCE_UNRECOVERABLE, persist_recover_decision(&in),
"row 4b: clean-but-behind + no peer + no catch-up must decide the LOUD "
"PERMANENT FENCE — never process exit, never a silent fence");
recover_decision_t d = persist_recover_fatal_if_unrecoverable(&in);
TEST_ASSERT_EQUAL_INT(RECOVER_DECISION_FATAL, d);
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, g_fatal_hook_calls,
"row 4b: a behind member with no reachable peer must FATAL once the "
"bounded window elapses (boundary logged before terminate)");
recover_decision_t d = persist_recover_loud_fence_if_unrecoverable(p, &in);
TEST_ASSERT_EQUAL_INT(RECOVER_DECISION_FENCE_UNRECOVERABLE, d);
TEST_ASSERT_TRUE_MESSAGE(
persist_recovery_unrecoverable(p),
"row 4b: a behind member with no peer is held non-promotable + ALIVE "
"(sticky flag set; boundary logged loudly)");
persist_fatal_hook = NULL;
printf("test_boot_policy_row4b_clean_below_no_peer_bounded_fatal: PASS\n");
persist_clear_recovery_unrecoverable(p);
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(p),
"row 4b: catch-up CLEARS the unrecoverable fence (Req 2.6)");
dur_free_flag_persist(p);
printf("test_boot_policy_row4b_clean_below_no_peer_loud_fence: PASS\n");
}
/* Row 5: CLEAN snapshot-sentinel (committed at sentinel LSN, NO
@ -1901,9 +1932,6 @@ static void test_boot_policy_row5_snapshot_sentinel_catchup_never_fatal(void)
ha_config_t cfg = dur_ha_cfg();
TEST_ASSERT_EQUAL_INT(0, ha_init(&ha, &cfg, 0));
g_fatal_hook_calls = 0;
persist_fatal_hook = dur_fatal_hook;
/* Sentinel at LSN 5, cluster has advanced to HWM 12, peer reachable. */
recover_decision_t d1 = ha_evaluate_promotion_fence(
&ha, /*truncated=*/false, /*durable_floor=*/5, /*safe_hwm=*/12,
@ -1911,10 +1939,7 @@ static void test_boot_policy_row5_snapshot_sentinel_catchup_never_fatal(void)
TEST_ASSERT_EQUAL_INT_MESSAGE(
RECOVER_DECISION_FENCE, d1,
"row 5: a snapshot-sentinel CLEAN-but-behind member must FENCE, never "
"FATAL while a peer is reachable");
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, g_fatal_hook_calls,
"row 5: a legitimate sentinel must NEVER trigger FATAL");
"the unrecoverable terminal while a peer is reachable");
recover_decision_t d2 = ha_evaluate_promotion_fence(
&ha, /*truncated=*/false, /*durable_floor=*/12, /*safe_hwm=*/12,
@ -1922,9 +1947,7 @@ static void test_boot_policy_row5_snapshot_sentinel_catchup_never_fatal(void)
TEST_ASSERT_EQUAL_INT_MESSAGE(
RECOVER_DECISION_ELIGIBLE, d2,
"row 5: after catch-up the sentinel member is eligible (Req 2.6)");
TEST_ASSERT_EQUAL_INT(0, g_fatal_hook_calls);
persist_fatal_hook = NULL;
ha_fini(&ha);
printf("test_boot_policy_row5_snapshot_sentinel_catchup_never_fatal: PASS\n");
}
@ -1943,9 +1966,6 @@ static void test_boot_policy_row6_empty_first_boot_eligible(void)
ha_config_t cfg = dur_ha_cfg();
TEST_ASSERT_EQUAL_INT(0, ha_init(&ha, &cfg, 0));
g_fatal_hook_calls = 0;
persist_fatal_hook = dur_fatal_hook;
recover_decision_t d = ha_evaluate_promotion_fence(
&ha, /*truncated=*/false, /*durable_floor=*/0, /*safe_hwm=*/0,
/*peer_reachable=*/false, /*catch_up_succeeded=*/false);
@ -1955,11 +1975,7 @@ static void test_boot_policy_row6_empty_first_boot_eligible(void)
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, atomic_load_explicit(&ha.in_catch_up, memory_order_acquire),
"row 6: a cold-start member must not be fenced");
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, g_fatal_hook_calls,
"row 6: a cold-start member must never FATAL");
persist_fatal_hook = NULL;
ha_fini(&ha);
printf("test_boot_policy_row6_empty_first_boot_eligible: PASS\n");
}
@ -1968,13 +1984,14 @@ static void test_boot_policy_row6_empty_first_boot_eligible(void)
* A hollow node (recovery-truncated OR durable below the HWM) must be refused
* promotion regardless of quorum size — combined with the 4.2 freshness floor
* it can neither win the vote nor self-promote. At minimal quorum with NO
* healthy peer, a hollow node FATALs (bounded-unrecoverable) rather than
* silently leading; with a peer it FENCEs. Either way it never becomes
* ELIGIBLE while hollow. */
* healthy peer, a hollow node is held in the LOUD PERMANENT FENCE
* (FENCE_UNRECOVERABLE, stays alive, non-promotable) rather than silently
* leading; with a peer it FENCEs. Either way it never becomes ELIGIBLE while
* hollow. */
static void test_promotion_gate_hollow_refused_at_minimal_quorum(void)
{
/* Truncated + below HWM + no peer (minimal/degraded quorum) -> NEVER
* ELIGIBLE (FATAL — fail loud rather than promote a hollow node). */
* ELIGIBLE (loud permanent fence — never promote a hollow node). */
TEST_ASSERT_NOT_EQUAL_INT_MESSAGE(
RECOVER_DECISION_ELIGIBLE,
ha_promotion_gate_decision(/*truncated=*/true, /*durable_floor=*/3,
@ -1983,7 +2000,7 @@ static void test_promotion_gate_hollow_refused_at_minimal_quorum(void)
"a recovery-truncated node must never be ELIGIBLE at minimal quorum "
"(Req 2.5)");
/* CLEAN but below HWM + no peer -> NEVER ELIGIBLE (FATAL). */
/* CLEAN but below HWM + no peer -> NEVER ELIGIBLE (loud permanent fence). */
TEST_ASSERT_NOT_EQUAL_INT_MESSAGE(
RECOVER_DECISION_ELIGIBLE,
ha_promotion_gate_decision(/*truncated=*/false, /*durable_floor=*/5,
@ -2003,19 +2020,22 @@ static void test_promotion_gate_hollow_refused_at_minimal_quorum(void)
printf("test_promotion_gate_hollow_refused_at_minimal_quorum: PASS\n");
}
/* ── task 4.4: bounded-window catch-up before FATAL ──────────────────────────
/* ── task 4.4: catch-up re-alert window (never-self-exit rework 2026-06-18) ───
*
* The 4.3 gate decided FATAL synchronously at the promotion instant when a
* recovery-truncated/behind member had no reachable peer right then. Task 4.4
* defers that FATAL to a one-shot bounded-window timer: the member is fenced
* (in_catch_up) and given DEFAULT_RECOVERY_CATCHUP_WINDOW_MS to find a peer /
* make progress; the 3.3 FATAL is reached ONLY from the timer after a real
* bounded failure (still hollow + no peer + no progress).
* A recovery-truncated/behind member with no reachable peer at the promotion
* instant is FENCED (in_catch_up) and STAYS ALIVE, retrying catch-up. A
* delaymgr re-alert timer fires after DEFAULT_RECOVERY_CATCHUP_WINDOW_MS: if the
* member is still unrecoverable it RE-EMITS the loud operator alert (sets the
* sticky persist_recovery_unrecoverable flag) and the process STAYS ALIVE —
* there is NO abort/exit anywhere. Catch-up releases the fence + clears the
* flag (Req 2.6); shutdown/cancel stops the re-alert.
*
* These tests drive ha_arm_promotion_fence_timer (the production deferral wired
* into raft_cb_become_leader) through a REAL delaymgr with a short window seam,
* so the timer fires fast and deterministically. A minimal cluster_t carries
* the delaymgr + persist + ha exactly as production does. */
* the delaymgr + persist + ha exactly as production does. They assert the
* member STAYS ALIVE + non-promotable (sticky flag) instead of asserting an
* abort/hook — the self-exit terminal is GONE. */
/* The delaymgr runner used by production (run_delay_tasks in cluster.c) is not
* linked into the unit-test target; provide the same trivial pass-through. */
@ -2084,28 +2104,31 @@ static void dur_destroy_fence_cluster(cluster_t* c)
free(c);
}
/* Spin until the fatal hook fires or a timeout elapses (the delaymgr fires the
* one-shot timer on its own worker thread). */
static bool dur_wait_fatal(int expect_calls, int timeout_ms)
/* Spin until the sticky unrecoverable flag is set on c->persist or a timeout
* elapses (the delaymgr fires the re-alert timer on its own worker thread). */
static bool dur_wait_unrecoverable(cluster_t* c, int timeout_ms)
{
int waited = 0;
while (g_fatal_hook_calls < expect_calls && waited < timeout_ms) {
while (!persist_recovery_unrecoverable(&c->persist) && waited < timeout_ms) {
usleep(2000);
waited += 2;
}
return g_fatal_hook_calls >= expect_calls;
return persist_recovery_unrecoverable(&c->persist);
}
/* 4.4-RED carrier + core deferral: a recovery-truncated/behind member with NO
* reachable peer must NOT FATAL synchronously at the promotion instant — it
* fences (in_catch_up set, process alive) and arms a bounded-window timer; the
* 3.3 FATAL fires ONLY from that timer after the window with no peer/progress.
/* 4.4 (never-self-exit): a recovery-truncated/behind member with NO reachable
* peer must NOT self-exit — it fences (in_catch_up set, process ALIVE) and arms
* the re-alert timer; when the window elapses with no peer/progress the timer
* RE-EMITS the loud alert (sets the sticky non-promotable flag) and the process
* STAYS ALIVE.
*
* RED (pre-4.4): the gate routed a no-peer truncated member straight through
* ha_evaluate_promotion_fence → persist_recover_fatal_if_unrecoverable, so the
* fatal hook was called SYNCHRONOUSLY (count==1 with zero delay) — there was no
* bounded window, which is exactly the task-5.2 soak collapse. (Req 2.2, 2.4) */
static void test_promotion_fence_defers_fatal_to_bounded_timer(void)
* RED→GREEN: pre-rework the no-peer truncated member was routed (synchronously
* or via the timer) to persist_recover_fatal_if_unrecoverable, which abort()ed
* (the old fatal-capture hook fired) — that self-exit was the task-5.2 soak
* quorum collapse. Post-rework there is NO abort: the member stays alive,
* fenced, and loudly non-promotable. Reaching the end of this test (the
* process did not die) IS part of the proof. (Req 2.2, 2.4) */
static void test_promotion_fence_unrecoverable_stays_alive_loud(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_fence_defer_XXXXXX");
@ -2115,46 +2138,41 @@ static void test_promotion_fence_defers_fatal_to_bounded_timer(void)
/*durable_floor=*/4);
TEST_ASSERT_NOT_NULL(c);
g_fatal_hook_calls = 0;
persist_fatal_hook = dur_fatal_hook;
ha_set_promotion_fence_window_ms(40); /* short bounded window for the test */
ha_set_promotion_fence_window_ms(40); /* short re-alert window for the test */
/* Arm the fence exactly as raft_cb_become_leader does for a truncated,
* below-HWM, no-peer winner. This must NOT FATAL synchronously. */
* below-HWM, no-peer winner. This must NOT self-exit and must fence. */
extern void ha_arm_promotion_fence_timer_for_test(cluster_t*, bool, uint64_t,
uint64_t);
ha_arm_promotion_fence_timer_for_test(c, /*truncated=*/true,
/*durable_floor=*/4, /*hwm=*/10);
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, g_fatal_hook_calls,
"4.4: the promotion gate must NOT FATAL synchronously — FATAL is "
"deferred to the bounded-window timer (Req 2.2/2.4)");
1, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire),
"4.4: a fenced member is held in_catch_up (refused promotion), ALIVE");
/* Let the re-alert window elapse: still truncated + no peer + no progress →
* the timer RE-EMITS the loud alert and SETS the sticky non-promotable flag.
* The process STAYS ALIVE (no abort/exit anywhere on this path). */
TEST_ASSERT_TRUE_MESSAGE(
dur_wait_unrecoverable(c, 1000),
"4.4: the re-alert timer must set the sticky unrecoverable flag after "
"the window with no peer/progress — staying ALIVE, not exiting (Req 2.4)");
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire),
"4.4: a fenced member is held in_catch_up (refused promotion), alive");
"4.4: an unrecoverable member stays fenced (non-promotable) and ALIVE");
/* Now let the bounded window elapse: still truncated + no peer + no
* progress → the 3.3 FATAL fires from the timer (exactly once). */
TEST_ASSERT_TRUE_MESSAGE(
dur_wait_fatal(1, 1000),
"4.4: the deferred FATAL must fire from the timer after the bounded "
"window with no peer and no progress (Req 2.4)");
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, g_fatal_hook_calls,
"4.4: the deferred FATAL fires exactly once from the timer");
persist_fatal_hook = NULL;
ha_set_promotion_fence_window_ms(0); /* restore default */
dur_destroy_fence_cluster(c);
printf("test_promotion_fence_defers_fatal_to_bounded_timer: PASS\n");
printf("test_promotion_fence_unrecoverable_stays_alive_loud: PASS\n");
}
/* 4.4 Req 2.6: catch-up within the window → fence released, FATAL NOT called.
* Arm the fence, then simulate catch-up (clear truncated + advance durable_floor
* past the HWM) before the timer fires; when the timer fires it observes a
* caught-up member, releases in_catch_up, and does NOT FATAL. */
static void test_promotion_fence_released_on_catchup_no_fatal(void)
/* 4.4 Req 2.6: catch-up within the window → fence + sticky flag released, and
* the member NEVER self-exits. Arm the fence, then simulate catch-up (clear
* truncated + advance durable_floor past the HWM) before the timer fires; when
* the timer fires it observes a caught-up member, releases in_catch_up, clears
* the sticky unrecoverable flag, and does NOT set the unrecoverable flag. */
static void test_promotion_fence_released_on_catchup(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_fence_catchup_XXXXXX");
@ -2164,15 +2182,12 @@ static void test_promotion_fence_released_on_catchup_no_fatal(void)
/*durable_floor=*/4);
TEST_ASSERT_NOT_NULL(c);
g_fatal_hook_calls = 0;
persist_fatal_hook = dur_fatal_hook;
ha_set_promotion_fence_window_ms(120); /* generous so we can mutate first */
extern void ha_arm_promotion_fence_timer_for_test(cluster_t*, bool, uint64_t,
uint64_t);
ha_arm_promotion_fence_timer_for_test(c, /*truncated=*/true,
/*durable_floor=*/4, /*hwm=*/10);
TEST_ASSERT_EQUAL_INT(0, g_fatal_hook_calls);
TEST_ASSERT_EQUAL_INT(
1, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire));
@ -2180,26 +2195,26 @@ static void test_promotion_fence_released_on_catchup_no_fatal(void)
c->persist.recovery_verdict = RECOVER_CLEAN;
atomic_store_explicit(&c->persist.durable_lsn, 10, memory_order_release);
/* Wait long enough for the window to fire; assert it did NOT FATAL and the
* fence was released (Req 2.6). */
/* Wait long enough for the window to fire; assert the fence was released and
* the sticky flag is NOT set (Req 2.6) — the member self-healed, alive. */
usleep(300 * 1000);
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, g_fatal_hook_calls,
"4.4: a member that caught up within the window must NOT FATAL (Req 2.6)");
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire),
"4.4: catch-up within the window RELEASES the in_catch_up fence (Req 2.6)");
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(&c->persist),
"4.4: a member that caught up must NOT be left in the unrecoverable "
"fence (Req 2.6)");
persist_fatal_hook = NULL;
ha_set_promotion_fence_window_ms(0);
dur_destroy_fence_cluster(c);
printf("test_promotion_fence_released_on_catchup_no_fatal: PASS\n");
printf("test_promotion_fence_released_on_catchup: PASS\n");
}
/* 4.4: cancelling the timer (shutdown / recovery) before it fires → NO FATAL.
* Proves the deferred FATAL never fires after a clean cancel — a stopped member
* must not self-terminate, and there is no use-after-free of the timer ctx. */
static void test_promotion_fence_cancel_prevents_fatal(void)
/* 4.4: cancelling the timer (shutdown / recovery) before it fires → the
* re-alert never fires, no use-after-free of the timer ctx, and the process
* stays alive (it never self-exited even before the cancel). */
static void test_promotion_fence_cancel_stops_realert(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_fence_cancel_XXXXXX");
@ -2209,29 +2224,27 @@ static void test_promotion_fence_cancel_prevents_fatal(void)
/*durable_floor=*/4);
TEST_ASSERT_NOT_NULL(c);
g_fatal_hook_calls = 0;
persist_fatal_hook = dur_fatal_hook;
ha_set_promotion_fence_window_ms(60);
extern void ha_arm_promotion_fence_timer_for_test(cluster_t*, bool, uint64_t,
uint64_t);
ha_arm_promotion_fence_timer_for_test(c, /*truncated=*/true,
/*durable_floor=*/4, /*hwm=*/10);
TEST_ASSERT_EQUAL_INT(0, g_fatal_hook_calls);
/* Cancel before the window elapses. */
ha_cancel_promotion_fence_timer(&c->ha);
/* Wait past the window — the cancelled timer must be a no-op, NO FATAL. */
/* Wait past the window — the cancelled timer must be a no-op (no re-alert,
* no UAF). The member stayed alive throughout (no abort anywhere). */
usleep(200 * 1000);
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, g_fatal_hook_calls,
"4.4: a cancelled fence timer must NEVER FATAL (shutdown/recovery)");
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(&c->persist),
"4.4: a cancelled fence timer must not fire the re-alert "
"(shutdown/recovery)");
persist_fatal_hook = NULL;
ha_set_promotion_fence_window_ms(0);
dur_destroy_fence_cluster(c);
printf("test_promotion_fence_cancel_prevents_fatal: PASS\n");
printf("test_promotion_fence_cancel_stops_realert: PASS\n");
}
#endif /* XCE_TESTING */
@ -2272,25 +2285,26 @@ void test_commit_durability(void)
/* task 3.2 — Boot Policy Decision Table skeletons (IGNORED, pending
* 3.3/4.3). Registered so each row is reported as IGNORED, not failed;
* the suite stays GREEN. See design.md "Boot Policy Decision Table". */
/* task 3.3 — FATAL rows (4, 4b) now ACTIVE (PASS), plus the never-FATAL
* invariant for the peer-reachable rows (3, 5). The full FENCE->ELIGIBLE
* eligibility outcome of rows 1,2,3,5,6 stays IGNORED pending task 4.3. */
/* task 3.3 — unrecoverable rows (4, 4b) = LOUD PERMANENT FENCE (stays alive,
* non-promotable, clears on catch-up), plus the never-unrecoverable
* invariant for the peer-reachable rows (3, 5) and the FENCE->ELIGIBLE
* eligibility outcome of rows 1,2,3,5,6. (never-self-exit rework) */
RUN_TEST(test_boot_policy_peer_reachable_never_fatal);
RUN_TEST(test_boot_policy_row1_clean_at_or_above_eligible);
RUN_TEST(test_boot_policy_row2_clean_below_peer_catchup_eligible);
RUN_TEST(test_boot_policy_row3_truncated_peer_catchup_eligible);
RUN_TEST(test_boot_policy_row4_truncated_no_peer_bounded_fatal);
RUN_TEST(test_boot_policy_row4b_clean_below_no_peer_bounded_fatal);
RUN_TEST(test_boot_policy_row4_truncated_no_peer_loud_fence);
RUN_TEST(test_boot_policy_row4b_clean_below_no_peer_loud_fence);
RUN_TEST(test_boot_policy_row5_snapshot_sentinel_catchup_never_fatal);
RUN_TEST(test_boot_policy_row6_empty_first_boot_eligible);
/* task 4.3 — hollow node refused even at minimal quorum (Req 2.5). */
RUN_TEST(test_promotion_gate_hollow_refused_at_minimal_quorum);
/* task 4.4 — bounded-window catch-up before FATAL (no synchronous FATAL;
* deferred-timer FATAL only after a real bounded failure; fence released on
* catch-up; cancelled timer never FATALs). (Req 2.2, 2.4, 2.6) */
RUN_TEST(test_promotion_fence_defers_fatal_to_bounded_timer);
RUN_TEST(test_promotion_fence_released_on_catchup_no_fatal);
RUN_TEST(test_promotion_fence_cancel_prevents_fatal);
/* task 4.4 — catch-up re-alert window (NEVER self-exits; stays alive +
* loudly non-promotable on the unrecoverable path; fence + sticky flag
* released on catch-up; cancelled timer stops the re-alert). (Req 2.2/2.4/2.6) */
RUN_TEST(test_promotion_fence_unrecoverable_stays_alive_loud);
RUN_TEST(test_promotion_fence_released_on_catchup);
RUN_TEST(test_promotion_fence_cancel_stops_realert);
#else
printf("test_commit_durability: skipped (no XCE_TESTING)\n");
#endif

125
src/include/persist.h
View file

@ -46,19 +46,29 @@ typedef enum {
* ELIGIBLE — may promote (no fence). Rows 1, 6.
* FENCE — ineligible to promote; catch up from a healthy peer, regain
* eligibility on success (Req 2.6). Rows 2, 3, 5. A reachable
* healthy peer NEVER yields FATAL.
* FATAL — fail loud: cannot recover locally AND cannot catch up within
* the bounded window. Terminate with the truncation/boundary
* logged rather than serve divergent state (Req 2.4). Rows 4, 4b.
* healthy peer NEVER yields FENCE_UNRECOVERABLE.
* FENCE_UNRECOVERABLE — loud, PERMANENT, NON-terminating fence: the member
* cannot recover locally AND cannot catch up (no healthy peer).
* It logs LOUDLY, stays PERMANENTLY non-promotable, and keeps
* retrying catch-up indefinitely — but the process STAYS ALIVE.
* Rows 4, 4b. This is NOT process exit: a controller MUST NEVER
* self-terminate as a designed response to an operational
* condition (user invariant, 2026-06-18). A self-exited controller
* needs MANUAL admin recovery and leaves quorum degraded
* meanwhile; a controller that stays alive and keeps retrying
* SELF-HEALS when a peer returns. The fence RELEASES (→ ELIGIBLE,
* Req 2.6) automatically when catch-up reaches a
* committed-consistent state.
*
* AUTHORITATIVE INVARIANT (design.md): the fence is never a silent indefinite
* wait — a truncated/behind member resolves to exactly one of FENCE→ELIGIBLE
* or a BOUNDED FATAL. There is no "fenced forever, silently divergent" state.
* AUTHORITATIVE INVARIANT (design.md): a truncated/behind member resolves to
* either FENCE→ELIGIBLE (caught up) or a loud PERMANENT FENCE_UNRECOVERABLE
* (stays alive, non-promotable, keeps retrying). It is NEVER silently divergent
* (it refuses to lead) and the process NEVER self-exits.
* ============================================================ */
typedef enum {
RECOVER_DECISION_ELIGIBLE = 0,
RECOVER_DECISION_FENCE = 1,
RECOVER_DECISION_FATAL = 2,
RECOVER_DECISION_ELIGIBLE = 0,
RECOVER_DECISION_FENCE = 1,
RECOVER_DECISION_FENCE_UNRECOVERABLE = 2,
} recover_decision_t;
/* Policy inputs evaluated at boot / promotion-eligibility (design.md table).
@ -177,6 +187,22 @@ struct persist_s {
* before the node participates; read by the promotion gate thereafter. */
recover_verdict_t recovery_verdict;
/* Sticky "unrecoverable, loudly fenced" flag (ha-commit-durability Req 2.4,
* NEVER-SELF-EXIT rework 2026-06-18). Set by
* persist_recover_loud_fence_if_unrecoverable when a member is
* recovery-truncated AND/OR durable below the cluster committed high-water
* mark with NO successful catch-up: the member stays ALIVE but is held
* PERMANENTLY non-promotable and keeps retrying catch-up. A controller MUST
* NEVER self-exit as a designed response to this operational condition; it
* self-heals automatically when a healthy peer returns and catch-up reaches
* a committed-consistent state — at which point this flag is CLEARED via
* persist_clear_recovery_unrecoverable (wired into the fence-release /
* catch-up path). Read by the promotion gate as belt-and-suspenders
* (a member must never lead while this is set). Atomic: set on the recovery
* path, cleared on the catch-up path, read by the promotion/heartbeat
* threads. Defaults to false before recovery runs. */
_Atomic bool recovery_unrecoverable;
/* Paths (derived from workdir + name via persist_set_workdir) */
char wal_path[4096];
char snap_path[4096];
@ -347,53 +373,64 @@ uint64_t persist_durable_floor(persist_t* p);
recover_verdict_t persist_recovery_verdict(persist_t* p);
bool persist_recovery_truncated(persist_t* p);
/* Bounded catch-up window (ha-commit-durability task 3.3, Req 2.4).
* The FINITE deadline a recovery-truncated / behind member is given to reach a
* committed-consistent state from a healthy peer before the boot policy FATALs.
* The fence must NEVER be an indefinite wait (design.md authoritative
* invariant), so the window is always finite. Default 30s — generous enough for
* a healthy-peer catch-up stream yet bounded; the verification mode overrides it
* via the inputs/timeout the caller (task 4.3 boot path) supplies. */
/* Bounded catch-up RE-ALERT window (ha-commit-durability task 3.3, Req 2.4;
* never-self-exit rework 2026-06-18).
* The interval after which a still-unrecoverable, loudly-fenced member RE-EMITS
* its operator alert (it does NOT terminate — see RECOVER_DECISION_FENCE_-
* UNRECOVERABLE). The member stays alive and keeps retrying catch-up
* indefinitely; this window only controls how often the loud operator-visible
* alert is repeated. Default 30s. */
#define DEFAULT_RECOVERY_CATCHUP_WINDOW_MS 30000u
/* persist_recover_decision (ha-commit-durability task 3.3, Req 2.4).
*
* Pure decision over the Boot Policy Decision Table: maps the policy inputs to
* RECOVER_DECISION_{ELIGIBLE,FENCE,FATAL}. No side effects, no termination — it
* only classifies, so it is unit-testable in isolation. The FATAL terminal is
* acted on by persist_recover_fatal_if_unrecoverable (which logs + terminates).
* RECOVER_DECISION_{ELIGIBLE,FENCE,FENCE_UNRECOVERABLE}. No side effects, no
* termination — it only classifies, so it is unit-testable in isolation. The
* FENCE_UNRECOVERABLE terminal is acted on by
* persist_recover_loud_fence_if_unrecoverable (which logs LOUDLY + sets the
* sticky non-promotable flag — it NEVER terminates the process).
*
* Contract (design.md rows): a reachable healthy peer NEVER returns FATAL
* (rows 3, 5); FATAL is returned ONLY when the member is truncated or behind
* the HWM, has NO healthy peer, and bounded catch-up did not reach a
* committed-consistent state (rows 4, 4b). */
* Contract (design.md rows): a reachable healthy peer NEVER returns
* FENCE_UNRECOVERABLE (rows 3, 5); it is returned ONLY when the member is
* truncated or behind the HWM, has NO healthy peer, and catch-up did not reach
* a committed-consistent state (rows 4, 4b). */
recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in);
/* persist_recover_fatal_if_unrecoverable (ha-commit-durability task 3.3,
* Req 2.4).
/* persist_recover_loud_fence_if_unrecoverable (ha-commit-durability task 3.3,
* Req 2.4; never-self-exit rework 2026-06-18).
*
* Evaluate persist_recover_decision over `in`; if the outcome is FATAL, log at
* ERROR LOUDLY — INCLUDING the truncation boundary (durable_floor_lsn) and the
* cluster high-water mark — and then TERMINATE (fail loud rather than serve
* divergent state). Returns the decision so a non-FATAL caller can fence/proceed
* (FENCE/ELIGIBLE are acted on by task 4.3).
* Evaluate persist_recover_decision over `in`; if the outcome is
* FENCE_UNRECOVERABLE, log LOUDLY at ERROR — INCLUDING the truncation boundary
* (durable_floor_lsn) and the cluster high-water mark — stating the member is
* staying fenced + non-promotable and will keep retrying catch-up, then SET the
* sticky `recovery_unrecoverable` flag on `p` (so the promotion gate refuses
* leadership as belt-and-suspenders). Returns the decision.
*
* Termination routes through persist_fatal_hook: in production the hook is NULL
* and the function flushes logs and abort()s (NEVER returns on FATAL); a test
* build may install a hook to capture the FATAL decision + boundary log without
* the process dying (mirrors the ha_fatal_hook seam in ha_convergence.c). */
* CRITICAL — the process STAYS ALIVE. There is NO abort()/exit()/tlog_exit()
* here: a controller MUST NEVER self-terminate as a designed response to an
* operational condition (user invariant, 2026-06-18). The member self-heals —
* the fence releases (→ ELIGIBLE, Req 2.6) and the flag clears via
* persist_clear_recovery_unrecoverable — automatically when a healthy peer
* returns and catch-up reaches a committed-consistent state.
*
* `p` may be NULL (pure-decision callers that only want the classification +
* loud log without touching a persist_t); the flag set is then skipped. */
recover_decision_t
persist_recover_fatal_if_unrecoverable(const recover_policy_inputs_t* in);
persist_recover_loud_fence_if_unrecoverable(persist_t* p,
const recover_policy_inputs_t* in);
/* persist_fatal_hook: test-only seam for the FATAL (abort) contract of
* persist_recover_fatal_if_unrecoverable (ha-commit-durability task 3.3,
* Req 2.4). Mirrors ha_fatal_hook (ha_convergence.c).
/* persist_recovery_unrecoverable / persist_clear_recovery_unrecoverable
* (ha-commit-durability Req 2.4/2.6, never-self-exit rework 2026-06-18).
*
* When non-NULL, the FATAL path invokes this INSTEAD OF abort(), so a test
* binary can assert the FATAL decision was taken and the truncation boundary
* was logged without the process dying. Production leaves it NULL and actually
* terminates. Signature: void hook(const char *msg). */
extern void (*persist_fatal_hook)(const char* msg);
* Accessor + clear for the sticky "unrecoverable, loudly fenced" flag. The flag
* is SET by persist_recover_loud_fence_if_unrecoverable and CLEARED on the
* catch-up / fence-release path once the member reaches a committed-consistent
* state (not truncated AND durable_floor >= HWM). While set, the member is held
* permanently non-promotable but stays alive. Safe on a NULL p (returns false /
* no-op). */
bool persist_recovery_unrecoverable(persist_t* p);
void persist_clear_recovery_unrecoverable(persist_t* p);
/* Register a durable-advance wake callback (ha-commit-durability task 2.2).
* The group-commit flusher calls fn(ctx) after it advances durable_lsn and