fix(ha-commit-durability)!: controller never self-terminates — loud non-promotable fence replaces recovery FATAL/abort
Per design invariant (a controller exits only on crash or admin kill), replace the self-abort recovery terminal (tasks 3.3/4.3/4.4) with a loud, permanent, non-promotable fence that stays alive and keeps retrying catch-up (self-heals on peer return; no manual admin recovery). Records the never-self-terminate invariant as project-wide guidance in src/controller/CLAUDE.md + .kiro/steering/ha-async-io.md.
This commit is contained in:
parent
addf53e3b4
commit
59d9df246c
10 changed files with 607 additions and 406 deletions
|
|
@ -162,12 +162,13 @@ graph TD
|
||||||
Floor --> Eligible[eligible to lead]
|
Floor --> Eligible[eligible to lead]
|
||||||
Truncated --> CanCatchUp{healthy peer reachable}
|
Truncated --> CanCatchUp{healthy peer reachable}
|
||||||
CanCatchUp -->|yes| Fence[fence from promotion until caught up]
|
CanCatchUp -->|yes| Fence[fence from promotion until caught up]
|
||||||
CanCatchUp -->|no and needed for quorum| Fatal[FATAL fail loud]
|
CanCatchUp -->|no| LoudFence[LOUD PERMANENT FENCE: non-promotable, STAYS ALIVE, keeps retrying catch-up — NEVER self-exit]
|
||||||
|
LoudFence --> Caught
|
||||||
Fence --> Caught{caught up past last committed boundary}
|
Fence --> Caught{caught up past last committed boundary}
|
||||||
Caught -->|yes| Eligible
|
Caught -->|yes| Eligible
|
||||||
```
|
```
|
||||||
|
|
||||||
Key decisions: corruption is never silently accepted (Req 2.1). A truncated member is fenced from promotion and catches up from a healthy primary (Req 2.2, 2.6); if it cannot reach a consistent committed state and the cluster needs it, the controller fails loudly (Req 2.4), honoring the project's fail-loud preference. A candidate never advertises or leads beyond its durable WAL tip (Req 2.3), so a hollow node cannot win even under degraded quorum (Req 2.5).
|
Key decisions: corruption is never silently accepted (Req 2.1). A truncated member is fenced from promotion and catches up from a healthy primary (Req 2.2, 2.6); if it cannot reach a consistent committed state, the controller logs a LOUD operator-visible alert and holds itself PERMANENTLY non-promotable while STAYING ALIVE and continuing to retry catch-up (Req 2.4) — it refuses to serve divergent state by refusing leadership, NOT by terminating. **A controller MUST NEVER self-exit** (`exit`/`abort`) as a designed response to an operational condition (user invariant, 2026-06-18): a self-exited controller needs manual admin recovery and degrades quorum meanwhile, whereas one that stays alive self-heals when a peer returns (Req 2.6). A candidate never advertises or leads beyond its durable WAL tip (Req 2.3), so a hollow node cannot win even under degraded quorum (Req 2.5).
|
||||||
|
|
||||||
## Requirements Traceability
|
## Requirements Traceability
|
||||||
|
|
||||||
|
|
@ -182,7 +183,7 @@ Key decisions: corruption is never silently accepted (Req 2.1). A truncated memb
|
||||||
| 2.1 | no silent truncate-accept | RecoveryIntegrity | `replay_wal` verdict | Recovery fence |
|
| 2.1 | no silent truncate-accept | RecoveryIntegrity | `replay_wal` verdict | Recovery fence |
|
||||||
| 2.2 | fence until caught up | PromotionFence | promotion gate, `in_catch_up` | Recovery fence |
|
| 2.2 | fence until caught up | PromotionFence | promotion gate, `in_catch_up` | Recovery fence |
|
||||||
| 2.3 | never lead below high-water | PromotionFence + FreshnessFloor | `raft_cb_become_leader`, `ha_vote_self_freshness` | Recovery fence |
|
| 2.3 | never lead below high-water | PromotionFence + FreshnessFloor | `raft_cb_become_leader`, `ha_vote_self_freshness` | Recovery fence |
|
||||||
| 2.4 | fail-loud when unrecoverable | RecoveryIntegrity | FATAL policy | Recovery fence |
|
| 2.4 | loud permanent non-promotable fence when unrecoverable (stays alive, NEVER self-exit) | RecoveryIntegrity | loud-fence policy + sticky `recovery_unrecoverable` flag | Recovery fence |
|
||||||
| 2.5 | degraded quorum preserves invariant | DurableCommit + PromotionFence | durable-majority + fence | Both |
|
| 2.5 | degraded quorum preserves invariant | DurableCommit + PromotionFence | durable-majority + fence | Both |
|
||||||
| 2.6 | re-eligibility on catch-up | PromotionFence | fence-release hook | Recovery fence |
|
| 2.6 | re-eligibility on catch-up | PromotionFence | fence-release hook | Recovery fence |
|
||||||
| 3.1 | configurable durability cadence | DurabilityConfig | `--wal-sync-ms` | — |
|
| 3.1 | configurable durability cadence | DurabilityConfig | `--wal-sync-ms` | — |
|
||||||
|
|
@ -194,7 +195,7 @@ Key decisions: corruption is never silently accepted (Req 2.1). A truncated memb
|
||||||
| Component | Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|
| Component | Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|
||||||
|-----------|-------|--------|--------------|--------------------------|-----------|
|
|-----------|-------|--------|--------------|--------------------------|-----------|
|
||||||
| DurableCommit | Persistence + Commit | Synchronous persist + self-counts-durable | 1.1–1.6, 2.5 | persist_t (P0), commit_engine (P0), repl ACK (P1) | State |
|
| DurableCommit | Persistence + Commit | Synchronous persist + self-counts-durable | 1.1–1.6, 2.5 | persist_t (P0), commit_engine (P0), repl ACK (P1) | State |
|
||||||
| RecoveryIntegrity | Persistence | Verdict on recovery; durable floor; FATAL policy | 2.1, 2.4 | replay_wal (P0) | State |
|
| RecoveryIntegrity | Persistence | Verdict on recovery; durable floor; loud permanent non-promotable fence policy (stays alive, NEVER self-exit) | 2.1, 2.4 | replay_wal (P0) | State |
|
||||||
| PromotionFence | Election | Eligibility gate + fence-release | 2.2, 2.3, 2.6, 2.5 | RecoveryIntegrity (P0), in_catch_up (P1) | State |
|
| PromotionFence | Election | Eligibility gate + fence-release | 2.2, 2.3, 2.6, 2.5 | RecoveryIntegrity (P0), in_catch_up (P1) | State |
|
||||||
| FreshnessFloor | Election | Durable-backed vote floor & advertisement | 2.3 | durable_lsn (P0), ha_state (P1) | State |
|
| FreshnessFloor | Election | Durable-backed vote floor & advertisement | 2.3 | durable_lsn (P0), ha_state (P1) | State |
|
||||||
| DurabilityConfig + TestSeam | Runtime / Test | Override cadence; verification hook | 3.1–3.3 | persist_config (P0), XCE_TESTING (P1) | State |
|
| DurabilityConfig + TestSeam | Runtime / Test | Override cadence; verification hook | 3.1–3.3 | persist_config (P0), XCE_TESTING (P1) | State |
|
||||||
|
|
@ -235,14 +236,14 @@ Key decisions: corruption is never silently accepted (Req 2.1). A truncated memb
|
||||||
|
|
||||||
| Field | Detail |
|
| Field | Detail |
|
||||||
|-------|--------|
|
|-------|--------|
|
||||||
| Intent | Distinguish clean log end from corruption; re-derive durable floor; fail loud when unrecoverable |
|
| Intent | Distinguish clean log end from corruption; re-derive durable floor; loud permanent non-promotable fence when unrecoverable (stays alive, NEVER self-exit) |
|
||||||
| Requirements | 2.1, 2.4 |
|
| Requirements | 2.1, 2.4 |
|
||||||
|
|
||||||
**Responsibilities & Constraints**
|
**Responsibilities & Constraints**
|
||||||
- `replay_wal`/`persist_recover` return a verdict: `RECOVER_CLEAN` (reached a clean entry boundary) vs `RECOVER_TRUNCATED` (stopped at CRC break / mismatch / short read).
|
- `replay_wal`/`persist_recover` return a verdict: `RECOVER_CLEAN` (reached a clean entry boundary) vs `RECOVER_TRUNCATED` (stopped at CRC break / mismatch / short read).
|
||||||
- On `RECOVER_TRUNCATED`, the member is marked recovery-truncated and is NOT treated as a clean recovery; it must not advertise or lead beyond the last intact LSN.
|
- On `RECOVER_TRUNCATED`, the member is marked recovery-truncated and is NOT treated as a clean recovery; it must not advertise or lead beyond the last intact LSN.
|
||||||
- The recovered durable floor (highest intact replayed LSN) is published so the freshness advertisement is durable-backed.
|
- The recovered durable floor (highest intact replayed LSN) is published so the freshness advertisement is durable-backed.
|
||||||
- FATAL policy: if a recovery-truncated member cannot reach a healthy peer to catch up and the cluster requires it, the controller terminates loudly rather than serve divergent state.
|
- Loud-fence policy (NEVER self-exit, user invariant 2026-06-18): if a recovery-truncated / behind member cannot reach a healthy peer to catch up, the controller logs a LOUD operator-visible alert (truncation boundary vs cluster HWM) via `persist_recover_loud_fence_if_unrecoverable`, sets a sticky `recovery_unrecoverable` flag, and holds itself PERMANENTLY non-promotable while STAYING ALIVE and continuing to retry catch-up. It refuses to serve divergent state by refusing leadership — it does NOT `exit`/`abort`. The flag clears (and the fence releases → eligible, Req 2.6) automatically when a peer returns and catch-up reaches a committed-consistent state.
|
||||||
|
|
||||||
**Contracts**: State [x]
|
**Contracts**: State [x]
|
||||||
|
|
||||||
|
|
@ -254,15 +255,15 @@ Key decisions: corruption is never silently accepted (Req 2.1). A truncated memb
|
||||||
**Implementation Notes**
|
**Implementation Notes**
|
||||||
- Integration: composes with `ha-convergence-recovery` — that spec truncates an **intact** log on demotion/rejoin; this spec refuses to accept a **corrupt** log silently.
|
- Integration: composes with `ha-convergence-recovery` — that spec truncates an **intact** log on demotion/rejoin; this spec refuses to accept a **corrupt** log silently.
|
||||||
- Validation: unit test injects a CRC break at LSN 1 and asserts verdict `RECOVER_TRUNCATED` + non-promotability.
|
- Validation: unit test injects a CRC break at LSN 1 and asserts verdict `RECOVER_TRUNCATED` + non-promotability.
|
||||||
- Risks: must not FATAL on a legitimately empty first-boot WAL (no entries ≠ corruption).
|
- Risks: must not loud-fence on a legitimately empty first-boot WAL (no entries ≠ corruption).
|
||||||
|
|
||||||
### Boot Policy Decision Table (task 3.2)
|
### Boot Policy Decision Table (task 3.2)
|
||||||
|
|
||||||
> **Status:** DECISION recorded (task 3.2). No production behavior change is made by 3.2 — this table is the contract that task **3.3** (the FATAL branch) and task **4.3** (the promotion fence + release) implement. The skeleton tests in `test_commit_durability.c` (one per row, `TEST_IGNORE`-skipped pending 3.3/4.3) encode each row's expected outcome.
|
> **Status:** DECISION recorded (task 3.2); **terminal action corrected by the never-self-exit rework (2026-06-18).** The original table's `FATAL` outcome (a self-`abort`) was REPLACED: the unrecoverable terminal is now a **LOUD PERMANENT FENCE** — non-promotable, **stays alive**, keeps retrying catch-up. This table is the contract that task **3.3** (the loud-fence terminal) and task **4.3** (the promotion fence + release) implement.
|
||||||
|
|
||||||
**Decision (Req 2.2, 2.4):** A member that boots with a structurally clean log but below the cluster's committed high-water mark, or with a recovery-**truncated** log, is **fenced from leadership** at boot (never promoted while behind/truncated). It then **attempts catch-up** from a healthy peer. It becomes eligible **only** once catch-up brings its durable position to a committed-consistent state at/above the cluster boundary. It terminates **loudly (FATAL)** only on a **bounded** failure to reach that state — i.e. it cannot recover locally **and** cannot catch up within the bounded window (no healthy peer reachable / cannot reach a committed-consistent state).
|
**Decision (Req 2.2, 2.4):** A member that boots with a structurally clean log but below the cluster's committed high-water mark, or with a recovery-**truncated** log, is **fenced from leadership** at boot (never promoted while behind/truncated). It then **attempts catch-up** from a healthy peer. It becomes eligible **only** once catch-up brings its durable position to a committed-consistent state at/above the cluster boundary. If it cannot reach that state (no healthy peer reachable / cannot reach a committed-consistent state), it enters a **LOUD PERMANENT FENCE**: it logs a loud, operator-visible alert (truncation boundary vs cluster HWM), holds itself **permanently non-promotable**, and **stays alive while continuing to retry catch-up** — it **NEVER self-terminates**. The fence releases (→ ELIGIBLE) automatically once catch-up reaches a committed-consistent state.
|
||||||
|
|
||||||
**Invariant (authoritative):** the fence is **never a silent indefinite wait**. A truncated/behind member always resolves to exactly one of two terminal outcomes: it **catches up → ELIGIBLE**, or it **FATALs (fail-loud)** on a bounded failure. There is no third "fenced forever, still running, silently divergent" state. Snapshot-sentinel recovery is treated as CLEAN-but-behind (catch-up), never FATAL on a legitimate sentinel.
|
**Invariant (authoritative):** A truncated/behind member always resolves to exactly one of two outcomes: it **catches up → ELIGIBLE**, or it enters a **LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying catch-up — NEVER self-exit)**. It is never silently divergent (it refuses to lead), and the process never self-`exit`s/`abort`s as a designed response to this operational condition (user invariant, 2026-06-18) — a self-exited controller needs manual admin recovery and degrades quorum, whereas one that stays alive self-heals when a peer returns (Req 2.6). Snapshot-sentinel recovery is treated as CLEAN-but-behind (catch-up), never the unrecoverable terminal on a legitimate sentinel.
|
||||||
|
|
||||||
**Inputs (evaluated at boot / promotion-eligibility):**
|
**Inputs (evaluated at boot / promotion-eligibility):**
|
||||||
- **verdict** — recovery integrity verdict from task 3.1: `CLEAN` vs `TRUNCATED`.
|
- **verdict** — recovery integrity verdict from task 3.1: `CLEAN` vs `TRUNCATED`.
|
||||||
|
|
@ -273,21 +274,21 @@ Key decisions: corruption is never silently accepted (Req 2.1). A truncated memb
|
||||||
**Outcomes:**
|
**Outcomes:**
|
||||||
- **ELIGIBLE** — may promote (no fence).
|
- **ELIGIBLE** — may promote (no fence).
|
||||||
- **FENCE+CATCH-UP → ELIGIBLE** — ineligible to promote; catch up from a healthy peer; regain eligibility on success (Req 2.6).
|
- **FENCE+CATCH-UP → ELIGIBLE** — ineligible to promote; catch up from a healthy peer; regain eligibility on success (Req 2.6).
|
||||||
- **FATAL** — fail loud; terminate with the truncation/boundary logged rather than serve divergent state (Req 2.4).
|
- **LOUD PERMANENT FENCE** — log a loud operator-visible alert (truncation/boundary), set the sticky `recovery_unrecoverable` flag, and hold the member permanently non-promotable while it **stays alive** and keeps retrying catch-up — **NEVER self-exit**; releases → ELIGIBLE on catch-up (Req 2.4, 2.6).
|
||||||
|
|
||||||
| # | verdict | durable-vs-HWM | healthy peer | catch-up result | Outcome | Rationale / Req |
|
| # | verdict | durable-vs-HWM | healthy peer | catch-up result | Outcome | Rationale / Req |
|
||||||
|---|---------|----------------|--------------|-----------------|---------|-----------------|
|
|---|---------|----------------|--------------|-----------------|---------|-----------------|
|
||||||
| 1 | CLEAN | at/above | n/a | n/a | **ELIGIBLE** | Log intact and not behind — normal promotable member. (Req 2.3) |
|
| 1 | CLEAN | at/above | n/a | n/a | **ELIGIBLE** | Log intact and not behind — normal promotable member. (Req 2.3) |
|
||||||
| 2 | CLEAN | below | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | Intact but behind the HWM; must not lead below the boundary; catch up then promote. (Req 2.2, 2.3, 2.6) |
|
| 2 | CLEAN | below | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | Intact but behind the HWM; must not lead below the boundary; catch up then promote. (Req 2.2, 2.3, 2.6) |
|
||||||
| 3 | TRUNCATED | (below — a truncated log is by definition not provably at/above) | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | Structurally truncated but a healthy peer exists; fence, catch up, regain eligibility. Never FATAL when a peer is reachable. (Req 2.2, 2.6) |
|
| 3 | TRUNCATED | (below — a truncated log is by definition not provably at/above) | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | Structurally truncated but a healthy peer exists; fence, catch up, regain eligibility. Never the unrecoverable terminal when a peer is reachable. (Req 2.2, 2.6) |
|
||||||
| 4 | TRUNCATED | below | no | fails (bounded window elapses / no committed-consistent state reachable) | **FATAL** | Cannot recover locally AND cannot catch up — bounded failure ⇒ fail loud, never silent indefinite fence. (Req 2.4) |
|
| 4 | TRUNCATED | below | no | fails (no committed-consistent state reachable) | **LOUD PERMANENT FENCE** (non-promotable, stays alive, keeps retrying catch-up — NEVER self-exit) | Cannot recover locally AND no peer to catch up from ⇒ log loudly + hold non-promotable + alive; releases → ELIGIBLE when a peer returns and it catches up. NEVER `exit`/`abort`. (Req 2.4, 2.6) |
|
||||||
| 4b | CLEAN | below | no | fails (bounded window elapses, no peer to advance from) | **FATAL** | Behind the HWM and no healthy peer to catch up from within the bounded window — same bounded-failure terminal as row 4; never a silent indefinite fence. (Req 2.4, 2.2) |
|
| 4b | CLEAN | below | no | fails (no peer to advance from) | **LOUD PERMANENT FENCE** (non-promotable, stays alive, keeps retrying catch-up — NEVER self-exit) | Behind the HWM and no healthy peer to catch up from — same non-terminating loud-fence as row 4; NEVER self-exit. (Req 2.4, 2.2, 2.6) |
|
||||||
| 5 | CLEAN (snapshot-sentinel: committed at sentinel LSN, no sub-sentinel entries) | below (no entries below the sentinel) | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | A legitimate sentinel snapshot is CLEAN-but-behind, NOT corruption; treat as must-catch-up, **never FATAL** on a legitimate sentinel. (Req 2.1, 2.2) |
|
| 5 | CLEAN (snapshot-sentinel: committed at sentinel LSN, no sub-sentinel entries) | below (no entries below the sentinel) | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | A legitimate sentinel snapshot is CLEAN-but-behind, NOT corruption; treat as must-catch-up, **never the unrecoverable terminal** on a legitimate sentinel. (Req 2.1, 2.2) |
|
||||||
| 6 | CLEAN | at/above (empty first-boot WAL, fresh cold start: durable=0, HWM=0) | n/a | n/a | **ELIGIBLE** | An empty first-boot WAL is not behind anything — cold-start members are promotable. (Req 2.1, 2.3) |
|
| 6 | CLEAN | at/above (empty first-boot WAL, fresh cold start: durable=0, HWM=0) | n/a | n/a | **ELIGIBLE** | An empty first-boot WAL is not behind anything — cold-start members are promotable. (Req 2.1, 2.3) |
|
||||||
|
|
||||||
**Notes on the dimensions:**
|
**Notes on the dimensions:**
|
||||||
- A `TRUNCATED` verdict is never simultaneously "provably at/above the HWM": truncation means the log stops below its last intact boundary, so a truncated member is treated as behind for fence purposes regardless of the volatile in-memory position (this is why 4.2/3.4 derive the freshness floor from the **durable** WAL tip, not a volatile value).
|
- A `TRUNCATED` verdict is never simultaneously "provably at/above the HWM": truncation means the log stops below its last intact boundary, so a truncated member is treated as behind for fence purposes regardless of the volatile in-memory position (this is why 4.2/3.4 derive the freshness floor from the **durable** WAL tip, not a volatile value).
|
||||||
- "bounded window" is the catch-up deadline pinned by task 3.3; the policy requires it to be **finite** so the fence cannot become a silent indefinite wait.
|
- the catch-up retry is **indefinite** (the member stays alive and keeps trying) and **loud** (a periodic operator-visible re-alert, controlled by `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS`); it is never silently divergent because the member refuses leadership the entire time. The previous "bounded window before FATAL" deadline was REMOVED by the never-self-exit rework — there is no deadline-driven termination.
|
||||||
- The `healthy peer = no` + `at/above` combination does not need catch-up and is covered by row 1 (ELIGIBLE) — a member already at/above the HWM is promotable without a peer.
|
- The `healthy peer = no` + `at/above` combination does not need catch-up and is covered by row 1 (ELIGIBLE) — a member already at/above the HWM is promotable without a peer.
|
||||||
|
|
||||||
### Election — PromotionFence + FreshnessFloor
|
### Election — PromotionFence + FreshnessFloor
|
||||||
|
|
@ -458,7 +459,7 @@ Walking the relevant items under the pinned formula:
|
||||||
|
|
||||||
### Error Strategy
|
### Error Strategy
|
||||||
- **Durability failure** (`fdatasync` error in the flusher): `durable_lsn` does not advance, so the entry is NOT counted durable, the commit does not advance, and the client does not receive *ok* (Req 1.4). The flusher surfaces the I/O error loudly (fail-loud); a persistent local sync failure is treated as a member fault, not silently retried into a false-durable state.
|
- **Durability failure** (`fdatasync` error in the flusher): `durable_lsn` does not advance, so the entry is NOT counted durable, the commit does not advance, and the client does not receive *ok* (Req 1.4). The flusher surfaces the I/O error loudly (fail-loud); a persistent local sync failure is treated as a member fault, not silently retried into a false-durable state.
|
||||||
- **Recovery corruption** (`RECOVER_TRUNCATED`): never silently accepted (Req 2.1). Fence from promotion + catch up; FATAL when catch-up is impossible and the node is required (Req 2.4) — fail-loud.
|
- **Recovery corruption** (`RECOVER_TRUNCATED`): never silently accepted (Req 2.1). Fence from promotion + catch up; when catch-up is impossible (no peer) the node enters a LOUD PERMANENT FENCE — non-promotable, **stays alive**, keeps retrying catch-up, NEVER self-exits (Req 2.4, 2.6).
|
||||||
- **Promotion while behind**: refused at the gate (Req 2.3); logged with the local durable tip vs the cluster high-water mark.
|
- **Promotion while behind**: refused at the gate (Req 2.3); logged with the local durable tip vs the cluster high-water mark.
|
||||||
|
|
||||||
### Monitoring
|
### Monitoring
|
||||||
|
|
@ -489,7 +490,7 @@ Walking the relevant items under the pinned formula:
|
||||||
|
|
||||||
## Open Questions / Gating Items (resolve in `/kiro-spec-tasks`)
|
## Open Questions / Gating Items (resolve in `/kiro-spec-tasks`)
|
||||||
- **Freshness-floor formula (gating, election-safety):** pin the exact reconciliation of the durable-WAL-derived floor vs the existing `max(applied_lsn, last_known_committed_lsn)`. Too low ⇒ a truncated node still wins (the bug); too high ⇒ election livelock. Make a `raft/raft.tla` Leader-Completeness **and liveness** re-check a blocking sub-task before the freshness change merges (Req 2.3).
|
- **Freshness-floor formula (gating, election-safety):** pin the exact reconciliation of the durable-WAL-derived floor vs the existing `max(applied_lsn, last_known_committed_lsn)`. Too low ⇒ a truncated node still wins (the bug); too high ⇒ election livelock. Make a `raft/raft.tla` Leader-Completeness **and liveness** re-check a blocking sub-task before the freshness change merges (Req 2.3).
|
||||||
- **FATAL-vs-fence decision point (gating):** a booting truncated member cannot know it is "required for quorum" at `cluster_init`. Define the concrete policy: fence at boot, attempt catch-up from a healthy primary, and FATAL only on a bounded failure to reach a committed-consistent state — never a silent indefinite fence (Req 2.2, 2.4). **RESOLVED (task 3.2):** see the **Boot Policy Decision Table (task 3.2)** above for the recorded input-state → outcome table; tasks 3.3 (FATAL branch) and 4.3 (fence + release) implement it.
|
- **Unrecoverable-terminal decision point (gating):** a booting truncated member cannot know it is "required for quorum" at `cluster_init`. Concrete policy: fence at boot, attempt catch-up from a healthy primary, and on a failure to reach a committed-consistent state (no peer) enter a **LOUD PERMANENT FENCE** — non-promotable, stays alive, keeps retrying catch-up, **NEVER self-exit** (Req 2.2, 2.4, 2.6). **RESOLVED (task 3.2; terminal corrected by the never-self-exit rework 2026-06-18 — the original self-`abort` FATAL was replaced by the loud permanent fence):** see the **Boot Policy Decision Table (task 3.2)** above; tasks 3.3 (loud-fence terminal) and 4.3 (fence + release) implement it.
|
||||||
- **Flusher lost-wakeup proof:** the append-signal → flush → commit-wake path must be shown free of lost wakeups (level-triggered on the appended-vs-synced gap) before relying on it for the durability invariant (Req 1.1, 1.2).
|
- **Flusher lost-wakeup proof:** the append-signal → flush → commit-wake path must be shown free of lost wakeups (level-triggered on the appended-vs-synced gap) before relying on it for the durability invariant (Req 1.1, 1.2).
|
||||||
- **Snapshot-recovery edge:** current snapshots are 16-byte LSN sentinels (no entity data), so a member that recovers "from snapshot" has a committed position at the sentinel LSN but **no log entries below it**. The recovery verdict (3.1) must treat snapshot-sentinel + clean replay as CLEAN (not TRUNCATED), and the promotion fence (4.3) must treat such a member as behind-and-must-catch-up (never FATAL on a legitimate sentinel snapshot). Interacts with the roadmap `entity-bearing snapshots` item; pin the classification before implementing 3.1/4.3 (Req 2.1, 2.2).
|
- **Snapshot-recovery edge:** current snapshots are 16-byte LSN sentinels (no entity data), so a member that recovers "from snapshot" has a committed position at the sentinel LSN but **no log entries below it**. The recovery verdict (3.1) must treat snapshot-sentinel + clean replay as CLEAN (not TRUNCATED), and the promotion fence (4.3) must treat such a member as behind-and-must-catch-up (never the unrecoverable terminal on a legitimate sentinel snapshot). Interacts with the roadmap `entity-bearing snapshots` item; pin the classification before implementing 3.1/4.3 (Req 2.1, 2.2).
|
||||||
- **Dormant synchronous-append path:** `persist_wal_sync_append` advances the commit position directly on majority ACK and assumes inline fsync — a premise group-commit removes (task 2.4). It has no production caller today but must be deleted or re-routed so it cannot reintroduce ACK-without-durability commit (Req 1.1, 1.2).
|
- **Dormant synchronous-append path:** `persist_wal_sync_append` advances the commit position directly on majority ACK and assumes inline fsync — a premise group-commit removes (task 2.4). It has no production caller today but must be deleted or re-routed so it cannot reintroduce ACK-without-durability commit (Req 1.1, 1.2).
|
||||||
|
|
|
||||||
61
.kiro/specs/ha-commit-durability/requirements.md
Normal file
61
.kiro/specs/ha-commit-durability/requirements.md
Normal file
|
|
@ -0,0 +1,61 @@
|
||||||
|
# Requirements Document
|
||||||
|
|
||||||
|
## Project Description (Input)
|
||||||
|
|
||||||
|
**Who has the problem:** Cluster operators running the XCE HA controller cluster — and every client whose acknowledged job/queue/worker operations must survive controller failover.
|
||||||
|
|
||||||
|
**Current situation:** In a 3-node failover test, committed jobs were permanently lost (`joblist` dropped 10→3; the 7 earliest, already acknowledged, vanished). Root cause (bug-661): **acknowledgment does not imply durability.** The controller advances its commit position on a majority of replication **acknowledgments**, but the primary counts its own contribution as soon as the entry is written to volatile buffers, not after it is persisted to stable storage — background persistence happens on a timer. A standby already persists before acknowledging, but the primary does not. So a "committed" operation can be durable on as few as one member; a single ill-timed crash + restart, or a restarted member with a truncated log winning leadership, discards it. This violates the documented assumption that a committed operation is held durably on a majority.
|
||||||
|
|
||||||
|
**What should change (chosen scope — durable commit + promotion fence):**
|
||||||
|
1. **Durable commit:** when the controller acknowledges an operation to a client, that operation's log entry must already be persisted to stable storage on a fault-tolerant majority (the acknowledging member plus ⌊N/2⌋ others). The controller must count its own copy toward the commit majority only once it is durably persisted.
|
||||||
|
2. **Recovery integrity and leadership fence (fail-loud):** a member that cannot reconstruct its committed state from persisted storage must be barred from leadership until it has caught up, and must never become authoritative while behind the cluster's known committed high-water mark — surfacing the failure loudly rather than silently discarding committed operations.
|
||||||
|
|
||||||
|
The mechanism (synchronous persistence vs. later group-commit batching) is a design decision; group-commit batching is deferred to the roadmap as a contract-preserving throughput optimization. See `.kiro/specs/ha-commit-durability/brief.md` for the full discovery analysis, evidence trail, and design decision.
|
||||||
|
|
||||||
|
## Introduction
|
||||||
|
|
||||||
|
This spec closes the durability gap beneath the HA consistency machinery: it makes acknowledgment imply durability. After this spec, any operation the controller reports as successful is guaranteed to be persisted on a majority of members at the moment of acknowledgment, and therefore survives the loss of any minority of members and any subsequent leadership change. It also fences a member that cannot recover its committed state — or that lags the cluster's committed high-water mark — from becoming the authoritative leader, so a truncated or corrupted log can never overwrite acknowledged operations. The client reply path already waits on the commit barrier; this spec strengthens the meaning of "committed" to "durably persisted on a majority," so the existing reply gate enforces the invariant. A testability provision makes the guarantee deterministically verifiable without depending on race timing.
|
||||||
|
|
||||||
|
## Boundary Context
|
||||||
|
|
||||||
|
- **In scope**: the durable-commit invariant on the primary's commit path (self contributes to the commit majority only when durably persisted); pinning the standby's persist-before-acknowledge behavior; the no-premature-acknowledgment guarantee; survival of acknowledged operations across the crash+restart of any minority; preserving cluster liveness (heartbeat/election) while performing durability work; recovery-integrity and leadership fencing for members that cannot reconstruct committed state or lag the committed high-water mark; degraded-quorum preservation of the invariant; a deterministic durability-verification provision; the soak E2E (`src/tools/test-commit-durability.sh`) plus unit coverage.
|
||||||
|
- **Out of scope**: the replication/acknowledgment wire format (`ha-wal-replication`); per-entry majority arithmetic and commit-index advancement (`ha-commit-engine` — consumed, this spec changes only the durability *precondition* on its inputs); demotion truncate/self-restart and rejoin reconciliation of an intact log (`ha-convergence-recovery`); promotion no-op and own-term gate (`ha-single-leader-role`); raw vote-grant arithmetic (`raft-library` — this spec adds a durability/high-water *precondition*, not new vote logic); WAL group-commit batching (roadmap — a contract-preserving throughput optimization, deferred); single-record job-array persistence (`job-array-single-wal-record`, roadmap); entity-bearing snapshots (roadmap); client-side controller failover (`libxce-failover`).
|
||||||
|
- **Adjacent expectations**: relies on `ha-commit-engine`'s commit-position semantics (monotonic, per-entry majority) and on `ha-wal-replication`'s acknowledgment and catch-up machinery; the client reply is already gated on the commit barrier (`ha-commit-barrier-rollout`), so strengthening "committed" to "durable-majority" enforces the invariant without changing the reply path. Downstream, `ha-convergence-recovery`'s single-outcome convergence depends on this spec making "persisted committed state" genuinely durable; a returning member that this spec fences from leadership until caught up complements that spec's truncate-and-restart reconciliation.
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
### Requirement 1: Durable Commit — Acknowledgment Implies Majority Persistence
|
||||||
|
|
||||||
|
**Objective:** As a cluster operator, on behalf of every client, I want any operation the controller acknowledges as successful to already be durably persisted on a majority of controllers, so that no acknowledged work is ever lost to a controller failure or failover.
|
||||||
|
|
||||||
|
#### Acceptance Criteria
|
||||||
|
|
||||||
|
1. When the controller acknowledges a state-changing operation to a client as successful, the controller shall have durably persisted that operation's log entry to stable storage on a majority of cluster members — the acknowledging member plus at least ⌊N/2⌋ other members.
|
||||||
|
2. While evaluating whether an operation is committed, the controller shall count its own log entry toward the commit majority only after that entry is durably persisted to its local stable storage, not when it is merely written to volatile buffers.
|
||||||
|
3. When a standby acknowledges a replicated log entry, the standby shall durably persist that entry to its local stable storage before sending the acknowledgment.
|
||||||
|
4. While a state-changing operation's log entry is not yet durably persisted on a majority of members, the controller shall not report that operation as successful to the requester.
|
||||||
|
5. If any minority of cluster members that contributed to an operation's commit are simultaneously lost and restarted, the acknowledged operation shall remain present and recoverable on the surviving cluster.
|
||||||
|
6. While performing durability persistence for a commit, the controller shall continue to service heartbeat and election processing without stalling, so that durability work does not induce spurious failover.
|
||||||
|
|
||||||
|
### Requirement 2: Recovery Integrity and Leadership Fence
|
||||||
|
|
||||||
|
**Objective:** As a cluster operator, I want a controller that cannot reconstruct its committed state to be barred from leadership until it has caught up, so that a member with a truncated or corrupted log can never become authoritative and discard acknowledged operations.
|
||||||
|
|
||||||
|
#### Acceptance Criteria
|
||||||
|
|
||||||
|
1. When the controller recovers its persisted log at startup, if the log cannot be reconstructed contiguously up to its last known committed boundary, the controller shall not silently discard committed entries and continue as if its committed state were complete.
|
||||||
|
2. If a controller's recovered log does not reach its last known committed boundary, the controller shall be ineligible to become primary until it has caught up to the cluster's committed boundary from a healthy member.
|
||||||
|
3. While a controller's committed position is below the cluster's known committed high-water mark, the controller shall not assume leadership in a way that would discard already-committed operations.
|
||||||
|
4. If a controller cannot recover to a consistent committed state and cannot catch up from a healthy member, the controller shall surface the condition as a loud, operator-visible alert and hold itself permanently non-promotable while staying alive and continuing to retry catch-up — refusing to serve potentially divergent state by refusing leadership, NOT by terminating the process. A controller MUST NEVER self-terminate (`exit`/`abort`) as a designed response to an operational condition: a self-exited controller requires manual admin recovery and degrades quorum meanwhile, whereas one that stays alive and keeps retrying self-heals automatically when a peer returns (see Requirement 2.6). (A genuine programming-invariant `assert`, or an external admin/supervisor kill, is not such a designed response.)
|
||||||
|
5. While the cluster operates with reduced redundancy or minimal quorum, the controller shall preserve the durable-commit guarantee of Requirement 1 across leadership changes, so that no acknowledged operation is lost even when quorum is temporarily degraded.
|
||||||
|
6. When a fenced member finishes catching up to the cluster's committed boundary, the controller shall restore that member's eligibility to participate in leadership.
|
||||||
|
|
||||||
|
### Requirement 3: Deterministic Durability Verification
|
||||||
|
|
||||||
|
**Objective:** As a developer verifying durability, I want the durability behavior to be deterministically controllable in a verification mode, so that the durable-before-acknowledge guarantee can be confirmed without depending on race timing.
|
||||||
|
|
||||||
|
#### Acceptance Criteria
|
||||||
|
|
||||||
|
1. Where a durability-verification mode is enabled, the controller shall make the durability behavior deterministically controllable (for example, by configuring or disabling background persistence) so that an acknowledged operation's durability can be asserted without relying on timing.
|
||||||
|
2. Where durability-verification mode removes reliance on background persistence, an operation that the controller acknowledged as successful shall be recoverable after an immediate, ungraceful termination of a contributing member.
|
||||||
|
3. The controller shall preserve the same externally observable acknowledgment semantics whether or not durability-verification mode is enabled, so that the mode changes timing controllability only, not the durability contract.
|
||||||
|
|
@ -143,3 +143,4 @@
|
||||||
- 4.3: Pre-promotion eligibility gate in `raft_cb_become_leader` (ha.c, post-truncate / pre-CTRL_PRIMARY, returns on refuse). `ha_promotion_gate_decision` (pure, delegates to 3.3's `persist_recover_decision` — table is single source of truth) + `ha_evaluate_promotion_fence` (live: ELIGIBLE clears `in_catch_up`, FENCE sets it, FATAL → 3.3 `persist_recover_fatal_if_unrecoverable`). REFUSE if `persist_recovery_truncated` OR `persist_durable_floor < promotion_safe_lsn` (the existing voter-reply-derived `safe_committed_lsn`, read AFTER `persist_truncate_wal_above` so it reflects 2.2's clamp). `peer_reachable` = some granting voter has a live endpoint (`ectx->votes[i].vote_granted && .ep != NULL`) — so a multi-node election winner ALWAYS has a peer ⇒ FENCE (never premature FATAL, honoring rows 3/5); FATAL only fires no-peer (single-node corrupt = genuinely unrecoverable, row 4). Reuses `in_catch_up` (NOT a parallel fence); release on a later promotion attempt once caught up (durable_floor ≥ HWM, not truncated) → ELIGIBLE clears the fence (Req 2.6, re-evaluation model). `safe_committed_lsn` arithmetic UNTOUCHED. Turned 3.2 rows 1/2/3/5/6 GREEN (0 ignored boot-policy rows remain) + minimal-quorum test (Req 2.5). DEFERRED to 5.1: the live bounded-window catch-up TIMER (grant `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS` to find a peer before FATAL) — here `catch_up_succeeded=false` at the gate, so it FENCEs (peer) or FATALs (no peer) immediately; fail-loud-conservative, never silently leads hollow. NOTE: reviewed in MAIN CONTEXT (review subagent hit a spend limit) — suite green ×1 (Fail:0, all 9 boot-policy/gate tests PASS, test_fact_hook_append passed too since this shell's /tmp is fsync-able). Phase 4 COMPLETE.
|
- 4.3: Pre-promotion eligibility gate in `raft_cb_become_leader` (ha.c, post-truncate / pre-CTRL_PRIMARY, returns on refuse). `ha_promotion_gate_decision` (pure, delegates to 3.3's `persist_recover_decision` — table is single source of truth) + `ha_evaluate_promotion_fence` (live: ELIGIBLE clears `in_catch_up`, FENCE sets it, FATAL → 3.3 `persist_recover_fatal_if_unrecoverable`). REFUSE if `persist_recovery_truncated` OR `persist_durable_floor < promotion_safe_lsn` (the existing voter-reply-derived `safe_committed_lsn`, read AFTER `persist_truncate_wal_above` so it reflects 2.2's clamp). `peer_reachable` = some granting voter has a live endpoint (`ectx->votes[i].vote_granted && .ep != NULL`) — so a multi-node election winner ALWAYS has a peer ⇒ FENCE (never premature FATAL, honoring rows 3/5); FATAL only fires no-peer (single-node corrupt = genuinely unrecoverable, row 4). Reuses `in_catch_up` (NOT a parallel fence); release on a later promotion attempt once caught up (durable_floor ≥ HWM, not truncated) → ELIGIBLE clears the fence (Req 2.6, re-evaluation model). `safe_committed_lsn` arithmetic UNTOUCHED. Turned 3.2 rows 1/2/3/5/6 GREEN (0 ignored boot-policy rows remain) + minimal-quorum test (Req 2.5). DEFERRED to 5.1: the live bounded-window catch-up TIMER (grant `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS` to find a peer before FATAL) — here `catch_up_succeeded=false` at the gate, so it FENCEs (peer) or FATALs (no peer) immediately; fail-loud-conservative, never silently leads hollow. NOTE: reviewed in MAIN CONTEXT (review subagent hit a spend limit) — suite green ×1 (Fail:0, all 9 boot-policy/gate tests PASS, test_fact_hook_append passed too since this shell's /tmp is fsync-able). Phase 4 COMPLETE.
|
||||||
- 5.1: INTEGRATION verification — the end-to-end durable-majority path was ALREADY wired by 2.1/2.2/2.4/4.x (no production gap found); 5.1 adds 3 integration tests + a tiny `#ifdef XCE_TESTING` liveness-observability seam (`persist_test_last_fdatasync_thread`, nm-excluded from prod). `test_durable_majority_commit_end_to_end`: holds durability off (forced fsync-fail), delivers BOTH standby ACKs, asserts committed_lsn/WAIT do NOT release (ACK-majority alone insufficient) — then flush makes self durable ⇒ committed advances + WAIT releases rc=0; asserts `committed_lsn <= durable_lsn`. RED proof: injecting the pre-2.2 `highest_appended_lsn` cap trips `assert(cand <= persist_durable_lsn)`. `test_sole_durable_contributor_loss_survives` (Req 1.5/2.5): property-asserted (committed ⇒ ≥⌊N/2⌋+1 durable copies, two persist dirs, drop one) — LIVE failover proof is 5.2. `test_durability_work_off_iomgr_preserves_liveness` (Req 1.6): seam records the flusher thread id; asserts fdatasync runs off the caller/iomgr thread + heartbeat reads stay <50ms during flushes. GOTCHA: a real persist with registered standbys needs the `XCE_TESTING repl_broadcast_send_hook` (drop) installed before append, else dummy endpoints trip an ASAN overflow in `dataq_enq`. CAVEATS (for 5.3/validation): (1) meson `Fail:0` masks Unity sub-group failures — gate on `grep -c :FAIL: testlog.txt`==0; (2) run on a fsync-able fs (the pre-existing test_fact_hook_append EIO failure is /tmp-overlay-only). 4.3 catch-up-timer-before-FATAL still a follow-up. Reviewed in MAIN CONTEXT (subagent spend limit).
|
- 5.1: INTEGRATION verification — the end-to-end durable-majority path was ALREADY wired by 2.1/2.2/2.4/4.x (no production gap found); 5.1 adds 3 integration tests + a tiny `#ifdef XCE_TESTING` liveness-observability seam (`persist_test_last_fdatasync_thread`, nm-excluded from prod). `test_durable_majority_commit_end_to_end`: holds durability off (forced fsync-fail), delivers BOTH standby ACKs, asserts committed_lsn/WAIT do NOT release (ACK-majority alone insufficient) — then flush makes self durable ⇒ committed advances + WAIT releases rc=0; asserts `committed_lsn <= durable_lsn`. RED proof: injecting the pre-2.2 `highest_appended_lsn` cap trips `assert(cand <= persist_durable_lsn)`. `test_sole_durable_contributor_loss_survives` (Req 1.5/2.5): property-asserted (committed ⇒ ≥⌊N/2⌋+1 durable copies, two persist dirs, drop one) — LIVE failover proof is 5.2. `test_durability_work_off_iomgr_preserves_liveness` (Req 1.6): seam records the flusher thread id; asserts fdatasync runs off the caller/iomgr thread + heartbeat reads stay <50ms during flushes. GOTCHA: a real persist with registered standbys needs the `XCE_TESTING repl_broadcast_send_hook` (drop) installed before append, else dummy endpoints trip an ASAN overflow in `dataq_enq`. CAVEATS (for 5.3/validation): (1) meson `Fail:0` masks Unity sub-group failures — gate on `grep -c :FAIL: testlog.txt`==0; (2) run on a fsync-able fs (the pre-existing test_fact_hook_append EIO failure is /tmp-overlay-only). 4.3 catch-up-timer-before-FATAL still a follow-up. Reviewed in MAIN CONTEXT (subagent spend limit).
|
||||||
- 4.4: Closed the 5.2-soak liveness gap. `raft_cb_become_leader` NO LONGER FATALs synchronously — on FENCE / would-be-FATAL (truncated or durable_floor < HWM) it calls `ha_arm_promotion_fence_timer` (sets `in_catch_up`, schedules a one-shot `delay_task_ms(&c->delaymgr, ha_promotion_fence_window_ms(), 1, promotion_fence_timeout_fn, ctx)`) and RETURNS without becoming primary. The timer callback (`promotion_fence_timeout_fn`, ha.c:702) re-reads LIVE state after the window: caught-up (`!truncated && durable_floor>=HWM`) → release fence, NO FATAL (Req 2.6); peer reachable (`repl_active_standby_count>0`) OR durable_floor advanced past the arm-time baseline → progress possible, leave fenced (existing catch-up machinery / next promotion-gate ELIGIBLE releases it), NO FATAL; ONLY still-truncated/behind + no-peer + no-progress → 3.3 `persist_recover_fatal_if_unrecoverable` (Req 2.4). UAF-safe cancel (`ha_cancel_promotion_fence_timer`: cancelled flag + atomic-exchange the stored ctx; handle cleared before the cancelled-check, mirrors `raft_election_timeout_fn`); cancelled in `ha_fini`; fail-loud if `delay_task_ms` itself fails (never fence indefinitely). Test window seam `ha_set_promotion_fence_window_ms` + driver `ha_arm_promotion_fence_timer_for_test`. 3 tests (defers-fatal-to-timer, released-on-catchup-no-fatal, cancel-prevents-fatal) with RED proof (re-injecting sync FATAL → 3 fails). Reviewed MAIN CONTEXT (subagent spend limit): suite :FAIL:==0, /tmp fsync-able this run. NOTE: this is the bounded-window timer 4.3/5.1 deferred; with it, 5.2's soak should keep the cluster available (truncated rejoiners catch up, not die).
|
- 4.4: Closed the 5.2-soak liveness gap. `raft_cb_become_leader` NO LONGER FATALs synchronously — on FENCE / would-be-FATAL (truncated or durable_floor < HWM) it calls `ha_arm_promotion_fence_timer` (sets `in_catch_up`, schedules a one-shot `delay_task_ms(&c->delaymgr, ha_promotion_fence_window_ms(), 1, promotion_fence_timeout_fn, ctx)`) and RETURNS without becoming primary. The timer callback (`promotion_fence_timeout_fn`, ha.c:702) re-reads LIVE state after the window: caught-up (`!truncated && durable_floor>=HWM`) → release fence, NO FATAL (Req 2.6); peer reachable (`repl_active_standby_count>0`) OR durable_floor advanced past the arm-time baseline → progress possible, leave fenced (existing catch-up machinery / next promotion-gate ELIGIBLE releases it), NO FATAL; ONLY still-truncated/behind + no-peer + no-progress → 3.3 `persist_recover_fatal_if_unrecoverable` (Req 2.4). UAF-safe cancel (`ha_cancel_promotion_fence_timer`: cancelled flag + atomic-exchange the stored ctx; handle cleared before the cancelled-check, mirrors `raft_election_timeout_fn`); cancelled in `ha_fini`; fail-loud if `delay_task_ms` itself fails (never fence indefinitely). Test window seam `ha_set_promotion_fence_window_ms` + driver `ha_arm_promotion_fence_timer_for_test`. 3 tests (defers-fatal-to-timer, released-on-catchup-no-fatal, cancel-prevents-fatal) with RED proof (re-injecting sync FATAL → 3 fails). Reviewed MAIN CONTEXT (subagent spend limit): suite :FAIL:==0, /tmp fsync-able this run. NOTE: this is the bounded-window timer 4.3/5.1 deferred; with it, 5.2's soak should keep the cluster available (truncated rejoiners catch up, not die).
|
||||||
|
- REWORK (2026-06-18, user invariant — a controller MUST NEVER self-terminate): tasks 3.3/4.3/4.4 had implemented Req 2.4 "fail loud" as a SELF-ABORT (`persist_recover_fatal_if_unrecoverable → tlog_exit(); abort()` + `persist_fatal_hook`). That violated the invariant (a self-exited controller needs MANUAL admin recovery + leaves quorum degraded; a live retrying node self-heals when a peer returns) and was the true root cause of the 5.2 soak quorum collapse. CORRECTED: removed the abort/exit terminal entirely. New: `persist_recover_loud_fence_if_unrecoverable(persist_t*)` (NO abort) logs a loud operator-visible ERROR (truncation boundary vs cluster HWM) + sets sticky `_Atomic bool recovery_unrecoverable` (accessors `persist_recovery_unrecoverable`/`persist_clear_recovery_unrecoverable`); the member STAYS ALIVE, permanently non-promotable/fenced (`in_catch_up` + the sticky flag both consulted by the promotion gate), and KEEPS RETRYING catch-up. Decision enum `RECOVER_DECISION_FATAL` → `RECOVER_DECISION_FENCE_UNRECOVERABLE` (non-terminating). 4.4 timer no longer aborts — it re-emits the loud operator alert + re-arms (operator visibility), never exits. Flag + fence CLEAR automatically on catch-up (not truncated AND durable_floor ≥ HWM) → re-eligible (Req 2.6), fully automatic. `persist_fatal_hook` removed. Verified: grep/nm show NO abort/exit/tlog_exit on the recovery path; ASAN suite :FAIL:==0; reworked tests (row4/4b → loud_fence, unrecoverable-stays-alive-loud, released-on-catchup, cancel-stops-realert) PASS. Spec updated: requirements.md Req 2.4 + design.md (decision table rows 4/4b, RecoveryIntegrity, traceability, FATAL-vs-fence open question) reworded "FATAL/terminate" → "LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — NEVER self-exit)". PROJECT-WIDE GUIDANCE added: `src/controller/CLAUDE.md` §"Process Lifecycle (DESIGN INVARIANT)" + `.kiro/steering/ha-async-io.md` — controllers never self-terminate. (`ha_self_restart` execv-restart in convergence/demotion is a SEPARATE primitive, untouched.)
|
||||||
|
|
|
||||||
|
|
@ -6,6 +6,8 @@ The HA layer combines Raft-style consensus election with a single-writer replica
|
||||||
|
|
||||||
**Single-writer constraint**: only the primary writes WAL entries. This avoids Raft log replication complexity while the election mechanism inherits Raft's safety guarantees (term-based voting, freshness gates, monotonic commits).
|
**Single-writer constraint**: only the primary writes WAL entries. This avoids Raft log replication complexity while the election mechanism inherits Raft's safety guarantees (term-based voting, freshness gates, monotonic commits).
|
||||||
|
|
||||||
|
**Process-lifecycle invariant (applies to ALL controller code)**: a controller MUST NEVER self-terminate. It exits only on a genuine crash or an admin/supervisor kill — never via `exit()`/`abort()`/`tlog_exit()` as a designed response to an operational condition (unrecoverable/corrupt WAL, lost quorum, unreachable peer, failed catch-up). "Fail loud" means log a loud operator-visible ERROR + enter a permanent non-promotable/fenced state + **stay alive and keep retrying catch-up** (so the node self-heals when a peer returns, without manual admin recovery) — it does NOT mean exit. See `src/controller/CLAUDE.md` → "Process Lifecycle (DESIGN INVARIANT)" for the authoritative statement.
|
||||||
|
|
||||||
## Async I/O Model
|
## Async I/O Model
|
||||||
|
|
||||||
Three components in `src/lib/`:
|
Three components in `src/lib/`:
|
||||||
|
|
|
||||||
|
|
@ -24,6 +24,15 @@ Links against `libxce` from `src/lib/`.
|
||||||
- Component-specific headers in `include/`
|
- Component-specific headers in `include/`
|
||||||
- Shared types defined in `src/include/xce/` (libxce)
|
- Shared types defined in `src/include/xce/` (libxce)
|
||||||
|
|
||||||
|
## Process Lifecycle (DESIGN INVARIANT — applies to ALL controller tasks)
|
||||||
|
|
||||||
|
**A controller process MUST NEVER self-terminate.** It exits ONLY on (a) a genuine crash (segfault/OOM) or (b) an admin/supervisor kill. A controller must NOT call `exit()`/`_exit()`/`abort()`/`tlog_exit()` (or any self-termination) as a **designed response to an operational condition** — e.g. a recovery-truncated/corrupt WAL, an inability to reach quorum, a peer being unreachable, a failed catch-up, or any cluster-state condition.
|
||||||
|
|
||||||
|
- **Rationale:** a self-exited controller requires **manual admin recovery** to come back and leaves quorum degraded in the meantime. A controller that **stays alive and keeps retrying self-heals automatically** when a peer returns — no admin in the loop.
|
||||||
|
- **"Fail loud" = refuse to participate, not exit.** When a controller detects it cannot safely act (e.g. it cannot reconstruct committed state, or would serve divergent state), it must: log a **loud, operator-visible ERROR** (with the actionable boundary, e.g. durable_floor vs cluster high-water), enter a **permanent non-promotable / fenced** state (never become authoritative, never serve divergent state), **stay alive**, and **keep retrying catch-up** so it recovers automatically when conditions allow. Clear the fenced state once a committed-consistent state is reached.
|
||||||
|
- **The only acceptable `assert()`/abort** is for a genuine *programming-invariant* violation (a bug that means the process is in an impossible state) — that is a "crash," not an operational response. Never use `assert()`/abort to handle expected operational/failure conditions.
|
||||||
|
- **When reviewing or writing controller code:** any new `exit`/`abort`/`tlog_exit`/`FATAL`-style terminal on an operational path is a design violation — replace it with the loud-fence-and-stay-alive pattern above. (Precedent: `ha-commit-durability` recovery-unrecoverable handling — `persist_recover_loud_fence_if_unrecoverable` + the `recovery_unrecoverable` sticky fence, which replaced an earlier incorrect `abort()` terminal.)
|
||||||
|
|
||||||
## HA Protocol
|
## HA Protocol
|
||||||
|
|
||||||
See `.kiro/steering/ha-async-io.md` for the full HA protocol reference: async I/O model (iomgr/vtaskmgr/delaymgr), connection ownership, Raft election, heartbeat model, two-tier replication consistency, dispatch gate, and TLA+ variable mapping. The TLA+ safety invariants and protocol verification checklist below are the authoritative controller-specific verification gates — keep them here.
|
See `.kiro/steering/ha-async-io.md` for the full HA protocol reference: async I/O model (iomgr/vtaskmgr/delaymgr), connection ownership, Raft election, heartbeat model, two-tier replication consistency, dispatch gate, and TLA+ variable mapping. The TLA+ safety invariants and protocol verification checklist below are the authoritative controller-specific verification gates — keep them here.
|
||||||
|
|
@ -67,6 +76,10 @@ The file `raft/raft.tla` contains Diego Ongaro's TLA+ specification of the Raft
|
||||||
|
|
||||||
**Commit Rule** (current-term entry required): Each WAL entry is committed individually via `repl_wait_majority_ack()`. Since entries are single-writer (only the current primary writes), every committed entry is implicitly from the current leader's term. Equivalent to Raft's batched commit rule.
|
**Commit Rule** (current-term entry required): Each WAL entry is committed individually via `repl_wait_majority_ack()`. Since entries are single-writer (only the current primary writes), every committed entry is implicitly from the current leader's term. Equivalent to Raft's batched commit rule.
|
||||||
|
|
||||||
|
**Commit Ordering** (contiguous, in log order — RULE): `committed_lsn` advances only as a contiguous prefix in LSN/append order; entries are never committed or reordered out of log order. The single-writer append lock (`p->lock`) is the serialization point. Durability flushing (incl. group-commit batching) makes a contiguous **prefix** durable only — `durable_lsn` is a prefix watermark, never a per-entry set, so a later entry can never be durable while an earlier one is not. Log order must preserve **causal dependencies** (append a cause before its effect — e.g. job submit before job start); independent concurrent events take an arbitrary but consistent total order, which is linearizable via the single serialization point plus commit-gated reply. Out-of-order / parallel commit is NOT supported by this single-log architecture (it would require per-entity logs with dependency tracking — explicitly out of scope).
|
||||||
|
|
||||||
|
**Commit Durability** (acknowledged ⇒ durable on a majority — TARGET RULE): when the controller replies success to a client, the operation's entry must be `fdatasync`'d on a fault-tolerant majority (the acking member + ⌊N/2⌋ others); a member counts toward the commit majority for an LSN only once that LSN is durable locally (self caps at `durable_lsn`, not the appended tip). **KNOWN GAP — bug-661, spec `ha-commit-durability` (in progress, 2026-06-18):** the commit majority currently counts the primary's *appended* (page-cache) tip via `majority_candidate_locked`, and primary fsync is async (`fsync_worker` timer), so "committed" does NOT yet imply durable-on-a-majority — an acknowledged entry can be durable on a single member and lost across failover. Until that spec lands, do not assume an acked operation survives a crash of the acking minority.
|
||||||
|
|
||||||
### Post-Election Safe Commit Point (Critical)
|
### Post-Election Safe Commit Point (Critical)
|
||||||
|
|
||||||
After winning election, the new primary determines `safe_committed_lsn` from voter state collected during the election. This eliminates a round-trip vs waiting for the first heartbeat reply:
|
After winning election, the new primary determines `safe_committed_lsn` from voter state collected during the election. This eliminates a round-trip vs waiting for the first heartbeat reply:
|
||||||
|
|
@ -89,6 +102,10 @@ When modifying HA protocol code (election, replication, heartbeat), verify:
|
||||||
- [ ] `safe_committed_lsn` is derived from voter replies, not from stale ctrlhello
|
- [ ] `safe_committed_lsn` is derived from voter replies, not from stale ctrlhello
|
||||||
- [ ] WAL truncation on promotion only removes entries above the safe commit point
|
- [ ] WAL truncation on promotion only removes entries above the safe commit point
|
||||||
- [ ] Freshness gate in RequestVote: candidate must have `lastLogTerm >= voter` AND `lastLogIndex >= voter`
|
- [ ] Freshness gate in RequestVote: candidate must have `lastLogTerm >= voter` AND `lastLogIndex >= voter`
|
||||||
|
- [ ] `committed_lsn` advances only as a contiguous LSN-ordered prefix — no out-of-order or per-entry commit
|
||||||
|
- [ ] Durability flushing makes a contiguous prefix durable; `durable_lsn` is a prefix watermark (never a per-entry set)
|
||||||
|
- [ ] Causal dependencies are appended in order (cause before effect, e.g. submit before start)
|
||||||
|
- [ ] (post `ha-commit-durability`) a member counts toward the commit majority only at its durably-persisted LSN; acknowledged ⇒ durable on a majority
|
||||||
|
|
||||||
## HA Design Constraints
|
## HA Design Constraints
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -572,8 +572,10 @@ recover_decision_t ha_promotion_gate_decision(bool recovery_truncated,
|
||||||
* the committed boundary regains eligibility, Req 2.6).
|
* the committed boundary regains eligibility, Req 2.6).
|
||||||
* - FENCE → set the fence, return FENCE (refuse promotion; catch up from a
|
* - FENCE → set the fence, return FENCE (refuse promotion; catch up from a
|
||||||
* healthy peer).
|
* healthy peer).
|
||||||
* - FATAL → run the 3.3 loud-terminate path (logs the truncation boundary,
|
* - FENCE_UNRECOVERABLE → run the 3.3 loud-fence action (logs LOUDLY + sets
|
||||||
* then terminates — or fires persist_fatal_hook in tests).
|
* the sticky non-promotable flag) and ALSO set the in_catch_up
|
||||||
|
* fence. The member STAYS ALIVE, non-promotable, retrying
|
||||||
|
* catch-up — it NEVER self-exits (Req 2.4).
|
||||||
*/
|
*/
|
||||||
recover_decision_t ha_evaluate_promotion_fence(ha_state_t *ha,
|
recover_decision_t ha_evaluate_promotion_fence(ha_state_t *ha,
|
||||||
bool recovery_truncated,
|
bool recovery_truncated,
|
||||||
|
|
@ -603,10 +605,11 @@ recover_decision_t ha_evaluate_promotion_fence(ha_state_t *ha,
|
||||||
"ha-commit-durability Req 2.2)",
|
"ha-commit-durability Req 2.2)",
|
||||||
recovery_truncated, durable_floor, safe_hwm);
|
recovery_truncated, durable_floor, safe_hwm);
|
||||||
break;
|
break;
|
||||||
case RECOVER_DECISION_FATAL: {
|
case RECOVER_DECISION_FENCE_UNRECOVERABLE: {
|
||||||
/* Bounded-unrecoverable: route through the 3.3 loud-terminate action,
|
/* Unrecoverable: route through the 3.3 loud-fence action — logs LOUDLY
|
||||||
* which logs the truncation boundary at ERROR then terminates (or fires
|
* and sets the sticky non-promotable flag. Also hold the in_catch_up
|
||||||
* persist_fatal_hook under test). Req 2.4. */
|
* fence so the member refuses promotion. The process STAYS ALIVE and
|
||||||
|
* keeps retrying catch-up; it NEVER self-exits (Req 2.4). */
|
||||||
recover_policy_inputs_t in = {
|
recover_policy_inputs_t in = {
|
||||||
.verdict = recovery_truncated ? RECOVER_TRUNCATED
|
.verdict = recovery_truncated ? RECOVER_TRUNCATED
|
||||||
: RECOVER_CLEAN,
|
: RECOVER_CLEAN,
|
||||||
|
|
@ -616,25 +619,36 @@ recover_decision_t ha_evaluate_promotion_fence(ha_state_t *ha,
|
||||||
.durable_floor_lsn = durable_floor,
|
.durable_floor_lsn = durable_floor,
|
||||||
.cluster_hwm_lsn = safe_hwm,
|
.cluster_hwm_lsn = safe_hwm,
|
||||||
};
|
};
|
||||||
persist_recover_fatal_if_unrecoverable(&in);
|
persist_t *p = (ha != NULL && ha->raft_c != NULL) ? &ha->raft_c->persist
|
||||||
|
: NULL;
|
||||||
|
persist_recover_loud_fence_if_unrecoverable(p, &in);
|
||||||
|
if (ha != NULL)
|
||||||
|
atomic_store_explicit(&ha->in_catch_up, 1, memory_order_release);
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return d;
|
return d;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* ── ha-commit-durability task 4.4: bounded-window catch-up before FATAL ─────
|
/* ── ha-commit-durability task 4.4: bounded-window catch-up re-alert ─────────
|
||||||
*
|
*
|
||||||
* The 4.3 gate above can DECIDE FATAL synchronously, but a member that won an
|
* (never-self-exit rework 2026-06-18.) A member that wins an election while
|
||||||
* election with no peer reachable AT THE PROMOTION INSTANT may simply be seeing
|
* recovery-truncated / behind the HWM with no peer reachable AT THE PROMOTION
|
||||||
* a transient no-peer snapshot during aggressive failover (task-5.2 soak: a
|
* INSTANT is FENCED (in_catch_up set, non-promotable) and STAYS ALIVE, retrying
|
||||||
* recovery-truncated winner self-FATAL'd immediately and repeatedly until
|
* catch-up indefinitely. This one-shot timer exists ONLY for operator
|
||||||
* quorum was lost — an availability collapse, not durable loss). Per the 3.2
|
* VISIBILITY: when it fires, if the member is STILL unrecoverable (still
|
||||||
* decision table the FATAL terminal must be reached ONLY after a BOUNDED
|
* truncated/behind AND no peer AND no progress) it RE-EMITS the loud,
|
||||||
* failure: the member is fenced (in_catch_up), given DEFAULT_RECOVERY_CATCHUP_-
|
* operator-visible alert via persist_recover_loud_fence_if_unrecoverable (which
|
||||||
* WINDOW_MS to find a peer / make catch-up progress, and FATAL'd from a one-shot
|
* also keeps the sticky non-promotable flag set) — it NEVER terminates the
|
||||||
* timer only if it is STILL hollow with no peer and no progress when the window
|
* process. If the member has caught up (not truncated AND durable_floor >=
|
||||||
* elapses.
|
* HWM) the timer RELEASES the fence (Req 2.6). If progress is possible (peer
|
||||||
|
* back, or durable floor advanced) it leaves the member fenced under the
|
||||||
|
* existing catch-up machinery.
|
||||||
|
*
|
||||||
|
* Earlier this timer DEFERRED-then-ABORTED. The abort is GONE: a controller
|
||||||
|
* MUST NEVER self-terminate as a designed response to an operational condition
|
||||||
|
* (user invariant, 2026-06-18). The task-5.2 soak quorum collapse was caused
|
||||||
|
* by that self-abort; staying alive + retrying self-heals when a peer returns.
|
||||||
*
|
*
|
||||||
* The window is a settable seam so the timer test runs fast. Default =
|
* The window is a settable seam so the timer test runs fast. Default =
|
||||||
* DEFAULT_RECOVERY_CATCHUP_WINDOW_MS (persist.h). */
|
* DEFAULT_RECOVERY_CATCHUP_WINDOW_MS (persist.h). */
|
||||||
|
|
@ -684,21 +698,26 @@ void ha_cancel_promotion_fence_timer(ha_state_t *ha)
|
||||||
atomic_store_explicit(&t->cancelled, 1, memory_order_release);
|
atomic_store_explicit(&t->cancelled, 1, memory_order_release);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* promotion_fence_timeout_fn: the bounded-window timer callback. Owns its ctx —
|
/* Forward decl: the timeout callback re-arms itself for periodic re-alert. */
|
||||||
* frees it on exit. Re-evaluates the LIVE recovery/durability state and reaches
|
static void ha_arm_promotion_fence_timer(ha_state_t *ha, cluster_t *c,
|
||||||
* the 3.3 FATAL ONLY on a real bounded failure (Req 2.4):
|
bool truncated, uint64_t durable_floor,
|
||||||
* - still recovery-truncated/behind AND no healthy peer reachable AND no
|
uint64_t target_hwm);
|
||||||
* catch-up progress (durable_floor has not advanced past the baseline)
|
|
||||||
* → persist_recover_fatal_if_unrecoverable (fail loud).
|
/* promotion_fence_timeout_fn: the bounded-window re-alert timer callback. Owns
|
||||||
|
* its ctx — frees it on exit. Re-evaluates the LIVE recovery/durability state.
|
||||||
|
* It NEVER terminates the process (never-self-exit invariant, 2026-06-18):
|
||||||
* - caught up (NOT truncated AND durable_floor >= target_hwm) → release the
|
* - caught up (NOT truncated AND durable_floor >= target_hwm) → release the
|
||||||
* in_catch_up fence; do NOT FATAL (Req 2.6 — eligible again).
|
* in_catch_up fence + clear the sticky unrecoverable flag; eligible again
|
||||||
* - a healthy peer is reachable now, OR catch-up is making progress but is not
|
* (Req 2.6).
|
||||||
* yet complete → leave the member fenced under the existing catch-up
|
* - a healthy peer is reachable now, OR catch-up is making progress → leave
|
||||||
* machinery and do NOT FATAL while progress is possible. The single bounded
|
* the member fenced under the existing catch-up machinery (it will release
|
||||||
* window has elapsed, so we do not re-arm: a member with a live peer is no
|
* via ha_evaluate_promotion_fence → ELIGIBLE when it completes); do not
|
||||||
* longer in the "no peer + no progress" bounded-failure state the FATAL
|
* re-arm the re-alert while progress is happening.
|
||||||
* guards, and the existing catch-up / re-election path will release the
|
* - STILL unrecoverable (truncated/behind AND no peer AND no progress) →
|
||||||
* fence (ha_evaluate_promotion_fence → ELIGIBLE) when it completes. */
|
* RE-EMIT the loud operator alert via persist_recover_loud_fence_if_-
|
||||||
|
* unrecoverable (keeps the sticky flag + in_catch_up set) and RE-ARM the
|
||||||
|
* timer so the alert keeps repeating until a peer returns. The process
|
||||||
|
* STAYS ALIVE and keeps retrying — it NEVER aborts. */
|
||||||
static int promotion_fence_timeout_fn(void *arg)
|
static int promotion_fence_timeout_fn(void *arg)
|
||||||
{
|
{
|
||||||
promotion_fence_ctx_t *t = (promotion_fence_ctx_t *)arg;
|
promotion_fence_ctx_t *t = (promotion_fence_ctx_t *)arg;
|
||||||
|
|
@ -721,7 +740,7 @@ static int promotion_fence_timeout_fn(void *arg)
|
||||||
|
|
||||||
if (!c) { free(t); return 0; } /* defensive: no persist to evaluate */
|
if (!c) { free(t); return 0; } /* defensive: no persist to evaluate */
|
||||||
|
|
||||||
/* Re-read the LIVE state after the bounded window. */
|
/* Re-read the LIVE state after the window. */
|
||||||
bool truncated = persist_recovery_truncated(&c->persist);
|
bool truncated = persist_recovery_truncated(&c->persist);
|
||||||
uint64_t durable_floor = persist_durable_floor(&c->persist);
|
uint64_t durable_floor = persist_durable_floor(&c->persist);
|
||||||
bool below_hwm = durable_floor < t->target_hwm;
|
bool below_hwm = durable_floor < t->target_hwm;
|
||||||
|
|
@ -731,9 +750,11 @@ static int promotion_fence_timeout_fn(void *arg)
|
||||||
: false;
|
: false;
|
||||||
|
|
||||||
/* Caught up: not truncated AND durable position reached the boundary →
|
/* Caught up: not truncated AND durable position reached the boundary →
|
||||||
* release the fence; the member is eligible on its next election (Req 2.6). */
|
* release the fence + clear the sticky flag; the member is eligible on its
|
||||||
|
* next election (Req 2.6). */
|
||||||
if (!truncated && !below_hwm) {
|
if (!truncated && !below_hwm) {
|
||||||
atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release);
|
atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release);
|
||||||
|
persist_clear_recovery_unrecoverable(&c->persist);
|
||||||
INFO("HA: promotion fence RELEASED after catch-up — durable_floor=%"
|
INFO("HA: promotion fence RELEASED after catch-up — durable_floor=%"
|
||||||
PRIu64 " >= cluster HWM=%" PRIu64 " (ha-commit-durability Req 2.6)",
|
PRIu64 " >= cluster HWM=%" PRIu64 " (ha-commit-durability Req 2.6)",
|
||||||
durable_floor, t->target_hwm);
|
durable_floor, t->target_hwm);
|
||||||
|
|
@ -742,14 +763,13 @@ static int promotion_fence_timeout_fn(void *arg)
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Still hollow, but a peer is reachable or catch-up advanced the durable
|
/* Still hollow, but a peer is reachable or catch-up advanced the durable
|
||||||
* floor: progress is possible — do NOT FATAL. Leave the member fenced
|
* floor: progress is possible. Leave the member fenced (in_catch_up stays
|
||||||
* (in_catch_up stays set) under the existing catch-up machinery; the bounded
|
* set) under the existing catch-up machinery; do not re-arm the re-alert
|
||||||
* window has elapsed so we do not re-arm. A member with a live peer is no
|
* while progress is happening. */
|
||||||
* longer in the no-peer bounded-failure state the FATAL guards. */
|
|
||||||
if (peer_reachable || made_progress) {
|
if (peer_reachable || made_progress) {
|
||||||
INFO("HA: promotion fence catch-up window elapsed but progress is "
|
INFO("HA: promotion fence catch-up window elapsed but progress is "
|
||||||
"possible (peer_reachable=%d durable_floor=%" PRIu64 " baseline=%"
|
"possible (peer_reachable=%d durable_floor=%" PRIu64 " baseline=%"
|
||||||
PRIu64 " HWM=%" PRIu64 ") — remaining fenced, NOT FATAL "
|
PRIu64 " HWM=%" PRIu64 ") — remaining fenced, catching up "
|
||||||
"(ha-commit-durability Req 2.2)",
|
"(ha-commit-durability Req 2.2)",
|
||||||
peer_reachable, durable_floor, t->baseline_durable_floor,
|
peer_reachable, durable_floor, t->baseline_durable_floor,
|
||||||
t->target_hwm);
|
t->target_hwm);
|
||||||
|
|
@ -757,9 +777,11 @@ static int promotion_fence_timeout_fn(void *arg)
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Bounded failure: STILL truncated/behind AND no healthy peer AND no
|
/* STILL unrecoverable: truncated/behind AND no healthy peer AND no catch-up
|
||||||
* catch-up progress after the window → fail loud (Req 2.4). This is the
|
* progress. RE-EMIT the loud operator alert and keep the member fenced +
|
||||||
* ONLY path that reaches the 3.3 FATAL — never synchronously at promotion. */
|
* non-promotable + ALIVE (Req 2.4) — NEVER terminate. Re-arm the re-alert
|
||||||
|
* timer so the operator keeps hearing about it until a peer returns. */
|
||||||
|
uint64_t target_hwm = t->target_hwm; /* capture before free (UAF guard) */
|
||||||
{
|
{
|
||||||
recover_policy_inputs_t in = {
|
recover_policy_inputs_t in = {
|
||||||
.verdict = truncated ? RECOVER_TRUNCATED
|
.verdict = truncated ? RECOVER_TRUNCATED
|
||||||
|
|
@ -768,27 +790,27 @@ static int promotion_fence_timeout_fn(void *arg)
|
||||||
.healthy_peer_reachable = false,
|
.healthy_peer_reachable = false,
|
||||||
.catch_up_succeeded = false,
|
.catch_up_succeeded = false,
|
||||||
.durable_floor_lsn = durable_floor,
|
.durable_floor_lsn = durable_floor,
|
||||||
.cluster_hwm_lsn = t->target_hwm,
|
.cluster_hwm_lsn = target_hwm,
|
||||||
};
|
};
|
||||||
ERROR("HA: promotion catch-up window (%u ms) elapsed with no peer and "
|
/* Loud, operator-visible alert + sticky non-promotable flag. STAYS
|
||||||
"no progress — recovery_truncated=%d durable_floor=%" PRIu64
|
* ALIVE: no abort/exit anywhere on this path. */
|
||||||
" < cluster HWM=%" PRIu64 "; failing loud rather than serving "
|
persist_recover_loud_fence_if_unrecoverable(&c->persist, &in);
|
||||||
"divergent state (ha-commit-durability Req 2.4)",
|
|
||||||
ha_promotion_fence_window_ms(), truncated, durable_floor,
|
|
||||||
t->target_hwm);
|
|
||||||
persist_recover_fatal_if_unrecoverable(&in);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
free(t);
|
free(t);
|
||||||
|
|
||||||
|
/* Re-arm so the loud alert repeats periodically while the member remains
|
||||||
|
* unrecoverable. The member keeps retrying catch-up; the next firing
|
||||||
|
* re-evaluates and releases the fence once a peer returns and it catches up. */
|
||||||
|
ha_arm_promotion_fence_timer(ha, c, truncated, durable_floor, target_hwm);
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* ha_arm_promotion_fence_timer: fence the member (set in_catch_up) and schedule
|
/* ha_arm_promotion_fence_timer: fence the member (set in_catch_up) and schedule
|
||||||
* the one-shot bounded-window FATAL timer. Cancels any previously-armed timer
|
* the one-shot bounded-window RE-ALERT timer. Cancels any previously-armed
|
||||||
* first so a re-evaluation never leaks or double-fires. Returns 0 on success.
|
* timer first so a re-evaluation never leaks or double-fires. On a failure to
|
||||||
* On a failure to schedule the timer it FATALs immediately via the 3.3 action so
|
* allocate/schedule the timer it emits the loud unrecoverable fence ONCE (stays
|
||||||
* the bounded-failure terminal is never silently lost (a member that could not
|
* alive, non-promotable) — it NEVER aborts (never-self-exit invariant). */
|
||||||
* arm its own catch-up deadline must not run on indefinitely). */
|
|
||||||
static void ha_arm_promotion_fence_timer(ha_state_t *ha, cluster_t *c,
|
static void ha_arm_promotion_fence_timer(ha_state_t *ha, cluster_t *c,
|
||||||
bool truncated, uint64_t durable_floor,
|
bool truncated, uint64_t durable_floor,
|
||||||
uint64_t target_hwm)
|
uint64_t target_hwm)
|
||||||
|
|
@ -801,7 +823,7 @@ static void ha_arm_promotion_fence_timer(ha_state_t *ha, cluster_t *c,
|
||||||
|
|
||||||
promotion_fence_ctx_t *t = calloc(1, sizeof(*t));
|
promotion_fence_ctx_t *t = calloc(1, sizeof(*t));
|
||||||
if (!t)
|
if (!t)
|
||||||
goto fail_loud;
|
goto loud_fence_no_timer;
|
||||||
|
|
||||||
atomic_init(&t->cancelled, 0);
|
atomic_init(&t->cancelled, 0);
|
||||||
t->ha = ha;
|
t->ha = ha;
|
||||||
|
|
@ -816,20 +838,22 @@ static void ha_arm_promotion_fence_timer(ha_state_t *ha, cluster_t *c,
|
||||||
if (rc < 0) {
|
if (rc < 0) {
|
||||||
WARN("ha_arm_promotion_fence_timer: delay_task_ms failed: %d", rc);
|
WARN("ha_arm_promotion_fence_timer: delay_task_ms failed: %d", rc);
|
||||||
free(t);
|
free(t);
|
||||||
goto fail_loud;
|
goto loud_fence_no_timer;
|
||||||
}
|
}
|
||||||
|
|
||||||
atomic_store_explicit(&ha->promotion_fence_ctx, t, memory_order_release);
|
atomic_store_explicit(&ha->promotion_fence_ctx, t, memory_order_release);
|
||||||
INFO("HA: promotion FENCED + bounded catch-up window armed (%u ms) — "
|
INFO("HA: promotion FENCED + catch-up re-alert window armed (%u ms) — "
|
||||||
"recovery_truncated=%d durable_floor=%" PRIu64 " < cluster HWM=%" PRIu64
|
"recovery_truncated=%d durable_floor=%" PRIu64 " < cluster HWM=%" PRIu64
|
||||||
"; FATAL deferred to the timer (ha-commit-durability Req 2.2/2.4)",
|
"; staying ALIVE + non-promotable, retrying catch-up "
|
||||||
|
"(ha-commit-durability Req 2.2/2.4)",
|
||||||
window_ms, truncated, durable_floor, target_hwm);
|
window_ms, truncated, durable_floor, target_hwm);
|
||||||
return;
|
return;
|
||||||
|
|
||||||
fail_loud:
|
loud_fence_no_timer:
|
||||||
{
|
{
|
||||||
/* Could not arm the bounded deadline — reach the FATAL terminal now
|
/* Could not arm the re-alert timer — still emit the loud unrecoverable
|
||||||
* rather than fence indefinitely. */
|
* fence once so the operator sees it; the member stays fenced
|
||||||
|
* (in_catch_up set above) + non-promotable + ALIVE. NO abort. */
|
||||||
recover_policy_inputs_t in = {
|
recover_policy_inputs_t in = {
|
||||||
.verdict = truncated ? RECOVER_TRUNCATED
|
.verdict = truncated ? RECOVER_TRUNCATED
|
||||||
: RECOVER_CLEAN,
|
: RECOVER_CLEAN,
|
||||||
|
|
@ -839,7 +863,7 @@ fail_loud:
|
||||||
.durable_floor_lsn = durable_floor,
|
.durable_floor_lsn = durable_floor,
|
||||||
.cluster_hwm_lsn = target_hwm,
|
.cluster_hwm_lsn = target_hwm,
|
||||||
};
|
};
|
||||||
persist_recover_fatal_if_unrecoverable(&in);
|
persist_recover_loud_fence_if_unrecoverable(&c->persist, &in);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -919,10 +943,16 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
|
||||||
* the in_catch_up fence and return WITHOUT becoming primary; the member
|
* the in_catch_up fence and return WITHOUT becoming primary; the member
|
||||||
* catches up from a healthy peer and a later promotion attempt passes the
|
* catches up from a healthy peer and a later promotion attempt passes the
|
||||||
* gate (Req 2.2/2.6). Only the production path (real persist) is gated; the
|
* gate (Req 2.2/2.6). Only the production path (real persist) is gated; the
|
||||||
* c==NULL unit shim has no persist and skips it. */
|
* c==NULL unit shim has no persist and skips it.
|
||||||
|
*
|
||||||
|
* Belt-and-suspenders (never-self-exit rework 2026-06-18): a member that has
|
||||||
|
* been marked sticky-unrecoverable (persist_recovery_unrecoverable) must
|
||||||
|
* NEVER become primary while that flag is set — it stays alive, fenced, and
|
||||||
|
* keeps retrying catch-up. The flag clears automatically on catch-up. */
|
||||||
if (c != NULL) {
|
if (c != NULL) {
|
||||||
bool truncated = persist_recovery_truncated(&c->persist);
|
bool truncated = persist_recovery_truncated(&c->persist);
|
||||||
uint64_t durable_floor = persist_durable_floor(&c->persist);
|
uint64_t durable_floor = persist_durable_floor(&c->persist);
|
||||||
|
bool unrecoverable = persist_recovery_unrecoverable(&c->persist);
|
||||||
|
|
||||||
/* A healthy peer is reachable for catch-up iff some voter granted with a
|
/* A healthy peer is reachable for catch-up iff some voter granted with a
|
||||||
* live endpoint survives this election (these become the standby
|
* live endpoint survives this election (these become the standby
|
||||||
|
|
@ -943,36 +973,36 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
|
||||||
|
|
||||||
/* catch_up_succeeded is false at the promotion-eligibility evaluation:
|
/* catch_up_succeeded is false at the promotion-eligibility evaluation:
|
||||||
* a hollow member never silently proceeds to leadership. We evaluate
|
* a hollow member never silently proceeds to leadership. We evaluate
|
||||||
* the PURE 3.2/3.3 decision here and DEFER its terminal — task 4.4 does
|
* the PURE 3.2/3.3 decision here. A FENCE or an UNRECOVERABLE
|
||||||
* NOT reach the 3.3 FATAL synchronously at the promotion instant. A
|
* (truncated/behind + no peer right now) both fence the member
|
||||||
* FENCE or a would-be-FATAL (truncated/behind + no peer right now) both
|
* (in_catch_up) and arm the catch-up re-alert timer; the member STAYS
|
||||||
* fence the member (in_catch_up) and arm the bounded catch-up window;
|
* ALIVE and keeps retrying — it NEVER self-exits (never-self-exit
|
||||||
* the FATAL is reached ONLY from that timer after a real bounded failure
|
* invariant, 2026-06-18; this is what fixes the task-5.2 soak collapse
|
||||||
* (still hollow + no peer + no progress). This avoids the task-5.2 soak
|
* where a self-aborting recovering winner repeatedly dropped quorum). */
|
||||||
* collapse where a transient no-peer snapshot during aggressive failover
|
|
||||||
* self-FATAL'd a recovering winner repeatedly until quorum was lost. */
|
|
||||||
recover_decision_t gate = ha_promotion_gate_decision(
|
recover_decision_t gate = ha_promotion_gate_decision(
|
||||||
truncated, durable_floor, promotion_safe_lsn,
|
truncated, durable_floor, promotion_safe_lsn,
|
||||||
peer_reachable, /*catch_up_succeeded=*/false);
|
peer_reachable, /*catch_up_succeeded=*/false);
|
||||||
|
|
||||||
if (gate == RECOVER_DECISION_ELIGIBLE) {
|
if (gate == RECOVER_DECISION_ELIGIBLE && !unrecoverable) {
|
||||||
/* ELIGIBLE: release the fence (a caught-up member regains
|
/* ELIGIBLE and not sticky-unrecoverable: release the fence (a
|
||||||
* eligibility, Req 2.6) and cancel any pending bounded-window FATAL
|
* caught-up member regains eligibility, Req 2.6), clear the sticky
|
||||||
* timer — a member that is now promotable must never be FATAL'd by a
|
* flag if set, and cancel any pending re-alert timer. Proceed to
|
||||||
* stale timer. Proceed to authoritative promotion below. */
|
* authoritative promotion below. */
|
||||||
atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release);
|
atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release);
|
||||||
|
persist_clear_recovery_unrecoverable(&c->persist);
|
||||||
ha_cancel_promotion_fence_timer(ha);
|
ha_cancel_promotion_fence_timer(ha);
|
||||||
} else {
|
} else {
|
||||||
/* FENCE or would-be-FATAL: refuse promotion — do NOT become
|
/* FENCE, UNRECOVERABLE, or sticky-unrecoverable: refuse promotion —
|
||||||
* CTRL_PRIMARY. Fence the member and arm the bounded catch-up
|
* do NOT become CTRL_PRIMARY. Fence the member and arm the catch-up
|
||||||
* window; the 3.3 FATAL is deferred to the timer (Req 2.2/2.4).
|
* re-alert timer. The member STAYS ALIVE, non-promotable, retrying
|
||||||
* Never FATAL synchronously, never fence indefinitely. */
|
* catch-up (Req 2.2/2.4). NEVER self-exit, never fence silently. */
|
||||||
WARN("HA: promotion REFUSED (decision=%d) — recovery_truncated=%d "
|
WARN("HA: promotion REFUSED (decision=%d unrecoverable=%d) — "
|
||||||
"durable_floor=%" PRIu64 " cluster_hwm=%" PRIu64
|
"recovery_truncated=%d durable_floor=%" PRIu64
|
||||||
" peer_reachable=%d; fencing + arming bounded catch-up window "
|
" cluster_hwm=%" PRIu64 " peer_reachable=%d; fencing + arming "
|
||||||
|
"catch-up re-alert window, staying ALIVE "
|
||||||
"(ha-commit-durability Req 2.2/2.4)",
|
"(ha-commit-durability Req 2.2/2.4)",
|
||||||
(int)gate, truncated, durable_floor, promotion_safe_lsn,
|
(int)gate, unrecoverable, truncated, durable_floor,
|
||||||
peer_reachable);
|
promotion_safe_lsn, peer_reachable);
|
||||||
ha_arm_promotion_fence_timer(ha, c, truncated, durable_floor,
|
ha_arm_promotion_fence_timer(ha, c, truncated, durable_floor,
|
||||||
promotion_safe_lsn);
|
promotion_safe_lsn);
|
||||||
return;
|
return;
|
||||||
|
|
|
||||||
|
|
@ -520,7 +520,10 @@ void ha_compute_safe_committed_lsn(ha_elect_ctx_t *ctx,
|
||||||
* 2.5, 2.6). Maps the promoting member's recovery/durability state to the Boot
|
* 2.5, 2.6). Maps the promoting member's recovery/durability state to the Boot
|
||||||
* Policy Decision Table outcome by building recover_policy_inputs_t and
|
* Policy Decision Table outcome by building recover_policy_inputs_t and
|
||||||
* delegating to persist_recover_decision (DRY — the 3.2/3.3 decision table is
|
* delegating to persist_recover_decision (DRY — the 3.2/3.3 decision table is
|
||||||
* the single source of truth; this gate does not re-derive it):
|
* the single source of truth; this gate does not re-derive it). Note the
|
||||||
|
* FENCE_UNRECOVERABLE terminal is a LOUD PERMANENT NON-promotable fence that
|
||||||
|
* STAYS ALIVE and keeps retrying catch-up — it is NEVER process exit
|
||||||
|
* (never-self-exit invariant, 2026-06-18):
|
||||||
* - recovery_truncated → the task-3.1 structural-corruption flag.
|
* - recovery_truncated → the task-3.1 structural-corruption flag.
|
||||||
* - durable_floor → the local CLAMPED durable position (read AFTER the
|
* - durable_floor → the local CLAMPED durable position (read AFTER the
|
||||||
* promotion-path persist_truncate_wal_above so the
|
* promotion-path persist_truncate_wal_above so the
|
||||||
|
|
@ -547,9 +550,12 @@ recover_decision_t ha_promotion_gate_decision(bool recovery_truncated,
|
||||||
* boundary regains eligibility — Req 2.6).
|
* boundary regains eligibility — Req 2.6).
|
||||||
* - FENCE → sets the in_catch_up fence (refuse promotion; catch up from a
|
* - FENCE → sets the in_catch_up fence (refuse promotion; catch up from a
|
||||||
* healthy peer) and returns FENCE.
|
* healthy peer) and returns FENCE.
|
||||||
* - FATAL → routes through persist_recover_fatal_if_unrecoverable (logs the
|
* - FENCE_UNRECOVERABLE → routes through
|
||||||
* truncation boundary at ERROR then terminates; the test seam
|
* persist_recover_loud_fence_if_unrecoverable (logs LOUDLY at
|
||||||
* persist_fatal_hook captures it) and returns FATAL.
|
* ERROR + sets the sticky persist recovery_unrecoverable flag),
|
||||||
|
* also sets the in_catch_up fence, and returns
|
||||||
|
* FENCE_UNRECOVERABLE. The process STAYS ALIVE, non-promotable,
|
||||||
|
* retrying catch-up — it NEVER self-exits (Req 2.4).
|
||||||
* Reuses the existing in_catch_up fence — does NOT invent a parallel fence. */
|
* Reuses the existing in_catch_up fence — does NOT invent a parallel fence. */
|
||||||
recover_decision_t ha_evaluate_promotion_fence(ha_state_t *ha,
|
recover_decision_t ha_evaluate_promotion_fence(ha_state_t *ha,
|
||||||
bool recovery_truncated,
|
bool recovery_truncated,
|
||||||
|
|
@ -558,34 +564,37 @@ recover_decision_t ha_evaluate_promotion_fence(ha_state_t *ha,
|
||||||
bool peer_reachable,
|
bool peer_reachable,
|
||||||
bool catch_up_succeeded);
|
bool catch_up_succeeded);
|
||||||
|
|
||||||
/* ── ha-commit-durability task 4.4: bounded-window catch-up before FATAL ─────
|
/* ── ha-commit-durability task 4.4: catch-up re-alert window ─────────────────
|
||||||
|
* (never-self-exit rework 2026-06-18.)
|
||||||
*
|
*
|
||||||
* The promotion gate (4.3) no longer reaches the 3.3 FATAL synchronously at the
|
* When a recovery-truncated / behind member wins an election with no reachable
|
||||||
* promotion instant. When a recovery-truncated / behind member wins an election
|
* peer AT THAT INSTANT (a transient no-peer snapshot during aggressive
|
||||||
* with no reachable peer AT THAT INSTANT (a transient no-peer snapshot during
|
* failover), it FENCES (sets in_catch_up, does NOT become primary) and arms a
|
||||||
* aggressive failover), it FENCES (sets in_catch_up, does NOT become primary)
|
* delaymgr re-alert timer for DEFAULT_RECOVERY_CATCHUP_WINDOW_MS. When the
|
||||||
* and arms a one-shot delaymgr timer for DEFAULT_RECOVERY_CATCHUP_WINDOW_MS.
|
* timer fires, if the member is STILL recovery-truncated/behind AND no healthy
|
||||||
* The 3.3 FATAL is reached ONLY from that timer, and ONLY if the member is STILL
|
* peer is reachable AND no catch-up progress was made, it RE-EMITS the loud
|
||||||
* recovery-truncated/behind AND no healthy peer is reachable AND no catch-up
|
* operator alert (persist_recover_loud_fence_if_unrecoverable) and RE-ARMS —
|
||||||
* progress was made when it fires (Req 2.4). Catch-up that brings durable_floor
|
* the process STAYS ALIVE and keeps retrying catch-up INDEFINITELY. It NEVER
|
||||||
* >= HWM releases the fence and cancels the timer (Req 2.6); shutdown cancels it
|
* terminates: a controller must never self-exit as a designed response to an
|
||||||
* too — never a synchronous FATAL, never an indefinite fence.
|
* operational condition. Catch-up that brings durable_floor >= HWM releases
|
||||||
|
* the fence + clears the sticky flag and stops the re-alert (Req 2.6); shutdown
|
||||||
|
* cancels it too.
|
||||||
*
|
*
|
||||||
* ha_promotion_fence_window_ms: the bounded catch-up window in milliseconds.
|
* ha_promotion_fence_window_ms: the re-alert interval in milliseconds.
|
||||||
* Defaults to DEFAULT_RECOVERY_CATCHUP_WINDOW_MS; tests override it (via
|
* Defaults to DEFAULT_RECOVERY_CATCHUP_WINDOW_MS; tests override it (via
|
||||||
* ha_set_promotion_fence_window_ms) to a small value so the timer fires fast. */
|
* ha_set_promotion_fence_window_ms) to a small value so the timer fires fast. */
|
||||||
uint32_t ha_promotion_fence_window_ms(void);
|
uint32_t ha_promotion_fence_window_ms(void);
|
||||||
|
|
||||||
/* ha_cancel_promotion_fence_timer: cancel a pending bounded-window FATAL timer
|
/* ha_cancel_promotion_fence_timer: cancel a pending re-alert timer (idempotent;
|
||||||
* (idempotent; no-op when none is armed). Called when the member catches up
|
* no-op when none is armed). Called when the member catches up (fence released)
|
||||||
* (fence released) or on shutdown, so neither a caught-up nor a stopped member
|
* or on shutdown, so a stopped member stops re-alerting. It never gated a
|
||||||
* ever reaches the deferred FATAL. */
|
* process exit — the timer never terminates the process. */
|
||||||
void ha_cancel_promotion_fence_timer(ha_state_t *ha);
|
void ha_cancel_promotion_fence_timer(ha_state_t *ha);
|
||||||
|
|
||||||
#ifdef XCE_TESTING
|
#ifdef XCE_TESTING
|
||||||
/* Test seam: override the bounded catch-up window so the deferred-FATAL timer
|
/* Test seam: override the re-alert window so the timer fires deterministically
|
||||||
* fires deterministically fast. 0 restores the DEFAULT_RECOVERY_CATCHUP_WINDOW_MS
|
* fast. 0 restores the DEFAULT_RECOVERY_CATCHUP_WINDOW_MS default. XCE_TESTING
|
||||||
* default. XCE_TESTING only — compiled out of production. */
|
* only — compiled out of production. */
|
||||||
void ha_set_promotion_fence_window_ms(uint32_t ms);
|
void ha_set_promotion_fence_window_ms(uint32_t ms);
|
||||||
#endif
|
#endif
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -449,6 +449,11 @@ int persist_init(persist_t* p, persist_config_t* cfg, cluster_t* c)
|
||||||
* replay outcome (RECOVER_TRUNCATED only on a structural log corruption). */
|
* replay outcome (RECOVER_TRUNCATED only on a structural log corruption). */
|
||||||
p->recovery_verdict = RECOVER_CLEAN;
|
p->recovery_verdict = RECOVER_CLEAN;
|
||||||
|
|
||||||
|
/* Sticky "unrecoverable, loudly fenced" flag starts clear (Req 2.4): a fresh
|
||||||
|
* store is promotable. Set by persist_recover_loud_fence_if_unrecoverable
|
||||||
|
* on an unrecoverable boot; cleared on catch-up (never self-exits). */
|
||||||
|
atomic_store(&p->recovery_unrecoverable, false);
|
||||||
|
|
||||||
/* Durable-advance wake hook starts unregistered (ha-commit-durability 2.2):
|
/* Durable-advance wake hook starts unregistered (ha-commit-durability 2.2):
|
||||||
* the commit engine registers it in commit_engine_init. Until then every
|
* the commit engine registers it in commit_engine_init. Until then every
|
||||||
* flusher durable advance is a no-op wake. */
|
* flusher durable advance is a no-op wake. */
|
||||||
|
|
@ -1700,21 +1705,25 @@ bool persist_recovery_truncated(persist_t* p)
|
||||||
}
|
}
|
||||||
|
|
||||||
/* ------------------------------------------------------------------ *
|
/* ------------------------------------------------------------------ *
|
||||||
* Boot-policy FATAL decision (ha-commit-durability task 3.3, Req 2.4).
|
* Boot-policy loud-fence decision (ha-commit-durability task 3.3, Req 2.4;
|
||||||
|
* never-self-exit rework 2026-06-18).
|
||||||
*
|
*
|
||||||
* Implements the FATAL terminal of the Boot Policy Decision Table
|
* Implements the FENCE_UNRECOVERABLE terminal of the Boot Policy Decision Table
|
||||||
* (design.md). The decision is a PURE function of its inputs; the FATAL
|
* (design.md). The decision is a PURE function of its inputs; the action (loud
|
||||||
* action (loud log + terminate) is separated into
|
* operator-visible log + set the sticky non-promotable flag) is separated into
|
||||||
* persist_recover_fatal_if_unrecoverable so the decision can be unit-tested
|
* persist_recover_loud_fence_if_unrecoverable so the decision can be unit-tested
|
||||||
* without aborting. A reachable healthy peer NEVER FATALs (rows 3, 5);
|
* in isolation. A reachable healthy peer NEVER yields FENCE_UNRECOVERABLE
|
||||||
* FATAL is reserved for a member that cannot recover locally AND cannot catch
|
* (rows 3, 5); it is reserved for a member that cannot recover locally AND
|
||||||
* up within the bounded window (rows 4, 4b) — never a silent indefinite fence.
|
* cannot catch up (no healthy peer, rows 4, 4b).
|
||||||
|
*
|
||||||
|
* CRITICAL: this terminal is a LOUD, PERMANENT, NON-promotable fence — the
|
||||||
|
* process STAYS ALIVE and keeps retrying catch-up. There is NO abort()/exit():
|
||||||
|
* a controller MUST NEVER self-terminate as a designed response to an
|
||||||
|
* operational condition (user invariant, 2026-06-18). A self-exited controller
|
||||||
|
* needs manual admin recovery and degrades quorum meanwhile; one that stays
|
||||||
|
* alive and keeps retrying self-heals when a peer returns (Req 2.6).
|
||||||
* ------------------------------------------------------------------ */
|
* ------------------------------------------------------------------ */
|
||||||
|
|
||||||
/* FATAL seam: NULL in production (real abort); a test may install a hook to
|
|
||||||
* capture the FATAL path without dying. Mirrors ha_fatal_hook. */
|
|
||||||
void (*persist_fatal_hook)(const char* msg) = NULL;
|
|
||||||
|
|
||||||
recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in)
|
recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in)
|
||||||
{
|
{
|
||||||
if (in == NULL)
|
if (in == NULL)
|
||||||
|
|
@ -1723,7 +1732,7 @@ recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in)
|
||||||
bool truncated = (in->verdict == RECOVER_TRUNCATED);
|
bool truncated = (in->verdict == RECOVER_TRUNCATED);
|
||||||
|
|
||||||
/* Rows 1 & 6: a structurally CLEAN log that is NOT behind the cluster
|
/* Rows 1 & 6: a structurally CLEAN log that is NOT behind the cluster
|
||||||
* high-water mark is a normal promotable member — no fence, no FATAL. */
|
* high-water mark is a normal promotable member — no fence. */
|
||||||
if (!truncated && !in->durable_below_hwm)
|
if (!truncated && !in->durable_below_hwm)
|
||||||
return RECOVER_DECISION_ELIGIBLE;
|
return RECOVER_DECISION_ELIGIBLE;
|
||||||
|
|
||||||
|
|
@ -1732,63 +1741,83 @@ recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in)
|
||||||
* (design.md note), so it is treated as behind for fence purposes. */
|
* (design.md note), so it is treated as behind for fence purposes. */
|
||||||
|
|
||||||
/* Rows 2, 3, 5: a healthy peer is reachable. Catch up from it and regain
|
/* Rows 2, 3, 5: a healthy peer is reachable. Catch up from it and regain
|
||||||
* eligibility (task 4.3). A reachable peer NEVER yields FATAL — even for a
|
* eligibility (task 4.3). A reachable peer NEVER yields the unrecoverable
|
||||||
* truncated log or a legitimate snapshot-sentinel. If catch-up has already
|
* fence — even for a truncated log or a legitimate snapshot-sentinel. If
|
||||||
* succeeded the member is eligible; otherwise it stays fenced (4.3 drives
|
* catch-up has already succeeded the member is eligible; otherwise it stays
|
||||||
* the catch-up + release; this task only refuses FATAL while a peer
|
* fenced (4.3 drives the catch-up + release). */
|
||||||
* exists). */
|
|
||||||
if (in->healthy_peer_reachable) {
|
if (in->healthy_peer_reachable) {
|
||||||
if (in->catch_up_succeeded)
|
if (in->catch_up_succeeded)
|
||||||
return RECOVER_DECISION_ELIGIBLE;
|
return RECOVER_DECISION_ELIGIBLE;
|
||||||
return RECOVER_DECISION_FENCE;
|
return RECOVER_DECISION_FENCE;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* No healthy peer. The bounded catch-up window is the member's only path to
|
/* No healthy peer. If catch-up did not reach a committed-consistent state,
|
||||||
* a committed-consistent state; if it did not reach one, the member can
|
* the member can neither recover locally nor catch up right now — rows 4 &
|
||||||
* neither recover locally nor catch up — rows 4 & 4b: FATAL (fail loud),
|
* 4b: a LOUD, PERMANENT, NON-promotable fence (NOT process exit). It stays
|
||||||
* never a silent indefinite fence (Req 2.4). */
|
* alive and keeps retrying; the fence releases when a peer returns
|
||||||
|
* (Req 2.4/2.6). */
|
||||||
if (!in->catch_up_succeeded)
|
if (!in->catch_up_succeeded)
|
||||||
return RECOVER_DECISION_FATAL;
|
return RECOVER_DECISION_FENCE_UNRECOVERABLE;
|
||||||
|
|
||||||
/* No peer but catch-up is reported succeeded (e.g. already at/above after a
|
/* No peer but catch-up is reported succeeded (e.g. already at/above after a
|
||||||
* local catch-up): eligible. Defensive — the boot path reaches the
|
* local catch-up): eligible. */
|
||||||
* no-peer branch with catch_up_succeeded=false in the bounded-failure case
|
|
||||||
* this task owns. */
|
|
||||||
return RECOVER_DECISION_ELIGIBLE;
|
return RECOVER_DECISION_ELIGIBLE;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
bool persist_recovery_unrecoverable(persist_t* p)
|
||||||
|
{
|
||||||
|
return p && atomic_load_explicit(&p->recovery_unrecoverable,
|
||||||
|
memory_order_acquire);
|
||||||
|
}
|
||||||
|
|
||||||
|
void persist_clear_recovery_unrecoverable(persist_t* p)
|
||||||
|
{
|
||||||
|
if (!p)
|
||||||
|
return;
|
||||||
|
/* Clear only on a genuine transition so the loud "fence released" log fires
|
||||||
|
* once, not on every catch-up tick. */
|
||||||
|
if (atomic_exchange_explicit(&p->recovery_unrecoverable, false,
|
||||||
|
memory_order_acq_rel)) {
|
||||||
|
INFO("persist_recover: unrecoverable fence RELEASED — member reached a "
|
||||||
|
"committed-consistent state via catch-up; promotable again "
|
||||||
|
"(ha-commit-durability Req 2.6)");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
recover_decision_t
|
recover_decision_t
|
||||||
persist_recover_fatal_if_unrecoverable(const recover_policy_inputs_t* in)
|
persist_recover_loud_fence_if_unrecoverable(persist_t* p,
|
||||||
|
const recover_policy_inputs_t* in)
|
||||||
{
|
{
|
||||||
recover_decision_t decision = persist_recover_decision(in);
|
recover_decision_t decision = persist_recover_decision(in);
|
||||||
|
|
||||||
if (decision != RECOVER_DECISION_FATAL)
|
if (decision != RECOVER_DECISION_FENCE_UNRECOVERABLE)
|
||||||
return decision;
|
return decision;
|
||||||
|
|
||||||
/* Fail loud: log the truncation boundary (durable_floor_lsn) and the
|
/* Loud, operator-visible alert: log the truncation boundary
|
||||||
* cluster high-water mark so an operator sees exactly where the local log
|
* (durable_floor_lsn) and the cluster high-water mark so an operator sees
|
||||||
* stops short of the cluster's committed state (Req 2.4). */
|
* exactly where the local log stops short of the cluster's committed state.
|
||||||
|
* The member STAYS ALIVE, stays non-promotable, and keeps retrying catch-up
|
||||||
|
* (Req 2.4) — it NEVER self-exits. */
|
||||||
const char* why = (in->verdict == RECOVER_TRUNCATED)
|
const char* why = (in->verdict == RECOVER_TRUNCATED)
|
||||||
? "recovery-truncated log (structural corruption)"
|
? "recovery-truncated log (structural corruption)"
|
||||||
: "log behind the cluster committed high-water mark";
|
: "log behind the cluster committed high-water mark";
|
||||||
ERROR("persist_recover: FATAL — %s and no healthy peer reachable to catch "
|
ERROR("persist_recover: UNRECOVERABLE — %s and no healthy peer reachable to "
|
||||||
"up within the bounded window (%u ms); truncation boundary "
|
"catch up; truncation boundary durable_floor_lsn=%" PRIu64
|
||||||
"durable_floor_lsn=%" PRIu64 " < cluster committed high-water "
|
" < cluster committed high-water hwm=%" PRIu64 ". Staying ALIVE and "
|
||||||
"hwm=%" PRIu64 " — terminating rather than serving divergent state "
|
"PERMANENTLY non-promotable (refusing to serve divergent state) and "
|
||||||
|
"RETRYING catch-up indefinitely — will self-heal when a healthy peer "
|
||||||
|
"returns; operator attention may be needed "
|
||||||
"(ha-commit-durability Req 2.4)",
|
"(ha-commit-durability Req 2.4)",
|
||||||
why, (unsigned)DEFAULT_RECOVERY_CATCHUP_WINDOW_MS,
|
why, in->durable_floor_lsn, in->cluster_hwm_lsn);
|
||||||
in->durable_floor_lsn, in->cluster_hwm_lsn);
|
|
||||||
|
|
||||||
if (persist_fatal_hook != NULL) {
|
/* Set the sticky non-promotable flag so the promotion gate refuses
|
||||||
/* Test seam: capture the FATAL decision + boundary log without dying.
|
* leadership as belt-and-suspenders until catch-up clears it (Req 2.4/2.6).
|
||||||
* Production leaves the hook NULL and aborts below. */
|
* p may be NULL for pure-classification callers that only want the loud log. */
|
||||||
persist_fatal_hook("persist_recover: bounded-unrecoverable (FATAL)");
|
if (p != NULL)
|
||||||
return RECOVER_DECISION_FATAL;
|
atomic_store_explicit(&p->recovery_unrecoverable, true,
|
||||||
}
|
memory_order_release);
|
||||||
|
|
||||||
tlog_exit(); /* flush the async log buffer before dying */
|
return RECOVER_DECISION_FENCE_UNRECOVERABLE;
|
||||||
abort(); /* NEVER returns in production */
|
|
||||||
return RECOVER_DECISION_FATAL; /* unreachable */
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Advance committed_lsn to `written_lsn` on a majority ACK — but ONLY if that
|
/* Advance committed_lsn to `written_lsn` on a majority ACK — but ONLY if that
|
||||||
|
|
|
||||||
|
|
@ -1587,19 +1587,36 @@ static void test_durable_floor_snapshot_sentinel_tip(void)
|
||||||
* failure. (Req 2.2, 2.4) *
|
* failure. (Req 2.2, 2.4) *
|
||||||
* ------------------------------------------------------------------ */
|
* ------------------------------------------------------------------ */
|
||||||
|
|
||||||
/* task 3.3 — FATAL-path capture seam. The production FATAL terminates via
|
/* task 3.3 — unrecoverable loud-fence verification (never-self-exit rework
|
||||||
* abort(); a test installs persist_fatal_hook so the FATAL decision + boundary
|
* 2026-06-18). The OLD behavior installed persist_fatal_hook to capture an
|
||||||
* log can be asserted without the process dying. */
|
* abort()/exit() without dying. That self-exit terminal is GONE: an
|
||||||
static int g_fatal_hook_calls = 0;
|
* unrecoverable member STAYS ALIVE and is held permanently non-promotable.
|
||||||
static char g_fatal_hook_msg[256];
|
* There is no hook to install — the tests assert the OBSERVABLE
|
||||||
|
* post-conditions instead:
|
||||||
static void dur_fatal_hook(const char* msg)
|
* - the pure decision returns RECOVER_DECISION_FENCE_UNRECOVERABLE,
|
||||||
|
* - persist_recover_loud_fence_if_unrecoverable RETURNS (no abort — the test
|
||||||
|
* simply continuing past the call proves the process stayed alive),
|
||||||
|
* - the sticky persist_recovery_unrecoverable flag is SET (member is
|
||||||
|
* permanently non-promotable until catch-up),
|
||||||
|
* - and the flag CLEARS on catch-up (Req 2.6).
|
||||||
|
* dur_persist_for_flag builds a real persist_t the loud-fence helper can set
|
||||||
|
* the sticky flag on. */
|
||||||
|
static persist_t* dur_flag_persist(const char* tmpdir)
|
||||||
{
|
{
|
||||||
g_fatal_hook_calls++;
|
persist_t* p = calloc(1, sizeof(*p));
|
||||||
if (msg != NULL) {
|
if (!p) return NULL;
|
||||||
strncpy(g_fatal_hook_msg, msg, sizeof(g_fatal_hook_msg) - 1);
|
persist_config_t cfg = persist_config_defaults();
|
||||||
g_fatal_hook_msg[sizeof(g_fatal_hook_msg) - 1] = '\0';
|
cfg.wal_sync_ms = 5000;
|
||||||
}
|
persist_set_workdir(p, tmpdir, "ctrl-flag");
|
||||||
|
if (persist_init(p, &cfg, NULL) != 0) { free(p); return NULL; }
|
||||||
|
return p;
|
||||||
|
}
|
||||||
|
|
||||||
|
static void dur_free_flag_persist(persist_t* p)
|
||||||
|
{
|
||||||
|
if (!p) return;
|
||||||
|
persist_fini(p);
|
||||||
|
free(p);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* task 3.3 — peer-reachable NEVER FATALs (the never-FATAL aspect of rows
|
/* task 3.3 — peer-reachable NEVER FATALs (the never-FATAL aspect of rows
|
||||||
|
|
@ -1613,10 +1630,13 @@ static void dur_fatal_hook(const char* msg)
|
||||||
* 2.4, design.md "never FATAL when a peer is reachable".) */
|
* 2.4, design.md "never FATAL when a peer is reachable".) */
|
||||||
static void test_boot_policy_peer_reachable_never_fatal(void)
|
static void test_boot_policy_peer_reachable_never_fatal(void)
|
||||||
{
|
{
|
||||||
g_fatal_hook_calls = 0;
|
char tmp[256];
|
||||||
persist_fatal_hook = dur_fatal_hook;
|
strcpy(tmp, "/tmp/xce_dur_peerfence_XXXXXX");
|
||||||
|
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
|
||||||
|
persist_t* p = dur_flag_persist(tmp);
|
||||||
|
TEST_ASSERT_NOT_NULL(p);
|
||||||
|
|
||||||
/* Row 3: TRUNCATED + healthy peer (catch-up not yet done) -> FENCE, no FATAL. */
|
/* Row 3: TRUNCATED + healthy peer (catch-up not yet done) -> FENCE. */
|
||||||
recover_policy_inputs_t truncated_peer = {
|
recover_policy_inputs_t truncated_peer = {
|
||||||
.verdict = RECOVER_TRUNCATED,
|
.verdict = RECOVER_TRUNCATED,
|
||||||
.durable_below_hwm = true,
|
.durable_below_hwm = true,
|
||||||
|
|
@ -1628,13 +1648,13 @@ static void test_boot_policy_peer_reachable_never_fatal(void)
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||||
RECOVER_DECISION_FENCE, persist_recover_decision(&truncated_peer),
|
RECOVER_DECISION_FENCE, persist_recover_decision(&truncated_peer),
|
||||||
"row 3: a truncated member with a reachable healthy peer must FENCE "
|
"row 3: a truncated member with a reachable healthy peer must FENCE "
|
||||||
"(catch up), NEVER FATAL");
|
"(catch up), NEVER the unrecoverable terminal");
|
||||||
TEST_ASSERT_EQUAL_INT(
|
TEST_ASSERT_EQUAL_INT(
|
||||||
RECOVER_DECISION_FENCE,
|
RECOVER_DECISION_FENCE,
|
||||||
persist_recover_fatal_if_unrecoverable(&truncated_peer));
|
persist_recover_loud_fence_if_unrecoverable(p, &truncated_peer));
|
||||||
|
|
||||||
/* Row 5: snapshot-sentinel CLEAN-but-behind + healthy peer -> FENCE, no
|
/* Row 5: snapshot-sentinel CLEAN-but-behind + healthy peer -> FENCE.
|
||||||
* FATAL. A legitimate sentinel is CLEAN-but-behind, not corruption. */
|
* A legitimate sentinel is CLEAN-but-behind, not corruption. */
|
||||||
recover_policy_inputs_t sentinel_peer = {
|
recover_policy_inputs_t sentinel_peer = {
|
||||||
.verdict = RECOVER_CLEAN,
|
.verdict = RECOVER_CLEAN,
|
||||||
.durable_below_hwm = true,
|
.durable_below_hwm = true,
|
||||||
|
|
@ -1646,18 +1666,19 @@ static void test_boot_policy_peer_reachable_never_fatal(void)
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||||
RECOVER_DECISION_FENCE, persist_recover_decision(&sentinel_peer),
|
RECOVER_DECISION_FENCE, persist_recover_decision(&sentinel_peer),
|
||||||
"row 5: a snapshot-sentinel CLEAN-but-behind member with a reachable "
|
"row 5: a snapshot-sentinel CLEAN-but-behind member with a reachable "
|
||||||
"peer must FENCE, NEVER FATAL");
|
"peer must FENCE, NEVER the unrecoverable terminal");
|
||||||
TEST_ASSERT_EQUAL_INT(
|
TEST_ASSERT_EQUAL_INT(
|
||||||
RECOVER_DECISION_FENCE,
|
RECOVER_DECISION_FENCE,
|
||||||
persist_recover_fatal_if_unrecoverable(&sentinel_peer));
|
persist_recover_loud_fence_if_unrecoverable(p, &sentinel_peer));
|
||||||
|
|
||||||
/* The FATAL terminate path must NOT have fired for either peer-reachable
|
/* The unrecoverable sticky flag must NOT have been set for either
|
||||||
* case — a reachable healthy peer never triggers FATAL. */
|
* peer-reachable case — a reachable healthy peer never reaches the
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
* unrecoverable terminal. */
|
||||||
0, g_fatal_hook_calls,
|
TEST_ASSERT_FALSE_MESSAGE(
|
||||||
"peer-reachable rows must NEVER fire the FATAL terminate path");
|
persist_recovery_unrecoverable(p),
|
||||||
|
"peer-reachable rows must NEVER set the unrecoverable sticky flag");
|
||||||
|
|
||||||
persist_fatal_hook = NULL;
|
dur_free_flag_persist(p);
|
||||||
printf("test_boot_policy_peer_reachable_never_fatal: PASS\n");
|
printf("test_boot_policy_peer_reachable_never_fatal: PASS\n");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -1763,9 +1784,6 @@ static void test_boot_policy_row3_truncated_peer_catchup_eligible(void)
|
||||||
ha_config_t cfg = dur_ha_cfg();
|
ha_config_t cfg = dur_ha_cfg();
|
||||||
TEST_ASSERT_EQUAL_INT(0, ha_init(&ha, &cfg, 0));
|
TEST_ASSERT_EQUAL_INT(0, ha_init(&ha, &cfg, 0));
|
||||||
|
|
||||||
g_fatal_hook_calls = 0;
|
|
||||||
persist_fatal_hook = dur_fatal_hook;
|
|
||||||
|
|
||||||
/* Truncated boundary at 3, cluster HWM 10, peer reachable, not caught up. */
|
/* Truncated boundary at 3, cluster HWM 10, peer reachable, not caught up. */
|
||||||
recover_decision_t d1 = ha_evaluate_promotion_fence(
|
recover_decision_t d1 = ha_evaluate_promotion_fence(
|
||||||
&ha, /*truncated=*/true, /*durable_floor=*/3, /*safe_hwm=*/10,
|
&ha, /*truncated=*/true, /*durable_floor=*/3, /*safe_hwm=*/10,
|
||||||
|
|
@ -1773,13 +1791,10 @@ static void test_boot_policy_row3_truncated_peer_catchup_eligible(void)
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||||
RECOVER_DECISION_FENCE, d1,
|
RECOVER_DECISION_FENCE, d1,
|
||||||
"row 3: a recovery-truncated member with a healthy peer must FENCE "
|
"row 3: a recovery-truncated member with a healthy peer must FENCE "
|
||||||
"(denied promotion), never FATAL");
|
"(denied promotion), never the unrecoverable terminal");
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||||
1, atomic_load_explicit(&ha.in_catch_up, memory_order_acquire),
|
1, atomic_load_explicit(&ha.in_catch_up, memory_order_acquire),
|
||||||
"row 3: a truncated member must be fenced (in_catch_up set)");
|
"row 3: a truncated member must be fenced (in_catch_up set)");
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
|
||||||
0, g_fatal_hook_calls,
|
|
||||||
"row 3: FATAL must NOT fire while a healthy peer is reachable");
|
|
||||||
|
|
||||||
/* After catch-up to/above the committed boundary -> ELIGIBLE, fence released. */
|
/* After catch-up to/above the committed boundary -> ELIGIBLE, fence released. */
|
||||||
recover_decision_t d2 = ha_evaluate_promotion_fence(
|
recover_decision_t d2 = ha_evaluate_promotion_fence(
|
||||||
|
|
@ -1791,28 +1806,29 @@ static void test_boot_policy_row3_truncated_peer_catchup_eligible(void)
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||||
0, atomic_load_explicit(&ha.in_catch_up, memory_order_acquire),
|
0, atomic_load_explicit(&ha.in_catch_up, memory_order_acquire),
|
||||||
"row 3: regained eligibility releases the fence (Req 2.6)");
|
"row 3: regained eligibility releases the fence (Req 2.6)");
|
||||||
TEST_ASSERT_EQUAL_INT(0, g_fatal_hook_calls);
|
|
||||||
|
|
||||||
persist_fatal_hook = NULL;
|
|
||||||
ha_fini(&ha);
|
ha_fini(&ha);
|
||||||
printf("test_boot_policy_row3_truncated_peer_catchup_eligible: PASS\n");
|
printf("test_boot_policy_row3_truncated_peer_catchup_eligible: PASS\n");
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Row 4: TRUNCATED + below HWM + NO healthy peer + catch-up fails
|
/* Row 4: TRUNCATED + below HWM + NO healthy peer + catch-up fails
|
||||||
* (bounded window elapses) -> FATAL.
|
* -> LOUD PERMANENT FENCE (non-promotable, STAYS ALIVE, keeps retrying), and
|
||||||
* Cannot recover locally AND cannot catch up within the bounded window:
|
* RELEASES on catch-up (Req 2.4, 2.6).
|
||||||
* fail loud rather than serve divergent state. WILL ASSERT (3.3): the
|
*
|
||||||
* member terminates loudly (FATAL) with the truncation boundary logged
|
* never-self-exit rework (2026-06-18): the member cannot recover locally AND
|
||||||
* when the bounded catch-up window elapses without a committed-consistent
|
* cannot catch up right now. It does NOT self-exit; it logs LOUDLY, sets the
|
||||||
* state. (Req 2.4) */
|
* sticky non-promotable flag, and stays alive. The flag clears once a peer
|
||||||
static void test_boot_policy_row4_truncated_no_peer_bounded_fatal(void)
|
* returns and it catches up. RED→GREEN: pre-rework
|
||||||
|
* persist_recover_fatal_if_unrecoverable aborted (or fired the abort-capture
|
||||||
|
* hook); post-rework persist_recover_loud_fence_if_unrecoverable RETURNS — the
|
||||||
|
* test continuing past the call IS the proof the process stayed alive. */
|
||||||
|
static void test_boot_policy_row4_truncated_no_peer_loud_fence(void)
|
||||||
{
|
{
|
||||||
/* Row 4: TRUNCATED + below HWM + NO healthy peer + bounded catch-up fails
|
char tmp[256];
|
||||||
* -> FATAL. Cannot recover locally AND cannot catch up within the bounded
|
strcpy(tmp, "/tmp/xce_dur_row4_XXXXXX");
|
||||||
* window: fail loud with the truncation boundary logged (Req 2.4). */
|
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
|
||||||
g_fatal_hook_calls = 0;
|
persist_t* p = dur_flag_persist(tmp);
|
||||||
g_fatal_hook_msg[0] = '\0';
|
TEST_ASSERT_NOT_NULL(p);
|
||||||
persist_fatal_hook = dur_fatal_hook;
|
|
||||||
|
|
||||||
recover_policy_inputs_t in = {
|
recover_policy_inputs_t in = {
|
||||||
.verdict = RECOVER_TRUNCATED,
|
.verdict = RECOVER_TRUNCATED,
|
||||||
|
|
@ -1823,39 +1839,49 @@ static void test_boot_policy_row4_truncated_no_peer_bounded_fatal(void)
|
||||||
.cluster_hwm_lsn = 12, /* cluster committed high-water */
|
.cluster_hwm_lsn = 12, /* cluster committed high-water */
|
||||||
};
|
};
|
||||||
|
|
||||||
/* Pure decision returns FATAL. */
|
/* Pure decision returns the non-terminating unrecoverable fence. */
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||||
RECOVER_DECISION_FATAL, persist_recover_decision(&in),
|
RECOVER_DECISION_FENCE_UNRECOVERABLE, persist_recover_decision(&in),
|
||||||
"row 4: truncated + below + no peer + bounded catch-up fails must "
|
"row 4: truncated + below + no peer + no catch-up must decide the LOUD "
|
||||||
"decide FATAL");
|
"PERMANENT FENCE — NOT process exit");
|
||||||
|
TEST_ASSERT_FALSE_MESSAGE(
|
||||||
|
persist_recovery_unrecoverable(p),
|
||||||
|
"row 4: the sticky flag is set by the ACTION helper, not the pure "
|
||||||
|
"decision");
|
||||||
|
|
||||||
/* The action helper routes FATAL through the hook (no abort in test) and
|
/* The action helper logs LOUDLY, sets the sticky flag, and RETURNS (no
|
||||||
* still returns FATAL — proving the loud-terminate path was taken. */
|
* abort/exit — the process stays alive; the test reaching the next line is
|
||||||
recover_decision_t d = persist_recover_fatal_if_unrecoverable(&in);
|
* the proof). */
|
||||||
TEST_ASSERT_EQUAL_INT(RECOVER_DECISION_FATAL, d);
|
recover_decision_t d = persist_recover_loud_fence_if_unrecoverable(p, &in);
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||||
1, g_fatal_hook_calls,
|
RECOVER_DECISION_FENCE_UNRECOVERABLE, d,
|
||||||
"row 4: the FATAL terminate path must fire exactly once (the "
|
"row 4: the loud-fence action returns FENCE_UNRECOVERABLE (stays alive)");
|
||||||
"truncation boundary is logged at ERROR just before it)");
|
TEST_ASSERT_TRUE_MESSAGE(
|
||||||
|
persist_recovery_unrecoverable(p),
|
||||||
|
"row 4: the member is held PERMANENTLY non-promotable (sticky flag set)");
|
||||||
|
|
||||||
persist_fatal_hook = NULL;
|
/* Req 2.6: once catch-up reaches a committed-consistent state the fence
|
||||||
printf("test_boot_policy_row4_truncated_no_peer_bounded_fatal: PASS\n");
|
* releases automatically — the flag clears, the member is promotable. */
|
||||||
|
persist_clear_recovery_unrecoverable(p);
|
||||||
|
TEST_ASSERT_FALSE_MESSAGE(
|
||||||
|
persist_recovery_unrecoverable(p),
|
||||||
|
"row 4: catch-up CLEARS the unrecoverable fence (Req 2.6)");
|
||||||
|
|
||||||
|
dur_free_flag_persist(p);
|
||||||
|
printf("test_boot_policy_row4_truncated_no_peer_loud_fence: PASS\n");
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Row 4b: CLEAN + below HWM + NO healthy peer + catch-up fails (bounded
|
/* Row 4b: CLEAN + below HWM + NO healthy peer + catch-up fails
|
||||||
* window elapses) -> FATAL.
|
* -> LOUD PERMANENT FENCE (non-promotable, STAYS ALIVE, keeps retrying).
|
||||||
* Behind the HWM with no peer to advance from within the bounded window:
|
* Behind the HWM with no peer to advance from — same non-terminating loud-fence
|
||||||
* same bounded-failure terminal as row 4 — never a silent indefinite
|
* terminal as row 4. NEVER self-exits. (Req 2.4, 2.2, 2.6) */
|
||||||
* fence. WILL ASSERT (3.3): a behind member with no reachable peer FATALs
|
static void test_boot_policy_row4b_clean_below_no_peer_loud_fence(void)
|
||||||
* once the bounded window elapses. (Req 2.4, 2.2) */
|
|
||||||
static void test_boot_policy_row4b_clean_below_no_peer_bounded_fatal(void)
|
|
||||||
{
|
{
|
||||||
/* Row 4b: CLEAN + below HWM + NO healthy peer + bounded catch-up fails
|
char tmp[256];
|
||||||
* -> FATAL. Behind the HWM with no peer to advance from within the bounded
|
strcpy(tmp, "/tmp/xce_dur_row4b_XXXXXX");
|
||||||
* window — same bounded-failure terminal as row 4 (Req 2.4, 2.2). */
|
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
|
||||||
g_fatal_hook_calls = 0;
|
persist_t* p = dur_flag_persist(tmp);
|
||||||
g_fatal_hook_msg[0] = '\0';
|
TEST_ASSERT_NOT_NULL(p);
|
||||||
persist_fatal_hook = dur_fatal_hook;
|
|
||||||
|
|
||||||
recover_policy_inputs_t in = {
|
recover_policy_inputs_t in = {
|
||||||
.verdict = RECOVER_CLEAN, /* structurally intact... */
|
.verdict = RECOVER_CLEAN, /* structurally intact... */
|
||||||
|
|
@ -1867,19 +1893,24 @@ static void test_boot_policy_row4b_clean_below_no_peer_bounded_fatal(void)
|
||||||
};
|
};
|
||||||
|
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||||
RECOVER_DECISION_FATAL, persist_recover_decision(&in),
|
RECOVER_DECISION_FENCE_UNRECOVERABLE, persist_recover_decision(&in),
|
||||||
"row 4b: clean-but-behind + no peer + bounded catch-up fails must "
|
"row 4b: clean-but-behind + no peer + no catch-up must decide the LOUD "
|
||||||
"decide FATAL — never a silent indefinite fence");
|
"PERMANENT FENCE — never process exit, never a silent fence");
|
||||||
|
|
||||||
recover_decision_t d = persist_recover_fatal_if_unrecoverable(&in);
|
recover_decision_t d = persist_recover_loud_fence_if_unrecoverable(p, &in);
|
||||||
TEST_ASSERT_EQUAL_INT(RECOVER_DECISION_FATAL, d);
|
TEST_ASSERT_EQUAL_INT(RECOVER_DECISION_FENCE_UNRECOVERABLE, d);
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
TEST_ASSERT_TRUE_MESSAGE(
|
||||||
1, g_fatal_hook_calls,
|
persist_recovery_unrecoverable(p),
|
||||||
"row 4b: a behind member with no reachable peer must FATAL once the "
|
"row 4b: a behind member with no peer is held non-promotable + ALIVE "
|
||||||
"bounded window elapses (boundary logged before terminate)");
|
"(sticky flag set; boundary logged loudly)");
|
||||||
|
|
||||||
persist_fatal_hook = NULL;
|
persist_clear_recovery_unrecoverable(p);
|
||||||
printf("test_boot_policy_row4b_clean_below_no_peer_bounded_fatal: PASS\n");
|
TEST_ASSERT_FALSE_MESSAGE(
|
||||||
|
persist_recovery_unrecoverable(p),
|
||||||
|
"row 4b: catch-up CLEARS the unrecoverable fence (Req 2.6)");
|
||||||
|
|
||||||
|
dur_free_flag_persist(p);
|
||||||
|
printf("test_boot_policy_row4b_clean_below_no_peer_loud_fence: PASS\n");
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Row 5: CLEAN snapshot-sentinel (committed at sentinel LSN, NO
|
/* Row 5: CLEAN snapshot-sentinel (committed at sentinel LSN, NO
|
||||||
|
|
@ -1901,9 +1932,6 @@ static void test_boot_policy_row5_snapshot_sentinel_catchup_never_fatal(void)
|
||||||
ha_config_t cfg = dur_ha_cfg();
|
ha_config_t cfg = dur_ha_cfg();
|
||||||
TEST_ASSERT_EQUAL_INT(0, ha_init(&ha, &cfg, 0));
|
TEST_ASSERT_EQUAL_INT(0, ha_init(&ha, &cfg, 0));
|
||||||
|
|
||||||
g_fatal_hook_calls = 0;
|
|
||||||
persist_fatal_hook = dur_fatal_hook;
|
|
||||||
|
|
||||||
/* Sentinel at LSN 5, cluster has advanced to HWM 12, peer reachable. */
|
/* Sentinel at LSN 5, cluster has advanced to HWM 12, peer reachable. */
|
||||||
recover_decision_t d1 = ha_evaluate_promotion_fence(
|
recover_decision_t d1 = ha_evaluate_promotion_fence(
|
||||||
&ha, /*truncated=*/false, /*durable_floor=*/5, /*safe_hwm=*/12,
|
&ha, /*truncated=*/false, /*durable_floor=*/5, /*safe_hwm=*/12,
|
||||||
|
|
@ -1911,10 +1939,7 @@ static void test_boot_policy_row5_snapshot_sentinel_catchup_never_fatal(void)
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||||
RECOVER_DECISION_FENCE, d1,
|
RECOVER_DECISION_FENCE, d1,
|
||||||
"row 5: a snapshot-sentinel CLEAN-but-behind member must FENCE, never "
|
"row 5: a snapshot-sentinel CLEAN-but-behind member must FENCE, never "
|
||||||
"FATAL while a peer is reachable");
|
"the unrecoverable terminal while a peer is reachable");
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
|
||||||
0, g_fatal_hook_calls,
|
|
||||||
"row 5: a legitimate sentinel must NEVER trigger FATAL");
|
|
||||||
|
|
||||||
recover_decision_t d2 = ha_evaluate_promotion_fence(
|
recover_decision_t d2 = ha_evaluate_promotion_fence(
|
||||||
&ha, /*truncated=*/false, /*durable_floor=*/12, /*safe_hwm=*/12,
|
&ha, /*truncated=*/false, /*durable_floor=*/12, /*safe_hwm=*/12,
|
||||||
|
|
@ -1922,9 +1947,7 @@ static void test_boot_policy_row5_snapshot_sentinel_catchup_never_fatal(void)
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||||
RECOVER_DECISION_ELIGIBLE, d2,
|
RECOVER_DECISION_ELIGIBLE, d2,
|
||||||
"row 5: after catch-up the sentinel member is eligible (Req 2.6)");
|
"row 5: after catch-up the sentinel member is eligible (Req 2.6)");
|
||||||
TEST_ASSERT_EQUAL_INT(0, g_fatal_hook_calls);
|
|
||||||
|
|
||||||
persist_fatal_hook = NULL;
|
|
||||||
ha_fini(&ha);
|
ha_fini(&ha);
|
||||||
printf("test_boot_policy_row5_snapshot_sentinel_catchup_never_fatal: PASS\n");
|
printf("test_boot_policy_row5_snapshot_sentinel_catchup_never_fatal: PASS\n");
|
||||||
}
|
}
|
||||||
|
|
@ -1943,9 +1966,6 @@ static void test_boot_policy_row6_empty_first_boot_eligible(void)
|
||||||
ha_config_t cfg = dur_ha_cfg();
|
ha_config_t cfg = dur_ha_cfg();
|
||||||
TEST_ASSERT_EQUAL_INT(0, ha_init(&ha, &cfg, 0));
|
TEST_ASSERT_EQUAL_INT(0, ha_init(&ha, &cfg, 0));
|
||||||
|
|
||||||
g_fatal_hook_calls = 0;
|
|
||||||
persist_fatal_hook = dur_fatal_hook;
|
|
||||||
|
|
||||||
recover_decision_t d = ha_evaluate_promotion_fence(
|
recover_decision_t d = ha_evaluate_promotion_fence(
|
||||||
&ha, /*truncated=*/false, /*durable_floor=*/0, /*safe_hwm=*/0,
|
&ha, /*truncated=*/false, /*durable_floor=*/0, /*safe_hwm=*/0,
|
||||||
/*peer_reachable=*/false, /*catch_up_succeeded=*/false);
|
/*peer_reachable=*/false, /*catch_up_succeeded=*/false);
|
||||||
|
|
@ -1955,11 +1975,7 @@ static void test_boot_policy_row6_empty_first_boot_eligible(void)
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||||
0, atomic_load_explicit(&ha.in_catch_up, memory_order_acquire),
|
0, atomic_load_explicit(&ha.in_catch_up, memory_order_acquire),
|
||||||
"row 6: a cold-start member must not be fenced");
|
"row 6: a cold-start member must not be fenced");
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
|
||||||
0, g_fatal_hook_calls,
|
|
||||||
"row 6: a cold-start member must never FATAL");
|
|
||||||
|
|
||||||
persist_fatal_hook = NULL;
|
|
||||||
ha_fini(&ha);
|
ha_fini(&ha);
|
||||||
printf("test_boot_policy_row6_empty_first_boot_eligible: PASS\n");
|
printf("test_boot_policy_row6_empty_first_boot_eligible: PASS\n");
|
||||||
}
|
}
|
||||||
|
|
@ -1968,13 +1984,14 @@ static void test_boot_policy_row6_empty_first_boot_eligible(void)
|
||||||
* A hollow node (recovery-truncated OR durable below the HWM) must be refused
|
* A hollow node (recovery-truncated OR durable below the HWM) must be refused
|
||||||
* promotion regardless of quorum size — combined with the 4.2 freshness floor
|
* promotion regardless of quorum size — combined with the 4.2 freshness floor
|
||||||
* it can neither win the vote nor self-promote. At minimal quorum with NO
|
* it can neither win the vote nor self-promote. At minimal quorum with NO
|
||||||
* healthy peer, a hollow node FATALs (bounded-unrecoverable) rather than
|
* healthy peer, a hollow node is held in the LOUD PERMANENT FENCE
|
||||||
* silently leading; with a peer it FENCEs. Either way it never becomes
|
* (FENCE_UNRECOVERABLE, stays alive, non-promotable) rather than silently
|
||||||
* ELIGIBLE while hollow. */
|
* leading; with a peer it FENCEs. Either way it never becomes ELIGIBLE while
|
||||||
|
* hollow. */
|
||||||
static void test_promotion_gate_hollow_refused_at_minimal_quorum(void)
|
static void test_promotion_gate_hollow_refused_at_minimal_quorum(void)
|
||||||
{
|
{
|
||||||
/* Truncated + below HWM + no peer (minimal/degraded quorum) -> NEVER
|
/* Truncated + below HWM + no peer (minimal/degraded quorum) -> NEVER
|
||||||
* ELIGIBLE (FATAL — fail loud rather than promote a hollow node). */
|
* ELIGIBLE (loud permanent fence — never promote a hollow node). */
|
||||||
TEST_ASSERT_NOT_EQUAL_INT_MESSAGE(
|
TEST_ASSERT_NOT_EQUAL_INT_MESSAGE(
|
||||||
RECOVER_DECISION_ELIGIBLE,
|
RECOVER_DECISION_ELIGIBLE,
|
||||||
ha_promotion_gate_decision(/*truncated=*/true, /*durable_floor=*/3,
|
ha_promotion_gate_decision(/*truncated=*/true, /*durable_floor=*/3,
|
||||||
|
|
@ -1983,7 +2000,7 @@ static void test_promotion_gate_hollow_refused_at_minimal_quorum(void)
|
||||||
"a recovery-truncated node must never be ELIGIBLE at minimal quorum "
|
"a recovery-truncated node must never be ELIGIBLE at minimal quorum "
|
||||||
"(Req 2.5)");
|
"(Req 2.5)");
|
||||||
|
|
||||||
/* CLEAN but below HWM + no peer -> NEVER ELIGIBLE (FATAL). */
|
/* CLEAN but below HWM + no peer -> NEVER ELIGIBLE (loud permanent fence). */
|
||||||
TEST_ASSERT_NOT_EQUAL_INT_MESSAGE(
|
TEST_ASSERT_NOT_EQUAL_INT_MESSAGE(
|
||||||
RECOVER_DECISION_ELIGIBLE,
|
RECOVER_DECISION_ELIGIBLE,
|
||||||
ha_promotion_gate_decision(/*truncated=*/false, /*durable_floor=*/5,
|
ha_promotion_gate_decision(/*truncated=*/false, /*durable_floor=*/5,
|
||||||
|
|
@ -2003,19 +2020,22 @@ static void test_promotion_gate_hollow_refused_at_minimal_quorum(void)
|
||||||
printf("test_promotion_gate_hollow_refused_at_minimal_quorum: PASS\n");
|
printf("test_promotion_gate_hollow_refused_at_minimal_quorum: PASS\n");
|
||||||
}
|
}
|
||||||
|
|
||||||
/* ── task 4.4: bounded-window catch-up before FATAL ──────────────────────────
|
/* ── task 4.4: catch-up re-alert window (never-self-exit rework 2026-06-18) ───
|
||||||
*
|
*
|
||||||
* The 4.3 gate decided FATAL synchronously at the promotion instant when a
|
* A recovery-truncated/behind member with no reachable peer at the promotion
|
||||||
* recovery-truncated/behind member had no reachable peer right then. Task 4.4
|
* instant is FENCED (in_catch_up) and STAYS ALIVE, retrying catch-up. A
|
||||||
* defers that FATAL to a one-shot bounded-window timer: the member is fenced
|
* delaymgr re-alert timer fires after DEFAULT_RECOVERY_CATCHUP_WINDOW_MS: if the
|
||||||
* (in_catch_up) and given DEFAULT_RECOVERY_CATCHUP_WINDOW_MS to find a peer /
|
* member is still unrecoverable it RE-EMITS the loud operator alert (sets the
|
||||||
* make progress; the 3.3 FATAL is reached ONLY from the timer after a real
|
* sticky persist_recovery_unrecoverable flag) and the process STAYS ALIVE —
|
||||||
* bounded failure (still hollow + no peer + no progress).
|
* there is NO abort/exit anywhere. Catch-up releases the fence + clears the
|
||||||
|
* flag (Req 2.6); shutdown/cancel stops the re-alert.
|
||||||
*
|
*
|
||||||
* These tests drive ha_arm_promotion_fence_timer (the production deferral wired
|
* These tests drive ha_arm_promotion_fence_timer (the production deferral wired
|
||||||
* into raft_cb_become_leader) through a REAL delaymgr with a short window seam,
|
* into raft_cb_become_leader) through a REAL delaymgr with a short window seam,
|
||||||
* so the timer fires fast and deterministically. A minimal cluster_t carries
|
* so the timer fires fast and deterministically. A minimal cluster_t carries
|
||||||
* the delaymgr + persist + ha exactly as production does. */
|
* the delaymgr + persist + ha exactly as production does. They assert the
|
||||||
|
* member STAYS ALIVE + non-promotable (sticky flag) instead of asserting an
|
||||||
|
* abort/hook — the self-exit terminal is GONE. */
|
||||||
|
|
||||||
/* The delaymgr runner used by production (run_delay_tasks in cluster.c) is not
|
/* The delaymgr runner used by production (run_delay_tasks in cluster.c) is not
|
||||||
* linked into the unit-test target; provide the same trivial pass-through. */
|
* linked into the unit-test target; provide the same trivial pass-through. */
|
||||||
|
|
@ -2084,28 +2104,31 @@ static void dur_destroy_fence_cluster(cluster_t* c)
|
||||||
free(c);
|
free(c);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Spin until the fatal hook fires or a timeout elapses (the delaymgr fires the
|
/* Spin until the sticky unrecoverable flag is set on c->persist or a timeout
|
||||||
* one-shot timer on its own worker thread). */
|
* elapses (the delaymgr fires the re-alert timer on its own worker thread). */
|
||||||
static bool dur_wait_fatal(int expect_calls, int timeout_ms)
|
static bool dur_wait_unrecoverable(cluster_t* c, int timeout_ms)
|
||||||
{
|
{
|
||||||
int waited = 0;
|
int waited = 0;
|
||||||
while (g_fatal_hook_calls < expect_calls && waited < timeout_ms) {
|
while (!persist_recovery_unrecoverable(&c->persist) && waited < timeout_ms) {
|
||||||
usleep(2000);
|
usleep(2000);
|
||||||
waited += 2;
|
waited += 2;
|
||||||
}
|
}
|
||||||
return g_fatal_hook_calls >= expect_calls;
|
return persist_recovery_unrecoverable(&c->persist);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* 4.4-RED carrier + core deferral: a recovery-truncated/behind member with NO
|
/* 4.4 (never-self-exit): a recovery-truncated/behind member with NO reachable
|
||||||
* reachable peer must NOT FATAL synchronously at the promotion instant — it
|
* peer must NOT self-exit — it fences (in_catch_up set, process ALIVE) and arms
|
||||||
* fences (in_catch_up set, process alive) and arms a bounded-window timer; the
|
* the re-alert timer; when the window elapses with no peer/progress the timer
|
||||||
* 3.3 FATAL fires ONLY from that timer after the window with no peer/progress.
|
* RE-EMITS the loud alert (sets the sticky non-promotable flag) and the process
|
||||||
|
* STAYS ALIVE.
|
||||||
*
|
*
|
||||||
* RED (pre-4.4): the gate routed a no-peer truncated member straight through
|
* RED→GREEN: pre-rework the no-peer truncated member was routed (synchronously
|
||||||
* ha_evaluate_promotion_fence → persist_recover_fatal_if_unrecoverable, so the
|
* or via the timer) to persist_recover_fatal_if_unrecoverable, which abort()ed
|
||||||
* fatal hook was called SYNCHRONOUSLY (count==1 with zero delay) — there was no
|
* (the old fatal-capture hook fired) — that self-exit was the task-5.2 soak
|
||||||
* bounded window, which is exactly the task-5.2 soak collapse. (Req 2.2, 2.4) */
|
* quorum collapse. Post-rework there is NO abort: the member stays alive,
|
||||||
static void test_promotion_fence_defers_fatal_to_bounded_timer(void)
|
* fenced, and loudly non-promotable. Reaching the end of this test (the
|
||||||
|
* process did not die) IS part of the proof. (Req 2.2, 2.4) */
|
||||||
|
static void test_promotion_fence_unrecoverable_stays_alive_loud(void)
|
||||||
{
|
{
|
||||||
char tmp[256];
|
char tmp[256];
|
||||||
strcpy(tmp, "/tmp/xce_fence_defer_XXXXXX");
|
strcpy(tmp, "/tmp/xce_fence_defer_XXXXXX");
|
||||||
|
|
@ -2115,46 +2138,41 @@ static void test_promotion_fence_defers_fatal_to_bounded_timer(void)
|
||||||
/*durable_floor=*/4);
|
/*durable_floor=*/4);
|
||||||
TEST_ASSERT_NOT_NULL(c);
|
TEST_ASSERT_NOT_NULL(c);
|
||||||
|
|
||||||
g_fatal_hook_calls = 0;
|
ha_set_promotion_fence_window_ms(40); /* short re-alert window for the test */
|
||||||
persist_fatal_hook = dur_fatal_hook;
|
|
||||||
ha_set_promotion_fence_window_ms(40); /* short bounded window for the test */
|
|
||||||
|
|
||||||
/* Arm the fence exactly as raft_cb_become_leader does for a truncated,
|
/* Arm the fence exactly as raft_cb_become_leader does for a truncated,
|
||||||
* below-HWM, no-peer winner. This must NOT FATAL synchronously. */
|
* below-HWM, no-peer winner. This must NOT self-exit and must fence. */
|
||||||
extern void ha_arm_promotion_fence_timer_for_test(cluster_t*, bool, uint64_t,
|
extern void ha_arm_promotion_fence_timer_for_test(cluster_t*, bool, uint64_t,
|
||||||
uint64_t);
|
uint64_t);
|
||||||
ha_arm_promotion_fence_timer_for_test(c, /*truncated=*/true,
|
ha_arm_promotion_fence_timer_for_test(c, /*truncated=*/true,
|
||||||
/*durable_floor=*/4, /*hwm=*/10);
|
/*durable_floor=*/4, /*hwm=*/10);
|
||||||
|
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||||
0, g_fatal_hook_calls,
|
1, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire),
|
||||||
"4.4: the promotion gate must NOT FATAL synchronously — FATAL is "
|
"4.4: a fenced member is held in_catch_up (refused promotion), ALIVE");
|
||||||
"deferred to the bounded-window timer (Req 2.2/2.4)");
|
|
||||||
|
/* Let the re-alert window elapse: still truncated + no peer + no progress →
|
||||||
|
* the timer RE-EMITS the loud alert and SETS the sticky non-promotable flag.
|
||||||
|
* The process STAYS ALIVE (no abort/exit anywhere on this path). */
|
||||||
|
TEST_ASSERT_TRUE_MESSAGE(
|
||||||
|
dur_wait_unrecoverable(c, 1000),
|
||||||
|
"4.4: the re-alert timer must set the sticky unrecoverable flag after "
|
||||||
|
"the window with no peer/progress — staying ALIVE, not exiting (Req 2.4)");
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||||
1, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire),
|
1, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire),
|
||||||
"4.4: a fenced member is held in_catch_up (refused promotion), alive");
|
"4.4: an unrecoverable member stays fenced (non-promotable) and ALIVE");
|
||||||
|
|
||||||
/* Now let the bounded window elapse: still truncated + no peer + no
|
|
||||||
* progress → the 3.3 FATAL fires from the timer (exactly once). */
|
|
||||||
TEST_ASSERT_TRUE_MESSAGE(
|
|
||||||
dur_wait_fatal(1, 1000),
|
|
||||||
"4.4: the deferred FATAL must fire from the timer after the bounded "
|
|
||||||
"window with no peer and no progress (Req 2.4)");
|
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
|
||||||
1, g_fatal_hook_calls,
|
|
||||||
"4.4: the deferred FATAL fires exactly once from the timer");
|
|
||||||
|
|
||||||
persist_fatal_hook = NULL;
|
|
||||||
ha_set_promotion_fence_window_ms(0); /* restore default */
|
ha_set_promotion_fence_window_ms(0); /* restore default */
|
||||||
dur_destroy_fence_cluster(c);
|
dur_destroy_fence_cluster(c);
|
||||||
printf("test_promotion_fence_defers_fatal_to_bounded_timer: PASS\n");
|
printf("test_promotion_fence_unrecoverable_stays_alive_loud: PASS\n");
|
||||||
}
|
}
|
||||||
|
|
||||||
/* 4.4 Req 2.6: catch-up within the window → fence released, FATAL NOT called.
|
/* 4.4 Req 2.6: catch-up within the window → fence + sticky flag released, and
|
||||||
* Arm the fence, then simulate catch-up (clear truncated + advance durable_floor
|
* the member NEVER self-exits. Arm the fence, then simulate catch-up (clear
|
||||||
* past the HWM) before the timer fires; when the timer fires it observes a
|
* truncated + advance durable_floor past the HWM) before the timer fires; when
|
||||||
* caught-up member, releases in_catch_up, and does NOT FATAL. */
|
* the timer fires it observes a caught-up member, releases in_catch_up, clears
|
||||||
static void test_promotion_fence_released_on_catchup_no_fatal(void)
|
* the sticky unrecoverable flag, and does NOT set the unrecoverable flag. */
|
||||||
|
static void test_promotion_fence_released_on_catchup(void)
|
||||||
{
|
{
|
||||||
char tmp[256];
|
char tmp[256];
|
||||||
strcpy(tmp, "/tmp/xce_fence_catchup_XXXXXX");
|
strcpy(tmp, "/tmp/xce_fence_catchup_XXXXXX");
|
||||||
|
|
@ -2164,15 +2182,12 @@ static void test_promotion_fence_released_on_catchup_no_fatal(void)
|
||||||
/*durable_floor=*/4);
|
/*durable_floor=*/4);
|
||||||
TEST_ASSERT_NOT_NULL(c);
|
TEST_ASSERT_NOT_NULL(c);
|
||||||
|
|
||||||
g_fatal_hook_calls = 0;
|
|
||||||
persist_fatal_hook = dur_fatal_hook;
|
|
||||||
ha_set_promotion_fence_window_ms(120); /* generous so we can mutate first */
|
ha_set_promotion_fence_window_ms(120); /* generous so we can mutate first */
|
||||||
|
|
||||||
extern void ha_arm_promotion_fence_timer_for_test(cluster_t*, bool, uint64_t,
|
extern void ha_arm_promotion_fence_timer_for_test(cluster_t*, bool, uint64_t,
|
||||||
uint64_t);
|
uint64_t);
|
||||||
ha_arm_promotion_fence_timer_for_test(c, /*truncated=*/true,
|
ha_arm_promotion_fence_timer_for_test(c, /*truncated=*/true,
|
||||||
/*durable_floor=*/4, /*hwm=*/10);
|
/*durable_floor=*/4, /*hwm=*/10);
|
||||||
TEST_ASSERT_EQUAL_INT(0, g_fatal_hook_calls);
|
|
||||||
TEST_ASSERT_EQUAL_INT(
|
TEST_ASSERT_EQUAL_INT(
|
||||||
1, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire));
|
1, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire));
|
||||||
|
|
||||||
|
|
@ -2180,26 +2195,26 @@ static void test_promotion_fence_released_on_catchup_no_fatal(void)
|
||||||
c->persist.recovery_verdict = RECOVER_CLEAN;
|
c->persist.recovery_verdict = RECOVER_CLEAN;
|
||||||
atomic_store_explicit(&c->persist.durable_lsn, 10, memory_order_release);
|
atomic_store_explicit(&c->persist.durable_lsn, 10, memory_order_release);
|
||||||
|
|
||||||
/* Wait long enough for the window to fire; assert it did NOT FATAL and the
|
/* Wait long enough for the window to fire; assert the fence was released and
|
||||||
* fence was released (Req 2.6). */
|
* the sticky flag is NOT set (Req 2.6) — the member self-healed, alive. */
|
||||||
usleep(300 * 1000);
|
usleep(300 * 1000);
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
|
||||||
0, g_fatal_hook_calls,
|
|
||||||
"4.4: a member that caught up within the window must NOT FATAL (Req 2.6)");
|
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||||
0, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire),
|
0, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire),
|
||||||
"4.4: catch-up within the window RELEASES the in_catch_up fence (Req 2.6)");
|
"4.4: catch-up within the window RELEASES the in_catch_up fence (Req 2.6)");
|
||||||
|
TEST_ASSERT_FALSE_MESSAGE(
|
||||||
|
persist_recovery_unrecoverable(&c->persist),
|
||||||
|
"4.4: a member that caught up must NOT be left in the unrecoverable "
|
||||||
|
"fence (Req 2.6)");
|
||||||
|
|
||||||
persist_fatal_hook = NULL;
|
|
||||||
ha_set_promotion_fence_window_ms(0);
|
ha_set_promotion_fence_window_ms(0);
|
||||||
dur_destroy_fence_cluster(c);
|
dur_destroy_fence_cluster(c);
|
||||||
printf("test_promotion_fence_released_on_catchup_no_fatal: PASS\n");
|
printf("test_promotion_fence_released_on_catchup: PASS\n");
|
||||||
}
|
}
|
||||||
|
|
||||||
/* 4.4: cancelling the timer (shutdown / recovery) before it fires → NO FATAL.
|
/* 4.4: cancelling the timer (shutdown / recovery) before it fires → the
|
||||||
* Proves the deferred FATAL never fires after a clean cancel — a stopped member
|
* re-alert never fires, no use-after-free of the timer ctx, and the process
|
||||||
* must not self-terminate, and there is no use-after-free of the timer ctx. */
|
* stays alive (it never self-exited even before the cancel). */
|
||||||
static void test_promotion_fence_cancel_prevents_fatal(void)
|
static void test_promotion_fence_cancel_stops_realert(void)
|
||||||
{
|
{
|
||||||
char tmp[256];
|
char tmp[256];
|
||||||
strcpy(tmp, "/tmp/xce_fence_cancel_XXXXXX");
|
strcpy(tmp, "/tmp/xce_fence_cancel_XXXXXX");
|
||||||
|
|
@ -2209,29 +2224,27 @@ static void test_promotion_fence_cancel_prevents_fatal(void)
|
||||||
/*durable_floor=*/4);
|
/*durable_floor=*/4);
|
||||||
TEST_ASSERT_NOT_NULL(c);
|
TEST_ASSERT_NOT_NULL(c);
|
||||||
|
|
||||||
g_fatal_hook_calls = 0;
|
|
||||||
persist_fatal_hook = dur_fatal_hook;
|
|
||||||
ha_set_promotion_fence_window_ms(60);
|
ha_set_promotion_fence_window_ms(60);
|
||||||
|
|
||||||
extern void ha_arm_promotion_fence_timer_for_test(cluster_t*, bool, uint64_t,
|
extern void ha_arm_promotion_fence_timer_for_test(cluster_t*, bool, uint64_t,
|
||||||
uint64_t);
|
uint64_t);
|
||||||
ha_arm_promotion_fence_timer_for_test(c, /*truncated=*/true,
|
ha_arm_promotion_fence_timer_for_test(c, /*truncated=*/true,
|
||||||
/*durable_floor=*/4, /*hwm=*/10);
|
/*durable_floor=*/4, /*hwm=*/10);
|
||||||
TEST_ASSERT_EQUAL_INT(0, g_fatal_hook_calls);
|
|
||||||
|
|
||||||
/* Cancel before the window elapses. */
|
/* Cancel before the window elapses. */
|
||||||
ha_cancel_promotion_fence_timer(&c->ha);
|
ha_cancel_promotion_fence_timer(&c->ha);
|
||||||
|
|
||||||
/* Wait past the window — the cancelled timer must be a no-op, NO FATAL. */
|
/* Wait past the window — the cancelled timer must be a no-op (no re-alert,
|
||||||
|
* no UAF). The member stayed alive throughout (no abort anywhere). */
|
||||||
usleep(200 * 1000);
|
usleep(200 * 1000);
|
||||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
TEST_ASSERT_FALSE_MESSAGE(
|
||||||
0, g_fatal_hook_calls,
|
persist_recovery_unrecoverable(&c->persist),
|
||||||
"4.4: a cancelled fence timer must NEVER FATAL (shutdown/recovery)");
|
"4.4: a cancelled fence timer must not fire the re-alert "
|
||||||
|
"(shutdown/recovery)");
|
||||||
|
|
||||||
persist_fatal_hook = NULL;
|
|
||||||
ha_set_promotion_fence_window_ms(0);
|
ha_set_promotion_fence_window_ms(0);
|
||||||
dur_destroy_fence_cluster(c);
|
dur_destroy_fence_cluster(c);
|
||||||
printf("test_promotion_fence_cancel_prevents_fatal: PASS\n");
|
printf("test_promotion_fence_cancel_stops_realert: PASS\n");
|
||||||
}
|
}
|
||||||
|
|
||||||
#endif /* XCE_TESTING */
|
#endif /* XCE_TESTING */
|
||||||
|
|
@ -2272,25 +2285,26 @@ void test_commit_durability(void)
|
||||||
/* task 3.2 — Boot Policy Decision Table skeletons (IGNORED, pending
|
/* task 3.2 — Boot Policy Decision Table skeletons (IGNORED, pending
|
||||||
* 3.3/4.3). Registered so each row is reported as IGNORED, not failed;
|
* 3.3/4.3). Registered so each row is reported as IGNORED, not failed;
|
||||||
* the suite stays GREEN. See design.md "Boot Policy Decision Table". */
|
* the suite stays GREEN. See design.md "Boot Policy Decision Table". */
|
||||||
/* task 3.3 — FATAL rows (4, 4b) now ACTIVE (PASS), plus the never-FATAL
|
/* task 3.3 — unrecoverable rows (4, 4b) = LOUD PERMANENT FENCE (stays alive,
|
||||||
* invariant for the peer-reachable rows (3, 5). The full FENCE->ELIGIBLE
|
* non-promotable, clears on catch-up), plus the never-unrecoverable
|
||||||
* eligibility outcome of rows 1,2,3,5,6 stays IGNORED pending task 4.3. */
|
* invariant for the peer-reachable rows (3, 5) and the FENCE->ELIGIBLE
|
||||||
|
* eligibility outcome of rows 1,2,3,5,6. (never-self-exit rework) */
|
||||||
RUN_TEST(test_boot_policy_peer_reachable_never_fatal);
|
RUN_TEST(test_boot_policy_peer_reachable_never_fatal);
|
||||||
RUN_TEST(test_boot_policy_row1_clean_at_or_above_eligible);
|
RUN_TEST(test_boot_policy_row1_clean_at_or_above_eligible);
|
||||||
RUN_TEST(test_boot_policy_row2_clean_below_peer_catchup_eligible);
|
RUN_TEST(test_boot_policy_row2_clean_below_peer_catchup_eligible);
|
||||||
RUN_TEST(test_boot_policy_row3_truncated_peer_catchup_eligible);
|
RUN_TEST(test_boot_policy_row3_truncated_peer_catchup_eligible);
|
||||||
RUN_TEST(test_boot_policy_row4_truncated_no_peer_bounded_fatal);
|
RUN_TEST(test_boot_policy_row4_truncated_no_peer_loud_fence);
|
||||||
RUN_TEST(test_boot_policy_row4b_clean_below_no_peer_bounded_fatal);
|
RUN_TEST(test_boot_policy_row4b_clean_below_no_peer_loud_fence);
|
||||||
RUN_TEST(test_boot_policy_row5_snapshot_sentinel_catchup_never_fatal);
|
RUN_TEST(test_boot_policy_row5_snapshot_sentinel_catchup_never_fatal);
|
||||||
RUN_TEST(test_boot_policy_row6_empty_first_boot_eligible);
|
RUN_TEST(test_boot_policy_row6_empty_first_boot_eligible);
|
||||||
/* task 4.3 — hollow node refused even at minimal quorum (Req 2.5). */
|
/* task 4.3 — hollow node refused even at minimal quorum (Req 2.5). */
|
||||||
RUN_TEST(test_promotion_gate_hollow_refused_at_minimal_quorum);
|
RUN_TEST(test_promotion_gate_hollow_refused_at_minimal_quorum);
|
||||||
/* task 4.4 — bounded-window catch-up before FATAL (no synchronous FATAL;
|
/* task 4.4 — catch-up re-alert window (NEVER self-exits; stays alive +
|
||||||
* deferred-timer FATAL only after a real bounded failure; fence released on
|
* loudly non-promotable on the unrecoverable path; fence + sticky flag
|
||||||
* catch-up; cancelled timer never FATALs). (Req 2.2, 2.4, 2.6) */
|
* released on catch-up; cancelled timer stops the re-alert). (Req 2.2/2.4/2.6) */
|
||||||
RUN_TEST(test_promotion_fence_defers_fatal_to_bounded_timer);
|
RUN_TEST(test_promotion_fence_unrecoverable_stays_alive_loud);
|
||||||
RUN_TEST(test_promotion_fence_released_on_catchup_no_fatal);
|
RUN_TEST(test_promotion_fence_released_on_catchup);
|
||||||
RUN_TEST(test_promotion_fence_cancel_prevents_fatal);
|
RUN_TEST(test_promotion_fence_cancel_stops_realert);
|
||||||
#else
|
#else
|
||||||
printf("test_commit_durability: skipped (no XCE_TESTING)\n");
|
printf("test_commit_durability: skipped (no XCE_TESTING)\n");
|
||||||
#endif
|
#endif
|
||||||
|
|
|
||||||
|
|
@ -46,19 +46,29 @@ typedef enum {
|
||||||
* ELIGIBLE — may promote (no fence). Rows 1, 6.
|
* ELIGIBLE — may promote (no fence). Rows 1, 6.
|
||||||
* FENCE — ineligible to promote; catch up from a healthy peer, regain
|
* FENCE — ineligible to promote; catch up from a healthy peer, regain
|
||||||
* eligibility on success (Req 2.6). Rows 2, 3, 5. A reachable
|
* eligibility on success (Req 2.6). Rows 2, 3, 5. A reachable
|
||||||
* healthy peer NEVER yields FATAL.
|
* healthy peer NEVER yields FENCE_UNRECOVERABLE.
|
||||||
* FATAL — fail loud: cannot recover locally AND cannot catch up within
|
* FENCE_UNRECOVERABLE — loud, PERMANENT, NON-terminating fence: the member
|
||||||
* the bounded window. Terminate with the truncation/boundary
|
* cannot recover locally AND cannot catch up (no healthy peer).
|
||||||
* logged rather than serve divergent state (Req 2.4). Rows 4, 4b.
|
* It logs LOUDLY, stays PERMANENTLY non-promotable, and keeps
|
||||||
|
* retrying catch-up indefinitely — but the process STAYS ALIVE.
|
||||||
|
* Rows 4, 4b. This is NOT process exit: a controller MUST NEVER
|
||||||
|
* self-terminate as a designed response to an operational
|
||||||
|
* condition (user invariant, 2026-06-18). A self-exited controller
|
||||||
|
* needs MANUAL admin recovery and leaves quorum degraded
|
||||||
|
* meanwhile; a controller that stays alive and keeps retrying
|
||||||
|
* SELF-HEALS when a peer returns. The fence RELEASES (→ ELIGIBLE,
|
||||||
|
* Req 2.6) automatically when catch-up reaches a
|
||||||
|
* committed-consistent state.
|
||||||
*
|
*
|
||||||
* AUTHORITATIVE INVARIANT (design.md): the fence is never a silent indefinite
|
* AUTHORITATIVE INVARIANT (design.md): a truncated/behind member resolves to
|
||||||
* wait — a truncated/behind member resolves to exactly one of FENCE→ELIGIBLE
|
* either FENCE→ELIGIBLE (caught up) or a loud PERMANENT FENCE_UNRECOVERABLE
|
||||||
* or a BOUNDED FATAL. There is no "fenced forever, silently divergent" state.
|
* (stays alive, non-promotable, keeps retrying). It is NEVER silently divergent
|
||||||
|
* (it refuses to lead) and the process NEVER self-exits.
|
||||||
* ============================================================ */
|
* ============================================================ */
|
||||||
typedef enum {
|
typedef enum {
|
||||||
RECOVER_DECISION_ELIGIBLE = 0,
|
RECOVER_DECISION_ELIGIBLE = 0,
|
||||||
RECOVER_DECISION_FENCE = 1,
|
RECOVER_DECISION_FENCE = 1,
|
||||||
RECOVER_DECISION_FATAL = 2,
|
RECOVER_DECISION_FENCE_UNRECOVERABLE = 2,
|
||||||
} recover_decision_t;
|
} recover_decision_t;
|
||||||
|
|
||||||
/* Policy inputs evaluated at boot / promotion-eligibility (design.md table).
|
/* Policy inputs evaluated at boot / promotion-eligibility (design.md table).
|
||||||
|
|
@ -177,6 +187,22 @@ struct persist_s {
|
||||||
* before the node participates; read by the promotion gate thereafter. */
|
* before the node participates; read by the promotion gate thereafter. */
|
||||||
recover_verdict_t recovery_verdict;
|
recover_verdict_t recovery_verdict;
|
||||||
|
|
||||||
|
/* Sticky "unrecoverable, loudly fenced" flag (ha-commit-durability Req 2.4,
|
||||||
|
* NEVER-SELF-EXIT rework 2026-06-18). Set by
|
||||||
|
* persist_recover_loud_fence_if_unrecoverable when a member is
|
||||||
|
* recovery-truncated AND/OR durable below the cluster committed high-water
|
||||||
|
* mark with NO successful catch-up: the member stays ALIVE but is held
|
||||||
|
* PERMANENTLY non-promotable and keeps retrying catch-up. A controller MUST
|
||||||
|
* NEVER self-exit as a designed response to this operational condition; it
|
||||||
|
* self-heals automatically when a healthy peer returns and catch-up reaches
|
||||||
|
* a committed-consistent state — at which point this flag is CLEARED via
|
||||||
|
* persist_clear_recovery_unrecoverable (wired into the fence-release /
|
||||||
|
* catch-up path). Read by the promotion gate as belt-and-suspenders
|
||||||
|
* (a member must never lead while this is set). Atomic: set on the recovery
|
||||||
|
* path, cleared on the catch-up path, read by the promotion/heartbeat
|
||||||
|
* threads. Defaults to false before recovery runs. */
|
||||||
|
_Atomic bool recovery_unrecoverable;
|
||||||
|
|
||||||
/* Paths (derived from workdir + name via persist_set_workdir) */
|
/* Paths (derived from workdir + name via persist_set_workdir) */
|
||||||
char wal_path[4096];
|
char wal_path[4096];
|
||||||
char snap_path[4096];
|
char snap_path[4096];
|
||||||
|
|
@ -347,53 +373,64 @@ uint64_t persist_durable_floor(persist_t* p);
|
||||||
recover_verdict_t persist_recovery_verdict(persist_t* p);
|
recover_verdict_t persist_recovery_verdict(persist_t* p);
|
||||||
bool persist_recovery_truncated(persist_t* p);
|
bool persist_recovery_truncated(persist_t* p);
|
||||||
|
|
||||||
/* Bounded catch-up window (ha-commit-durability task 3.3, Req 2.4).
|
/* Bounded catch-up RE-ALERT window (ha-commit-durability task 3.3, Req 2.4;
|
||||||
* The FINITE deadline a recovery-truncated / behind member is given to reach a
|
* never-self-exit rework 2026-06-18).
|
||||||
* committed-consistent state from a healthy peer before the boot policy FATALs.
|
* The interval after which a still-unrecoverable, loudly-fenced member RE-EMITS
|
||||||
* The fence must NEVER be an indefinite wait (design.md authoritative
|
* its operator alert (it does NOT terminate — see RECOVER_DECISION_FENCE_-
|
||||||
* invariant), so the window is always finite. Default 30s — generous enough for
|
* UNRECOVERABLE). The member stays alive and keeps retrying catch-up
|
||||||
* a healthy-peer catch-up stream yet bounded; the verification mode overrides it
|
* indefinitely; this window only controls how often the loud operator-visible
|
||||||
* via the inputs/timeout the caller (task 4.3 boot path) supplies. */
|
* alert is repeated. Default 30s. */
|
||||||
#define DEFAULT_RECOVERY_CATCHUP_WINDOW_MS 30000u
|
#define DEFAULT_RECOVERY_CATCHUP_WINDOW_MS 30000u
|
||||||
|
|
||||||
/* persist_recover_decision (ha-commit-durability task 3.3, Req 2.4).
|
/* persist_recover_decision (ha-commit-durability task 3.3, Req 2.4).
|
||||||
*
|
*
|
||||||
* Pure decision over the Boot Policy Decision Table: maps the policy inputs to
|
* Pure decision over the Boot Policy Decision Table: maps the policy inputs to
|
||||||
* RECOVER_DECISION_{ELIGIBLE,FENCE,FATAL}. No side effects, no termination — it
|
* RECOVER_DECISION_{ELIGIBLE,FENCE,FENCE_UNRECOVERABLE}. No side effects, no
|
||||||
* only classifies, so it is unit-testable in isolation. The FATAL terminal is
|
* termination — it only classifies, so it is unit-testable in isolation. The
|
||||||
* acted on by persist_recover_fatal_if_unrecoverable (which logs + terminates).
|
* FENCE_UNRECOVERABLE terminal is acted on by
|
||||||
|
* persist_recover_loud_fence_if_unrecoverable (which logs LOUDLY + sets the
|
||||||
|
* sticky non-promotable flag — it NEVER terminates the process).
|
||||||
*
|
*
|
||||||
* Contract (design.md rows): a reachable healthy peer NEVER returns FATAL
|
* Contract (design.md rows): a reachable healthy peer NEVER returns
|
||||||
* (rows 3, 5); FATAL is returned ONLY when the member is truncated or behind
|
* FENCE_UNRECOVERABLE (rows 3, 5); it is returned ONLY when the member is
|
||||||
* the HWM, has NO healthy peer, and bounded catch-up did not reach a
|
* truncated or behind the HWM, has NO healthy peer, and catch-up did not reach
|
||||||
* committed-consistent state (rows 4, 4b). */
|
* a committed-consistent state (rows 4, 4b). */
|
||||||
recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in);
|
recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in);
|
||||||
|
|
||||||
/* persist_recover_fatal_if_unrecoverable (ha-commit-durability task 3.3,
|
/* persist_recover_loud_fence_if_unrecoverable (ha-commit-durability task 3.3,
|
||||||
* Req 2.4).
|
* Req 2.4; never-self-exit rework 2026-06-18).
|
||||||
*
|
*
|
||||||
* Evaluate persist_recover_decision over `in`; if the outcome is FATAL, log at
|
* Evaluate persist_recover_decision over `in`; if the outcome is
|
||||||
* ERROR LOUDLY — INCLUDING the truncation boundary (durable_floor_lsn) and the
|
* FENCE_UNRECOVERABLE, log LOUDLY at ERROR — INCLUDING the truncation boundary
|
||||||
* cluster high-water mark — and then TERMINATE (fail loud rather than serve
|
* (durable_floor_lsn) and the cluster high-water mark — stating the member is
|
||||||
* divergent state). Returns the decision so a non-FATAL caller can fence/proceed
|
* staying fenced + non-promotable and will keep retrying catch-up, then SET the
|
||||||
* (FENCE/ELIGIBLE are acted on by task 4.3).
|
* sticky `recovery_unrecoverable` flag on `p` (so the promotion gate refuses
|
||||||
|
* leadership as belt-and-suspenders). Returns the decision.
|
||||||
*
|
*
|
||||||
* Termination routes through persist_fatal_hook: in production the hook is NULL
|
* CRITICAL — the process STAYS ALIVE. There is NO abort()/exit()/tlog_exit()
|
||||||
* and the function flushes logs and abort()s (NEVER returns on FATAL); a test
|
* here: a controller MUST NEVER self-terminate as a designed response to an
|
||||||
* build may install a hook to capture the FATAL decision + boundary log without
|
* operational condition (user invariant, 2026-06-18). The member self-heals —
|
||||||
* the process dying (mirrors the ha_fatal_hook seam in ha_convergence.c). */
|
* the fence releases (→ ELIGIBLE, Req 2.6) and the flag clears via
|
||||||
|
* persist_clear_recovery_unrecoverable — automatically when a healthy peer
|
||||||
|
* returns and catch-up reaches a committed-consistent state.
|
||||||
|
*
|
||||||
|
* `p` may be NULL (pure-decision callers that only want the classification +
|
||||||
|
* loud log without touching a persist_t); the flag set is then skipped. */
|
||||||
recover_decision_t
|
recover_decision_t
|
||||||
persist_recover_fatal_if_unrecoverable(const recover_policy_inputs_t* in);
|
persist_recover_loud_fence_if_unrecoverable(persist_t* p,
|
||||||
|
const recover_policy_inputs_t* in);
|
||||||
|
|
||||||
/* persist_fatal_hook: test-only seam for the FATAL (abort) contract of
|
/* persist_recovery_unrecoverable / persist_clear_recovery_unrecoverable
|
||||||
* persist_recover_fatal_if_unrecoverable (ha-commit-durability task 3.3,
|
* (ha-commit-durability Req 2.4/2.6, never-self-exit rework 2026-06-18).
|
||||||
* Req 2.4). Mirrors ha_fatal_hook (ha_convergence.c).
|
|
||||||
*
|
*
|
||||||
* When non-NULL, the FATAL path invokes this INSTEAD OF abort(), so a test
|
* Accessor + clear for the sticky "unrecoverable, loudly fenced" flag. The flag
|
||||||
* binary can assert the FATAL decision was taken and the truncation boundary
|
* is SET by persist_recover_loud_fence_if_unrecoverable and CLEARED on the
|
||||||
* was logged without the process dying. Production leaves it NULL and actually
|
* catch-up / fence-release path once the member reaches a committed-consistent
|
||||||
* terminates. Signature: void hook(const char *msg). */
|
* state (not truncated AND durable_floor >= HWM). While set, the member is held
|
||||||
extern void (*persist_fatal_hook)(const char* msg);
|
* permanently non-promotable but stays alive. Safe on a NULL p (returns false /
|
||||||
|
* no-op). */
|
||||||
|
bool persist_recovery_unrecoverable(persist_t* p);
|
||||||
|
void persist_clear_recovery_unrecoverable(persist_t* p);
|
||||||
|
|
||||||
/* Register a durable-advance wake callback (ha-commit-durability task 2.2).
|
/* Register a durable-advance wake callback (ha-commit-durability task 2.2).
|
||||||
* The group-commit flusher calls fn(ctx) after it advances durable_lsn and
|
* The group-commit flusher calls fn(ctx) after it advances durable_lsn and
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue