lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha-commit-durability)!: controller never self-terminates — loud non-promotable fence replaces recovery FATAL/abort

Browse source 
Per design invariant (a controller exits only on crash or admin kill), replace the
self-abort recovery terminal (tasks 3.3/4.3/4.4) with a loud, permanent, non-promotable
fence that stays alive and keeps retrying catch-up (self-heals on peer return; no manual
admin recovery). Records the never-self-terminate invariant as project-wide guidance in
src/controller/CLAUDE.md + .kiro/steering/ha-async-io.md.
This commit is contained in:
Rongsong Shen 2026-06-18 11:10:19 +00:00
parent addf53e3b4
commit 59d9df246c
10 changed files with 607 additions and 406 deletions
Download patch file Download diff file Expand all files Collapse all files

17
src/controller/CLAUDE.md
View file

@ -24,6 +24,15 @@ Links against `libxce` from `src/lib/`.
- Component-specific headers in `include/`
- Shared types defined in `src/include/xce/` (libxce)
## Process Lifecycle (DESIGN INVARIANT — applies to ALL controller tasks)
**A controller process MUST NEVER self-terminate.** It exits ONLY on (a) a genuine crash (segfault/OOM) or (b) an admin/supervisor kill. A controller must NOT call `exit()`/`_exit()`/`abort()`/`tlog_exit()` (or any self-termination) as a **designed response to an operational condition** — e.g. a recovery-truncated/corrupt WAL, an inability to reach quorum, a peer being unreachable, a failed catch-up, or any cluster-state condition.
- **Rationale:** a self-exited controller requires **manual admin recovery** to come back and leaves quorum degraded in the meantime. A controller that **stays alive and keeps retrying self-heals automatically** when a peer returns — no admin in the loop.
- **"Fail loud" = refuse to participate, not exit.** When a controller detects it cannot safely act (e.g. it cannot reconstruct committed state, or would serve divergent state), it must: log a **loud, operator-visible ERROR** (with the actionable boundary, e.g. durable_floor vs cluster high-water), enter a **permanent non-promotable / fenced** state (never become authoritative, never serve divergent state), **stay alive**, and **keep retrying catch-up** so it recovers automatically when conditions allow. Clear the fenced state once a committed-consistent state is reached.
- **The only acceptable `assert()`/abort** is for a genuine *programming-invariant* violation (a bug that means the process is in an impossible state) — that is a "crash," not an operational response. Never use `assert()`/abort to handle expected operational/failure conditions.
- **When reviewing or writing controller code:** any new `exit`/`abort`/`tlog_exit`/`FATAL`-style terminal on an operational path is a design violation — replace it with the loud-fence-and-stay-alive pattern above. (Precedent: `ha-commit-durability` recovery-unrecoverable handling — `persist_recover_loud_fence_if_unrecoverable` + the `recovery_unrecoverable` sticky fence, which replaced an earlier incorrect `abort()` terminal.)
## HA Protocol
See `.kiro/steering/ha-async-io.md` for the full HA protocol reference: async I/O model (iomgr/vtaskmgr/delaymgr), connection ownership, Raft election, heartbeat model, two-tier replication consistency, dispatch gate, and TLA+ variable mapping. The TLA+ safety invariants and protocol verification checklist below are the authoritative controller-specific verification gates — keep them here.
@ -67,6 +76,10 @@ The file `raft/raft.tla` contains Diego Ongaro's TLA+ specification of the Raft
**Commit Rule** (current-term entry required): Each WAL entry is committed individually via `repl_wait_majority_ack()`. Since entries are single-writer (only the current primary writes), every committed entry is implicitly from the current leader's term. Equivalent to Raft's batched commit rule.
**Commit Ordering** (contiguous, in log order — RULE): `committed_lsn` advances only as a contiguous prefix in LSN/append order; entries are never committed or reordered out of log order. The single-writer append lock (`p->lock`) is the serialization point. Durability flushing (incl. group-commit batching) makes a contiguous **prefix** durable only — `durable_lsn` is a prefix watermark, never a per-entry set, so a later entry can never be durable while an earlier one is not. Log order must preserve **causal dependencies** (append a cause before its effect — e.g. job submit before job start); independent concurrent events take an arbitrary but consistent total order, which is linearizable via the single serialization point plus commit-gated reply. Out-of-order / parallel commit is NOT supported by this single-log architecture (it would require per-entity logs with dependency tracking — explicitly out of scope).
**Commit Durability** (acknowledged ⇒ durable on a majority — TARGET RULE): when the controller replies success to a client, the operation's entry must be `fdatasync`'d on a fault-tolerant majority (the acking member + ⌊N/2⌋ others); a member counts toward the commit majority for an LSN only once that LSN is durable locally (self caps at `durable_lsn`, not the appended tip). **KNOWN GAP — bug-661, spec `ha-commit-durability` (in progress, 2026-06-18):** the commit majority currently counts the primary's *appended* (page-cache) tip via `majority_candidate_locked`, and primary fsync is async (`fsync_worker` timer), so "committed" does NOT yet imply durable-on-a-majority — an acknowledged entry can be durable on a single member and lost across failover. Until that spec lands, do not assume an acked operation survives a crash of the acking minority.
### Post-Election Safe Commit Point (Critical)
After winning election, the new primary determines `safe_committed_lsn` from voter state collected during the election. This eliminates a round-trip vs waiting for the first heartbeat reply:
@ -89,6 +102,10 @@ When modifying HA protocol code (election, replication, heartbeat), verify:
- [ ] `safe_committed_lsn` is derived from voter replies, not from stale ctrlhello
- [ ] WAL truncation on promotion only removes entries above the safe commit point
- [ ] Freshness gate in RequestVote: candidate must have `lastLogTerm >= voter` AND `lastLogIndex >= voter`
- [ ] `committed_lsn` advances only as a contiguous LSN-ordered prefix — no out-of-order or per-entry commit
- [ ] Durability flushing makes a contiguous prefix durable; `durable_lsn` is a prefix watermark (never a per-entry set)
- [ ] Causal dependencies are appended in order (cause before effect, e.g. submit before start)
- [ ] (post `ha-commit-durability`) a member counts toward the commit majority only at its durably-persisted LSN; acknowledged ⇒ durable on a majority
## HA Design Constraints

202
src/controller/ha.c
View file

@ -572,8 +572,10 @@ recover_decision_t ha_promotion_gate_decision(bool recovery_truncated,
* the committed boundary regains eligibility, Req 2.6).
* - FENCE → set the fence, return FENCE (refuse promotion; catch up from a
* healthy peer).
* - FATAL → run the 3.3 loud-terminate path (logs the truncation boundary,
* then terminates — or fires persist_fatal_hook in tests).
* - FENCE_UNRECOVERABLE → run the 3.3 loud-fence action (logs LOUDLY + sets
* the sticky non-promotable flag) and ALSO set the in_catch_up
* fence. The member STAYS ALIVE, non-promotable, retrying
* catch-up — it NEVER self-exits (Req 2.4).
*/
recover_decision_t ha_evaluate_promotion_fence(ha_state_t *ha,
bool recovery_truncated,
@ -603,10 +605,11 @@ recover_decision_t ha_evaluate_promotion_fence(ha_state_t *ha,
"ha-commit-durability Req 2.2)",
recovery_truncated, durable_floor, safe_hwm);
break;
case RECOVER_DECISION_FATAL: {
/* Bounded-unrecoverable: route through the 3.3 loud-terminate action,
* which logs the truncation boundary at ERROR then terminates (or fires
* persist_fatal_hook under test). Req 2.4. */
case RECOVER_DECISION_FENCE_UNRECOVERABLE: {
/* Unrecoverable: route through the 3.3 loud-fence action — logs LOUDLY
* and sets the sticky non-promotable flag. Also hold the in_catch_up
* fence so the member refuses promotion. The process STAYS ALIVE and
* keeps retrying catch-up; it NEVER self-exits (Req 2.4). */
recover_policy_inputs_t in = {
.verdict = recovery_truncated ? RECOVER_TRUNCATED
: RECOVER_CLEAN,
@ -616,25 +619,36 @@ recover_decision_t ha_evaluate_promotion_fence(ha_state_t *ha,
.durable_floor_lsn = durable_floor,
.cluster_hwm_lsn = safe_hwm,
};
persist_recover_fatal_if_unrecoverable(&in);
persist_t *p = (ha != NULL && ha->raft_c != NULL) ? &ha->raft_c->persist
: NULL;
persist_recover_loud_fence_if_unrecoverable(p, &in);
if (ha != NULL)
atomic_store_explicit(&ha->in_catch_up, 1, memory_order_release);
break;
}
}
return d;
}
/* ── ha-commit-durability task 4.4: bounded-window catch-up before FATAL ─────
/* ── ha-commit-durability task 4.4: bounded-window catch-up re-alert ─────────
*
* The 4.3 gate above can DECIDE FATAL synchronously, but a member that won an
* election with no peer reachable AT THE PROMOTION INSTANT may simply be seeing
* a transient no-peer snapshot during aggressive failover (task-5.2 soak: a
* recovery-truncated winner self-FATAL'd immediately and repeatedly until
* quorum was lost — an availability collapse, not durable loss). Per the 3.2
* decision table the FATAL terminal must be reached ONLY after a BOUNDED
* failure: the member is fenced (in_catch_up), given DEFAULT_RECOVERY_CATCHUP_-
* WINDOW_MS to find a peer / make catch-up progress, and FATAL'd from a one-shot
* timer only if it is STILL hollow with no peer and no progress when the window
* elapses.
* (never-self-exit rework 2026-06-18.) A member that wins an election while
* recovery-truncated / behind the HWM with no peer reachable AT THE PROMOTION
* INSTANT is FENCED (in_catch_up set, non-promotable) and STAYS ALIVE, retrying
* catch-up indefinitely. This one-shot timer exists ONLY for operator
* VISIBILITY: when it fires, if the member is STILL unrecoverable (still
* truncated/behind AND no peer AND no progress) it RE-EMITS the loud,
* operator-visible alert via persist_recover_loud_fence_if_unrecoverable (which
* also keeps the sticky non-promotable flag set) — it NEVER terminates the
* process. If the member has caught up (not truncated AND durable_floor >=
* HWM) the timer RELEASES the fence (Req 2.6). If progress is possible (peer
* back, or durable floor advanced) it leaves the member fenced under the
* existing catch-up machinery.
*
* Earlier this timer DEFERRED-then-ABORTED. The abort is GONE: a controller
* MUST NEVER self-terminate as a designed response to an operational condition
* (user invariant, 2026-06-18). The task-5.2 soak quorum collapse was caused
* by that self-abort; staying alive + retrying self-heals when a peer returns.
*
* The window is a settable seam so the timer test runs fast. Default =
* DEFAULT_RECOVERY_CATCHUP_WINDOW_MS (persist.h). */
@ -684,21 +698,26 @@ void ha_cancel_promotion_fence_timer(ha_state_t *ha)
atomic_store_explicit(&t->cancelled, 1, memory_order_release);
}
/* promotion_fence_timeout_fn: the bounded-window timer callback. Owns its ctx —
* frees it on exit. Re-evaluates the LIVE recovery/durability state and reaches
* the 3.3 FATAL ONLY on a real bounded failure (Req 2.4):
* - still recovery-truncated/behind AND no healthy peer reachable AND no
* catch-up progress (durable_floor has not advanced past the baseline)
* → persist_recover_fatal_if_unrecoverable (fail loud).
/* Forward decl: the timeout callback re-arms itself for periodic re-alert. */
static void ha_arm_promotion_fence_timer(ha_state_t *ha, cluster_t *c,
bool truncated, uint64_t durable_floor,
uint64_t target_hwm);
/* promotion_fence_timeout_fn: the bounded-window re-alert timer callback. Owns
* its ctx — frees it on exit. Re-evaluates the LIVE recovery/durability state.
* It NEVER terminates the process (never-self-exit invariant, 2026-06-18):
* - caught up (NOT truncated AND durable_floor >= target_hwm) → release the
* in_catch_up fence; do NOT FATAL (Req 2.6 — eligible again).
* - a healthy peer is reachable now, OR catch-up is making progress but is not
* yet complete → leave the member fenced under the existing catch-up
* machinery and do NOT FATAL while progress is possible. The single bounded
* window has elapsed, so we do not re-arm: a member with a live peer is no
* longer in the "no peer + no progress" bounded-failure state the FATAL
* guards, and the existing catch-up / re-election path will release the
* fence (ha_evaluate_promotion_fence → ELIGIBLE) when it completes. */
* in_catch_up fence + clear the sticky unrecoverable flag; eligible again
* (Req 2.6).
* - a healthy peer is reachable now, OR catch-up is making progress → leave
* the member fenced under the existing catch-up machinery (it will release
* via ha_evaluate_promotion_fence → ELIGIBLE when it completes); do not
* re-arm the re-alert while progress is happening.
* - STILL unrecoverable (truncated/behind AND no peer AND no progress) →
* RE-EMIT the loud operator alert via persist_recover_loud_fence_if_-
* unrecoverable (keeps the sticky flag + in_catch_up set) and RE-ARM the
* timer so the alert keeps repeating until a peer returns. The process
* STAYS ALIVE and keeps retrying — it NEVER aborts. */
static int promotion_fence_timeout_fn(void *arg)
{
promotion_fence_ctx_t *t = (promotion_fence_ctx_t *)arg;
@ -721,7 +740,7 @@ static int promotion_fence_timeout_fn(void *arg)
if (!c) { free(t); return 0; } /* defensive: no persist to evaluate */
/* Re-read the LIVE state after the bounded window. */
/* Re-read the LIVE state after the window. */
bool truncated = persist_recovery_truncated(&c->persist);
uint64_t durable_floor = persist_durable_floor(&c->persist);
bool below_hwm = durable_floor < t->target_hwm;
@ -731,9 +750,11 @@ static int promotion_fence_timeout_fn(void *arg)
: false;
/* Caught up: not truncated AND durable position reached the boundary →
* release the fence; the member is eligible on its next election (Req 2.6). */
* release the fence + clear the sticky flag; the member is eligible on its
* next election (Req 2.6). */
if (!truncated && !below_hwm) {
atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release);
persist_clear_recovery_unrecoverable(&c->persist);
INFO("HA: promotion fence RELEASED after catch-up — durable_floor=%"
PRIu64 " >= cluster HWM=%" PRIu64 " (ha-commit-durability Req 2.6)",
durable_floor, t->target_hwm);
@ -742,14 +763,13 @@ static int promotion_fence_timeout_fn(void *arg)
}
/* Still hollow, but a peer is reachable or catch-up advanced the durable
* floor: progress is possible — do NOT FATAL. Leave the member fenced
* (in_catch_up stays set) under the existing catch-up machinery; the bounded
* window has elapsed so we do not re-arm. A member with a live peer is no
* longer in the no-peer bounded-failure state the FATAL guards. */
* floor: progress is possible. Leave the member fenced (in_catch_up stays
* set) under the existing catch-up machinery; do not re-arm the re-alert
* while progress is happening. */
if (peer_reachable || made_progress) {
INFO("HA: promotion fence catch-up window elapsed but progress is "
"possible (peer_reachable=%d durable_floor=%" PRIu64 " baseline=%"
PRIu64 " HWM=%" PRIu64 ") — remaining fenced, NOT FATAL "
PRIu64 " HWM=%" PRIu64 ") — remaining fenced, catching up "
"(ha-commit-durability Req 2.2)",
peer_reachable, durable_floor, t->baseline_durable_floor,
t->target_hwm);
@ -757,9 +777,11 @@ static int promotion_fence_timeout_fn(void *arg)
return 0;
}
/* Bounded failure: STILL truncated/behind AND no healthy peer AND no
* catch-up progress after the window → fail loud (Req 2.4). This is the
* ONLY path that reaches the 3.3 FATAL — never synchronously at promotion. */
/* STILL unrecoverable: truncated/behind AND no healthy peer AND no catch-up
* progress. RE-EMIT the loud operator alert and keep the member fenced +
* non-promotable + ALIVE (Req 2.4) — NEVER terminate. Re-arm the re-alert
* timer so the operator keeps hearing about it until a peer returns. */
uint64_t target_hwm = t->target_hwm; /* capture before free (UAF guard) */
{
recover_policy_inputs_t in = {
.verdict = truncated ? RECOVER_TRUNCATED
@ -768,27 +790,27 @@ static int promotion_fence_timeout_fn(void *arg)
.healthy_peer_reachable = false,
.catch_up_succeeded = false,
.durable_floor_lsn = durable_floor,
.cluster_hwm_lsn = t->target_hwm,
.cluster_hwm_lsn = target_hwm,
};
ERROR("HA: promotion catch-up window (%u ms) elapsed with no peer and "
"no progress — recovery_truncated=%d durable_floor=%" PRIu64
" < cluster HWM=%" PRIu64 "; failing loud rather than serving "
"divergent state (ha-commit-durability Req 2.4)",
ha_promotion_fence_window_ms(), truncated, durable_floor,
t->target_hwm);
persist_recover_fatal_if_unrecoverable(&in);
/* Loud, operator-visible alert + sticky non-promotable flag. STAYS
* ALIVE: no abort/exit anywhere on this path. */
persist_recover_loud_fence_if_unrecoverable(&c->persist, &in);
}
free(t);
/* Re-arm so the loud alert repeats periodically while the member remains
* unrecoverable. The member keeps retrying catch-up; the next firing
* re-evaluates and releases the fence once a peer returns and it catches up. */
ha_arm_promotion_fence_timer(ha, c, truncated, durable_floor, target_hwm);
return 0;
}
/* ha_arm_promotion_fence_timer: fence the member (set in_catch_up) and schedule
* the one-shot bounded-window FATAL timer. Cancels any previously-armed timer
* first so a re-evaluation never leaks or double-fires. Returns 0 on success.
* On a failure to schedule the timer it FATALs immediately via the 3.3 action so
* the bounded-failure terminal is never silently lost (a member that could not
* arm its own catch-up deadline must not run on indefinitely). */
* the one-shot bounded-window RE-ALERT timer. Cancels any previously-armed
* timer first so a re-evaluation never leaks or double-fires. On a failure to
* allocate/schedule the timer it emits the loud unrecoverable fence ONCE (stays
* alive, non-promotable) — it NEVER aborts (never-self-exit invariant). */
static void ha_arm_promotion_fence_timer(ha_state_t *ha, cluster_t *c,
bool truncated, uint64_t durable_floor,
uint64_t target_hwm)
@ -801,7 +823,7 @@ static void ha_arm_promotion_fence_timer(ha_state_t *ha, cluster_t *c,
promotion_fence_ctx_t *t = calloc(1, sizeof(*t));
if (!t)
goto fail_loud;
goto loud_fence_no_timer;
atomic_init(&t->cancelled, 0);
t->ha = ha;
@ -816,20 +838,22 @@ static void ha_arm_promotion_fence_timer(ha_state_t *ha, cluster_t *c,
if (rc < 0) {
WARN("ha_arm_promotion_fence_timer: delay_task_ms failed: %d", rc);
free(t);
goto fail_loud;
goto loud_fence_no_timer;
}
atomic_store_explicit(&ha->promotion_fence_ctx, t, memory_order_release);
INFO("HA: promotion FENCED + bounded catch-up window armed (%u ms) — "
INFO("HA: promotion FENCED + catch-up re-alert window armed (%u ms) — "
"recovery_truncated=%d durable_floor=%" PRIu64 " < cluster HWM=%" PRIu64
"; FATAL deferred to the timer (ha-commit-durability Req 2.2/2.4)",
"; staying ALIVE + non-promotable, retrying catch-up "
"(ha-commit-durability Req 2.2/2.4)",
window_ms, truncated, durable_floor, target_hwm);
return;
fail_loud:
loud_fence_no_timer:
{
/* Could not arm the bounded deadline — reach the FATAL terminal now
* rather than fence indefinitely. */
/* Could not arm the re-alert timer — still emit the loud unrecoverable
* fence once so the operator sees it; the member stays fenced
* (in_catch_up set above) + non-promotable + ALIVE. NO abort. */
recover_policy_inputs_t in = {
.verdict = truncated ? RECOVER_TRUNCATED
: RECOVER_CLEAN,
@ -839,7 +863,7 @@ fail_loud:
.durable_floor_lsn = durable_floor,
.cluster_hwm_lsn = target_hwm,
};
persist_recover_fatal_if_unrecoverable(&in);
persist_recover_loud_fence_if_unrecoverable(&c->persist, &in);
}
}
@ -919,10 +943,16 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
* the in_catch_up fence and return WITHOUT becoming primary; the member
* catches up from a healthy peer and a later promotion attempt passes the
* gate (Req 2.2/2.6). Only the production path (real persist) is gated; the
* c==NULL unit shim has no persist and skips it. */
* c==NULL unit shim has no persist and skips it.
*
* Belt-and-suspenders (never-self-exit rework 2026-06-18): a member that has
* been marked sticky-unrecoverable (persist_recovery_unrecoverable) must
* NEVER become primary while that flag is set — it stays alive, fenced, and
* keeps retrying catch-up. The flag clears automatically on catch-up. */
if (c != NULL) {
bool truncated = persist_recovery_truncated(&c->persist);
uint64_t durable_floor = persist_durable_floor(&c->persist);
bool unrecoverable = persist_recovery_unrecoverable(&c->persist);
/* A healthy peer is reachable for catch-up iff some voter granted with a
* live endpoint survives this election (these become the standby
@ -943,36 +973,36 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
/* catch_up_succeeded is false at the promotion-eligibility evaluation:
* a hollow member never silently proceeds to leadership. We evaluate
* the PURE 3.2/3.3 decision here and DEFER its terminal — task 4.4 does
* NOT reach the 3.3 FATAL synchronously at the promotion instant. A
* FENCE or a would-be-FATAL (truncated/behind + no peer right now) both
* fence the member (in_catch_up) and arm the bounded catch-up window;
* the FATAL is reached ONLY from that timer after a real bounded failure
* (still hollow + no peer + no progress). This avoids the task-5.2 soak
* collapse where a transient no-peer snapshot during aggressive failover
* self-FATAL'd a recovering winner repeatedly until quorum was lost. */
* the PURE 3.2/3.3 decision here. A FENCE or an UNRECOVERABLE
* (truncated/behind + no peer right now) both fence the member
* (in_catch_up) and arm the catch-up re-alert timer; the member STAYS
* ALIVE and keeps retrying — it NEVER self-exits (never-self-exit
* invariant, 2026-06-18; this is what fixes the task-5.2 soak collapse
* where a self-aborting recovering winner repeatedly dropped quorum). */
recover_decision_t gate = ha_promotion_gate_decision(
truncated, durable_floor, promotion_safe_lsn,
peer_reachable, /*catch_up_succeeded=*/false);
if (gate == RECOVER_DECISION_ELIGIBLE) {
/* ELIGIBLE: release the fence (a caught-up member regains
* eligibility, Req 2.6) and cancel any pending bounded-window FATAL
* timer — a member that is now promotable must never be FATAL'd by a
* stale timer. Proceed to authoritative promotion below. */
if (gate == RECOVER_DECISION_ELIGIBLE && !unrecoverable) {
/* ELIGIBLE and not sticky-unrecoverable: release the fence (a
* caught-up member regains eligibility, Req 2.6), clear the sticky
* flag if set, and cancel any pending re-alert timer. Proceed to
* authoritative promotion below. */
atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release);
persist_clear_recovery_unrecoverable(&c->persist);
ha_cancel_promotion_fence_timer(ha);
} else {
/* FENCE or would-be-FATAL: refuse promotion — do NOT become
* CTRL_PRIMARY. Fence the member and arm the bounded catch-up
* window; the 3.3 FATAL is deferred to the timer (Req 2.2/2.4).
* Never FATAL synchronously, never fence indefinitely. */
WARN("HA: promotion REFUSED (decision=%d) — recovery_truncated=%d "
"durable_floor=%" PRIu64 " cluster_hwm=%" PRIu64
" peer_reachable=%d; fencing + arming bounded catch-up window "
/* FENCE, UNRECOVERABLE, or sticky-unrecoverable: refuse promotion —
* do NOT become CTRL_PRIMARY. Fence the member and arm the catch-up
* re-alert timer. The member STAYS ALIVE, non-promotable, retrying
* catch-up (Req 2.2/2.4). NEVER self-exit, never fence silently. */
WARN("HA: promotion REFUSED (decision=%d unrecoverable=%d) — "
"recovery_truncated=%d durable_floor=%" PRIu64
" cluster_hwm=%" PRIu64 " peer_reachable=%d; fencing + arming "
"catch-up re-alert window, staying ALIVE "
"(ha-commit-durability Req 2.2/2.4)",
(int)gate, truncated, durable_floor, promotion_safe_lsn,
peer_reachable);
(int)gate, unrecoverable, truncated, durable_floor,
promotion_safe_lsn, peer_reachable);
ha_arm_promotion_fence_timer(ha, c, truncated, durable_floor,
promotion_safe_lsn);
return;

55
src/controller/ha.h
View file

@ -520,7 +520,10 @@ void ha_compute_safe_committed_lsn(ha_elect_ctx_t *ctx,
* 2.5, 2.6). Maps the promoting member's recovery/durability state to the Boot
* Policy Decision Table outcome by building recover_policy_inputs_t and
* delegating to persist_recover_decision (DRY — the 3.2/3.3 decision table is
* the single source of truth; this gate does not re-derive it):
* the single source of truth; this gate does not re-derive it). Note the
* FENCE_UNRECOVERABLE terminal is a LOUD PERMANENT NON-promotable fence that
* STAYS ALIVE and keeps retrying catch-up — it is NEVER process exit
* (never-self-exit invariant, 2026-06-18):
* - recovery_truncated → the task-3.1 structural-corruption flag.
* - durable_floor → the local CLAMPED durable position (read AFTER the
* promotion-path persist_truncate_wal_above so the
@ -547,9 +550,12 @@ recover_decision_t ha_promotion_gate_decision(bool recovery_truncated,
* boundary regains eligibility — Req 2.6).
* - FENCE → sets the in_catch_up fence (refuse promotion; catch up from a
* healthy peer) and returns FENCE.
* - FATAL → routes through persist_recover_fatal_if_unrecoverable (logs the
* truncation boundary at ERROR then terminates; the test seam
* persist_fatal_hook captures it) and returns FATAL.
* - FENCE_UNRECOVERABLE → routes through
* persist_recover_loud_fence_if_unrecoverable (logs LOUDLY at
* ERROR + sets the sticky persist recovery_unrecoverable flag),
* also sets the in_catch_up fence, and returns
* FENCE_UNRECOVERABLE. The process STAYS ALIVE, non-promotable,
* retrying catch-up — it NEVER self-exits (Req 2.4).
* Reuses the existing in_catch_up fence — does NOT invent a parallel fence. */
recover_decision_t ha_evaluate_promotion_fence(ha_state_t *ha,
bool recovery_truncated,
@ -558,34 +564,37 @@ recover_decision_t ha_evaluate_promotion_fence(ha_state_t *ha,
bool peer_reachable,
bool catch_up_succeeded);
/* ── ha-commit-durability task 4.4: bounded-window catch-up before FATAL ─────
/* ── ha-commit-durability task 4.4: catch-up re-alert window ─────────────────
* (never-self-exit rework 2026-06-18.)
*
* The promotion gate (4.3) no longer reaches the 3.3 FATAL synchronously at the
* promotion instant. When a recovery-truncated / behind member wins an election
* with no reachable peer AT THAT INSTANT (a transient no-peer snapshot during
* aggressive failover), it FENCES (sets in_catch_up, does NOT become primary)
* and arms a one-shot delaymgr timer for DEFAULT_RECOVERY_CATCHUP_WINDOW_MS.
* The 3.3 FATAL is reached ONLY from that timer, and ONLY if the member is STILL
* recovery-truncated/behind AND no healthy peer is reachable AND no catch-up
* progress was made when it fires (Req 2.4). Catch-up that brings durable_floor
* >= HWM releases the fence and cancels the timer (Req 2.6); shutdown cancels it
* too — never a synchronous FATAL, never an indefinite fence.
* When a recovery-truncated / behind member wins an election with no reachable
* peer AT THAT INSTANT (a transient no-peer snapshot during aggressive
* failover), it FENCES (sets in_catch_up, does NOT become primary) and arms a
* delaymgr re-alert timer for DEFAULT_RECOVERY_CATCHUP_WINDOW_MS. When the
* timer fires, if the member is STILL recovery-truncated/behind AND no healthy
* peer is reachable AND no catch-up progress was made, it RE-EMITS the loud
* operator alert (persist_recover_loud_fence_if_unrecoverable) and RE-ARMS —
* the process STAYS ALIVE and keeps retrying catch-up INDEFINITELY. It NEVER
* terminates: a controller must never self-exit as a designed response to an
* operational condition. Catch-up that brings durable_floor >= HWM releases
* the fence + clears the sticky flag and stops the re-alert (Req 2.6); shutdown
* cancels it too.
*
* ha_promotion_fence_window_ms: the bounded catch-up window in milliseconds.
* ha_promotion_fence_window_ms: the re-alert interval in milliseconds.
* Defaults to DEFAULT_RECOVERY_CATCHUP_WINDOW_MS; tests override it (via
* ha_set_promotion_fence_window_ms) to a small value so the timer fires fast. */
uint32_t ha_promotion_fence_window_ms(void);
/* ha_cancel_promotion_fence_timer: cancel a pending bounded-window FATAL timer
* (idempotent; no-op when none is armed). Called when the member catches up
* (fence released) or on shutdown, so neither a caught-up nor a stopped member
* ever reaches the deferred FATAL. */
/* ha_cancel_promotion_fence_timer: cancel a pending re-alert timer (idempotent;
* no-op when none is armed). Called when the member catches up (fence released)
* or on shutdown, so a stopped member stops re-alerting. It never gated a
* process exit — the timer never terminates the process. */
void ha_cancel_promotion_fence_timer(ha_state_t *ha);
#ifdef XCE_TESTING
/* Test seam: override the bounded catch-up window so the deferred-FATAL timer
* fires deterministically fast. 0 restores the DEFAULT_RECOVERY_CATCHUP_WINDOW_MS
* default. XCE_TESTING only — compiled out of production. */
/* Test seam: override the re-alert window so the timer fires deterministically
* fast. 0 restores the DEFAULT_RECOVERY_CATCHUP_WINDOW_MS default. XCE_TESTING
* only — compiled out of production. */
void ha_set_promotion_fence_window_ms(uint32_t ms);
#endif

121
src/controller/persist.c
View file

@ -449,6 +449,11 @@ int persist_init(persist_t* p, persist_config_t* cfg, cluster_t* c)
* replay outcome (RECOVER_TRUNCATED only on a structural log corruption). */
p->recovery_verdict = RECOVER_CLEAN;
/* Sticky "unrecoverable, loudly fenced" flag starts clear (Req 2.4): a fresh
* store is promotable. Set by persist_recover_loud_fence_if_unrecoverable
* on an unrecoverable boot; cleared on catch-up (never self-exits). */
atomic_store(&p->recovery_unrecoverable, false);
/* Durable-advance wake hook starts unregistered (ha-commit-durability 2.2):
* the commit engine registers it in commit_engine_init. Until then every
* flusher durable advance is a no-op wake. */
@ -1700,21 +1705,25 @@ bool persist_recovery_truncated(persist_t* p)
}
/* ------------------------------------------------------------------ *
* Boot-policy FATAL decision (ha-commit-durability task 3.3, Req 2.4).
* Boot-policy loud-fence decision (ha-commit-durability task 3.3, Req 2.4;
* never-self-exit rework 2026-06-18).
*
* Implements the FATAL terminal of the Boot Policy Decision Table
* (design.md). The decision is a PURE function of its inputs; the FATAL
* action (loud log + terminate) is separated into
* persist_recover_fatal_if_unrecoverable so the decision can be unit-tested
* without aborting. A reachable healthy peer NEVER FATALs (rows 3, 5);
* FATAL is reserved for a member that cannot recover locally AND cannot catch
* up within the bounded window (rows 4, 4b) — never a silent indefinite fence.
* Implements the FENCE_UNRECOVERABLE terminal of the Boot Policy Decision Table
* (design.md). The decision is a PURE function of its inputs; the action (loud
* operator-visible log + set the sticky non-promotable flag) is separated into
* persist_recover_loud_fence_if_unrecoverable so the decision can be unit-tested
* in isolation. A reachable healthy peer NEVER yields FENCE_UNRECOVERABLE
* (rows 3, 5); it is reserved for a member that cannot recover locally AND
* cannot catch up (no healthy peer, rows 4, 4b).
*
* CRITICAL: this terminal is a LOUD, PERMANENT, NON-promotable fence — the
* process STAYS ALIVE and keeps retrying catch-up. There is NO abort()/exit():
* a controller MUST NEVER self-terminate as a designed response to an
* operational condition (user invariant, 2026-06-18). A self-exited controller
* needs manual admin recovery and degrades quorum meanwhile; one that stays
* alive and keeps retrying self-heals when a peer returns (Req 2.6).
* ------------------------------------------------------------------ */
/* FATAL seam: NULL in production (real abort); a test may install a hook to
* capture the FATAL path without dying. Mirrors ha_fatal_hook. */
void (*persist_fatal_hook)(const char* msg) = NULL;
recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in)
{
if (in == NULL)
@ -1723,7 +1732,7 @@ recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in)
bool truncated = (in->verdict == RECOVER_TRUNCATED);
/* Rows 1 & 6: a structurally CLEAN log that is NOT behind the cluster
* high-water mark is a normal promotable member — no fence, no FATAL. */
* high-water mark is a normal promotable member — no fence. */
if (!truncated && !in->durable_below_hwm)
return RECOVER_DECISION_ELIGIBLE;
@ -1732,63 +1741,83 @@ recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in)
* (design.md note), so it is treated as behind for fence purposes. */
/* Rows 2, 3, 5: a healthy peer is reachable. Catch up from it and regain
* eligibility (task 4.3). A reachable peer NEVER yields FATAL — even for a
* truncated log or a legitimate snapshot-sentinel. If catch-up has already
* succeeded the member is eligible; otherwise it stays fenced (4.3 drives
* the catch-up + release; this task only refuses FATAL while a peer
* exists). */
* eligibility (task 4.3). A reachable peer NEVER yields the unrecoverable
* fence — even for a truncated log or a legitimate snapshot-sentinel. If
* catch-up has already succeeded the member is eligible; otherwise it stays
* fenced (4.3 drives the catch-up + release). */
if (in->healthy_peer_reachable) {
if (in->catch_up_succeeded)
return RECOVER_DECISION_ELIGIBLE;
return RECOVER_DECISION_FENCE;
}
/* No healthy peer. The bounded catch-up window is the member's only path to
* a committed-consistent state; if it did not reach one, the member can
* neither recover locally nor catch up — rows 4 & 4b: FATAL (fail loud),
* never a silent indefinite fence (Req 2.4). */
/* No healthy peer. If catch-up did not reach a committed-consistent state,
* the member can neither recover locally nor catch up right now — rows 4 &
* 4b: a LOUD, PERMANENT, NON-promotable fence (NOT process exit). It stays
* alive and keeps retrying; the fence releases when a peer returns
* (Req 2.4/2.6). */
if (!in->catch_up_succeeded)
return RECOVER_DECISION_FATAL;
return RECOVER_DECISION_FENCE_UNRECOVERABLE;
/* No peer but catch-up is reported succeeded (e.g. already at/above after a
* local catch-up): eligible. Defensive — the boot path reaches the
* no-peer branch with catch_up_succeeded=false in the bounded-failure case
* this task owns. */
* local catch-up): eligible. */
return RECOVER_DECISION_ELIGIBLE;
}
bool persist_recovery_unrecoverable(persist_t* p)
{
return p && atomic_load_explicit(&p->recovery_unrecoverable,
memory_order_acquire);
}
void persist_clear_recovery_unrecoverable(persist_t* p)
{
if (!p)
return;
/* Clear only on a genuine transition so the loud "fence released" log fires
* once, not on every catch-up tick. */
if (atomic_exchange_explicit(&p->recovery_unrecoverable, false,
memory_order_acq_rel)) {
INFO("persist_recover: unrecoverable fence RELEASED — member reached a "
"committed-consistent state via catch-up; promotable again "
"(ha-commit-durability Req 2.6)");
}
}
recover_decision_t
persist_recover_fatal_if_unrecoverable(const recover_policy_inputs_t* in)
persist_recover_loud_fence_if_unrecoverable(persist_t* p,
const recover_policy_inputs_t* in)
{
recover_decision_t decision = persist_recover_decision(in);
if (decision != RECOVER_DECISION_FATAL)
if (decision != RECOVER_DECISION_FENCE_UNRECOVERABLE)
return decision;
/* Fail loud: log the truncation boundary (durable_floor_lsn) and the
* cluster high-water mark so an operator sees exactly where the local log
* stops short of the cluster's committed state (Req 2.4). */
/* Loud, operator-visible alert: log the truncation boundary
* (durable_floor_lsn) and the cluster high-water mark so an operator sees
* exactly where the local log stops short of the cluster's committed state.
* The member STAYS ALIVE, stays non-promotable, and keeps retrying catch-up
* (Req 2.4) — it NEVER self-exits. */
const char* why = (in->verdict == RECOVER_TRUNCATED)
? "recovery-truncated log (structural corruption)"
: "log behind the cluster committed high-water mark";
ERROR("persist_recover: FATAL — %s and no healthy peer reachable to catch "
"up within the bounded window (%u ms); truncation boundary "
"durable_floor_lsn=%" PRIu64 " < cluster committed high-water "
"hwm=%" PRIu64 " — terminating rather than serving divergent state "
ERROR("persist_recover: UNRECOVERABLE — %s and no healthy peer reachable to "
"catch up; truncation boundary durable_floor_lsn=%" PRIu64
" < cluster committed high-water hwm=%" PRIu64 ". Staying ALIVE and "
"PERMANENTLY non-promotable (refusing to serve divergent state) and "
"RETRYING catch-up indefinitely — will self-heal when a healthy peer "
"returns; operator attention may be needed "
"(ha-commit-durability Req 2.4)",
why, (unsigned)DEFAULT_RECOVERY_CATCHUP_WINDOW_MS,
in->durable_floor_lsn, in->cluster_hwm_lsn);
why, in->durable_floor_lsn, in->cluster_hwm_lsn);
if (persist_fatal_hook != NULL) {
/* Test seam: capture the FATAL decision + boundary log without dying.
* Production leaves the hook NULL and aborts below. */
persist_fatal_hook("persist_recover: bounded-unrecoverable (FATAL)");
return RECOVER_DECISION_FATAL;
}
/* Set the sticky non-promotable flag so the promotion gate refuses
* leadership as belt-and-suspenders until catch-up clears it (Req 2.4/2.6).
* p may be NULL for pure-classification callers that only want the loud log. */
if (p != NULL)
atomic_store_explicit(&p->recovery_unrecoverable, true,
memory_order_release);
tlog_exit(); /* flush the async log buffer before dying */
abort(); /* NEVER returns in production */
return RECOVER_DECISION_FATAL; /* unreachable */
return RECOVER_DECISION_FENCE_UNRECOVERABLE;
}
/* Advance committed_lsn to `written_lsn` on a majority ACK — but ONLY if that

390
src/controller/testsuite/test_commit_durability.c
View file

@ -1587,19 +1587,36 @@ static void test_durable_floor_snapshot_sentinel_tip(void)
* failure. (Req 2.2, 2.4) *
* ------------------------------------------------------------------ */
/* task 3.3 — FATAL-path capture seam. The production FATAL terminates via
* abort(); a test installs persist_fatal_hook so the FATAL decision + boundary
* log can be asserted without the process dying. */
static int g_fatal_hook_calls = 0;
static char g_fatal_hook_msg[256];
static void dur_fatal_hook(const char* msg)
/* task 3.3 — unrecoverable loud-fence verification (never-self-exit rework
* 2026-06-18). The OLD behavior installed persist_fatal_hook to capture an
* abort()/exit() without dying. That self-exit terminal is GONE: an
* unrecoverable member STAYS ALIVE and is held permanently non-promotable.
* There is no hook to install — the tests assert the OBSERVABLE
* post-conditions instead:
* - the pure decision returns RECOVER_DECISION_FENCE_UNRECOVERABLE,
* - persist_recover_loud_fence_if_unrecoverable RETURNS (no abort — the test
* simply continuing past the call proves the process stayed alive),
* - the sticky persist_recovery_unrecoverable flag is SET (member is
* permanently non-promotable until catch-up),
* - and the flag CLEARS on catch-up (Req 2.6).
* dur_persist_for_flag builds a real persist_t the loud-fence helper can set
* the sticky flag on. */
static persist_t* dur_flag_persist(const char* tmpdir)
{
g_fatal_hook_calls++;
if (msg != NULL) {
strncpy(g_fatal_hook_msg, msg, sizeof(g_fatal_hook_msg) - 1);
g_fatal_hook_msg[sizeof(g_fatal_hook_msg) - 1] = '\0';
}
persist_t* p = calloc(1, sizeof(*p));
if (!p) return NULL;
persist_config_t cfg = persist_config_defaults();
cfg.wal_sync_ms = 5000;
persist_set_workdir(p, tmpdir, "ctrl-flag");
if (persist_init(p, &cfg, NULL) != 0) { free(p); return NULL; }
return p;
}
static void dur_free_flag_persist(persist_t* p)
{
if (!p) return;
persist_fini(p);
free(p);
}
/* task 3.3 — peer-reachable NEVER FATALs (the never-FATAL aspect of rows
@ -1613,10 +1630,13 @@ static void dur_fatal_hook(const char* msg)
* 2.4, design.md "never FATAL when a peer is reachable".) */
static void test_boot_policy_peer_reachable_never_fatal(void)
{
g_fatal_hook_calls = 0;
persist_fatal_hook = dur_fatal_hook;
char tmp[256];
strcpy(tmp, "/tmp/xce_dur_peerfence_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
persist_t* p = dur_flag_persist(tmp);
TEST_ASSERT_NOT_NULL(p);
/* Row 3: TRUNCATED + healthy peer (catch-up not yet done) -> FENCE, no FATAL. */
/* Row 3: TRUNCATED + healthy peer (catch-up not yet done) -> FENCE. */
recover_policy_inputs_t truncated_peer = {
.verdict = RECOVER_TRUNCATED,
.durable_below_hwm = true,
@ -1628,13 +1648,13 @@ static void test_boot_policy_peer_reachable_never_fatal(void)
TEST_ASSERT_EQUAL_INT_MESSAGE(
RECOVER_DECISION_FENCE, persist_recover_decision(&truncated_peer),
"row 3: a truncated member with a reachable healthy peer must FENCE "
"(catch up), NEVER FATAL");
"(catch up), NEVER the unrecoverable terminal");
TEST_ASSERT_EQUAL_INT(
RECOVER_DECISION_FENCE,
persist_recover_fatal_if_unrecoverable(&truncated_peer));
persist_recover_loud_fence_if_unrecoverable(p, &truncated_peer));
/* Row 5: snapshot-sentinel CLEAN-but-behind + healthy peer -> FENCE, no
* FATAL. A legitimate sentinel is CLEAN-but-behind, not corruption. */
/* Row 5: snapshot-sentinel CLEAN-but-behind + healthy peer -> FENCE.
* A legitimate sentinel is CLEAN-but-behind, not corruption. */
recover_policy_inputs_t sentinel_peer = {
.verdict = RECOVER_CLEAN,
.durable_below_hwm = true,
@ -1646,18 +1666,19 @@ static void test_boot_policy_peer_reachable_never_fatal(void)
TEST_ASSERT_EQUAL_INT_MESSAGE(
RECOVER_DECISION_FENCE, persist_recover_decision(&sentinel_peer),
"row 5: a snapshot-sentinel CLEAN-but-behind member with a reachable "
"peer must FENCE, NEVER FATAL");
"peer must FENCE, NEVER the unrecoverable terminal");
TEST_ASSERT_EQUAL_INT(
RECOVER_DECISION_FENCE,
persist_recover_fatal_if_unrecoverable(&sentinel_peer));
persist_recover_loud_fence_if_unrecoverable(p, &sentinel_peer));
/* The FATAL terminate path must NOT have fired for either peer-reachable
* case — a reachable healthy peer never triggers FATAL. */
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, g_fatal_hook_calls,
"peer-reachable rows must NEVER fire the FATAL terminate path");
/* The unrecoverable sticky flag must NOT have been set for either
* peer-reachable case — a reachable healthy peer never reaches the
* unrecoverable terminal. */
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(p),
"peer-reachable rows must NEVER set the unrecoverable sticky flag");
persist_fatal_hook = NULL;
dur_free_flag_persist(p);
printf("test_boot_policy_peer_reachable_never_fatal: PASS\n");
}
@ -1763,9 +1784,6 @@ static void test_boot_policy_row3_truncated_peer_catchup_eligible(void)
ha_config_t cfg = dur_ha_cfg();
TEST_ASSERT_EQUAL_INT(0, ha_init(&ha, &cfg, 0));
g_fatal_hook_calls = 0;
persist_fatal_hook = dur_fatal_hook;
/* Truncated boundary at 3, cluster HWM 10, peer reachable, not caught up. */
recover_decision_t d1 = ha_evaluate_promotion_fence(
&ha, /*truncated=*/true, /*durable_floor=*/3, /*safe_hwm=*/10,
@ -1773,13 +1791,10 @@ static void test_boot_policy_row3_truncated_peer_catchup_eligible(void)
TEST_ASSERT_EQUAL_INT_MESSAGE(
RECOVER_DECISION_FENCE, d1,
"row 3: a recovery-truncated member with a healthy peer must FENCE "
"(denied promotion), never FATAL");
"(denied promotion), never the unrecoverable terminal");
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, atomic_load_explicit(&ha.in_catch_up, memory_order_acquire),
"row 3: a truncated member must be fenced (in_catch_up set)");
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, g_fatal_hook_calls,
"row 3: FATAL must NOT fire while a healthy peer is reachable");
/* After catch-up to/above the committed boundary -> ELIGIBLE, fence released. */
recover_decision_t d2 = ha_evaluate_promotion_fence(
@ -1791,28 +1806,29 @@ static void test_boot_policy_row3_truncated_peer_catchup_eligible(void)
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, atomic_load_explicit(&ha.in_catch_up, memory_order_acquire),
"row 3: regained eligibility releases the fence (Req 2.6)");
TEST_ASSERT_EQUAL_INT(0, g_fatal_hook_calls);
persist_fatal_hook = NULL;
ha_fini(&ha);
printf("test_boot_policy_row3_truncated_peer_catchup_eligible: PASS\n");
}
/* Row 4: TRUNCATED + below HWM + NO healthy peer + catch-up fails
* (bounded window elapses) -> FATAL.
* Cannot recover locally AND cannot catch up within the bounded window:
* fail loud rather than serve divergent state. WILL ASSERT (3.3): the
* member terminates loudly (FATAL) with the truncation boundary logged
* when the bounded catch-up window elapses without a committed-consistent
* state. (Req 2.4) */
static void test_boot_policy_row4_truncated_no_peer_bounded_fatal(void)
* -> LOUD PERMANENT FENCE (non-promotable, STAYS ALIVE, keeps retrying), and
* RELEASES on catch-up (Req 2.4, 2.6).
*
* never-self-exit rework (2026-06-18): the member cannot recover locally AND
* cannot catch up right now. It does NOT self-exit; it logs LOUDLY, sets the
* sticky non-promotable flag, and stays alive. The flag clears once a peer
* returns and it catches up. RED→GREEN: pre-rework
* persist_recover_fatal_if_unrecoverable aborted (or fired the abort-capture
* hook); post-rework persist_recover_loud_fence_if_unrecoverable RETURNS — the
* test continuing past the call IS the proof the process stayed alive. */
static void test_boot_policy_row4_truncated_no_peer_loud_fence(void)
{
/* Row 4: TRUNCATED + below HWM + NO healthy peer + bounded catch-up fails
* -> FATAL. Cannot recover locally AND cannot catch up within the bounded
* window: fail loud with the truncation boundary logged (Req 2.4). */
g_fatal_hook_calls = 0;
g_fatal_hook_msg[0] = '\0';
persist_fatal_hook = dur_fatal_hook;
char tmp[256];
strcpy(tmp, "/tmp/xce_dur_row4_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
persist_t* p = dur_flag_persist(tmp);
TEST_ASSERT_NOT_NULL(p);
recover_policy_inputs_t in = {
.verdict = RECOVER_TRUNCATED,
@ -1823,39 +1839,49 @@ static void test_boot_policy_row4_truncated_no_peer_bounded_fatal(void)
.cluster_hwm_lsn = 12, /* cluster committed high-water */
};
/* Pure decision returns FATAL. */
/* Pure decision returns the non-terminating unrecoverable fence. */
TEST_ASSERT_EQUAL_INT_MESSAGE(
RECOVER_DECISION_FATAL, persist_recover_decision(&in),
"row 4: truncated + below + no peer + bounded catch-up fails must "
"decide FATAL");
RECOVER_DECISION_FENCE_UNRECOVERABLE, persist_recover_decision(&in),
"row 4: truncated + below + no peer + no catch-up must decide the LOUD "
"PERMANENT FENCE — NOT process exit");
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(p),
"row 4: the sticky flag is set by the ACTION helper, not the pure "
"decision");
/* The action helper routes FATAL through the hook (no abort in test) and
* still returns FATAL — proving the loud-terminate path was taken. */
recover_decision_t d = persist_recover_fatal_if_unrecoverable(&in);
TEST_ASSERT_EQUAL_INT(RECOVER_DECISION_FATAL, d);
/* The action helper logs LOUDLY, sets the sticky flag, and RETURNS (no
* abort/exit — the process stays alive; the test reaching the next line is
* the proof). */
recover_decision_t d = persist_recover_loud_fence_if_unrecoverable(p, &in);
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, g_fatal_hook_calls,
"row 4: the FATAL terminate path must fire exactly once (the "
"truncation boundary is logged at ERROR just before it)");
RECOVER_DECISION_FENCE_UNRECOVERABLE, d,
"row 4: the loud-fence action returns FENCE_UNRECOVERABLE (stays alive)");
TEST_ASSERT_TRUE_MESSAGE(
persist_recovery_unrecoverable(p),
"row 4: the member is held PERMANENTLY non-promotable (sticky flag set)");
persist_fatal_hook = NULL;
printf("test_boot_policy_row4_truncated_no_peer_bounded_fatal: PASS\n");
/* Req 2.6: once catch-up reaches a committed-consistent state the fence
* releases automatically — the flag clears, the member is promotable. */
persist_clear_recovery_unrecoverable(p);
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(p),
"row 4: catch-up CLEARS the unrecoverable fence (Req 2.6)");
dur_free_flag_persist(p);
printf("test_boot_policy_row4_truncated_no_peer_loud_fence: PASS\n");
}
/* Row 4b: CLEAN + below HWM + NO healthy peer + catch-up fails (bounded
* window elapses) -> FATAL.
* Behind the HWM with no peer to advance from within the bounded window:
* same bounded-failure terminal as row 4 — never a silent indefinite
* fence. WILL ASSERT (3.3): a behind member with no reachable peer FATALs
* once the bounded window elapses. (Req 2.4, 2.2) */
static void test_boot_policy_row4b_clean_below_no_peer_bounded_fatal(void)
/* Row 4b: CLEAN + below HWM + NO healthy peer + catch-up fails
* -> LOUD PERMANENT FENCE (non-promotable, STAYS ALIVE, keeps retrying).
* Behind the HWM with no peer to advance from — same non-terminating loud-fence
* terminal as row 4. NEVER self-exits. (Req 2.4, 2.2, 2.6) */
static void test_boot_policy_row4b_clean_below_no_peer_loud_fence(void)
{
/* Row 4b: CLEAN + below HWM + NO healthy peer + bounded catch-up fails
* -> FATAL. Behind the HWM with no peer to advance from within the bounded
* window — same bounded-failure terminal as row 4 (Req 2.4, 2.2). */
g_fatal_hook_calls = 0;
g_fatal_hook_msg[0] = '\0';
persist_fatal_hook = dur_fatal_hook;
char tmp[256];
strcpy(tmp, "/tmp/xce_dur_row4b_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
persist_t* p = dur_flag_persist(tmp);
TEST_ASSERT_NOT_NULL(p);
recover_policy_inputs_t in = {
.verdict = RECOVER_CLEAN, /* structurally intact... */
@ -1867,19 +1893,24 @@ static void test_boot_policy_row4b_clean_below_no_peer_bounded_fatal(void)
};
TEST_ASSERT_EQUAL_INT_MESSAGE(
RECOVER_DECISION_FATAL, persist_recover_decision(&in),
"row 4b: clean-but-behind + no peer + bounded catch-up fails must "
"decide FATAL — never a silent indefinite fence");
RECOVER_DECISION_FENCE_UNRECOVERABLE, persist_recover_decision(&in),
"row 4b: clean-but-behind + no peer + no catch-up must decide the LOUD "
"PERMANENT FENCE — never process exit, never a silent fence");
recover_decision_t d = persist_recover_fatal_if_unrecoverable(&in);
TEST_ASSERT_EQUAL_INT(RECOVER_DECISION_FATAL, d);
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, g_fatal_hook_calls,
"row 4b: a behind member with no reachable peer must FATAL once the "
"bounded window elapses (boundary logged before terminate)");
recover_decision_t d = persist_recover_loud_fence_if_unrecoverable(p, &in);
TEST_ASSERT_EQUAL_INT(RECOVER_DECISION_FENCE_UNRECOVERABLE, d);
TEST_ASSERT_TRUE_MESSAGE(
persist_recovery_unrecoverable(p),
"row 4b: a behind member with no peer is held non-promotable + ALIVE "
"(sticky flag set; boundary logged loudly)");
persist_fatal_hook = NULL;
printf("test_boot_policy_row4b_clean_below_no_peer_bounded_fatal: PASS\n");
persist_clear_recovery_unrecoverable(p);
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(p),
"row 4b: catch-up CLEARS the unrecoverable fence (Req 2.6)");
dur_free_flag_persist(p);
printf("test_boot_policy_row4b_clean_below_no_peer_loud_fence: PASS\n");
}
/* Row 5: CLEAN snapshot-sentinel (committed at sentinel LSN, NO
@ -1901,9 +1932,6 @@ static void test_boot_policy_row5_snapshot_sentinel_catchup_never_fatal(void)
ha_config_t cfg = dur_ha_cfg();
TEST_ASSERT_EQUAL_INT(0, ha_init(&ha, &cfg, 0));
g_fatal_hook_calls = 0;
persist_fatal_hook = dur_fatal_hook;
/* Sentinel at LSN 5, cluster has advanced to HWM 12, peer reachable. */
recover_decision_t d1 = ha_evaluate_promotion_fence(
&ha, /*truncated=*/false, /*durable_floor=*/5, /*safe_hwm=*/12,
@ -1911,10 +1939,7 @@ static void test_boot_policy_row5_snapshot_sentinel_catchup_never_fatal(void)
TEST_ASSERT_EQUAL_INT_MESSAGE(
RECOVER_DECISION_FENCE, d1,
"row 5: a snapshot-sentinel CLEAN-but-behind member must FENCE, never "
"FATAL while a peer is reachable");
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, g_fatal_hook_calls,
"row 5: a legitimate sentinel must NEVER trigger FATAL");
"the unrecoverable terminal while a peer is reachable");
recover_decision_t d2 = ha_evaluate_promotion_fence(
&ha, /*truncated=*/false, /*durable_floor=*/12, /*safe_hwm=*/12,
@ -1922,9 +1947,7 @@ static void test_boot_policy_row5_snapshot_sentinel_catchup_never_fatal(void)
TEST_ASSERT_EQUAL_INT_MESSAGE(
RECOVER_DECISION_ELIGIBLE, d2,
"row 5: after catch-up the sentinel member is eligible (Req 2.6)");
TEST_ASSERT_EQUAL_INT(0, g_fatal_hook_calls);
persist_fatal_hook = NULL;
ha_fini(&ha);
printf("test_boot_policy_row5_snapshot_sentinel_catchup_never_fatal: PASS\n");
}
@ -1943,9 +1966,6 @@ static void test_boot_policy_row6_empty_first_boot_eligible(void)
ha_config_t cfg = dur_ha_cfg();
TEST_ASSERT_EQUAL_INT(0, ha_init(&ha, &cfg, 0));
g_fatal_hook_calls = 0;
persist_fatal_hook = dur_fatal_hook;
recover_decision_t d = ha_evaluate_promotion_fence(
&ha, /*truncated=*/false, /*durable_floor=*/0, /*safe_hwm=*/0,
/*peer_reachable=*/false, /*catch_up_succeeded=*/false);
@ -1955,11 +1975,7 @@ static void test_boot_policy_row6_empty_first_boot_eligible(void)
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, atomic_load_explicit(&ha.in_catch_up, memory_order_acquire),
"row 6: a cold-start member must not be fenced");
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, g_fatal_hook_calls,
"row 6: a cold-start member must never FATAL");
persist_fatal_hook = NULL;
ha_fini(&ha);
printf("test_boot_policy_row6_empty_first_boot_eligible: PASS\n");
}
@ -1968,13 +1984,14 @@ static void test_boot_policy_row6_empty_first_boot_eligible(void)
* A hollow node (recovery-truncated OR durable below the HWM) must be refused
* promotion regardless of quorum size — combined with the 4.2 freshness floor
* it can neither win the vote nor self-promote. At minimal quorum with NO
* healthy peer, a hollow node FATALs (bounded-unrecoverable) rather than
* silently leading; with a peer it FENCEs. Either way it never becomes
* ELIGIBLE while hollow. */
* healthy peer, a hollow node is held in the LOUD PERMANENT FENCE
* (FENCE_UNRECOVERABLE, stays alive, non-promotable) rather than silently
* leading; with a peer it FENCEs. Either way it never becomes ELIGIBLE while
* hollow. */
static void test_promotion_gate_hollow_refused_at_minimal_quorum(void)
{
/* Truncated + below HWM + no peer (minimal/degraded quorum) -> NEVER
* ELIGIBLE (FATAL — fail loud rather than promote a hollow node). */
* ELIGIBLE (loud permanent fence — never promote a hollow node). */
TEST_ASSERT_NOT_EQUAL_INT_MESSAGE(
RECOVER_DECISION_ELIGIBLE,
ha_promotion_gate_decision(/*truncated=*/true, /*durable_floor=*/3,
@ -1983,7 +2000,7 @@ static void test_promotion_gate_hollow_refused_at_minimal_quorum(void)
"a recovery-truncated node must never be ELIGIBLE at minimal quorum "
"(Req 2.5)");
/* CLEAN but below HWM + no peer -> NEVER ELIGIBLE (FATAL). */
/* CLEAN but below HWM + no peer -> NEVER ELIGIBLE (loud permanent fence). */
TEST_ASSERT_NOT_EQUAL_INT_MESSAGE(
RECOVER_DECISION_ELIGIBLE,
ha_promotion_gate_decision(/*truncated=*/false, /*durable_floor=*/5,
@ -2003,19 +2020,22 @@ static void test_promotion_gate_hollow_refused_at_minimal_quorum(void)
printf("test_promotion_gate_hollow_refused_at_minimal_quorum: PASS\n");
}
/* ── task 4.4: bounded-window catch-up before FATAL ──────────────────────────
/* ── task 4.4: catch-up re-alert window (never-self-exit rework 2026-06-18) ───
*
* The 4.3 gate decided FATAL synchronously at the promotion instant when a
* recovery-truncated/behind member had no reachable peer right then. Task 4.4
* defers that FATAL to a one-shot bounded-window timer: the member is fenced
* (in_catch_up) and given DEFAULT_RECOVERY_CATCHUP_WINDOW_MS to find a peer /
* make progress; the 3.3 FATAL is reached ONLY from the timer after a real
* bounded failure (still hollow + no peer + no progress).
* A recovery-truncated/behind member with no reachable peer at the promotion
* instant is FENCED (in_catch_up) and STAYS ALIVE, retrying catch-up. A
* delaymgr re-alert timer fires after DEFAULT_RECOVERY_CATCHUP_WINDOW_MS: if the
* member is still unrecoverable it RE-EMITS the loud operator alert (sets the
* sticky persist_recovery_unrecoverable flag) and the process STAYS ALIVE —
* there is NO abort/exit anywhere. Catch-up releases the fence + clears the
* flag (Req 2.6); shutdown/cancel stops the re-alert.
*
* These tests drive ha_arm_promotion_fence_timer (the production deferral wired
* into raft_cb_become_leader) through a REAL delaymgr with a short window seam,
* so the timer fires fast and deterministically. A minimal cluster_t carries
* the delaymgr + persist + ha exactly as production does. */
* the delaymgr + persist + ha exactly as production does. They assert the
* member STAYS ALIVE + non-promotable (sticky flag) instead of asserting an
* abort/hook — the self-exit terminal is GONE. */
/* The delaymgr runner used by production (run_delay_tasks in cluster.c) is not
* linked into the unit-test target; provide the same trivial pass-through. */
@ -2084,28 +2104,31 @@ static void dur_destroy_fence_cluster(cluster_t* c)
free(c);
}
/* Spin until the fatal hook fires or a timeout elapses (the delaymgr fires the
* one-shot timer on its own worker thread). */
static bool dur_wait_fatal(int expect_calls, int timeout_ms)
/* Spin until the sticky unrecoverable flag is set on c->persist or a timeout
* elapses (the delaymgr fires the re-alert timer on its own worker thread). */
static bool dur_wait_unrecoverable(cluster_t* c, int timeout_ms)
{
int waited = 0;
while (g_fatal_hook_calls < expect_calls && waited < timeout_ms) {
while (!persist_recovery_unrecoverable(&c->persist) && waited < timeout_ms) {
usleep(2000);
waited += 2;
}
return g_fatal_hook_calls >= expect_calls;
return persist_recovery_unrecoverable(&c->persist);
}
/* 4.4-RED carrier + core deferral: a recovery-truncated/behind member with NO
* reachable peer must NOT FATAL synchronously at the promotion instant — it
* fences (in_catch_up set, process alive) and arms a bounded-window timer; the
* 3.3 FATAL fires ONLY from that timer after the window with no peer/progress.
/* 4.4 (never-self-exit): a recovery-truncated/behind member with NO reachable
* peer must NOT self-exit — it fences (in_catch_up set, process ALIVE) and arms
* the re-alert timer; when the window elapses with no peer/progress the timer
* RE-EMITS the loud alert (sets the sticky non-promotable flag) and the process
* STAYS ALIVE.
*
* RED (pre-4.4): the gate routed a no-peer truncated member straight through
* ha_evaluate_promotion_fence → persist_recover_fatal_if_unrecoverable, so the
* fatal hook was called SYNCHRONOUSLY (count==1 with zero delay) — there was no
* bounded window, which is exactly the task-5.2 soak collapse. (Req 2.2, 2.4) */
static void test_promotion_fence_defers_fatal_to_bounded_timer(void)
* RED→GREEN: pre-rework the no-peer truncated member was routed (synchronously
* or via the timer) to persist_recover_fatal_if_unrecoverable, which abort()ed
* (the old fatal-capture hook fired) — that self-exit was the task-5.2 soak
* quorum collapse. Post-rework there is NO abort: the member stays alive,
* fenced, and loudly non-promotable. Reaching the end of this test (the
* process did not die) IS part of the proof. (Req 2.2, 2.4) */
static void test_promotion_fence_unrecoverable_stays_alive_loud(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_fence_defer_XXXXXX");
@ -2115,46 +2138,41 @@ static void test_promotion_fence_defers_fatal_to_bounded_timer(void)
/*durable_floor=*/4);
TEST_ASSERT_NOT_NULL(c);
g_fatal_hook_calls = 0;
persist_fatal_hook = dur_fatal_hook;
ha_set_promotion_fence_window_ms(40); /* short bounded window for the test */
ha_set_promotion_fence_window_ms(40); /* short re-alert window for the test */
/* Arm the fence exactly as raft_cb_become_leader does for a truncated,
* below-HWM, no-peer winner. This must NOT FATAL synchronously. */
* below-HWM, no-peer winner. This must NOT self-exit and must fence. */
extern void ha_arm_promotion_fence_timer_for_test(cluster_t*, bool, uint64_t,
uint64_t);
ha_arm_promotion_fence_timer_for_test(c, /*truncated=*/true,
/*durable_floor=*/4, /*hwm=*/10);
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, g_fatal_hook_calls,
"4.4: the promotion gate must NOT FATAL synchronously — FATAL is "
"deferred to the bounded-window timer (Req 2.2/2.4)");
1, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire),
"4.4: a fenced member is held in_catch_up (refused promotion), ALIVE");
/* Let the re-alert window elapse: still truncated + no peer + no progress →
* the timer RE-EMITS the loud alert and SETS the sticky non-promotable flag.
* The process STAYS ALIVE (no abort/exit anywhere on this path). */
TEST_ASSERT_TRUE_MESSAGE(
dur_wait_unrecoverable(c, 1000),
"4.4: the re-alert timer must set the sticky unrecoverable flag after "
"the window with no peer/progress — staying ALIVE, not exiting (Req 2.4)");
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire),
"4.4: a fenced member is held in_catch_up (refused promotion), alive");
"4.4: an unrecoverable member stays fenced (non-promotable) and ALIVE");
/* Now let the bounded window elapse: still truncated + no peer + no
* progress → the 3.3 FATAL fires from the timer (exactly once). */
TEST_ASSERT_TRUE_MESSAGE(
dur_wait_fatal(1, 1000),
"4.4: the deferred FATAL must fire from the timer after the bounded "
"window with no peer and no progress (Req 2.4)");
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, g_fatal_hook_calls,
"4.4: the deferred FATAL fires exactly once from the timer");
persist_fatal_hook = NULL;
ha_set_promotion_fence_window_ms(0); /* restore default */
dur_destroy_fence_cluster(c);
printf("test_promotion_fence_defers_fatal_to_bounded_timer: PASS\n");
printf("test_promotion_fence_unrecoverable_stays_alive_loud: PASS\n");
}
/* 4.4 Req 2.6: catch-up within the window → fence released, FATAL NOT called.
* Arm the fence, then simulate catch-up (clear truncated + advance durable_floor
* past the HWM) before the timer fires; when the timer fires it observes a
* caught-up member, releases in_catch_up, and does NOT FATAL. */
static void test_promotion_fence_released_on_catchup_no_fatal(void)
/* 4.4 Req 2.6: catch-up within the window → fence + sticky flag released, and
* the member NEVER self-exits. Arm the fence, then simulate catch-up (clear
* truncated + advance durable_floor past the HWM) before the timer fires; when
* the timer fires it observes a caught-up member, releases in_catch_up, clears
* the sticky unrecoverable flag, and does NOT set the unrecoverable flag. */
static void test_promotion_fence_released_on_catchup(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_fence_catchup_XXXXXX");
@ -2164,15 +2182,12 @@ static void test_promotion_fence_released_on_catchup_no_fatal(void)
/*durable_floor=*/4);
TEST_ASSERT_NOT_NULL(c);
g_fatal_hook_calls = 0;
persist_fatal_hook = dur_fatal_hook;
ha_set_promotion_fence_window_ms(120); /* generous so we can mutate first */
extern void ha_arm_promotion_fence_timer_for_test(cluster_t*, bool, uint64_t,
uint64_t);
ha_arm_promotion_fence_timer_for_test(c, /*truncated=*/true,
/*durable_floor=*/4, /*hwm=*/10);
TEST_ASSERT_EQUAL_INT(0, g_fatal_hook_calls);
TEST_ASSERT_EQUAL_INT(
1, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire));
@ -2180,26 +2195,26 @@ static void test_promotion_fence_released_on_catchup_no_fatal(void)
c->persist.recovery_verdict = RECOVER_CLEAN;
atomic_store_explicit(&c->persist.durable_lsn, 10, memory_order_release);
/* Wait long enough for the window to fire; assert it did NOT FATAL and the
* fence was released (Req 2.6). */
/* Wait long enough for the window to fire; assert the fence was released and
* the sticky flag is NOT set (Req 2.6) — the member self-healed, alive. */
usleep(300 * 1000);
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, g_fatal_hook_calls,
"4.4: a member that caught up within the window must NOT FATAL (Req 2.6)");
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire),
"4.4: catch-up within the window RELEASES the in_catch_up fence (Req 2.6)");
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(&c->persist),
"4.4: a member that caught up must NOT be left in the unrecoverable "
"fence (Req 2.6)");
persist_fatal_hook = NULL;
ha_set_promotion_fence_window_ms(0);
dur_destroy_fence_cluster(c);
printf("test_promotion_fence_released_on_catchup_no_fatal: PASS\n");
printf("test_promotion_fence_released_on_catchup: PASS\n");
}
/* 4.4: cancelling the timer (shutdown / recovery) before it fires → NO FATAL.
* Proves the deferred FATAL never fires after a clean cancel — a stopped member
* must not self-terminate, and there is no use-after-free of the timer ctx. */
static void test_promotion_fence_cancel_prevents_fatal(void)
/* 4.4: cancelling the timer (shutdown / recovery) before it fires → the
* re-alert never fires, no use-after-free of the timer ctx, and the process
* stays alive (it never self-exited even before the cancel). */
static void test_promotion_fence_cancel_stops_realert(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_fence_cancel_XXXXXX");
@ -2209,29 +2224,27 @@ static void test_promotion_fence_cancel_prevents_fatal(void)
/*durable_floor=*/4);
TEST_ASSERT_NOT_NULL(c);
g_fatal_hook_calls = 0;
persist_fatal_hook = dur_fatal_hook;
ha_set_promotion_fence_window_ms(60);
extern void ha_arm_promotion_fence_timer_for_test(cluster_t*, bool, uint64_t,
uint64_t);
ha_arm_promotion_fence_timer_for_test(c, /*truncated=*/true,
/*durable_floor=*/4, /*hwm=*/10);
TEST_ASSERT_EQUAL_INT(0, g_fatal_hook_calls);
/* Cancel before the window elapses. */
ha_cancel_promotion_fence_timer(&c->ha);
/* Wait past the window — the cancelled timer must be a no-op, NO FATAL. */
/* Wait past the window — the cancelled timer must be a no-op (no re-alert,
* no UAF). The member stayed alive throughout (no abort anywhere). */
usleep(200 * 1000);
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, g_fatal_hook_calls,
"4.4: a cancelled fence timer must NEVER FATAL (shutdown/recovery)");
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(&c->persist),
"4.4: a cancelled fence timer must not fire the re-alert "
"(shutdown/recovery)");
persist_fatal_hook = NULL;
ha_set_promotion_fence_window_ms(0);
dur_destroy_fence_cluster(c);
printf("test_promotion_fence_cancel_prevents_fatal: PASS\n");
printf("test_promotion_fence_cancel_stops_realert: PASS\n");
}
#endif /* XCE_TESTING */
@ -2272,25 +2285,26 @@ void test_commit_durability(void)
/* task 3.2 — Boot Policy Decision Table skeletons (IGNORED, pending
* 3.3/4.3). Registered so each row is reported as IGNORED, not failed;
* the suite stays GREEN. See design.md "Boot Policy Decision Table". */
/* task 3.3 — FATAL rows (4, 4b) now ACTIVE (PASS), plus the never-FATAL
* invariant for the peer-reachable rows (3, 5). The full FENCE->ELIGIBLE
* eligibility outcome of rows 1,2,3,5,6 stays IGNORED pending task 4.3. */
/* task 3.3 — unrecoverable rows (4, 4b) = LOUD PERMANENT FENCE (stays alive,
* non-promotable, clears on catch-up), plus the never-unrecoverable
* invariant for the peer-reachable rows (3, 5) and the FENCE->ELIGIBLE
* eligibility outcome of rows 1,2,3,5,6. (never-self-exit rework) */
RUN_TEST(test_boot_policy_peer_reachable_never_fatal);
RUN_TEST(test_boot_policy_row1_clean_at_or_above_eligible);
RUN_TEST(test_boot_policy_row2_clean_below_peer_catchup_eligible);
RUN_TEST(test_boot_policy_row3_truncated_peer_catchup_eligible);
RUN_TEST(test_boot_policy_row4_truncated_no_peer_bounded_fatal);
RUN_TEST(test_boot_policy_row4b_clean_below_no_peer_bounded_fatal);
RUN_TEST(test_boot_policy_row4_truncated_no_peer_loud_fence);
RUN_TEST(test_boot_policy_row4b_clean_below_no_peer_loud_fence);
RUN_TEST(test_boot_policy_row5_snapshot_sentinel_catchup_never_fatal);
RUN_TEST(test_boot_policy_row6_empty_first_boot_eligible);
/* task 4.3 — hollow node refused even at minimal quorum (Req 2.5). */
RUN_TEST(test_promotion_gate_hollow_refused_at_minimal_quorum);
/* task 4.4 — bounded-window catch-up before FATAL (no synchronous FATAL;
* deferred-timer FATAL only after a real bounded failure; fence released on
* catch-up; cancelled timer never FATALs). (Req 2.2, 2.4, 2.6) */
RUN_TEST(test_promotion_fence_defers_fatal_to_bounded_timer);
RUN_TEST(test_promotion_fence_released_on_catchup_no_fatal);
RUN_TEST(test_promotion_fence_cancel_prevents_fatal);
/* task 4.4 — catch-up re-alert window (NEVER self-exits; stays alive +
* loudly non-promotable on the unrecoverable path; fence + sticky flag
* released on catch-up; cancelled timer stops the re-alert). (Req 2.2/2.4/2.6) */
RUN_TEST(test_promotion_fence_unrecoverable_stays_alive_loud);
RUN_TEST(test_promotion_fence_released_on_catchup);
RUN_TEST(test_promotion_fence_cancel_stops_realert);
#else
printf("test_commit_durability: skipped (no XCE_TESTING)\n");
#endif

125
src/include/persist.h
View file

@ -46,19 +46,29 @@ typedef enum {
* ELIGIBLE — may promote (no fence). Rows 1, 6.
* FENCE — ineligible to promote; catch up from a healthy peer, regain
* eligibility on success (Req 2.6). Rows 2, 3, 5. A reachable
* healthy peer NEVER yields FATAL.
* FATAL — fail loud: cannot recover locally AND cannot catch up within
* the bounded window. Terminate with the truncation/boundary
* logged rather than serve divergent state (Req 2.4). Rows 4, 4b.
* healthy peer NEVER yields FENCE_UNRECOVERABLE.
* FENCE_UNRECOVERABLE — loud, PERMANENT, NON-terminating fence: the member
* cannot recover locally AND cannot catch up (no healthy peer).
* It logs LOUDLY, stays PERMANENTLY non-promotable, and keeps
* retrying catch-up indefinitely — but the process STAYS ALIVE.
* Rows 4, 4b. This is NOT process exit: a controller MUST NEVER
* self-terminate as a designed response to an operational
* condition (user invariant, 2026-06-18). A self-exited controller
* needs MANUAL admin recovery and leaves quorum degraded
* meanwhile; a controller that stays alive and keeps retrying
* SELF-HEALS when a peer returns. The fence RELEASES (→ ELIGIBLE,
* Req 2.6) automatically when catch-up reaches a
* committed-consistent state.
*
* AUTHORITATIVE INVARIANT (design.md): the fence is never a silent indefinite
* wait — a truncated/behind member resolves to exactly one of FENCE→ELIGIBLE
* or a BOUNDED FATAL. There is no "fenced forever, silently divergent" state.
* AUTHORITATIVE INVARIANT (design.md): a truncated/behind member resolves to
* either FENCE→ELIGIBLE (caught up) or a loud PERMANENT FENCE_UNRECOVERABLE
* (stays alive, non-promotable, keeps retrying). It is NEVER silently divergent
* (it refuses to lead) and the process NEVER self-exits.
* ============================================================ */
typedef enum {
RECOVER_DECISION_ELIGIBLE = 0,
RECOVER_DECISION_FENCE = 1,
RECOVER_DECISION_FATAL = 2,
RECOVER_DECISION_ELIGIBLE = 0,
RECOVER_DECISION_FENCE = 1,
RECOVER_DECISION_FENCE_UNRECOVERABLE = 2,
} recover_decision_t;
/* Policy inputs evaluated at boot / promotion-eligibility (design.md table).
@ -177,6 +187,22 @@ struct persist_s {
* before the node participates; read by the promotion gate thereafter. */
recover_verdict_t recovery_verdict;
/* Sticky "unrecoverable, loudly fenced" flag (ha-commit-durability Req 2.4,
* NEVER-SELF-EXIT rework 2026-06-18). Set by
* persist_recover_loud_fence_if_unrecoverable when a member is
* recovery-truncated AND/OR durable below the cluster committed high-water
* mark with NO successful catch-up: the member stays ALIVE but is held
* PERMANENTLY non-promotable and keeps retrying catch-up. A controller MUST
* NEVER self-exit as a designed response to this operational condition; it
* self-heals automatically when a healthy peer returns and catch-up reaches
* a committed-consistent state — at which point this flag is CLEARED via
* persist_clear_recovery_unrecoverable (wired into the fence-release /
* catch-up path). Read by the promotion gate as belt-and-suspenders
* (a member must never lead while this is set). Atomic: set on the recovery
* path, cleared on the catch-up path, read by the promotion/heartbeat
* threads. Defaults to false before recovery runs. */
_Atomic bool recovery_unrecoverable;
/* Paths (derived from workdir + name via persist_set_workdir) */
char wal_path[4096];
char snap_path[4096];
@ -347,53 +373,64 @@ uint64_t persist_durable_floor(persist_t* p);
recover_verdict_t persist_recovery_verdict(persist_t* p);
bool persist_recovery_truncated(persist_t* p);
/* Bounded catch-up window (ha-commit-durability task 3.3, Req 2.4).
* The FINITE deadline a recovery-truncated / behind member is given to reach a
* committed-consistent state from a healthy peer before the boot policy FATALs.
* The fence must NEVER be an indefinite wait (design.md authoritative
* invariant), so the window is always finite. Default 30s — generous enough for
* a healthy-peer catch-up stream yet bounded; the verification mode overrides it
* via the inputs/timeout the caller (task 4.3 boot path) supplies. */
/* Bounded catch-up RE-ALERT window (ha-commit-durability task 3.3, Req 2.4;
* never-self-exit rework 2026-06-18).
* The interval after which a still-unrecoverable, loudly-fenced member RE-EMITS
* its operator alert (it does NOT terminate — see RECOVER_DECISION_FENCE_-
* UNRECOVERABLE). The member stays alive and keeps retrying catch-up
* indefinitely; this window only controls how often the loud operator-visible
* alert is repeated. Default 30s. */
#define DEFAULT_RECOVERY_CATCHUP_WINDOW_MS 30000u
/* persist_recover_decision (ha-commit-durability task 3.3, Req 2.4).
*
* Pure decision over the Boot Policy Decision Table: maps the policy inputs to
* RECOVER_DECISION_{ELIGIBLE,FENCE,FATAL}. No side effects, no termination — it
* only classifies, so it is unit-testable in isolation. The FATAL terminal is
* acted on by persist_recover_fatal_if_unrecoverable (which logs + terminates).
* RECOVER_DECISION_{ELIGIBLE,FENCE,FENCE_UNRECOVERABLE}. No side effects, no
* termination — it only classifies, so it is unit-testable in isolation. The
* FENCE_UNRECOVERABLE terminal is acted on by
* persist_recover_loud_fence_if_unrecoverable (which logs LOUDLY + sets the
* sticky non-promotable flag — it NEVER terminates the process).
*
* Contract (design.md rows): a reachable healthy peer NEVER returns FATAL
* (rows 3, 5); FATAL is returned ONLY when the member is truncated or behind
* the HWM, has NO healthy peer, and bounded catch-up did not reach a
* committed-consistent state (rows 4, 4b). */
* Contract (design.md rows): a reachable healthy peer NEVER returns
* FENCE_UNRECOVERABLE (rows 3, 5); it is returned ONLY when the member is
* truncated or behind the HWM, has NO healthy peer, and catch-up did not reach
* a committed-consistent state (rows 4, 4b). */
recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in);
/* persist_recover_fatal_if_unrecoverable (ha-commit-durability task 3.3,
* Req 2.4).
/* persist_recover_loud_fence_if_unrecoverable (ha-commit-durability task 3.3,
* Req 2.4; never-self-exit rework 2026-06-18).
*
* Evaluate persist_recover_decision over `in`; if the outcome is FATAL, log at
* ERROR LOUDLY — INCLUDING the truncation boundary (durable_floor_lsn) and the
* cluster high-water mark — and then TERMINATE (fail loud rather than serve
* divergent state). Returns the decision so a non-FATAL caller can fence/proceed
* (FENCE/ELIGIBLE are acted on by task 4.3).
* Evaluate persist_recover_decision over `in`; if the outcome is
* FENCE_UNRECOVERABLE, log LOUDLY at ERROR — INCLUDING the truncation boundary
* (durable_floor_lsn) and the cluster high-water mark — stating the member is
* staying fenced + non-promotable and will keep retrying catch-up, then SET the
* sticky `recovery_unrecoverable` flag on `p` (so the promotion gate refuses
* leadership as belt-and-suspenders). Returns the decision.
*
* Termination routes through persist_fatal_hook: in production the hook is NULL
* and the function flushes logs and abort()s (NEVER returns on FATAL); a test
* build may install a hook to capture the FATAL decision + boundary log without
* the process dying (mirrors the ha_fatal_hook seam in ha_convergence.c). */
* CRITICAL — the process STAYS ALIVE. There is NO abort()/exit()/tlog_exit()
* here: a controller MUST NEVER self-terminate as a designed response to an
* operational condition (user invariant, 2026-06-18). The member self-heals —
* the fence releases (→ ELIGIBLE, Req 2.6) and the flag clears via
* persist_clear_recovery_unrecoverable — automatically when a healthy peer
* returns and catch-up reaches a committed-consistent state.
*
* `p` may be NULL (pure-decision callers that only want the classification +
* loud log without touching a persist_t); the flag set is then skipped. */
recover_decision_t
persist_recover_fatal_if_unrecoverable(const recover_policy_inputs_t* in);
persist_recover_loud_fence_if_unrecoverable(persist_t* p,
const recover_policy_inputs_t* in);
/* persist_fatal_hook: test-only seam for the FATAL (abort) contract of
* persist_recover_fatal_if_unrecoverable (ha-commit-durability task 3.3,
* Req 2.4). Mirrors ha_fatal_hook (ha_convergence.c).
/* persist_recovery_unrecoverable / persist_clear_recovery_unrecoverable
* (ha-commit-durability Req 2.4/2.6, never-self-exit rework 2026-06-18).
*
* When non-NULL, the FATAL path invokes this INSTEAD OF abort(), so a test
* binary can assert the FATAL decision was taken and the truncation boundary
* was logged without the process dying. Production leaves it NULL and actually
* terminates. Signature: void hook(const char *msg). */
extern void (*persist_fatal_hook)(const char* msg);
* Accessor + clear for the sticky "unrecoverable, loudly fenced" flag. The flag
* is SET by persist_recover_loud_fence_if_unrecoverable and CLEARED on the
* catch-up / fence-release path once the member reaches a committed-consistent
* state (not truncated AND durable_floor >= HWM). While set, the member is held
* permanently non-promotable but stays alive. Safe on a NULL p (returns false /
* no-op). */
bool persist_recovery_unrecoverable(persist_t* p);
void persist_clear_recovery_unrecoverable(persist_t* p);
/* Register a durable-advance wake callback (ha-commit-durability task 2.2).
* The group-commit flusher calls fn(ctx) after it advances durable_lsn and