lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha-commit-durability)!: controller never self-terminates — loud non-promotable fence replaces recovery FATAL/abort

Browse source 
Per design invariant (a controller exits only on crash or admin kill), replace the
self-abort recovery terminal (tasks 3.3/4.3/4.4) with a loud, permanent, non-promotable
fence that stays alive and keeps retrying catch-up (self-heals on peer return; no manual
admin recovery). Records the never-self-terminate invariant as project-wide guidance in
src/controller/CLAUDE.md + .kiro/steering/ha-async-io.md.
This commit is contained in:
Rongsong Shen 2026-06-18 11:10:19 +00:00
parent addf53e3b4
commit 59d9df246c
10 changed files with 607 additions and 406 deletions
Download patch file Download diff file Expand all files Collapse all files

121
src/controller/persist.c
View file

@ -449,6 +449,11 @@ int persist_init(persist_t* p, persist_config_t* cfg, cluster_t* c)
* replay outcome (RECOVER_TRUNCATED only on a structural log corruption). */
p->recovery_verdict = RECOVER_CLEAN;
/* Sticky "unrecoverable, loudly fenced" flag starts clear (Req 2.4): a fresh
* store is promotable. Set by persist_recover_loud_fence_if_unrecoverable
* on an unrecoverable boot; cleared on catch-up (never self-exits). */
atomic_store(&p->recovery_unrecoverable, false);
/* Durable-advance wake hook starts unregistered (ha-commit-durability 2.2):
* the commit engine registers it in commit_engine_init. Until then every
* flusher durable advance is a no-op wake. */
@ -1700,21 +1705,25 @@ bool persist_recovery_truncated(persist_t* p)
}
/* ------------------------------------------------------------------ *
* Boot-policy FATAL decision (ha-commit-durability task 3.3, Req 2.4).
* Boot-policy loud-fence decision (ha-commit-durability task 3.3, Req 2.4;
* never-self-exit rework 2026-06-18).
*
* Implements the FATAL terminal of the Boot Policy Decision Table
* (design.md). The decision is a PURE function of its inputs; the FATAL
* action (loud log + terminate) is separated into
* persist_recover_fatal_if_unrecoverable so the decision can be unit-tested
* without aborting. A reachable healthy peer NEVER FATALs (rows 3, 5);
* FATAL is reserved for a member that cannot recover locally AND cannot catch
* up within the bounded window (rows 4, 4b) — never a silent indefinite fence.
* Implements the FENCE_UNRECOVERABLE terminal of the Boot Policy Decision Table
* (design.md). The decision is a PURE function of its inputs; the action (loud
* operator-visible log + set the sticky non-promotable flag) is separated into
* persist_recover_loud_fence_if_unrecoverable so the decision can be unit-tested
* in isolation. A reachable healthy peer NEVER yields FENCE_UNRECOVERABLE
* (rows 3, 5); it is reserved for a member that cannot recover locally AND
* cannot catch up (no healthy peer, rows 4, 4b).
*
* CRITICAL: this terminal is a LOUD, PERMANENT, NON-promotable fence — the
* process STAYS ALIVE and keeps retrying catch-up. There is NO abort()/exit():
* a controller MUST NEVER self-terminate as a designed response to an
* operational condition (user invariant, 2026-06-18). A self-exited controller
* needs manual admin recovery and degrades quorum meanwhile; one that stays
* alive and keeps retrying self-heals when a peer returns (Req 2.6).
* ------------------------------------------------------------------ */
/* FATAL seam: NULL in production (real abort); a test may install a hook to
* capture the FATAL path without dying. Mirrors ha_fatal_hook. */
void (*persist_fatal_hook)(const char* msg) = NULL;
recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in)
{
if (in == NULL)
@ -1723,7 +1732,7 @@ recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in)
bool truncated = (in->verdict == RECOVER_TRUNCATED);
/* Rows 1 & 6: a structurally CLEAN log that is NOT behind the cluster
* high-water mark is a normal promotable member — no fence, no FATAL. */
* high-water mark is a normal promotable member — no fence. */
if (!truncated && !in->durable_below_hwm)
return RECOVER_DECISION_ELIGIBLE;
@ -1732,63 +1741,83 @@ recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in)
* (design.md note), so it is treated as behind for fence purposes. */
/* Rows 2, 3, 5: a healthy peer is reachable. Catch up from it and regain
* eligibility (task 4.3). A reachable peer NEVER yields FATAL — even for a
* truncated log or a legitimate snapshot-sentinel. If catch-up has already
* succeeded the member is eligible; otherwise it stays fenced (4.3 drives
* the catch-up + release; this task only refuses FATAL while a peer
* exists). */
* eligibility (task 4.3). A reachable peer NEVER yields the unrecoverable
* fence — even for a truncated log or a legitimate snapshot-sentinel. If
* catch-up has already succeeded the member is eligible; otherwise it stays
* fenced (4.3 drives the catch-up + release). */
if (in->healthy_peer_reachable) {
if (in->catch_up_succeeded)
return RECOVER_DECISION_ELIGIBLE;
return RECOVER_DECISION_FENCE;
}
/* No healthy peer. The bounded catch-up window is the member's only path to
* a committed-consistent state; if it did not reach one, the member can
* neither recover locally nor catch up — rows 4 & 4b: FATAL (fail loud),
* never a silent indefinite fence (Req 2.4). */
/* No healthy peer. If catch-up did not reach a committed-consistent state,
* the member can neither recover locally nor catch up right now — rows 4 &
* 4b: a LOUD, PERMANENT, NON-promotable fence (NOT process exit). It stays
* alive and keeps retrying; the fence releases when a peer returns
* (Req 2.4/2.6). */
if (!in->catch_up_succeeded)
return RECOVER_DECISION_FATAL;
return RECOVER_DECISION_FENCE_UNRECOVERABLE;
/* No peer but catch-up is reported succeeded (e.g. already at/above after a
* local catch-up): eligible. Defensive — the boot path reaches the
* no-peer branch with catch_up_succeeded=false in the bounded-failure case
* this task owns. */
* local catch-up): eligible. */
return RECOVER_DECISION_ELIGIBLE;
}
bool persist_recovery_unrecoverable(persist_t* p)
{
return p && atomic_load_explicit(&p->recovery_unrecoverable,
memory_order_acquire);
}
void persist_clear_recovery_unrecoverable(persist_t* p)
{
if (!p)
return;
/* Clear only on a genuine transition so the loud "fence released" log fires
* once, not on every catch-up tick. */
if (atomic_exchange_explicit(&p->recovery_unrecoverable, false,
memory_order_acq_rel)) {
INFO("persist_recover: unrecoverable fence RELEASED — member reached a "
"committed-consistent state via catch-up; promotable again "
"(ha-commit-durability Req 2.6)");
}
}
recover_decision_t
persist_recover_fatal_if_unrecoverable(const recover_policy_inputs_t* in)
persist_recover_loud_fence_if_unrecoverable(persist_t* p,
const recover_policy_inputs_t* in)
{
recover_decision_t decision = persist_recover_decision(in);
if (decision != RECOVER_DECISION_FATAL)
if (decision != RECOVER_DECISION_FENCE_UNRECOVERABLE)
return decision;
/* Fail loud: log the truncation boundary (durable_floor_lsn) and the
* cluster high-water mark so an operator sees exactly where the local log
* stops short of the cluster's committed state (Req 2.4). */
/* Loud, operator-visible alert: log the truncation boundary
* (durable_floor_lsn) and the cluster high-water mark so an operator sees
* exactly where the local log stops short of the cluster's committed state.
* The member STAYS ALIVE, stays non-promotable, and keeps retrying catch-up
* (Req 2.4) — it NEVER self-exits. */
const char* why = (in->verdict == RECOVER_TRUNCATED)
? "recovery-truncated log (structural corruption)"
: "log behind the cluster committed high-water mark";
ERROR("persist_recover: FATAL — %s and no healthy peer reachable to catch "
"up within the bounded window (%u ms); truncation boundary "
"durable_floor_lsn=%" PRIu64 " < cluster committed high-water "
"hwm=%" PRIu64 " — terminating rather than serving divergent state "
ERROR("persist_recover: UNRECOVERABLE — %s and no healthy peer reachable to "
"catch up; truncation boundary durable_floor_lsn=%" PRIu64
" < cluster committed high-water hwm=%" PRIu64 ". Staying ALIVE and "
"PERMANENTLY non-promotable (refusing to serve divergent state) and "
"RETRYING catch-up indefinitely — will self-heal when a healthy peer "
"returns; operator attention may be needed "
"(ha-commit-durability Req 2.4)",
why, (unsigned)DEFAULT_RECOVERY_CATCHUP_WINDOW_MS,
in->durable_floor_lsn, in->cluster_hwm_lsn);
why, in->durable_floor_lsn, in->cluster_hwm_lsn);
if (persist_fatal_hook != NULL) {
/* Test seam: capture the FATAL decision + boundary log without dying.
* Production leaves the hook NULL and aborts below. */
persist_fatal_hook("persist_recover: bounded-unrecoverable (FATAL)");
return RECOVER_DECISION_FATAL;
}
/* Set the sticky non-promotable flag so the promotion gate refuses
* leadership as belt-and-suspenders until catch-up clears it (Req 2.4/2.6).
* p may be NULL for pure-classification callers that only want the loud log. */
if (p != NULL)
atomic_store_explicit(&p->recovery_unrecoverable, true,
memory_order_release);
tlog_exit(); /* flush the async log buffer before dying */
abort(); /* NEVER returns in production */
return RECOVER_DECISION_FATAL; /* unreachable */
return RECOVER_DECISION_FENCE_UNRECOVERABLE;
}
/* Advance committed_lsn to `written_lsn` on a majority ACK — but ONLY if that