From e3fc42c11ee2e8580e60bb1f6c1b19060aa2559e Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Thu, 18 Jun 2026 08:15:22 +0000 Subject: [PATCH] feat(ha-commit-durability): loud bounded FATAL on unrecoverable recovery (task 3.3) --- .kiro/specs/ha-commit-durability/tasks.md | 3 +- src/controller/persist.c | 92 +++++++++++ .../testsuite/test_commit_durability.c | 149 +++++++++++++++++- src/include/persist.h | 89 +++++++++++ 4 files changed, 324 insertions(+), 9 deletions(-) diff --git a/.kiro/specs/ha-commit-durability/tasks.md b/.kiro/specs/ha-commit-durability/tasks.md index ed13c569..e87b9027 100644 --- a/.kiro/specs/ha-commit-durability/tasks.md +++ b/.kiro/specs/ha-commit-durability/tasks.md @@ -63,7 +63,7 @@ - _Requirements: 2.2, 2.4_ - _Boundary: RecoveryIntegrity_ -- [ ] 3.3 Implement loud FATAL on bounded-unrecoverable recovery +- [x] 3.3 Implement loud FATAL on bounded-unrecoverable recovery - Implement the FATAL branch of the 3.2 policy: when a recovery-truncated member cannot reach a committed-consistent state within the bounded catch-up window, terminate loudly with the truncation boundary logged; a reachable healthy peer never triggers FATAL. - Observable: the 3.2 decision-table rows for the FATAL/no-FATAL cases pass; logs show the truncation boundary on FATAL. - _Requirements: 2.4_ @@ -129,3 +129,4 @@ - 2.4: SPEC INACCURACY corrected — `persist_wal_sync_append` is NOT dormant; it is the LIVE job-submit path (`persist_job_hook_fun` calls it at persist.c:1911). Took the "route through durable-majority eval" option (delete infeasible). Production path (ce!=NULL, lsn!=0) was already durable-safe via 2.2's `majority_candidate_locked` self-cap. Closed the hazard in the LEGACY branch: the former unconditional `atomic_store(committed_lsn, written_lsn)` on majority ACK is now `sync_append_legacy_commit_store()` which advances ONLY when `written_lsn <= persist_durable_lsn(p)` (LSN-0 edge: store of 0 valid). Added the design's debug invariant `committed_lsn <= durable_lsn` (`#ifdef XCE_TESTING` assert at commit.c:162 + inside the legacy store) as the future-proof guard. Fixed the stale "after local fsync via persist_wal_async" comment (2.1 deferred the fsync). CODE-SEARCH (authoritative): the ONLY `committed_lsn` stores are — engine advance (commit.c, durable-capped by 2.2), legacy sync_append (now gated), and the two `persist_truncate_wal_above` cutoff stores (NOT ACK-driven; each immediately `clamp_durable_lsn_locked`). ha.c/ha_convergence.c stores are `last_known_committed_lsn` (DIFFERENT field). So no path advances commit on ACK without the durable precondition. Seam `persist_test_sync_append_legacy_commit_store` is production-excluded. - 3.1: `recover_verdict_t {RECOVER_CLEAN, RECOVER_TRUNCATED}` (persist.h) + `persist_t.recovery_verdict` field + accessors `persist_recovery_verdict()`/`persist_recovery_truncated()` (NOT XCE_TESTING-gated — 4.3 consumes them in production). `replay_wal` classifies: CRC chain break / CRC mismatch / short header/payload read → TRUNCATED; clean entry boundary / pre-allocated-zero tail / empty WAL / snapshot-sentinel+clean-replay → CLEAN. Verdict is monotonic (init CLEAN once; only ever set TRUNCATED; a later clean segment can't un-set it). Verdict = STRUCTURAL integrity, NOT "behind the cluster" (the behind-high-water case is 4.2/4.3). 3.1 ONLY sets state + WARNs — does NOT FATAL (3.3) or fence (4.3). DRY: did NOT extract a shared scanner (replay_wal mutates persist_t; the 1.2 seam is read-only) — no third copy added. ENV FINDING (see cerebrum 2026-06-18): this sandbox's /tmp can be an overlay that EIOs on fdatasync; because committed_lsn is now durable-gated (2.x), an fdatasync EIO correctly HOLDS committed_lsn, which makes `test_fact_hook_append_is_retained_and_nonblocking` (commit_barrier_rollout) FAIL on overlay /tmp — confirmed pre-existing (fails identically on the clean pre-spec baseline), NOT a regression. Final validation (5.x) must run on a real fsync-able fs. - 3.2: DECISION-ONLY (no production change). Boot Policy Decision Table recorded in design.md §"Boot Policy Decision Table (task 3.2)" — 7 rows over {verdict CLEAN/TRUNCATED × at-above/below HWM × peer reachable × bounded-catch-up succeeds/fails} + snapshot-sentinel + empty-first-boot. Outcomes: ELIGIBLE / FENCE+CATCH-UP→ELIGIBLE / bounded FATAL. AUTHORITATIVE INVARIANT: never a silent indefinite fence — every truncated/behind member resolves to exactly one terminal (catch-up→ELIGIBLE, or BOUNDED FATAL). Snapshot-sentinel is NEVER FATAL on a legitimate sentinel. 7 skeleton tests in test_commit_durability.c (row1,2,3,4,4b,5,6) are `TEST_IGNORE_MESSAGE`-skipped (PENDING 3.3/4.3) so the suite stays GREEN — they are the RED contract 3.3 (FATAL rows) and 4.3 (fence/eligible rows) turn green. NOTE: pre-existing ASAN leaks in unrelated tests (test_demotion_convergence, ha_start_election_async, ha_rdhup_integration) make the STANDALONE controller_unit_test exit=1, but meson reports Fail:0 — don't be fooled by standalone exit code. +- 3.3: FATAL branch = pure decision `persist_recover_decision(recover_policy_inputs_t) → recover_decision_t {ELIGIBLE, FENCE, FATAL}` (persist.c) + action `persist_recover_fatal_if_unrecoverable()` which logs the truncation boundary (`durable_floor_lsn`/`cluster_hwm_lsn` + window ms) at ERROR then terminates via `persist_fatal_hook` (extern `void(*)(const char*)`, NULL in prod → `tlog_exit(); abort()`; mirrors `ha_fatal_hook`; tests install it to capture instead of abort). FATAL ONLY for bounded-unrecoverable + NO healthy peer (rows 4/4b); a reachable peer NEVER FATALs (gated behind `if (!healthy_peer_reachable)`); eligibility rows 1/2/6 never FATAL. Bound = FINITE `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS=30000` (persist.h) — keys off the `catch_up_succeeded` bounded-window result, never an indefinite wait. Turned 3.2 skeleton rows 4/4b GREEN + added `test_boot_policy_peer_reachable_never_fatal`; rows 1/2/3/5/6 stay IGNORE (FENCE→ELIGIBLE is 4.3). DEFERRED to 4.3/5.x: wiring this decision into the live boot/promotion path (3.3 is the unit-tested decision + terminate only; no ha.c/cluster.c edits). diff --git a/src/controller/persist.c b/src/controller/persist.c index 82088358..ef0de997 100644 --- a/src/controller/persist.c +++ b/src/controller/persist.c @@ -1642,6 +1642,98 @@ bool persist_recovery_truncated(persist_t* p) return p && p->recovery_verdict == RECOVER_TRUNCATED; } +/* ------------------------------------------------------------------ * + * Boot-policy FATAL decision (ha-commit-durability task 3.3, Req 2.4). + * + * Implements the FATAL terminal of the Boot Policy Decision Table + * (design.md). The decision is a PURE function of its inputs; the FATAL + * action (loud log + terminate) is separated into + * persist_recover_fatal_if_unrecoverable so the decision can be unit-tested + * without aborting. A reachable healthy peer NEVER FATALs (rows 3, 5); + * FATAL is reserved for a member that cannot recover locally AND cannot catch + * up within the bounded window (rows 4, 4b) — never a silent indefinite fence. + * ------------------------------------------------------------------ */ + +/* FATAL seam: NULL in production (real abort); a test may install a hook to + * capture the FATAL path without dying. Mirrors ha_fatal_hook. */ +void (*persist_fatal_hook)(const char* msg) = NULL; + +recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in) +{ + if (in == NULL) + return RECOVER_DECISION_ELIGIBLE; /* defensive: nothing to fence */ + + bool truncated = (in->verdict == RECOVER_TRUNCATED); + + /* Rows 1 & 6: a structurally CLEAN log that is NOT behind the cluster + * high-water mark is a normal promotable member — no fence, no FATAL. */ + if (!truncated && !in->durable_below_hwm) + return RECOVER_DECISION_ELIGIBLE; + + /* From here the member is truncated OR behind the HWM — it must NOT lead + * as-is. A truncated log is by definition not provably at/above the HWM + * (design.md note), so it is treated as behind for fence purposes. */ + + /* Rows 2, 3, 5: a healthy peer is reachable. Catch up from it and regain + * eligibility (task 4.3). A reachable peer NEVER yields FATAL — even for a + * truncated log or a legitimate snapshot-sentinel. If catch-up has already + * succeeded the member is eligible; otherwise it stays fenced (4.3 drives + * the catch-up + release; this task only refuses FATAL while a peer + * exists). */ + if (in->healthy_peer_reachable) { + if (in->catch_up_succeeded) + return RECOVER_DECISION_ELIGIBLE; + return RECOVER_DECISION_FENCE; + } + + /* No healthy peer. The bounded catch-up window is the member's only path to + * a committed-consistent state; if it did not reach one, the member can + * neither recover locally nor catch up — rows 4 & 4b: FATAL (fail loud), + * never a silent indefinite fence (Req 2.4). */ + if (!in->catch_up_succeeded) + return RECOVER_DECISION_FATAL; + + /* No peer but catch-up is reported succeeded (e.g. already at/above after a + * local catch-up): eligible. Defensive — the boot path reaches the + * no-peer branch with catch_up_succeeded=false in the bounded-failure case + * this task owns. */ + return RECOVER_DECISION_ELIGIBLE; +} + +recover_decision_t +persist_recover_fatal_if_unrecoverable(const recover_policy_inputs_t* in) +{ + recover_decision_t decision = persist_recover_decision(in); + + if (decision != RECOVER_DECISION_FATAL) + return decision; + + /* Fail loud: log the truncation boundary (durable_floor_lsn) and the + * cluster high-water mark so an operator sees exactly where the local log + * stops short of the cluster's committed state (Req 2.4). */ + const char* why = (in->verdict == RECOVER_TRUNCATED) + ? "recovery-truncated log (structural corruption)" + : "log behind the cluster committed high-water mark"; + ERROR("persist_recover: FATAL — %s and no healthy peer reachable to catch " + "up within the bounded window (%u ms); truncation boundary " + "durable_floor_lsn=%" PRIu64 " < cluster committed high-water " + "hwm=%" PRIu64 " — terminating rather than serving divergent state " + "(ha-commit-durability Req 2.4)", + why, (unsigned)DEFAULT_RECOVERY_CATCHUP_WINDOW_MS, + in->durable_floor_lsn, in->cluster_hwm_lsn); + + if (persist_fatal_hook != NULL) { + /* Test seam: capture the FATAL decision + boundary log without dying. + * Production leaves the hook NULL and aborts below. */ + persist_fatal_hook("persist_recover: bounded-unrecoverable (FATAL)"); + return RECOVER_DECISION_FATAL; + } + + tlog_exit(); /* flush the async log buffer before dying */ + abort(); /* NEVER returns in production */ + return RECOVER_DECISION_FATAL; /* unreachable */ +} + /* Advance committed_lsn to `written_lsn` on a majority ACK — but ONLY if that * entry is already durable on local stable storage (ha-commit-durability task * 2.4, Req 1.1/1.2). This is the legacy (engine-less / LSN-0) sibling of the diff --git a/src/controller/testsuite/test_commit_durability.c b/src/controller/testsuite/test_commit_durability.c index f8980f78..103ef59a 100644 --- a/src/controller/testsuite/test_commit_durability.c +++ b/src/controller/testsuite/test_commit_durability.c @@ -1017,6 +1017,80 @@ static void test_recover_verdict_snapshot_sentinel_clean(void) * failure. (Req 2.2, 2.4) * * ------------------------------------------------------------------ */ +/* task 3.3 — FATAL-path capture seam. The production FATAL terminates via + * abort(); a test installs persist_fatal_hook so the FATAL decision + boundary + * log can be asserted without the process dying. */ +static int g_fatal_hook_calls = 0; +static char g_fatal_hook_msg[256]; + +static void dur_fatal_hook(const char* msg) +{ + g_fatal_hook_calls++; + if (msg != NULL) { + strncpy(g_fatal_hook_msg, msg, sizeof(g_fatal_hook_msg) - 1); + g_fatal_hook_msg[sizeof(g_fatal_hook_msg) - 1] = '\0'; + } +} + +/* task 3.3 — peer-reachable NEVER FATALs (the never-FATAL aspect of rows + * 3 & 5). This task (3.3) owns only the FATAL terminal; the full FENCE+ + * CATCH-UP -> ELIGIBLE eligibility outcome of these rows is task 4.3 (so the + * row3/row5 skeletons below stay IGNORED for 4.3). Here we pin ONLY the + * invariant 3.3 is responsible for: while a healthy peer is reachable, the + * decision is NEVER FATAL — neither for a TRUNCATED log (row 3) nor for a + * legitimate snapshot-sentinel CLEAN-but-behind member (row 5). It is FENCE + * (catch-up pending) and the FATAL terminate path is never fired. (Req 2.2, + * 2.4, design.md "never FATAL when a peer is reachable".) */ +static void test_boot_policy_peer_reachable_never_fatal(void) +{ + g_fatal_hook_calls = 0; + persist_fatal_hook = dur_fatal_hook; + + /* Row 3: TRUNCATED + healthy peer (catch-up not yet done) -> FENCE, no FATAL. */ + recover_policy_inputs_t truncated_peer = { + .verdict = RECOVER_TRUNCATED, + .durable_below_hwm = true, + .healthy_peer_reachable = true, + .catch_up_succeeded = false, + .durable_floor_lsn = 3, + .cluster_hwm_lsn = 10, + }; + TEST_ASSERT_EQUAL_INT_MESSAGE( + RECOVER_DECISION_FENCE, persist_recover_decision(&truncated_peer), + "row 3: a truncated member with a reachable healthy peer must FENCE " + "(catch up), NEVER FATAL"); + TEST_ASSERT_EQUAL_INT( + RECOVER_DECISION_FENCE, + persist_recover_fatal_if_unrecoverable(&truncated_peer)); + + /* Row 5: snapshot-sentinel CLEAN-but-behind + healthy peer -> FENCE, no + * FATAL. A legitimate sentinel is CLEAN-but-behind, not corruption. */ + recover_policy_inputs_t sentinel_peer = { + .verdict = RECOVER_CLEAN, + .durable_below_hwm = true, + .healthy_peer_reachable = true, + .catch_up_succeeded = false, + .durable_floor_lsn = 5, + .cluster_hwm_lsn = 5, + }; + TEST_ASSERT_EQUAL_INT_MESSAGE( + RECOVER_DECISION_FENCE, persist_recover_decision(&sentinel_peer), + "row 5: a snapshot-sentinel CLEAN-but-behind member with a reachable " + "peer must FENCE, NEVER FATAL"); + TEST_ASSERT_EQUAL_INT( + RECOVER_DECISION_FENCE, + persist_recover_fatal_if_unrecoverable(&sentinel_peer)); + + /* The FATAL terminate path must NOT have fired for either peer-reachable + * case — a reachable healthy peer never triggers FATAL. */ + TEST_ASSERT_EQUAL_INT_MESSAGE( + 0, g_fatal_hook_calls, + "peer-reachable rows must NEVER fire the FATAL terminate path"); + + persist_fatal_hook = NULL; + printf("test_boot_policy_peer_reachable_never_fatal: PASS\n"); +} + /* Row 1: CLEAN + durable at/above HWM -> ELIGIBLE. * A structurally intact log that is not behind the cluster boundary is * a normal promotable member: the promotion gate (4.3) must NOT fence @@ -1068,10 +1142,39 @@ static void test_boot_policy_row3_truncated_peer_catchup_eligible(void) * state. (Req 2.4) */ static void test_boot_policy_row4_truncated_no_peer_bounded_fatal(void) { - TEST_IGNORE_MESSAGE( - "row 4 (TRUNCATED + below + no peer + bounded catch-up fails -> " - "FATAL): pending task 3.3 (FATAL branch). Will assert loud " - "termination with the truncation boundary logged on bounded failure."); + /* Row 4: TRUNCATED + below HWM + NO healthy peer + bounded catch-up fails + * -> FATAL. Cannot recover locally AND cannot catch up within the bounded + * window: fail loud with the truncation boundary logged (Req 2.4). */ + g_fatal_hook_calls = 0; + g_fatal_hook_msg[0] = '\0'; + persist_fatal_hook = dur_fatal_hook; + + recover_policy_inputs_t in = { + .verdict = RECOVER_TRUNCATED, + .durable_below_hwm = true, + .healthy_peer_reachable = false, + .catch_up_succeeded = false, + .durable_floor_lsn = 7, /* truncation boundary */ + .cluster_hwm_lsn = 12, /* cluster committed high-water */ + }; + + /* Pure decision returns FATAL. */ + TEST_ASSERT_EQUAL_INT_MESSAGE( + RECOVER_DECISION_FATAL, persist_recover_decision(&in), + "row 4: truncated + below + no peer + bounded catch-up fails must " + "decide FATAL"); + + /* The action helper routes FATAL through the hook (no abort in test) and + * still returns FATAL — proving the loud-terminate path was taken. */ + recover_decision_t d = persist_recover_fatal_if_unrecoverable(&in); + TEST_ASSERT_EQUAL_INT(RECOVER_DECISION_FATAL, d); + TEST_ASSERT_EQUAL_INT_MESSAGE( + 1, g_fatal_hook_calls, + "row 4: the FATAL terminate path must fire exactly once (the " + "truncation boundary is logged at ERROR just before it)"); + + persist_fatal_hook = NULL; + printf("test_boot_policy_row4_truncated_no_peer_bounded_fatal: PASS\n"); } /* Row 4b: CLEAN + below HWM + NO healthy peer + catch-up fails (bounded @@ -1082,10 +1185,36 @@ static void test_boot_policy_row4_truncated_no_peer_bounded_fatal(void) * once the bounded window elapses. (Req 2.4, 2.2) */ static void test_boot_policy_row4b_clean_below_no_peer_bounded_fatal(void) { - TEST_IGNORE_MESSAGE( - "row 4b (CLEAN + below + no peer + bounded catch-up fails -> FATAL): " - "pending task 3.3. Will assert the behind-no-peer member FATALs on " - "bounded failure — never a silent indefinite fence."); + /* Row 4b: CLEAN + below HWM + NO healthy peer + bounded catch-up fails + * -> FATAL. Behind the HWM with no peer to advance from within the bounded + * window — same bounded-failure terminal as row 4 (Req 2.4, 2.2). */ + g_fatal_hook_calls = 0; + g_fatal_hook_msg[0] = '\0'; + persist_fatal_hook = dur_fatal_hook; + + recover_policy_inputs_t in = { + .verdict = RECOVER_CLEAN, /* structurally intact... */ + .durable_below_hwm = true, /* ...but behind the HWM */ + .healthy_peer_reachable = false, + .catch_up_succeeded = false, + .durable_floor_lsn = 9, + .cluster_hwm_lsn = 20, + }; + + TEST_ASSERT_EQUAL_INT_MESSAGE( + RECOVER_DECISION_FATAL, persist_recover_decision(&in), + "row 4b: clean-but-behind + no peer + bounded catch-up fails must " + "decide FATAL — never a silent indefinite fence"); + + recover_decision_t d = persist_recover_fatal_if_unrecoverable(&in); + TEST_ASSERT_EQUAL_INT(RECOVER_DECISION_FATAL, d); + TEST_ASSERT_EQUAL_INT_MESSAGE( + 1, g_fatal_hook_calls, + "row 4b: a behind member with no reachable peer must FATAL once the " + "bounded window elapses (boundary logged before terminate)"); + + persist_fatal_hook = NULL; + printf("test_boot_policy_row4b_clean_below_no_peer_bounded_fatal: PASS\n"); } /* Row 5: CLEAN snapshot-sentinel (committed at sentinel LSN, NO @@ -1148,6 +1277,10 @@ void test_commit_durability(void) /* task 3.2 — Boot Policy Decision Table skeletons (IGNORED, pending * 3.3/4.3). Registered so each row is reported as IGNORED, not failed; * the suite stays GREEN. See design.md "Boot Policy Decision Table". */ + /* task 3.3 — FATAL rows (4, 4b) now ACTIVE (PASS), plus the never-FATAL + * invariant for the peer-reachable rows (3, 5). The full FENCE->ELIGIBLE + * eligibility outcome of rows 1,2,3,5,6 stays IGNORED pending task 4.3. */ + RUN_TEST(test_boot_policy_peer_reachable_never_fatal); RUN_TEST(test_boot_policy_row1_clean_at_or_above_eligible); RUN_TEST(test_boot_policy_row2_clean_below_peer_catchup_eligible); RUN_TEST(test_boot_policy_row3_truncated_peer_catchup_eligible); diff --git a/src/include/persist.h b/src/include/persist.h index 3fa69b08..7244e257 100644 --- a/src/include/persist.h +++ b/src/include/persist.h @@ -35,6 +35,47 @@ typedef enum { * (consumed by the promotion gate, task 4.3). */ } recover_verdict_t; +/* ============================================================ + * Boot-policy recovery decision (ha-commit-durability task 3.3, Req 2.4) + * + * The outcome of evaluating a member's boot/promotion-eligibility state + * against the Boot Policy Decision Table (design.md "Boot Policy Decision + * Table (task 3.2)"). This task (3.3) owns the FATAL terminal; the fence + + * catch-up + release path is task 4.3. + * + * ELIGIBLE — may promote (no fence). Rows 1, 6. + * FENCE — ineligible to promote; catch up from a healthy peer, regain + * eligibility on success (Req 2.6). Rows 2, 3, 5. A reachable + * healthy peer NEVER yields FATAL. + * FATAL — fail loud: cannot recover locally AND cannot catch up within + * the bounded window. Terminate with the truncation/boundary + * logged rather than serve divergent state (Req 2.4). Rows 4, 4b. + * + * AUTHORITATIVE INVARIANT (design.md): the fence is never a silent indefinite + * wait — a truncated/behind member resolves to exactly one of FENCE→ELIGIBLE + * or a BOUNDED FATAL. There is no "fenced forever, silently divergent" state. + * ============================================================ */ +typedef enum { + RECOVER_DECISION_ELIGIBLE = 0, + RECOVER_DECISION_FENCE = 1, + RECOVER_DECISION_FATAL = 2, +} recover_decision_t; + +/* Policy inputs evaluated at boot / promotion-eligibility (design.md table). + * Modelled as a plain value so the decision is a pure, unit-testable function + * of its inputs — the live catch-up machinery (replication / task 4.3) supplies + * these; this task does not build it. */ +typedef struct { + recover_verdict_t verdict; /* RECOVER_CLEAN / RECOVER_TRUNCATED (task 3.1) */ + bool durable_below_hwm; /* local durable position < cluster committed HWM */ + bool healthy_peer_reachable; /* a healthy peer is reachable for catch-up */ + bool catch_up_succeeded; /* catch-up reached a committed-consistent state + * within the bounded window (only meaningful + * when a catch-up was attempted) */ + uint64_t durable_floor_lsn; /* highest intact replayed LSN (truncation boundary) */ + uint64_t cluster_hwm_lsn; /* cluster committed high-water mark, for logging */ +} recover_policy_inputs_t; + /* ============================================================ * Persistence configuration (Part 2.6) * ============================================================ */ @@ -276,6 +317,54 @@ uint64_t persist_durable_lsn(persist_t* p); recover_verdict_t persist_recovery_verdict(persist_t* p); bool persist_recovery_truncated(persist_t* p); +/* Bounded catch-up window (ha-commit-durability task 3.3, Req 2.4). + * The FINITE deadline a recovery-truncated / behind member is given to reach a + * committed-consistent state from a healthy peer before the boot policy FATALs. + * The fence must NEVER be an indefinite wait (design.md authoritative + * invariant), so the window is always finite. Default 30s — generous enough for + * a healthy-peer catch-up stream yet bounded; the verification mode overrides it + * via the inputs/timeout the caller (task 4.3 boot path) supplies. */ +#define DEFAULT_RECOVERY_CATCHUP_WINDOW_MS 30000u + +/* persist_recover_decision (ha-commit-durability task 3.3, Req 2.4). + * + * Pure decision over the Boot Policy Decision Table: maps the policy inputs to + * RECOVER_DECISION_{ELIGIBLE,FENCE,FATAL}. No side effects, no termination — it + * only classifies, so it is unit-testable in isolation. The FATAL terminal is + * acted on by persist_recover_fatal_if_unrecoverable (which logs + terminates). + * + * Contract (design.md rows): a reachable healthy peer NEVER returns FATAL + * (rows 3, 5); FATAL is returned ONLY when the member is truncated or behind + * the HWM, has NO healthy peer, and bounded catch-up did not reach a + * committed-consistent state (rows 4, 4b). */ +recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in); + +/* persist_recover_fatal_if_unrecoverable (ha-commit-durability task 3.3, + * Req 2.4). + * + * Evaluate persist_recover_decision over `in`; if the outcome is FATAL, log at + * ERROR LOUDLY — INCLUDING the truncation boundary (durable_floor_lsn) and the + * cluster high-water mark — and then TERMINATE (fail loud rather than serve + * divergent state). Returns the decision so a non-FATAL caller can fence/proceed + * (FENCE/ELIGIBLE are acted on by task 4.3). + * + * Termination routes through persist_fatal_hook: in production the hook is NULL + * and the function flushes logs and abort()s (NEVER returns on FATAL); a test + * build may install a hook to capture the FATAL decision + boundary log without + * the process dying (mirrors the ha_fatal_hook seam in ha_convergence.c). */ +recover_decision_t +persist_recover_fatal_if_unrecoverable(const recover_policy_inputs_t* in); + +/* persist_fatal_hook: test-only seam for the FATAL (abort) contract of + * persist_recover_fatal_if_unrecoverable (ha-commit-durability task 3.3, + * Req 2.4). Mirrors ha_fatal_hook (ha_convergence.c). + * + * When non-NULL, the FATAL path invokes this INSTEAD OF abort(), so a test + * binary can assert the FATAL decision was taken and the truncation boundary + * was logged without the process dying. Production leaves it NULL and actually + * terminates. Signature: void hook(const char *msg). */ +extern void (*persist_fatal_hook)(const char* msg); + /* Register a durable-advance wake callback (ha-commit-durability task 2.2). * The group-commit flusher calls fn(ctx) after it advances durable_lsn and * after releasing p->lock, so a consumer (the commit engine) can re-evaluate