feat(ha-commit-durability): loud bounded FATAL on unrecoverable recovery (task 3.3)

.kiro/specs/ha-commit-durability/tasks.md

@@ -63,7 +63,7 @@
   - _Requirements: 2.2, 2.4_
   - _Boundary: RecoveryIntegrity_
 
-- [ ] 3.3 Implement loud FATAL on bounded-unrecoverable recovery
+- [x] 3.3 Implement loud FATAL on bounded-unrecoverable recovery
   - Implement the FATAL branch of the 3.2 policy: when a recovery-truncated member cannot reach a committed-consistent state within the bounded catch-up window, terminate loudly with the truncation boundary logged; a reachable healthy peer never triggers FATAL.
   - Observable: the 3.2 decision-table rows for the FATAL/no-FATAL cases pass; logs show the truncation boundary on FATAL.
   - _Requirements: 2.4_
@@ -129,3 +129,4 @@
 - 2.4: SPEC INACCURACY corrected — `persist_wal_sync_append` is NOT dormant; it is the LIVE job-submit path (`persist_job_hook_fun` calls it at persist.c:1911). Took the "route through durable-majority eval" option (delete infeasible). Production path (ce!=NULL, lsn!=0) was already durable-safe via 2.2's `majority_candidate_locked` self-cap. Closed the hazard in the LEGACY branch: the former unconditional `atomic_store(committed_lsn, written_lsn)` on majority ACK is now `sync_append_legacy_commit_store()` which advances ONLY when `written_lsn <= persist_durable_lsn(p)` (LSN-0 edge: store of 0 valid). Added the design's debug invariant `committed_lsn <= durable_lsn` (`#ifdef XCE_TESTING` assert at commit.c:162 + inside the legacy store) as the future-proof guard. Fixed the stale "after local fsync via persist_wal_async" comment (2.1 deferred the fsync). CODE-SEARCH (authoritative): the ONLY `committed_lsn` stores are — engine advance (commit.c, durable-capped by 2.2), legacy sync_append (now gated), and the two `persist_truncate_wal_above` cutoff stores (NOT ACK-driven; each immediately `clamp_durable_lsn_locked`). ha.c/ha_convergence.c stores are `last_known_committed_lsn` (DIFFERENT field). So no path advances commit on ACK without the durable precondition. Seam `persist_test_sync_append_legacy_commit_store` is production-excluded.
 - 3.1: `recover_verdict_t {RECOVER_CLEAN, RECOVER_TRUNCATED}` (persist.h) + `persist_t.recovery_verdict` field + accessors `persist_recovery_verdict()`/`persist_recovery_truncated()` (NOT XCE_TESTING-gated — 4.3 consumes them in production). `replay_wal` classifies: CRC chain break / CRC mismatch / short header/payload read → TRUNCATED; clean entry boundary / pre-allocated-zero tail / empty WAL / snapshot-sentinel+clean-replay → CLEAN. Verdict is monotonic (init CLEAN once; only ever set TRUNCATED; a later clean segment can't un-set it). Verdict = STRUCTURAL integrity, NOT "behind the cluster" (the behind-high-water case is 4.2/4.3). 3.1 ONLY sets state + WARNs — does NOT FATAL (3.3) or fence (4.3). DRY: did NOT extract a shared scanner (replay_wal mutates persist_t; the 1.2 seam is read-only) — no third copy added. ENV FINDING (see cerebrum 2026-06-18): this sandbox's /tmp can be an overlay that EIOs on fdatasync; because committed_lsn is now durable-gated (2.x), an fdatasync EIO correctly HOLDS committed_lsn, which makes `test_fact_hook_append_is_retained_and_nonblocking` (commit_barrier_rollout) FAIL on overlay /tmp — confirmed pre-existing (fails identically on the clean pre-spec baseline), NOT a regression. Final validation (5.x) must run on a real fsync-able fs.
 - 3.2: DECISION-ONLY (no production change). Boot Policy Decision Table recorded in design.md §"Boot Policy Decision Table (task 3.2)" — 7 rows over {verdict CLEAN/TRUNCATED × at-above/below HWM × peer reachable × bounded-catch-up succeeds/fails} + snapshot-sentinel + empty-first-boot. Outcomes: ELIGIBLE / FENCE+CATCH-UP→ELIGIBLE / bounded FATAL. AUTHORITATIVE INVARIANT: never a silent indefinite fence — every truncated/behind member resolves to exactly one terminal (catch-up→ELIGIBLE, or BOUNDED FATAL). Snapshot-sentinel is NEVER FATAL on a legitimate sentinel. 7 skeleton tests in test_commit_durability.c (row1,2,3,4,4b,5,6) are `TEST_IGNORE_MESSAGE`-skipped (PENDING 3.3/4.3) so the suite stays GREEN — they are the RED contract 3.3 (FATAL rows) and 4.3 (fence/eligible rows) turn green. NOTE: pre-existing ASAN leaks in unrelated tests (test_demotion_convergence, ha_start_election_async, ha_rdhup_integration) make the STANDALONE controller_unit_test exit=1, but meson reports Fail:0 — don't be fooled by standalone exit code.
+- 3.3: FATAL branch = pure decision `persist_recover_decision(recover_policy_inputs_t) → recover_decision_t {ELIGIBLE, FENCE, FATAL}` (persist.c) + action `persist_recover_fatal_if_unrecoverable()` which logs the truncation boundary (`durable_floor_lsn`/`cluster_hwm_lsn` + window ms) at ERROR then terminates via `persist_fatal_hook` (extern `void(*)(const char*)`, NULL in prod → `tlog_exit(); abort()`; mirrors `ha_fatal_hook`; tests install it to capture instead of abort). FATAL ONLY for bounded-unrecoverable + NO healthy peer (rows 4/4b); a reachable peer NEVER FATALs (gated behind `if (!healthy_peer_reachable)`); eligibility rows 1/2/6 never FATAL. Bound = FINITE `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS=30000` (persist.h) — keys off the `catch_up_succeeded` bounded-window result, never an indefinite wait. Turned 3.2 skeleton rows 4/4b GREEN + added `test_boot_policy_peer_reachable_never_fatal`; rows 1/2/3/5/6 stay IGNORE (FENCE→ELIGIBLE is 4.3). DEFERRED to 4.3/5.x: wiring this decision into the live boot/promotion path (3.3 is the unit-tested decision + terminate only; no ha.c/cluster.c edits).

src/controller/persist.c

@@ -1642,6 +1642,98 @@ bool persist_recovery_truncated(persist_t* p)
     return p && p->recovery_verdict == RECOVER_TRUNCATED;
 }
 
+/* ------------------------------------------------------------------ *
+ * Boot-policy FATAL decision (ha-commit-durability task 3.3, Req 2.4).
+ *
+ * Implements the FATAL terminal of the Boot Policy Decision Table
+ * (design.md). The decision is a PURE function of its inputs; the FATAL
+ * action (loud log + terminate) is separated into
+ * persist_recover_fatal_if_unrecoverable so the decision can be unit-tested
+ * without aborting. A reachable healthy peer NEVER FATALs (rows 3, 5);
+ * FATAL is reserved for a member that cannot recover locally AND cannot catch
+ * up within the bounded window (rows 4, 4b) — never a silent indefinite fence.
+ * ------------------------------------------------------------------ */
+
+/* FATAL seam: NULL in production (real abort); a test may install a hook to
+ * capture the FATAL path without dying. Mirrors ha_fatal_hook. */
+void (*persist_fatal_hook)(const char* msg) = NULL;
+
+recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in)
+{
+    if (in == NULL)
+        return RECOVER_DECISION_ELIGIBLE; /* defensive: nothing to fence */
+
+    bool truncated = (in->verdict == RECOVER_TRUNCATED);
+
+    /* Rows 1 & 6: a structurally CLEAN log that is NOT behind the cluster
+     * high-water mark is a normal promotable member — no fence, no FATAL. */
+    if (!truncated && !in->durable_below_hwm)
+        return RECOVER_DECISION_ELIGIBLE;
+
+    /* From here the member is truncated OR behind the HWM — it must NOT lead
+     * as-is. A truncated log is by definition not provably at/above the HWM
+     * (design.md note), so it is treated as behind for fence purposes. */
+
+    /* Rows 2, 3, 5: a healthy peer is reachable. Catch up from it and regain
+     * eligibility (task 4.3). A reachable peer NEVER yields FATAL — even for a
+     * truncated log or a legitimate snapshot-sentinel. If catch-up has already
+     * succeeded the member is eligible; otherwise it stays fenced (4.3 drives
+     * the catch-up + release; this task only refuses FATAL while a peer
+     * exists). */
+    if (in->healthy_peer_reachable) {
+        if (in->catch_up_succeeded)
+            return RECOVER_DECISION_ELIGIBLE;
+        return RECOVER_DECISION_FENCE;
+    }
+
+    /* No healthy peer. The bounded catch-up window is the member's only path to
+     * a committed-consistent state; if it did not reach one, the member can
+     * neither recover locally nor catch up — rows 4 & 4b: FATAL (fail loud),
+     * never a silent indefinite fence (Req 2.4). */
+    if (!in->catch_up_succeeded)
+        return RECOVER_DECISION_FATAL;
+
+    /* No peer but catch-up is reported succeeded (e.g. already at/above after a
+     * local catch-up): eligible. Defensive — the boot path reaches the
+     * no-peer branch with catch_up_succeeded=false in the bounded-failure case
+     * this task owns. */
+    return RECOVER_DECISION_ELIGIBLE;
+}
+
+recover_decision_t
+persist_recover_fatal_if_unrecoverable(const recover_policy_inputs_t* in)
+{
+    recover_decision_t decision = persist_recover_decision(in);
+
+    if (decision != RECOVER_DECISION_FATAL)
+        return decision;
+
+    /* Fail loud: log the truncation boundary (durable_floor_lsn) and the
+     * cluster high-water mark so an operator sees exactly where the local log
+     * stops short of the cluster's committed state (Req 2.4). */
+    const char* why = (in->verdict == RECOVER_TRUNCATED)
+                          ? "recovery-truncated log (structural corruption)"
+                          : "log behind the cluster committed high-water mark";
+    ERROR("persist_recover: FATAL — %s and no healthy peer reachable to catch "
+          "up within the bounded window (%u ms); truncation boundary "
+          "durable_floor_lsn=%" PRIu64 " < cluster committed high-water "
+          "hwm=%" PRIu64 " — terminating rather than serving divergent state "
+          "(ha-commit-durability Req 2.4)",
+          why, (unsigned)DEFAULT_RECOVERY_CATCHUP_WINDOW_MS,
+          in->durable_floor_lsn, in->cluster_hwm_lsn);
+
+    if (persist_fatal_hook != NULL) {
+        /* Test seam: capture the FATAL decision + boundary log without dying.
+         * Production leaves the hook NULL and aborts below. */
+        persist_fatal_hook("persist_recover: bounded-unrecoverable (FATAL)");
+        return RECOVER_DECISION_FATAL;
+    }
+
+    tlog_exit(); /* flush the async log buffer before dying */
+    abort();     /* NEVER returns in production */
+    return RECOVER_DECISION_FATAL; /* unreachable */
+}
+
 /* Advance committed_lsn to `written_lsn` on a majority ACK — but ONLY if that
  * entry is already durable on local stable storage (ha-commit-durability task
  * 2.4, Req 1.1/1.2). This is the legacy (engine-less / LSN-0) sibling of the

src/controller/testsuite/test_commit_durability.c

@@ -1017,6 +1017,80 @@ static void test_recover_verdict_snapshot_sentinel_clean(void)
  * failure. (Req 2.2, 2.4)                                             *
  * ------------------------------------------------------------------ */
 
+/* task 3.3 — FATAL-path capture seam. The production FATAL terminates via
+ * abort(); a test installs persist_fatal_hook so the FATAL decision + boundary
+ * log can be asserted without the process dying. */
+static int  g_fatal_hook_calls = 0;
+static char g_fatal_hook_msg[256];
+
+static void dur_fatal_hook(const char* msg)
+{
+    g_fatal_hook_calls++;
+    if (msg != NULL) {
+        strncpy(g_fatal_hook_msg, msg, sizeof(g_fatal_hook_msg) - 1);
+        g_fatal_hook_msg[sizeof(g_fatal_hook_msg) - 1] = '\0';
+    }
+}
+
+/* task 3.3 — peer-reachable NEVER FATALs (the never-FATAL aspect of rows
+ * 3 & 5). This task (3.3) owns only the FATAL terminal; the full FENCE+
+ * CATCH-UP -> ELIGIBLE eligibility outcome of these rows is task 4.3 (so the
+ * row3/row5 skeletons below stay IGNORED for 4.3). Here we pin ONLY the
+ * invariant 3.3 is responsible for: while a healthy peer is reachable, the
+ * decision is NEVER FATAL — neither for a TRUNCATED log (row 3) nor for a
+ * legitimate snapshot-sentinel CLEAN-but-behind member (row 5). It is FENCE
+ * (catch-up pending) and the FATAL terminate path is never fired. (Req 2.2,
+ * 2.4, design.md "never FATAL when a peer is reachable".) */
+static void test_boot_policy_peer_reachable_never_fatal(void)
+{
+    g_fatal_hook_calls = 0;
+    persist_fatal_hook = dur_fatal_hook;
+
+    /* Row 3: TRUNCATED + healthy peer (catch-up not yet done) -> FENCE, no FATAL. */
+    recover_policy_inputs_t truncated_peer = {
+        .verdict                = RECOVER_TRUNCATED,
+        .durable_below_hwm      = true,
+        .healthy_peer_reachable = true,
+        .catch_up_succeeded     = false,
+        .durable_floor_lsn      = 3,
+        .cluster_hwm_lsn        = 10,
+    };
+    TEST_ASSERT_EQUAL_INT_MESSAGE(
+        RECOVER_DECISION_FENCE, persist_recover_decision(&truncated_peer),
+        "row 3: a truncated member with a reachable healthy peer must FENCE "
+        "(catch up), NEVER FATAL");
+    TEST_ASSERT_EQUAL_INT(
+        RECOVER_DECISION_FENCE,
+        persist_recover_fatal_if_unrecoverable(&truncated_peer));
+
+    /* Row 5: snapshot-sentinel CLEAN-but-behind + healthy peer -> FENCE, no
+     * FATAL. A legitimate sentinel is CLEAN-but-behind, not corruption. */
+    recover_policy_inputs_t sentinel_peer = {
+        .verdict                = RECOVER_CLEAN,
+        .durable_below_hwm      = true,
+        .healthy_peer_reachable = true,
+        .catch_up_succeeded     = false,
+        .durable_floor_lsn      = 5,
+        .cluster_hwm_lsn        = 5,
+    };
+    TEST_ASSERT_EQUAL_INT_MESSAGE(
+        RECOVER_DECISION_FENCE, persist_recover_decision(&sentinel_peer),
+        "row 5: a snapshot-sentinel CLEAN-but-behind member with a reachable "
+        "peer must FENCE, NEVER FATAL");
+    TEST_ASSERT_EQUAL_INT(
+        RECOVER_DECISION_FENCE,
+        persist_recover_fatal_if_unrecoverable(&sentinel_peer));
+
+    /* The FATAL terminate path must NOT have fired for either peer-reachable
+     * case — a reachable healthy peer never triggers FATAL. */
+    TEST_ASSERT_EQUAL_INT_MESSAGE(
+        0, g_fatal_hook_calls,
+        "peer-reachable rows must NEVER fire the FATAL terminate path");
+
+    persist_fatal_hook = NULL;
+    printf("test_boot_policy_peer_reachable_never_fatal: PASS\n");
+}
+
 /* Row 1: CLEAN + durable at/above HWM -> ELIGIBLE.
  * A structurally intact log that is not behind the cluster boundary is
  * a normal promotable member: the promotion gate (4.3) must NOT fence
@@ -1068,10 +1142,39 @@ static void test_boot_policy_row3_truncated_peer_catchup_eligible(void)
  * state. (Req 2.4) */
 static void test_boot_policy_row4_truncated_no_peer_bounded_fatal(void)
 {
-    TEST_IGNORE_MESSAGE(
+    /* Row 4: TRUNCATED + below HWM + NO healthy peer + bounded catch-up fails
-        "row 4 (TRUNCATED + below + no peer + bounded catch-up fails -> "
+     * -> FATAL. Cannot recover locally AND cannot catch up within the bounded
-        "FATAL): pending task 3.3 (FATAL branch). Will assert loud "
+     * window: fail loud with the truncation boundary logged (Req 2.4). */
-        "termination with the truncation boundary logged on bounded failure.");
+    g_fatal_hook_calls = 0;
+    g_fatal_hook_msg[0] = '\0';
+    persist_fatal_hook = dur_fatal_hook;
+
+    recover_policy_inputs_t in = {
+        .verdict                = RECOVER_TRUNCATED,
+        .durable_below_hwm      = true,
+        .healthy_peer_reachable = false,
+        .catch_up_succeeded     = false,
+        .durable_floor_lsn      = 7,  /* truncation boundary */
+        .cluster_hwm_lsn        = 12, /* cluster committed high-water */
+    };
+
+    /* Pure decision returns FATAL. */
+    TEST_ASSERT_EQUAL_INT_MESSAGE(
+        RECOVER_DECISION_FATAL, persist_recover_decision(&in),
+        "row 4: truncated + below + no peer + bounded catch-up fails must "
+        "decide FATAL");
+
+    /* The action helper routes FATAL through the hook (no abort in test) and
+     * still returns FATAL — proving the loud-terminate path was taken. */
+    recover_decision_t d = persist_recover_fatal_if_unrecoverable(&in);
+    TEST_ASSERT_EQUAL_INT(RECOVER_DECISION_FATAL, d);
+    TEST_ASSERT_EQUAL_INT_MESSAGE(
+        1, g_fatal_hook_calls,
+        "row 4: the FATAL terminate path must fire exactly once (the "
+        "truncation boundary is logged at ERROR just before it)");
+
+    persist_fatal_hook = NULL;
+    printf("test_boot_policy_row4_truncated_no_peer_bounded_fatal: PASS\n");
 }
 
 /* Row 4b: CLEAN + below HWM + NO healthy peer + catch-up fails (bounded
@@ -1082,10 +1185,36 @@ static void test_boot_policy_row4_truncated_no_peer_bounded_fatal(void)
  * once the bounded window elapses. (Req 2.4, 2.2) */
 static void test_boot_policy_row4b_clean_below_no_peer_bounded_fatal(void)
 {
-    TEST_IGNORE_MESSAGE(
+    /* Row 4b: CLEAN + below HWM + NO healthy peer + bounded catch-up fails
-        "row 4b (CLEAN + below + no peer + bounded catch-up fails -> FATAL): "
+     * -> FATAL. Behind the HWM with no peer to advance from within the bounded
-        "pending task 3.3. Will assert the behind-no-peer member FATALs on "
+     * window — same bounded-failure terminal as row 4 (Req 2.4, 2.2). */
-        "bounded failure — never a silent indefinite fence.");
+    g_fatal_hook_calls = 0;
+    g_fatal_hook_msg[0] = '\0';
+    persist_fatal_hook = dur_fatal_hook;
+
+    recover_policy_inputs_t in = {
+        .verdict                = RECOVER_CLEAN, /* structurally intact... */
+        .durable_below_hwm      = true,          /* ...but behind the HWM */
+        .healthy_peer_reachable = false,
+        .catch_up_succeeded     = false,
+        .durable_floor_lsn      = 9,
+        .cluster_hwm_lsn        = 20,
+    };
+
+    TEST_ASSERT_EQUAL_INT_MESSAGE(
+        RECOVER_DECISION_FATAL, persist_recover_decision(&in),
+        "row 4b: clean-but-behind + no peer + bounded catch-up fails must "
+        "decide FATAL — never a silent indefinite fence");
+
+    recover_decision_t d = persist_recover_fatal_if_unrecoverable(&in);
+    TEST_ASSERT_EQUAL_INT(RECOVER_DECISION_FATAL, d);
+    TEST_ASSERT_EQUAL_INT_MESSAGE(
+        1, g_fatal_hook_calls,
+        "row 4b: a behind member with no reachable peer must FATAL once the "
+        "bounded window elapses (boundary logged before terminate)");
+
+    persist_fatal_hook = NULL;
+    printf("test_boot_policy_row4b_clean_below_no_peer_bounded_fatal: PASS\n");
 }
 
 /* Row 5: CLEAN snapshot-sentinel (committed at sentinel LSN, NO
@@ -1148,6 +1277,10 @@ void test_commit_durability(void)
     /* task 3.2 — Boot Policy Decision Table skeletons (IGNORED, pending
      * 3.3/4.3). Registered so each row is reported as IGNORED, not failed;
      * the suite stays GREEN. See design.md "Boot Policy Decision Table". */
+    /* task 3.3 — FATAL rows (4, 4b) now ACTIVE (PASS), plus the never-FATAL
+     * invariant for the peer-reachable rows (3, 5). The full FENCE->ELIGIBLE
+     * eligibility outcome of rows 1,2,3,5,6 stays IGNORED pending task 4.3. */
+    RUN_TEST(test_boot_policy_peer_reachable_never_fatal);
     RUN_TEST(test_boot_policy_row1_clean_at_or_above_eligible);
     RUN_TEST(test_boot_policy_row2_clean_below_peer_catchup_eligible);
     RUN_TEST(test_boot_policy_row3_truncated_peer_catchup_eligible);

src/include/persist.h

@@ -35,6 +35,47 @@ typedef enum {
                             * (consumed by the promotion gate, task 4.3). */
 } recover_verdict_t;
 
+/* ============================================================
+ * Boot-policy recovery decision (ha-commit-durability task 3.3, Req 2.4)
+ *
+ * The outcome of evaluating a member's boot/promotion-eligibility state
+ * against the Boot Policy Decision Table (design.md "Boot Policy Decision
+ * Table (task 3.2)"). This task (3.3) owns the FATAL terminal; the fence +
+ * catch-up + release path is task 4.3.
+ *
+ *   ELIGIBLE — may promote (no fence). Rows 1, 6.
+ *   FENCE    — ineligible to promote; catch up from a healthy peer, regain
+ *              eligibility on success (Req 2.6). Rows 2, 3, 5. A reachable
+ *              healthy peer NEVER yields FATAL.
+ *   FATAL    — fail loud: cannot recover locally AND cannot catch up within
+ *              the bounded window. Terminate with the truncation/boundary
+ *              logged rather than serve divergent state (Req 2.4). Rows 4, 4b.
+ *
+ * AUTHORITATIVE INVARIANT (design.md): the fence is never a silent indefinite
+ * wait — a truncated/behind member resolves to exactly one of FENCE→ELIGIBLE
+ * or a BOUNDED FATAL. There is no "fenced forever, silently divergent" state.
+ * ============================================================ */
+typedef enum {
+    RECOVER_DECISION_ELIGIBLE = 0,
+    RECOVER_DECISION_FENCE    = 1,
+    RECOVER_DECISION_FATAL    = 2,
+} recover_decision_t;
+
+/* Policy inputs evaluated at boot / promotion-eligibility (design.md table).
+ * Modelled as a plain value so the decision is a pure, unit-testable function
+ * of its inputs — the live catch-up machinery (replication / task 4.3) supplies
+ * these; this task does not build it. */
+typedef struct {
+    recover_verdict_t verdict;          /* RECOVER_CLEAN / RECOVER_TRUNCATED (task 3.1) */
+    bool     durable_below_hwm;         /* local durable position < cluster committed HWM */
+    bool     healthy_peer_reachable;    /* a healthy peer is reachable for catch-up */
+    bool     catch_up_succeeded;        /* catch-up reached a committed-consistent state
+                                         * within the bounded window (only meaningful
+                                         * when a catch-up was attempted) */
+    uint64_t durable_floor_lsn;         /* highest intact replayed LSN (truncation boundary) */
+    uint64_t cluster_hwm_lsn;           /* cluster committed high-water mark, for logging */
+} recover_policy_inputs_t;
+
 /* ============================================================
  * Persistence configuration (Part 2.6)
  * ============================================================ */
@@ -276,6 +317,54 @@ uint64_t persist_durable_lsn(persist_t* p);
 recover_verdict_t persist_recovery_verdict(persist_t* p);
 bool persist_recovery_truncated(persist_t* p);
 
+/* Bounded catch-up window (ha-commit-durability task 3.3, Req 2.4).
+ * The FINITE deadline a recovery-truncated / behind member is given to reach a
+ * committed-consistent state from a healthy peer before the boot policy FATALs.
+ * The fence must NEVER be an indefinite wait (design.md authoritative
+ * invariant), so the window is always finite. Default 30s — generous enough for
+ * a healthy-peer catch-up stream yet bounded; the verification mode overrides it
+ * via the inputs/timeout the caller (task 4.3 boot path) supplies. */
+#define DEFAULT_RECOVERY_CATCHUP_WINDOW_MS 30000u
+
+/* persist_recover_decision (ha-commit-durability task 3.3, Req 2.4).
+ *
+ * Pure decision over the Boot Policy Decision Table: maps the policy inputs to
+ * RECOVER_DECISION_{ELIGIBLE,FENCE,FATAL}. No side effects, no termination — it
+ * only classifies, so it is unit-testable in isolation. The FATAL terminal is
+ * acted on by persist_recover_fatal_if_unrecoverable (which logs + terminates).
+ *
+ * Contract (design.md rows): a reachable healthy peer NEVER returns FATAL
+ * (rows 3, 5); FATAL is returned ONLY when the member is truncated or behind
+ * the HWM, has NO healthy peer, and bounded catch-up did not reach a
+ * committed-consistent state (rows 4, 4b). */
+recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in);
+
+/* persist_recover_fatal_if_unrecoverable (ha-commit-durability task 3.3,
+ * Req 2.4).
+ *
+ * Evaluate persist_recover_decision over `in`; if the outcome is FATAL, log at
+ * ERROR LOUDLY — INCLUDING the truncation boundary (durable_floor_lsn) and the
+ * cluster high-water mark — and then TERMINATE (fail loud rather than serve
+ * divergent state). Returns the decision so a non-FATAL caller can fence/proceed
+ * (FENCE/ELIGIBLE are acted on by task 4.3).
+ *
+ * Termination routes through persist_fatal_hook: in production the hook is NULL
+ * and the function flushes logs and abort()s (NEVER returns on FATAL); a test
+ * build may install a hook to capture the FATAL decision + boundary log without
+ * the process dying (mirrors the ha_fatal_hook seam in ha_convergence.c). */
+recover_decision_t
+persist_recover_fatal_if_unrecoverable(const recover_policy_inputs_t* in);
+
+/* persist_fatal_hook: test-only seam for the FATAL (abort) contract of
+ * persist_recover_fatal_if_unrecoverable (ha-commit-durability task 3.3,
+ * Req 2.4). Mirrors ha_fatal_hook (ha_convergence.c).
+ *
+ * When non-NULL, the FATAL path invokes this INSTEAD OF abort(), so a test
+ * binary can assert the FATAL decision was taken and the truncation boundary
+ * was logged without the process dying. Production leaves it NULL and actually
+ * terminates. Signature: void hook(const char *msg). */
+extern void (*persist_fatal_hook)(const char* msg);
+
 /* Register a durable-advance wake callback (ha-commit-durability task 2.2).
  * The group-commit flusher calls fn(ctx) after it advances durable_lsn and
  * after releasing p->lock, so a consumer (the commit engine) can re-evaluate