test(config-upgrade-snapshot): fault-inject crash/restart + leadership change (task 8.7)

Prove Requirements 5.2/5.4/5.5/6.3 via deterministic fault injection at
the two critical upgrade windows, entirely at the unit level using the
XCE_TESTING hooks earlier tasks built for exactly this purpose -- no
live soak needed.

test_marker_checkpoint_fault_hook_crash_then_restart_resumes_materialization
extends the existing fault-hook seam test: after simulating a crash
before the checkpoint write durably completes, it boots a genuinely
fresh persist_t/cluster_t against the same on-disk WAL via
persist_recover() (a real boot-replay simulation, not a same-process
re-call) and confirms materialization resumes and completes correctly.

test_leadership_change_mid_materialization_completes_via_same_generic_apply_drain
proves a promoted member's materialization completes identically to a
non-promoted control member's, by draining each through its actual
production entry point and comparing checkpoints byte-for-byte -- a
regression guard against any future role-conditional branch creeping
into the marker-apply path.

.kiro/specs/config-upgrade-snapshot/tasks.md

@@ -129,7 +129,7 @@
   - _Depends: 7, 8.1, 8.2, 8.3, 8.4_
   - _Blocked: bug-1472 — pre-existing, out-of-spec-boundary HA worker-control replication defect (`worker_from_capnp`, `worker.c:827-856`, introduced 2026-05-08, commit `9b6a91a8`, ~2.5 months before this spec). `WAL_WORKER_CTRL` apply treats "-EEXIST" (target worker already exists) as a benign no-op for ALL hook details — correct for JOIN replay, but wrong for UPDATE/DEAD, where the target already existing is the normal case. Consequence: a host admin-status change (`hostctrl -c`/`-o`) never propagates past the member that directly processed the RPC — completely general, unrelated to config upgrades or checkpoints. Live soak infra + task 8.6's own bug-1468/bug-1469 fixes are committed (959b0cd) and independently reviewed; only the Requirement 6.4 host-admin-status-survival assertion is blocked, on a bug this spec does not own. See `.wolf/buglog.json` bug-1471 (byte-precise diagnosis) and bug-1472 (root cause) for full forensic detail. Unblocks when bug-1472 is fixed by whoever owns worker/HA replication — no further config-upgrade-snapshot work needed once that lands (re-run `run-ha-soak.sh --config-upgrade` to confirm clean PASS)._
 
-- [ ] 8.7 Inject a crash and a leadership change at the two critical upgrade windows
+- [x] 8.7 Inject a crash and a leadership change at the two critical upgrade windows
   - Force a controller crash/restart between a configuration-change marker's commit and that member's own snapshot write becoming durable, and assert materialization resumes correctly from the marker on restart
   - Force a leadership change between a configuration-change marker's commit and every member completing its snapshot install, and assert the new leader continues driving materialization to completion without any special resume logic
   - Observable: both fault-injection scenarios complete with the post-upgrade consistency invariant and the crash-safety invariant intact
@@ -150,6 +150,8 @@
 
 ## Implementation Notes
 
+- **Task 8.7 (both scenarios, unit-level, no live soak needed):** design.md's Testing Strategy was explicit this task should use deterministic fault injection rather than chaotic-soak timing probability, and both hooks needed already existed from earlier tasks (`persist_marker_checkpoint_fault_hook`, task 8.1; `xce_upgrade_marker_committed_hook`, task 7) — task 8.7 was purely additive test work, zero production code touched. **Scenario 1** (`test_marker_checkpoint_fault_hook_crash_then_restart_resumes_materialization`, `test_upgrade_snapshot_apply.c`) builds on the existing seam test (which only proved the fault hook fires and gates the checkpoint call) by clearing the fault, tearing down without deleting the on-disk WAL, and booting a genuinely fresh `persist_t`/`cluster_t` via `persist_recover()` — a real second-process boot-replay simulation, deliberately not a same-process re-call (which the test's own comment argues would be a weaker, trivially-passing proof). **Scenario 2** (`test_leadership_change_mid_materialization_completes_via_same_generic_apply_drain`, `test_config_repl.c`) proves Req 5.4 by building a promoted member and a non-promoted control member from byte-identical committed-but-unapplied markers, draining each via the respective real production path (`persist_apply_pump_run_sync`, which production promotion — `raft_cb_become_leader`, `ha.c` — calls unconditionally right after the role flip; vs. plain `persist_apply_pump_run` for the control), and asserting the resulting checkpoints are byte-identical via `snapdiff_paths` (task 8.4's tool). Reviewer independently verified (not trusted) both design claims by reading `ha.c` directly (`upgrade_materialization_fenced` only refuses an *already*-fenced member; the sync-drain call is unconditional and unguarded by role-specific logic) and specifically interrogated whether this was a tautology (calling the same underlying function twice) — concluded it genuinely is not, since the real variable under test is role state at drain time plus the production entry point, and the test separately asserts promotion alone has zero effect on the fence. Also confirmed via direct comparison against the existing 6.2/8.3 fence tests that this is new *continuation* coverage (fence unset → promotion succeeds → drain completes → config activates), not a duplicate of the existing *gating* coverage (fence set → promotion refused, never reaching the drain).
+
 - **Task 8.6 progress (infrastructure + two real production bugs found and fixed; task NOT YET complete — one more bug blocks a clean live-soak PASS):** building this task's live 3-node chaotic-upgrade soak required substantial new infrastructure that never existed anywhere in this codebase: a client-side send path for `OP_upgradeConfig` (`xce_upgradeconfig_req_raw`/`xce_upgradeconfig`/`xcecli_upgradeconfig` — the auto-generated capnp encoder for this request type is a literal no-op for its `Data` fields, worked around via a hand-rolled bypass mirroring the existing `xce_replconfigpush_req_raw` idiom), a `cmd_hostctrl`/`cmd_upgradeconfig` CLI pair in `test.c`, and a new `--config-upgrade` mode in `run-ha-soak.sh` (+ `SKILL.md` docs). Building and running this surfaced two real, independently-confirmed production defects:
   - **bug-1468** (`cluster.c`): `myreq_handler`'s `OPCode_upgradeConfig` dispatch guard was inverted by an unrelated commit (`fd93997f`, 2026-05-31) — it rejected the only role (primary) that could ever legitimately reach it, making config upgrades unreachable from any network client at all. Fixed: restored correct polarity.
   - **bug-1469** (`persist.c`): `apply_committed_entry`'s `WAL_CONFIG_MARKER` case never called `config_activate_committed()` — so the node that *originates* a config upgrade never refreshed its own in-memory config/queue state as part of committing it (only receiving standbys did, via a separate out-of-band push path). If that node was killed before some later, independent event happened to refresh it, its checkpoint froze with stale config content — a genuine, **live-reproduced** instance of bug-857's originally-hypothesized failure mode (task 8.5 could only prove a different, unit-level mechanism; this is the first real empirical case). Fixed: added `config_activate_committed(c)` inside the marker-apply case, positioned *before* the marker-triggered checkpoint write (checkpoint content is built from live runtime tables that only reflect the reload after activation runs, not from `myconf` directly) and guarded against `c->myconf == NULL` (a real crash risk hit in an existing test fixture). Fixing this also exposed and required fixing a genuine race in a pre-existing test (`test_commit_advance_retry_clears_push_before_commit_fence`) that the new activation side-effect made observable for the first time — narrowed the race window, did not weaken any assertion. Reviewer independently re-derived the lock-safety claim from scratch (traced every mutex site in `config_repl.c`, confirmed none reachable from `config_activate_committed`) and the thread-class claim (`apply_committed_entry` runs on a `VTASK_LIVE_BLOCK` pool worker, confirmed via `persist_apply_pump_kick`) rather than trusting the implementer's account — both held.

src/controller/testsuite/test_config_repl.c

@@ -29,6 +29,10 @@
 #include "persist.h"
 #include "repl.h"
 #include "request.h"
+#include "snapdiff.h" /* snapdiff_paths -- task 8.7 Scenario 2: prove a
+                       * leadership-change-driven materialization drain
+                       * produces a byte-identical checkpoint to an ordinary,
+                       * non-promoted drain of the same marker */
 #include "typemap.h"
 #include "wal_types.h"
 #include "xce/cluster_def.h"
@@ -2487,6 +2491,257 @@ static void test_wal_config_marker_apply_activates_originating_member_config(voi
     printf("test_wal_config_marker_apply_activates_originating_member_config: PASS\n");
 }
 
+/* ── config-upgrade-snapshot task 8.7 Scenario 2: leadership change mid-
+ * materialization does not special-case, and completes via the SAME generic
+ * apply-drain path (Req 5.4) ────────────────────────────────────────────
+ *
+ * design.md's own "Key simplification found during design" (persist.c's
+ * WAL_CONFIG_MARKER case doc comment, apply_committed_entry): "no new
+ * leader-specific 'resume' logic is needed either, since a newly-elected
+ * leader satisfies this the same way, by continuing to apply its own
+ * committed log." This test proves that claim structurally rather than
+ * asserting it by fiat.
+ *
+ * The concrete mechanism (read directly out of raft_cb_become_leader,
+ * ha.c:1516-1979): the promotion-time eligibility gate
+ * (upgrade_materialization_fenced(c), ha.c:1763) only refuses promotion when
+ * the fence has ALREADY been set by a prior upgrade_materialization_verify()
+ * call. A marker that is committed but has never yet been through
+ * apply_committed_entry() at all -- a genuine apply backlog, not a checkpoint
+ * failure -- leaves the fence at its zero-initialized (unfenced) default,
+ * because nothing has evaluated it yet. So promotion is NOT refused in this
+ * case. What actually drains the backlog is ha.c:1878's
+ * persist_apply_pump_run_sync(&c->persist) call, unconditionally issued right
+ * after role flips to CTRL_PRIMARY -- the SAME synchronous drain function
+ * bug-1043's job-backlog fix added for an entirely unrelated reason (a
+ * promoted standby's un-applied committed jobs must be visible immediately).
+ * The WAL_CONFIG_MARKER case (persist.c) rides that pre-existing, generic
+ * drain for free: no leadership-specific config-upgrade code exists anywhere
+ * in the promotion path.
+ *
+ * This test drives that exact mechanism directly
+ * (persist_apply_pump_run_sync) rather than the full
+ * raft_cb_become_leader_with_cluster_for_test() tail, deliberately: the rest
+ * of that function's post-drain tail (quorum-transition logging, config push,
+ * ctrlhello, persist-hook registration, heartbeat start, own-term gate arm)
+ * is orthogonal to Requirement 5.4's materialization-continuation claim and
+ * would require a substantially larger, novel fixture (full ha_init +
+ * delaymgr_init, mirroring test_convergence_verification.c's cv_setup --
+ * this file's own t42_/t43_ fixtures deliberately do NOT wire that, since no
+ * other test in this file drives promotion past the fence-refusal point) to
+ * exercise safely -- a scope expansion this test-only task should not take
+ * on. The GATING half of Req 5.4 (a member whose fence IS already set
+ * refuses promotion) is already exhaustively covered by
+ * test_upgrade_materialization_fence_enforced_by_dispatch_gate_and_promotion
+ * (6.2-a) and
+ * test_wal_config_marker_apply_sets_fence_via_real_checkpoint_miss (6.2
+ * remediation) above, both via the real raft_cb_become_leader_with_cluster_
+ * for_test() gate -- this test does not repeat that proof, it closes the
+ * CONTINUATION half neither of those tests reach (both stop at the refusal;
+ * neither goes on to clear the condition and confirm materialization
+ * completes normally afterward).
+ *
+ * Rigor: builds TWO independent members (A: promoted mid-backlog: role set
+ * CTRL_PRIMARY the same way every other test in this file simulates
+ * promotion (t42_make_leader), THEN drained via persist_apply_pump_run_sync
+ * -- the same function production promotion invokes; B: never promoted,
+ * drained via the ordinary persist_apply_pump_run()) starting from the
+ * IDENTICAL committed marker + config bytes, and asserts their resulting
+ * checkpoints are BYTE-IDENTICAL (snapdiff_paths) -- not merely that both
+ * "also complete," but that they complete IDENTICALLY, mirroring bug-857's
+ * own two-member convergence-proof style (test_bug857_regression.c).
+ */
+static void test_leadership_change_mid_materialization_completes_via_same_generic_apply_drain(void)
+{
+    char tmp_a[256], tmp_b[256];
+    strcpy(tmp_a, "/tmp/xce_cr87s2_a_XXXXXX");
+    strcpy(tmp_b, "/tmp/xce_cr87s2_b_XXXXXX");
+    TEST_ASSERT_NOT_NULL(mkdtemp(tmp_a));
+    TEST_ASSERT_NOT_NULL(mkdtemp(tmp_b));
+
+    cluster_t *c_a = t42_make_cluster(tmp_a);
+    cluster_t *c_b = t42_make_cluster(tmp_b);
+    TEST_ASSERT_NOT_NULL(c_a);
+    TEST_ASSERT_NOT_NULL(c_b);
+
+    myconf_t *mc_a = t43_attach_myconf(c_a, tmp_a);
+    myconf_t *mc_b = t43_attach_myconf(c_b, tmp_b);
+
+    /* Identical config bytes on both members (t43_make_config_blob is
+     * documented byte-identical across calls -- exactly what this test
+     * wants, since A and B must start from the SAME upgrade content for a
+     * byte-identical-outcome comparison to be meaningful). */
+    size_t   blen = 0;
+    uint8_t *bytes = t43_make_config_blob(&blen);
+    uint8_t  sha[32];
+    SHA256(bytes, blen, sha);
+    uint64_t tag; { uint64_t tmp; memcpy(&tmp, sha, 8); tag = be64toh(tmp); }
+
+    int rc_save_a = conf_save_config(mc_a, bytes, blen, sha);
+    int rc_save_b = conf_save_config(mc_b, bytes, blen, sha);
+    free(bytes);
+
+    uint8_t marker_payload[WAL_CONFIG_MARKER_PAYLOAD_LEN];
+    size_t  out_len = 0;
+    int rc_encode = config_marker_encode(tag, sha, marker_payload,
+                                         WAL_CONFIG_MARKER_PAYLOAD_LEN,
+                                         &out_len);
+
+    /* Append the SAME marker directly to each member's physical WAL
+     * (persist_wal_async, bypassing config_append_marker/commit_submit's
+     * commit-engine/fsync-worker machinery entirely) and advance
+     * committed_lsn directly -- mirrors test_bug857_regression.c's own
+     * Scenario 2 technique exactly, chosen for the identical reason: it is
+     * the only DETERMINISTIC way to leave a marker genuinely committed but
+     * NOT YET applied (a real apply backlog) without racing this fixture's
+     * background fsync_worker thread, which a commit_submit(WAIT) round
+     * trip cannot guarantee (see t42_wait_applied's own doc comment on this
+     * exact race). */
+    int rc_wal_a = persist_wal_async(&c_a->persist, WAL_CONFIG_MARKER,
+                                     marker_payload,
+                                     WAL_CONFIG_MARKER_PAYLOAD_LEN);
+    uint64_t marker_lsn_a = c_a->persist.current_lsn - 1;
+    uint64_t committed_a  = persist_committed_lsn_advance_atomic(
+        &c_a->persist, marker_lsn_a);
+
+    int rc_wal_b = persist_wal_async(&c_b->persist, WAL_CONFIG_MARKER,
+                                     marker_payload,
+                                     WAL_CONFIG_MARKER_PAYLOAD_LEN);
+    uint64_t marker_lsn_b = c_b->persist.current_lsn - 1;
+    uint64_t committed_b  = persist_committed_lsn_advance_atomic(
+        &c_b->persist, marker_lsn_b);
+
+    /* Precondition (validates this is genuinely the "backlog, never yet
+     * evaluated" case, not the "already fenced" case 6.2-a covers): neither
+     * member has applied anything yet, and neither fence has ever been
+     * evaluated for this marker, so both read unfenced by construction. */
+    uint64_t last_applied_before_a = persist_last_applied_lsn(&c_a->persist);
+    uint64_t last_applied_before_b = persist_last_applied_lsn(&c_b->persist);
+    int      fenced_before_a = upgrade_materialization_fenced(c_a);
+    int      fenced_before_b = upgrade_materialization_fenced(c_b);
+
+    /* A: "leadership change" -- promoted via this fixture family's own
+     * established promotion-simulation call (t42_make_leader; every test in
+     * this file uses it for this purpose except the ones specifically
+     * proving the real election-time GATE itself, which this test does not
+     * re-prove -- see header comment). Confirm becoming primary alone has NO
+     * bearing on the fence (still reads unfenced immediately after the role
+     * flip -- nothing leader-specific touches it), THEN drive the exact
+     * function production promotion uses to drain a backlog
+     * (persist_apply_pump_run_sync, ha.c:1878). */
+    t42_make_leader(c_a, /*term*/60, /*n_peers*/1);
+    int32_t role_after_promotion =
+        atomic_load_explicit(&c_a->ha.role, memory_order_acquire);
+    int fenced_immediately_after_promotion = upgrade_materialization_fenced(c_a);
+    persist_apply_pump_run_sync(&c_a->persist);
+
+    /* B: no leadership change at all -- drained via the ordinary,
+     * always-available apply-pump entry point every non-promoted member
+     * (standby or an already-serving primary catching up its own backlog)
+     * uses identically. */
+    int applied_b = persist_apply_pump_run(&c_b->persist);
+
+    uint64_t last_applied_after_a = persist_last_applied_lsn(&c_a->persist);
+    uint64_t last_applied_after_b = persist_last_applied_lsn(&c_b->persist);
+    int      fenced_after_a = upgrade_materialization_fenced(c_a);
+    int      fenced_after_b = upgrade_materialization_fenced(c_b);
+
+    uint64_t gen_a = mc_a->cluster_config ? mc_a->cluster_config->config_generation : 0;
+    uint64_t gen_b = mc_b->cluster_config ? mc_b->cluster_config->config_generation : 0;
+    bool     cluster_config_a_present = (mc_a->cluster_config != NULL);
+    bool     cluster_config_b_present = (mc_b->cluster_config != NULL);
+
+    char snap_a[4096], snap_b[4096];
+    snprintf(snap_a, sizeof(snap_a), "%s/snap-%016" PRIx64 ".capnp",
+             c_a->persist.snap_path, marker_lsn_a);
+    snprintf(snap_b, sizeof(snap_b), "%s/snap-%016" PRIx64 ".capnp",
+             c_b->persist.snap_path, marker_lsn_b);
+    bool snap_a_exists = (access(snap_a, F_OK) == 0);
+    bool snap_b_exists = (access(snap_b, F_OK) == 0);
+
+    snapdiff_report_t diff_rep;
+    memset(&diff_rep, 0, sizeof(diff_rep));
+    int diff_rc = snapdiff_paths(snap_a, snap_b, NULL, &diff_rep);
+
+    t43_free_myconf(c_a);
+    t43_free_myconf(c_b);
+    t42_teardown(c_a);
+    t42_teardown(c_b);
+
+    /* ── Assertions ──────────────────────────────────────────────────── */
+
+    TEST_ASSERT_EQUAL_INT(0, rc_save_a);
+    TEST_ASSERT_EQUAL_INT(0, rc_save_b);
+    TEST_ASSERT_EQUAL_INT(0, rc_encode);
+    TEST_ASSERT_EQUAL_INT(0, rc_wal_a);
+    TEST_ASSERT_EQUAL_INT(0, rc_wal_b);
+    TEST_ASSERT_EQUAL_UINT64(marker_lsn_a, committed_a);
+    TEST_ASSERT_EQUAL_UINT64(marker_lsn_b, committed_b);
+    TEST_ASSERT_EQUAL_UINT64_MESSAGE(marker_lsn_a, marker_lsn_b,
+        "both members must start from the SAME marker lsn for a "
+        "byte-identical-outcome comparison to be meaningful");
+
+    TEST_ASSERT_EQUAL_UINT64_MESSAGE(0, last_applied_before_a,
+        "precondition: member A must not have applied anything yet");
+    TEST_ASSERT_EQUAL_UINT64_MESSAGE(0, last_applied_before_b,
+        "precondition: member B must not have applied anything yet");
+    TEST_ASSERT_EQUAL_INT_MESSAGE(0, fenced_before_a,
+        "precondition: an unevaluated backlog must NOT already be fenced -- "
+        "this is genuinely the case existing 6.2 tests don't cover, "
+        "distinct from a fence already set by a prior checkpoint miss");
+    TEST_ASSERT_EQUAL_INT(0, fenced_before_b);
+
+    TEST_ASSERT_EQUAL_INT_MESSAGE((int32_t)CTRL_PRIMARY, role_after_promotion,
+        "the promotion simulation must actually take effect");
+    TEST_ASSERT_EQUAL_INT_MESSAGE(0, fenced_immediately_after_promotion,
+        "becoming primary must have NO bearing on the fence by itself -- no "
+        "leader-specific code sets or clears it as a side effect of the "
+        "role transition");
+
+    TEST_ASSERT_TRUE_MESSAGE(last_applied_after_a >= marker_lsn_a,
+        "Req 5.4: the SAME generic sync-drain function production "
+        "promotion invokes (persist_apply_pump_run_sync) must complete "
+        "materialization for the backlogged marker -- no special resume "
+        "logic needed, none exists");
+    TEST_ASSERT_EQUAL_INT_MESSAGE(1, applied_b,
+        "control: the non-promoted member's ordinary apply pump must "
+        "apply exactly the one backlogged marker entry");
+    TEST_ASSERT_TRUE_MESSAGE(last_applied_after_b >= marker_lsn_b,
+        "control: the non-promoted member's ordinary apply pump must "
+        "complete materialization identically");
+    TEST_ASSERT_EQUAL_INT_MESSAGE(0, fenced_after_a,
+        "member A's materialization must be genuinely verified complete "
+        "(fence clears), not merely attempted");
+    TEST_ASSERT_EQUAL_INT(0, fenced_after_b);
+
+    TEST_ASSERT_TRUE_MESSAGE(cluster_config_a_present,
+        "member A's config must have activated as part of the same "
+        "marker-apply dispatch, exactly as it does for any non-promoted "
+        "member (bug-1469)");
+    TEST_ASSERT_TRUE(cluster_config_b_present);
+    TEST_ASSERT_EQUAL_UINT64(1, gen_a); /* t43_make_config_blob's hardcoded generation */
+    TEST_ASSERT_EQUAL_UINT64(gen_b, gen_a);
+
+    TEST_ASSERT_TRUE_MESSAGE(snap_a_exists,
+        "member A must have produced a checkpoint at the marker's own lsn");
+    TEST_ASSERT_TRUE_MESSAGE(snap_b_exists,
+        "member B must have produced a checkpoint at the marker's own lsn");
+
+    /* THE decisive proof: materialization triggered by a leadership change
+     * mid-backlog produces EXACTLY the same artifact as an ordinary,
+     * never-promoted member's continued apply of the identical marker --
+     * "completes identically to the non-promoted case," not merely "also
+     * completes." */
+    TEST_ASSERT_EQUAL_INT_MESSAGE(0, diff_rc,
+        "Req 5.4: a leadership-change-driven materialization drain must "
+        "produce a checkpoint byte-identical to the non-promoted member's");
+    TEST_ASSERT_TRUE_MESSAGE(diff_rep.identical,
+        "no leader-specific divergence exists in the materialized output");
+
+    printf("test_leadership_change_mid_materialization_completes_via_"
+           "same_generic_apply_drain: PASS\n");
+}
+
 /* ── wait_all_with_timeout (XCE_TESTING shim) ───── */
 
 #ifdef XCE_TESTING
@@ -5305,6 +5560,9 @@ void test_config_repl(void)
     /* bug-1469: WAL_CONFIG_MARKER apply must activate config on the
      * ORIGINATING member too, not just receiving standbys. */
     RUN_TEST(test_wal_config_marker_apply_activates_originating_member_config);
+    /* config-upgrade-snapshot task 8.7 Scenario 2: leadership change
+     * mid-materialization completes via the same generic apply-drain path. */
+    RUN_TEST(test_leadership_change_mid_materialization_completes_via_same_generic_apply_drain);
 #ifdef XCE_TESTING
     RUN_TEST(test_load_primary_config_missing_returns_error);
     RUN_TEST(test_wait_all_with_timeout_slow_threads);

src/controller/testsuite/test_upgrade_snapshot_apply.c

@@ -78,6 +78,9 @@
 #include "repl.h"     /* repl_add_standby / repl_on_ack -- Gap-1 WAL-trim-
                         * under-marker-trigger test (task 8.1, Req 7.4),
                         * mirrors test_e2e_checkpoint.c's own include */
+#include "snapverify.h" /* snapverify_path -- task 8.7 Scenario 1: verify the
+                        * restart-produced checkpoint is structurally
+                        * well-formed, not merely present */
 #include "typemap.h"  /* wal_catchup_entry_t -- repl_broadcast_send_hook's
                         * signature */
 #include "wal_types.h"
@@ -852,6 +855,204 @@ static void test_marker_checkpoint_fault_hook_gates_checkpoint_call(void)
            "PASS\n");
 }
 
+/* ── 9: crash before checkpoint durable, restart resumes materialization
+ *      (task 8.7 Scenario 1, Req 5.2, 5.3, 5.5) ─────────────────────────
+ *
+ * Builds directly on test 8's fault-hook seam above, whose own header
+ * comment is explicit that it proves ONLY the seam itself -- "not the full
+ * crash/restart regression -- that is task 8.7's job, which depends on
+ * this seam landing first." This test is that job.
+ *
+ * Phase 1 ("crash"): drive the marker's apply through the live pump with
+ * the fault hook forcing "the checkpoint write never completed" -- the
+ * same mechanics test 8 already proves in isolation (apply_rc ==
+ * -ECANCELED, no snapshot at the marker's lsn, last_applied_lsn stuck
+ * below it). This test re-confirms that baseline lightly (not exhaustively
+ * -- test 8 already owns that) and then goes one step further: physically
+ * tear down this persist_t/cluster_t WITHOUT deleting the on-disk tmpdir,
+ * mirroring test 6's own phase-1/phase-2 split exactly (test 6 is this
+ * same file's established pattern for simulating "the process exits and a
+ * fresh one boots against the same on-disk state" -- persist_stop/
+ * persist_fini inline, no rm -rf until the very end).
+ *
+ * Phase 2 ("restart succeeds this time"): clear the fault hook (the
+ * transient condition that caused the crash is gone), boot a genuinely
+ * FRESH persist_t + cluster_t against the SAME tmpdir/name, and call
+ * persist_recover() -- boot replay, the SECOND real production call site
+ * of apply_committed_entry() (test 6 above already proves this is a valid,
+ * lock-free calling convention for the marker case). A bare re-call of
+ * persist_apply_pump_run() on the SAME in-memory instance was deliberately
+ * rejected in favor of this: last_applied_lsn never advanced past the
+ * marker in phase 1, so a same-process re-call would only prove "this
+ * process retries," not "a genuinely restarted process resumes from
+ * durable state" -- persist_recover() against a fresh instance is the
+ * faithful simulation Requirement 5.2 actually describes. Because
+ * checkpoint_lsn was never advanced past 0 in phase 1 (no checkpoint ever
+ * wrote), replay restarts from lsn 1 and re-drives both the noop and the
+ * marker through apply_committed_entry() fresh -- this time succeeding.
+ *
+ * Phase 3 (idempotent re-restart, Req 5.3 in this exact crash/retry
+ * shape): call persist_recover() a SECOND time on the already-recovered
+ * instance (test_persist_recovery.c's own test_recover_from_empty_state
+ * documents this as valid: "persist_recover() is valid to call on a
+ * running instance too") -- proves a second, redundant "restart" after
+ * materialization already succeeded does not duplicate or corrupt the
+ * checkpoint. test_marker_apply_retry_is_idempotent (test 3 above) already
+ * covers a DIFFERENT retry shape -- a bare, same-process direct
+ * re-dispatch of apply_committed_entry(), never a real boot replay -- so
+ * this is not a duplicate of that coverage, it closes the boot-replay-
+ * specific shape that test doesn't touch.
+ *
+ * Discipline note: per this file's own header, every assertion-relevant
+ * observable across all three phases is gathered into a local variable
+ * FIRST; no TEST_ASSERT runs until every phase's teardown has already
+ * happened (avoids the documented longjmp-past-teardown pathology).
+ */
+
+static void
+test_marker_checkpoint_fault_hook_crash_then_restart_resumes_materialization(void)
+{
+    extern int (*persist_marker_checkpoint_fault_hook)(persist_t* p,
+                                                        uint64_t marker_lsn);
+    tusa_t t;
+    tusa_setup(&t);
+
+    memset(&g_tusa_fault_obs, 0, sizeof(g_tusa_fault_obs));
+    persist_marker_checkpoint_fault_hook = tusa_marker_fault_hook;
+
+    tusa_append_result_t ar = tusa_append_noop_then_marker(&t.p, 0x8A);
+    int crash_apply_rc = persist_apply_pump_run(&t.p);
+
+    bool     crash_snap_exists  = tusa_snapshot_exists(&t.p, ar.marker_lsn);
+    uint64_t crash_last_applied = atomic_load_explicit(
+        &t.p.last_applied_lsn, memory_order_acquire);
+    int      crash_fault_calls  = g_tusa_fault_obs.calls;
+
+    /* "Crash": tear down this instance WITHOUT deleting the on-disk tmpdir
+     * -- mirrors test 6's own phase-1 teardown exactly (persist_stop/
+     * persist_fini + mgr cleanup inline, no tusa_teardown/rm -rf until the
+     * very end of this test, after phase 2/3 have had a chance to read the
+     * same directory). */
+    persist_stop(&t.p);
+    persist_fini(&t.p);
+    zarray_fini(&t.c.sharedinst);
+    if (t.c.resmgr)    dereference(t.c.resmgr);
+    if (t.c.workermgr) dereference(t.c.workermgr);
+    if (t.c.qmgr)      dereference(t.c.qmgr);
+    if (t.c.jobmgr)    dereference(t.c.jobmgr);
+    xce_hook_fini(&t.c.hooksys);
+
+    /* "Restart succeeds this time": the transient condition that caused
+     * the crash is gone. */
+    persist_marker_checkpoint_fault_hook = NULL;
+
+    /* Phase 2: fresh persist_t + fresh cluster_t, SAME tmpdir/name as phase
+     * 1 -- boot replay (persist_recover -> replay_wal), mirrors test 6's
+     * own phase-2 construction exactly. */
+    kvmgr_params_t int_params = { .key_type = KVMGR_KEY_INT,
+                                  .nslots   = 32, .delta = 16 };
+    kvmgr_params_t str_params = { .key_type = KVMGR_KEY_STRING,
+                                  .nslots   = 32, .delta = 16 };
+    tusa_t t2;
+    memset(&t2, 0, sizeof(t2));
+    strncpy(t2.tmpdir, t.tmpdir, sizeof(t2.tmpdir) - 1);
+    t2.c.myhost           = &t2.c.myhost_info;
+    t2.c.myhost_info.name = (char*)"ctrl-test-restart";
+    t2.c.myhost_info.key  = (char*)"";
+    int rc_hook2 = xce_hook_init(&t2.c.hooksys, XCE_CONTROLLER_HOOK_MAX);
+    t2.c.jobmgr    = makeInstance(&xkvmgr_type, &int_params);
+    t2.c.qmgr      = makeInstance(&xkvmgr_type, &str_params);
+    t2.c.workermgr = makeInstance(&xkvmgr_type, &str_params);
+    t2.c.resmgr    = makeInstance(&xkvmgr_type, &str_params);
+    zarray_init(&t2.c.sharedinst, 16, NULL);
+
+    persist_config_t cfg2 = persist_config_defaults();
+    cfg2.wal_sync_ms = 5000;
+    cfg2.snap_keep_n = 4;
+    persist_set_workdir(&t2.p, t.tmpdir, "tusa");
+    int rc_init2  = persist_init(&t2.p, &cfg2, &t2.c);
+    int rc_start2 = persist_start(&t2.p);
+
+    int rc_recover = persist_recover(&t2.p);
+
+    uint64_t restart_checkpoint_lsn = t2.p.checkpoint_lsn;
+    bool     restart_snap_exists    = tusa_snapshot_exists(&t2.p, ar.marker_lsn);
+    uint64_t restart_last_applied   = atomic_load_explicit(
+        &t2.p.last_applied_lsn, memory_order_acquire);
+    int      restart_fault_calls    = g_tusa_fault_obs.calls;
+
+    char snap_path[4096];
+    snprintf(snap_path, sizeof(snap_path), "%s/snap-%016" PRIx64 ".capnp",
+             t2.p.snap_path, ar.marker_lsn);
+    snapverify_report_t verify_rep;
+    memset(&verify_rep, 0, sizeof(verify_rep));
+    int verify_rc = snapverify_path(snap_path, NULL, &verify_rep);
+
+    /* Phase 3: a SECOND, redundant "restart" after materialization already
+     * succeeded -- proves idempotency in this exact crash/retry shape. */
+    int      rc_recover_again = persist_recover(&t2.p);
+    uint64_t checkpoint_lsn_after_second_recover = t2.p.checkpoint_lsn;
+    int      snap_count_after_second_recover = tusa_count_snapshots(&t2.p);
+
+    tusa_teardown(&t2);
+
+    /* ── Assertions (only now, after every phase's teardown) ───────────── */
+
+    TEST_ASSERT_EQUAL_INT(0, ar.rc_encode);
+    TEST_ASSERT_EQUAL_INT(0, ar.rc_noop);
+    TEST_ASSERT_EQUAL_INT(0, ar.rc_marker);
+    TEST_ASSERT_EQUAL_INT_MESSAGE(1, crash_fault_calls,
+        "fault hook must have fired exactly once to force the simulated "
+        "crash");
+    TEST_ASSERT_EQUAL_INT_MESSAGE(-ECANCELED, crash_apply_rc,
+        "the simulated crash must surface as the marker's apply failure");
+    TEST_ASSERT_FALSE_MESSAGE(crash_snap_exists,
+        "crash state: no checkpoint must exist at the marker's lsn yet");
+    TEST_ASSERT_EQUAL_UINT64_MESSAGE(ar.noop_lsn, crash_last_applied,
+        "crash state: last_applied_lsn must be stuck below the marker");
+
+    TEST_ASSERT_EQUAL_INT(0, rc_hook2);
+    TEST_ASSERT_NOT_NULL(t2.c.jobmgr);
+    TEST_ASSERT_NOT_NULL(t2.c.qmgr);
+    TEST_ASSERT_NOT_NULL(t2.c.workermgr);
+    TEST_ASSERT_NOT_NULL(t2.c.resmgr);
+    TEST_ASSERT_EQUAL_INT(0, rc_init2);
+    TEST_ASSERT_EQUAL_INT(0, rc_start2);
+    TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc_recover,
+        "restart's boot replay must succeed now that the fault clears");
+    TEST_ASSERT_EQUAL_INT_MESSAGE(1, restart_fault_calls,
+        "the fault hook must NOT fire again during restart -- it was "
+        "cleared, simulating the transient condition having resolved");
+
+    TEST_ASSERT_EQUAL_UINT64_MESSAGE(ar.marker_lsn, restart_checkpoint_lsn,
+        "Req 5.2: restart must resume materialization AT THE MARKER, not "
+        "some other position");
+    TEST_ASSERT_TRUE_MESSAGE(restart_snap_exists,
+        "Req 5.2/5.5: restart must produce a durable checkpoint at the "
+        "marker's own lsn");
+    TEST_ASSERT_TRUE_MESSAGE(restart_last_applied >= ar.marker_lsn,
+        "Req 5.2: restart must advance last_applied_lsn past the marker "
+        "-- materialization resumed and completed, not re-stuck");
+
+    TEST_ASSERT_EQUAL_INT_MESSAGE(0, verify_rc,
+        "Req 5.5: the restart-produced checkpoint must be structurally "
+        "well-formed, not a partial/corrupt write");
+    TEST_ASSERT_TRUE_MESSAGE(verify_rep.ok, verify_rep.fail_reason);
+    TEST_ASSERT_EQUAL_UINT64(ar.marker_lsn, verify_rep.checkpoint_lsn);
+
+    TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc_recover_again,
+        "Req 5.3 in the crash/retry shape: a second, redundant restart "
+        "after materialization already succeeded must not error");
+    TEST_ASSERT_EQUAL_UINT64_MESSAGE(ar.marker_lsn,
+        checkpoint_lsn_after_second_recover,
+        "a redundant restart must not relabel or move the checkpoint");
+    TEST_ASSERT_EQUAL_INT_MESSAGE(1, snap_count_after_second_recover,
+        "a redundant restart must not duplicate the checkpoint file");
+
+    printf("test_marker_checkpoint_fault_hook_crash_then_restart_resumes_"
+           "materialization: PASS\n");
+}
+
 /* ── Dispatch ────────────────────────────────────────────────────────── */
 
 void test_upgrade_snapshot_apply(void)
@@ -867,4 +1068,5 @@ void test_upgrade_snapshot_apply(void)
     RUN_TEST(test_marker_apply_via_replay_checkpoints_at_marker_own_lsn);
     RUN_TEST(test_marker_triggered_checkpoint_trim_preserves_standby_mid_catchup_segments);
     RUN_TEST(test_marker_checkpoint_fault_hook_gates_checkpoint_call);
+    RUN_TEST(test_marker_checkpoint_fault_hook_crash_then_restart_resumes_materialization);
 }