lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

test(config-upgrade-snapshot): fault-inject crash/restart + leadership change (task 8.7)

Browse source 
Prove Requirements 5.2/5.4/5.5/6.3 via deterministic fault injection at
the two critical upgrade windows, entirely at the unit level using the
XCE_TESTING hooks earlier tasks built for exactly this purpose -- no
live soak needed.

test_marker_checkpoint_fault_hook_crash_then_restart_resumes_materialization
extends the existing fault-hook seam test: after simulating a crash
before the checkpoint write durably completes, it boots a genuinely
fresh persist_t/cluster_t against the same on-disk WAL via
persist_recover() (a real boot-replay simulation, not a same-process
re-call) and confirms materialization resumes and completes correctly.

test_leadership_change_mid_materialization_completes_via_same_generic_apply_drain
proves a promoted member's materialization completes identically to a
non-promoted control member's, by draining each through its actual
production entry point and comparing checkpoints byte-for-byte -- a
regression guard against any future role-conditional branch creeping
into the marker-apply path.
This commit is contained in:
Rongsong Shen 2026-07-25 18:20:43 +00:00
parent b5fde32033
commit d4af605fa6
3 changed files with 463 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

258
src/controller/testsuite/test_config_repl.c
View file

@ -29,6 +29,10 @@
#include "persist.h"
#include "repl.h"
#include "request.h"
#include "snapdiff.h" /* snapdiff_paths -- task 8.7 Scenario 2: prove a
* leadership-change-driven materialization drain
* produces a byte-identical checkpoint to an ordinary,
* non-promoted drain of the same marker */
#include "typemap.h"
#include "wal_types.h"
#include "xce/cluster_def.h"
@ -2487,6 +2491,257 @@ static void test_wal_config_marker_apply_activates_originating_member_config(voi
printf("test_wal_config_marker_apply_activates_originating_member_config: PASS\n");
}
/* ── config-upgrade-snapshot task 8.7 Scenario 2: leadership change mid-
* materialization does not special-case, and completes via the SAME generic
* apply-drain path (Req 5.4) ────────────────────────────────────────────
*
* design.md's own "Key simplification found during design" (persist.c's
* WAL_CONFIG_MARKER case doc comment, apply_committed_entry): "no new
* leader-specific 'resume' logic is needed either, since a newly-elected
* leader satisfies this the same way, by continuing to apply its own
* committed log." This test proves that claim structurally rather than
* asserting it by fiat.
*
* The concrete mechanism (read directly out of raft_cb_become_leader,
* ha.c:1516-1979): the promotion-time eligibility gate
* (upgrade_materialization_fenced(c), ha.c:1763) only refuses promotion when
* the fence has ALREADY been set by a prior upgrade_materialization_verify()
* call. A marker that is committed but has never yet been through
* apply_committed_entry() at all -- a genuine apply backlog, not a checkpoint
* failure -- leaves the fence at its zero-initialized (unfenced) default,
* because nothing has evaluated it yet. So promotion is NOT refused in this
* case. What actually drains the backlog is ha.c:1878's
* persist_apply_pump_run_sync(&c->persist) call, unconditionally issued right
* after role flips to CTRL_PRIMARY -- the SAME synchronous drain function
* bug-1043's job-backlog fix added for an entirely unrelated reason (a
* promoted standby's un-applied committed jobs must be visible immediately).
* The WAL_CONFIG_MARKER case (persist.c) rides that pre-existing, generic
* drain for free: no leadership-specific config-upgrade code exists anywhere
* in the promotion path.
*
* This test drives that exact mechanism directly
* (persist_apply_pump_run_sync) rather than the full
* raft_cb_become_leader_with_cluster_for_test() tail, deliberately: the rest
* of that function's post-drain tail (quorum-transition logging, config push,
* ctrlhello, persist-hook registration, heartbeat start, own-term gate arm)
* is orthogonal to Requirement 5.4's materialization-continuation claim and
* would require a substantially larger, novel fixture (full ha_init +
* delaymgr_init, mirroring test_convergence_verification.c's cv_setup --
* this file's own t42_/t43_ fixtures deliberately do NOT wire that, since no
* other test in this file drives promotion past the fence-refusal point) to
* exercise safely -- a scope expansion this test-only task should not take
* on. The GATING half of Req 5.4 (a member whose fence IS already set
* refuses promotion) is already exhaustively covered by
* test_upgrade_materialization_fence_enforced_by_dispatch_gate_and_promotion
* (6.2-a) and
* test_wal_config_marker_apply_sets_fence_via_real_checkpoint_miss (6.2
* remediation) above, both via the real raft_cb_become_leader_with_cluster_
* for_test() gate -- this test does not repeat that proof, it closes the
* CONTINUATION half neither of those tests reach (both stop at the refusal;
* neither goes on to clear the condition and confirm materialization
* completes normally afterward).
*
* Rigor: builds TWO independent members (A: promoted mid-backlog: role set
* CTRL_PRIMARY the same way every other test in this file simulates
* promotion (t42_make_leader), THEN drained via persist_apply_pump_run_sync
* -- the same function production promotion invokes; B: never promoted,
* drained via the ordinary persist_apply_pump_run()) starting from the
* IDENTICAL committed marker + config bytes, and asserts their resulting
* checkpoints are BYTE-IDENTICAL (snapdiff_paths) -- not merely that both
* "also complete," but that they complete IDENTICALLY, mirroring bug-857's
* own two-member convergence-proof style (test_bug857_regression.c).
*/
static void test_leadership_change_mid_materialization_completes_via_same_generic_apply_drain(void)
{
char tmp_a[256], tmp_b[256];
strcpy(tmp_a, "/tmp/xce_cr87s2_a_XXXXXX");
strcpy(tmp_b, "/tmp/xce_cr87s2_b_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp_a));
TEST_ASSERT_NOT_NULL(mkdtemp(tmp_b));
cluster_t *c_a = t42_make_cluster(tmp_a);
cluster_t *c_b = t42_make_cluster(tmp_b);
TEST_ASSERT_NOT_NULL(c_a);
TEST_ASSERT_NOT_NULL(c_b);
myconf_t *mc_a = t43_attach_myconf(c_a, tmp_a);
myconf_t *mc_b = t43_attach_myconf(c_b, tmp_b);
/* Identical config bytes on both members (t43_make_config_blob is
* documented byte-identical across calls -- exactly what this test
* wants, since A and B must start from the SAME upgrade content for a
* byte-identical-outcome comparison to be meaningful). */
size_t blen = 0;
uint8_t *bytes = t43_make_config_blob(&blen);
uint8_t sha[32];
SHA256(bytes, blen, sha);
uint64_t tag; { uint64_t tmp; memcpy(&tmp, sha, 8); tag = be64toh(tmp); }
int rc_save_a = conf_save_config(mc_a, bytes, blen, sha);
int rc_save_b = conf_save_config(mc_b, bytes, blen, sha);
free(bytes);
uint8_t marker_payload[WAL_CONFIG_MARKER_PAYLOAD_LEN];
size_t out_len = 0;
int rc_encode = config_marker_encode(tag, sha, marker_payload,
WAL_CONFIG_MARKER_PAYLOAD_LEN,
&out_len);
/* Append the SAME marker directly to each member's physical WAL
* (persist_wal_async, bypassing config_append_marker/commit_submit's
* commit-engine/fsync-worker machinery entirely) and advance
* committed_lsn directly -- mirrors test_bug857_regression.c's own
* Scenario 2 technique exactly, chosen for the identical reason: it is
* the only DETERMINISTIC way to leave a marker genuinely committed but
* NOT YET applied (a real apply backlog) without racing this fixture's
* background fsync_worker thread, which a commit_submit(WAIT) round
* trip cannot guarantee (see t42_wait_applied's own doc comment on this
* exact race). */
int rc_wal_a = persist_wal_async(&c_a->persist, WAL_CONFIG_MARKER,
marker_payload,
WAL_CONFIG_MARKER_PAYLOAD_LEN);
uint64_t marker_lsn_a = c_a->persist.current_lsn - 1;
uint64_t committed_a = persist_committed_lsn_advance_atomic(
&c_a->persist, marker_lsn_a);
int rc_wal_b = persist_wal_async(&c_b->persist, WAL_CONFIG_MARKER,
marker_payload,
WAL_CONFIG_MARKER_PAYLOAD_LEN);
uint64_t marker_lsn_b = c_b->persist.current_lsn - 1;
uint64_t committed_b = persist_committed_lsn_advance_atomic(
&c_b->persist, marker_lsn_b);
/* Precondition (validates this is genuinely the "backlog, never yet
* evaluated" case, not the "already fenced" case 6.2-a covers): neither
* member has applied anything yet, and neither fence has ever been
* evaluated for this marker, so both read unfenced by construction. */
uint64_t last_applied_before_a = persist_last_applied_lsn(&c_a->persist);
uint64_t last_applied_before_b = persist_last_applied_lsn(&c_b->persist);
int fenced_before_a = upgrade_materialization_fenced(c_a);
int fenced_before_b = upgrade_materialization_fenced(c_b);
/* A: "leadership change" -- promoted via this fixture family's own
* established promotion-simulation call (t42_make_leader; every test in
* this file uses it for this purpose except the ones specifically
* proving the real election-time GATE itself, which this test does not
* re-prove -- see header comment). Confirm becoming primary alone has NO
* bearing on the fence (still reads unfenced immediately after the role
* flip -- nothing leader-specific touches it), THEN drive the exact
* function production promotion uses to drain a backlog
* (persist_apply_pump_run_sync, ha.c:1878). */
t42_make_leader(c_a, /*term*/60, /*n_peers*/1);
int32_t role_after_promotion =
atomic_load_explicit(&c_a->ha.role, memory_order_acquire);
int fenced_immediately_after_promotion = upgrade_materialization_fenced(c_a);
persist_apply_pump_run_sync(&c_a->persist);
/* B: no leadership change at all -- drained via the ordinary,
* always-available apply-pump entry point every non-promoted member
* (standby or an already-serving primary catching up its own backlog)
* uses identically. */
int applied_b = persist_apply_pump_run(&c_b->persist);
uint64_t last_applied_after_a = persist_last_applied_lsn(&c_a->persist);
uint64_t last_applied_after_b = persist_last_applied_lsn(&c_b->persist);
int fenced_after_a = upgrade_materialization_fenced(c_a);
int fenced_after_b = upgrade_materialization_fenced(c_b);
uint64_t gen_a = mc_a->cluster_config ? mc_a->cluster_config->config_generation : 0;
uint64_t gen_b = mc_b->cluster_config ? mc_b->cluster_config->config_generation : 0;
bool cluster_config_a_present = (mc_a->cluster_config != NULL);
bool cluster_config_b_present = (mc_b->cluster_config != NULL);
char snap_a[4096], snap_b[4096];
snprintf(snap_a, sizeof(snap_a), "%s/snap-%016" PRIx64 ".capnp",
c_a->persist.snap_path, marker_lsn_a);
snprintf(snap_b, sizeof(snap_b), "%s/snap-%016" PRIx64 ".capnp",
c_b->persist.snap_path, marker_lsn_b);
bool snap_a_exists = (access(snap_a, F_OK) == 0);
bool snap_b_exists = (access(snap_b, F_OK) == 0);
snapdiff_report_t diff_rep;
memset(&diff_rep, 0, sizeof(diff_rep));
int diff_rc = snapdiff_paths(snap_a, snap_b, NULL, &diff_rep);
t43_free_myconf(c_a);
t43_free_myconf(c_b);
t42_teardown(c_a);
t42_teardown(c_b);
/* ── Assertions ──────────────────────────────────────────────────── */
TEST_ASSERT_EQUAL_INT(0, rc_save_a);
TEST_ASSERT_EQUAL_INT(0, rc_save_b);
TEST_ASSERT_EQUAL_INT(0, rc_encode);
TEST_ASSERT_EQUAL_INT(0, rc_wal_a);
TEST_ASSERT_EQUAL_INT(0, rc_wal_b);
TEST_ASSERT_EQUAL_UINT64(marker_lsn_a, committed_a);
TEST_ASSERT_EQUAL_UINT64(marker_lsn_b, committed_b);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(marker_lsn_a, marker_lsn_b,
"both members must start from the SAME marker lsn for a "
"byte-identical-outcome comparison to be meaningful");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(0, last_applied_before_a,
"precondition: member A must not have applied anything yet");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(0, last_applied_before_b,
"precondition: member B must not have applied anything yet");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, fenced_before_a,
"precondition: an unevaluated backlog must NOT already be fenced -- "
"this is genuinely the case existing 6.2 tests don't cover, "
"distinct from a fence already set by a prior checkpoint miss");
TEST_ASSERT_EQUAL_INT(0, fenced_before_b);
TEST_ASSERT_EQUAL_INT_MESSAGE((int32_t)CTRL_PRIMARY, role_after_promotion,
"the promotion simulation must actually take effect");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, fenced_immediately_after_promotion,
"becoming primary must have NO bearing on the fence by itself -- no "
"leader-specific code sets or clears it as a side effect of the "
"role transition");
TEST_ASSERT_TRUE_MESSAGE(last_applied_after_a >= marker_lsn_a,
"Req 5.4: the SAME generic sync-drain function production "
"promotion invokes (persist_apply_pump_run_sync) must complete "
"materialization for the backlogged marker -- no special resume "
"logic needed, none exists");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, applied_b,
"control: the non-promoted member's ordinary apply pump must "
"apply exactly the one backlogged marker entry");
TEST_ASSERT_TRUE_MESSAGE(last_applied_after_b >= marker_lsn_b,
"control: the non-promoted member's ordinary apply pump must "
"complete materialization identically");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, fenced_after_a,
"member A's materialization must be genuinely verified complete "
"(fence clears), not merely attempted");
TEST_ASSERT_EQUAL_INT(0, fenced_after_b);
TEST_ASSERT_TRUE_MESSAGE(cluster_config_a_present,
"member A's config must have activated as part of the same "
"marker-apply dispatch, exactly as it does for any non-promoted "
"member (bug-1469)");
TEST_ASSERT_TRUE(cluster_config_b_present);
TEST_ASSERT_EQUAL_UINT64(1, gen_a); /* t43_make_config_blob's hardcoded generation */
TEST_ASSERT_EQUAL_UINT64(gen_b, gen_a);
TEST_ASSERT_TRUE_MESSAGE(snap_a_exists,
"member A must have produced a checkpoint at the marker's own lsn");
TEST_ASSERT_TRUE_MESSAGE(snap_b_exists,
"member B must have produced a checkpoint at the marker's own lsn");
/* THE decisive proof: materialization triggered by a leadership change
* mid-backlog produces EXACTLY the same artifact as an ordinary,
* never-promoted member's continued apply of the identical marker --
* "completes identically to the non-promoted case," not merely "also
* completes." */
TEST_ASSERT_EQUAL_INT_MESSAGE(0, diff_rc,
"Req 5.4: a leadership-change-driven materialization drain must "
"produce a checkpoint byte-identical to the non-promoted member's");
TEST_ASSERT_TRUE_MESSAGE(diff_rep.identical,
"no leader-specific divergence exists in the materialized output");
printf("test_leadership_change_mid_materialization_completes_via_"
"same_generic_apply_drain: PASS\n");
}
/* ── wait_all_with_timeout (XCE_TESTING shim) ───── */
#ifdef XCE_TESTING
@ -5305,6 +5560,9 @@ void test_config_repl(void)
/* bug-1469: WAL_CONFIG_MARKER apply must activate config on the
* ORIGINATING member too, not just receiving standbys. */
RUN_TEST(test_wal_config_marker_apply_activates_originating_member_config);
/* config-upgrade-snapshot task 8.7 Scenario 2: leadership change
* mid-materialization completes via the same generic apply-drain path. */
RUN_TEST(test_leadership_change_mid_materialization_completes_via_same_generic_apply_drain);
#ifdef XCE_TESTING
RUN_TEST(test_load_primary_config_missing_returns_error);
RUN_TEST(test_wait_all_with_timeout_slow_threads);

202
src/controller/testsuite/test_upgrade_snapshot_apply.c
View file

@ -78,6 +78,9 @@
#include "repl.h" /* repl_add_standby / repl_on_ack -- Gap-1 WAL-trim-
* under-marker-trigger test (task 8.1, Req 7.4),
* mirrors test_e2e_checkpoint.c's own include */
#include "snapverify.h" /* snapverify_path -- task 8.7 Scenario 1: verify the
* restart-produced checkpoint is structurally
* well-formed, not merely present */
#include "typemap.h" /* wal_catchup_entry_t -- repl_broadcast_send_hook's
* signature */
#include "wal_types.h"
@ -852,6 +855,204 @@ static void test_marker_checkpoint_fault_hook_gates_checkpoint_call(void)
"PASS\n");
}
/* ── 9: crash before checkpoint durable, restart resumes materialization
* (task 8.7 Scenario 1, Req 5.2, 5.3, 5.5) ─────────────────────────
*
* Builds directly on test 8's fault-hook seam above, whose own header
* comment is explicit that it proves ONLY the seam itself -- "not the full
* crash/restart regression -- that is task 8.7's job, which depends on
* this seam landing first." This test is that job.
*
* Phase 1 ("crash"): drive the marker's apply through the live pump with
* the fault hook forcing "the checkpoint write never completed" -- the
* same mechanics test 8 already proves in isolation (apply_rc ==
* -ECANCELED, no snapshot at the marker's lsn, last_applied_lsn stuck
* below it). This test re-confirms that baseline lightly (not exhaustively
* -- test 8 already owns that) and then goes one step further: physically
* tear down this persist_t/cluster_t WITHOUT deleting the on-disk tmpdir,
* mirroring test 6's own phase-1/phase-2 split exactly (test 6 is this
* same file's established pattern for simulating "the process exits and a
* fresh one boots against the same on-disk state" -- persist_stop/
* persist_fini inline, no rm -rf until the very end).
*
* Phase 2 ("restart succeeds this time"): clear the fault hook (the
* transient condition that caused the crash is gone), boot a genuinely
* FRESH persist_t + cluster_t against the SAME tmpdir/name, and call
* persist_recover() -- boot replay, the SECOND real production call site
* of apply_committed_entry() (test 6 above already proves this is a valid,
* lock-free calling convention for the marker case). A bare re-call of
* persist_apply_pump_run() on the SAME in-memory instance was deliberately
* rejected in favor of this: last_applied_lsn never advanced past the
* marker in phase 1, so a same-process re-call would only prove "this
* process retries," not "a genuinely restarted process resumes from
* durable state" -- persist_recover() against a fresh instance is the
* faithful simulation Requirement 5.2 actually describes. Because
* checkpoint_lsn was never advanced past 0 in phase 1 (no checkpoint ever
* wrote), replay restarts from lsn 1 and re-drives both the noop and the
* marker through apply_committed_entry() fresh -- this time succeeding.
*
* Phase 3 (idempotent re-restart, Req 5.3 in this exact crash/retry
* shape): call persist_recover() a SECOND time on the already-recovered
* instance (test_persist_recovery.c's own test_recover_from_empty_state
* documents this as valid: "persist_recover() is valid to call on a
* running instance too") -- proves a second, redundant "restart" after
* materialization already succeeded does not duplicate or corrupt the
* checkpoint. test_marker_apply_retry_is_idempotent (test 3 above) already
* covers a DIFFERENT retry shape -- a bare, same-process direct
* re-dispatch of apply_committed_entry(), never a real boot replay -- so
* this is not a duplicate of that coverage, it closes the boot-replay-
* specific shape that test doesn't touch.
*
* Discipline note: per this file's own header, every assertion-relevant
* observable across all three phases is gathered into a local variable
* FIRST; no TEST_ASSERT runs until every phase's teardown has already
* happened (avoids the documented longjmp-past-teardown pathology).
*/
static void
test_marker_checkpoint_fault_hook_crash_then_restart_resumes_materialization(void)
{
extern int (*persist_marker_checkpoint_fault_hook)(persist_t* p,
uint64_t marker_lsn);
tusa_t t;
tusa_setup(&t);
memset(&g_tusa_fault_obs, 0, sizeof(g_tusa_fault_obs));
persist_marker_checkpoint_fault_hook = tusa_marker_fault_hook;
tusa_append_result_t ar = tusa_append_noop_then_marker(&t.p, 0x8A);
int crash_apply_rc = persist_apply_pump_run(&t.p);
bool crash_snap_exists = tusa_snapshot_exists(&t.p, ar.marker_lsn);
uint64_t crash_last_applied = atomic_load_explicit(
&t.p.last_applied_lsn, memory_order_acquire);
int crash_fault_calls = g_tusa_fault_obs.calls;
/* "Crash": tear down this instance WITHOUT deleting the on-disk tmpdir
* -- mirrors test 6's own phase-1 teardown exactly (persist_stop/
* persist_fini + mgr cleanup inline, no tusa_teardown/rm -rf until the
* very end of this test, after phase 2/3 have had a chance to read the
* same directory). */
persist_stop(&t.p);
persist_fini(&t.p);
zarray_fini(&t.c.sharedinst);
if (t.c.resmgr) dereference(t.c.resmgr);
if (t.c.workermgr) dereference(t.c.workermgr);
if (t.c.qmgr) dereference(t.c.qmgr);
if (t.c.jobmgr) dereference(t.c.jobmgr);
xce_hook_fini(&t.c.hooksys);
/* "Restart succeeds this time": the transient condition that caused
* the crash is gone. */
persist_marker_checkpoint_fault_hook = NULL;
/* Phase 2: fresh persist_t + fresh cluster_t, SAME tmpdir/name as phase
* 1 -- boot replay (persist_recover -> replay_wal), mirrors test 6's
* own phase-2 construction exactly. */
kvmgr_params_t int_params = { .key_type = KVMGR_KEY_INT,
.nslots = 32, .delta = 16 };
kvmgr_params_t str_params = { .key_type = KVMGR_KEY_STRING,
.nslots = 32, .delta = 16 };
tusa_t t2;
memset(&t2, 0, sizeof(t2));
strncpy(t2.tmpdir, t.tmpdir, sizeof(t2.tmpdir) - 1);
t2.c.myhost = &t2.c.myhost_info;
t2.c.myhost_info.name = (char*)"ctrl-test-restart";
t2.c.myhost_info.key = (char*)"";
int rc_hook2 = xce_hook_init(&t2.c.hooksys, XCE_CONTROLLER_HOOK_MAX);
t2.c.jobmgr = makeInstance(&xkvmgr_type, &int_params);
t2.c.qmgr = makeInstance(&xkvmgr_type, &str_params);
t2.c.workermgr = makeInstance(&xkvmgr_type, &str_params);
t2.c.resmgr = makeInstance(&xkvmgr_type, &str_params);
zarray_init(&t2.c.sharedinst, 16, NULL);
persist_config_t cfg2 = persist_config_defaults();
cfg2.wal_sync_ms = 5000;
cfg2.snap_keep_n = 4;
persist_set_workdir(&t2.p, t.tmpdir, "tusa");
int rc_init2 = persist_init(&t2.p, &cfg2, &t2.c);
int rc_start2 = persist_start(&t2.p);
int rc_recover = persist_recover(&t2.p);
uint64_t restart_checkpoint_lsn = t2.p.checkpoint_lsn;
bool restart_snap_exists = tusa_snapshot_exists(&t2.p, ar.marker_lsn);
uint64_t restart_last_applied = atomic_load_explicit(
&t2.p.last_applied_lsn, memory_order_acquire);
int restart_fault_calls = g_tusa_fault_obs.calls;
char snap_path[4096];
snprintf(snap_path, sizeof(snap_path), "%s/snap-%016" PRIx64 ".capnp",
t2.p.snap_path, ar.marker_lsn);
snapverify_report_t verify_rep;
memset(&verify_rep, 0, sizeof(verify_rep));
int verify_rc = snapverify_path(snap_path, NULL, &verify_rep);
/* Phase 3: a SECOND, redundant "restart" after materialization already
* succeeded -- proves idempotency in this exact crash/retry shape. */
int rc_recover_again = persist_recover(&t2.p);
uint64_t checkpoint_lsn_after_second_recover = t2.p.checkpoint_lsn;
int snap_count_after_second_recover = tusa_count_snapshots(&t2.p);
tusa_teardown(&t2);
/* ── Assertions (only now, after every phase's teardown) ───────────── */
TEST_ASSERT_EQUAL_INT(0, ar.rc_encode);
TEST_ASSERT_EQUAL_INT(0, ar.rc_noop);
TEST_ASSERT_EQUAL_INT(0, ar.rc_marker);
TEST_ASSERT_EQUAL_INT_MESSAGE(1, crash_fault_calls,
"fault hook must have fired exactly once to force the simulated "
"crash");
TEST_ASSERT_EQUAL_INT_MESSAGE(-ECANCELED, crash_apply_rc,
"the simulated crash must surface as the marker's apply failure");
TEST_ASSERT_FALSE_MESSAGE(crash_snap_exists,
"crash state: no checkpoint must exist at the marker's lsn yet");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(ar.noop_lsn, crash_last_applied,
"crash state: last_applied_lsn must be stuck below the marker");
TEST_ASSERT_EQUAL_INT(0, rc_hook2);
TEST_ASSERT_NOT_NULL(t2.c.jobmgr);
TEST_ASSERT_NOT_NULL(t2.c.qmgr);
TEST_ASSERT_NOT_NULL(t2.c.workermgr);
TEST_ASSERT_NOT_NULL(t2.c.resmgr);
TEST_ASSERT_EQUAL_INT(0, rc_init2);
TEST_ASSERT_EQUAL_INT(0, rc_start2);
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc_recover,
"restart's boot replay must succeed now that the fault clears");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, restart_fault_calls,
"the fault hook must NOT fire again during restart -- it was "
"cleared, simulating the transient condition having resolved");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(ar.marker_lsn, restart_checkpoint_lsn,
"Req 5.2: restart must resume materialization AT THE MARKER, not "
"some other position");
TEST_ASSERT_TRUE_MESSAGE(restart_snap_exists,
"Req 5.2/5.5: restart must produce a durable checkpoint at the "
"marker's own lsn");
TEST_ASSERT_TRUE_MESSAGE(restart_last_applied >= ar.marker_lsn,
"Req 5.2: restart must advance last_applied_lsn past the marker "
"-- materialization resumed and completed, not re-stuck");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, verify_rc,
"Req 5.5: the restart-produced checkpoint must be structurally "
"well-formed, not a partial/corrupt write");
TEST_ASSERT_TRUE_MESSAGE(verify_rep.ok, verify_rep.fail_reason);
TEST_ASSERT_EQUAL_UINT64(ar.marker_lsn, verify_rep.checkpoint_lsn);
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc_recover_again,
"Req 5.3 in the crash/retry shape: a second, redundant restart "
"after materialization already succeeded must not error");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(ar.marker_lsn,
checkpoint_lsn_after_second_recover,
"a redundant restart must not relabel or move the checkpoint");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, snap_count_after_second_recover,
"a redundant restart must not duplicate the checkpoint file");
printf("test_marker_checkpoint_fault_hook_crash_then_restart_resumes_"
"materialization: PASS\n");
}
/* ── Dispatch ────────────────────────────────────────────────────────── */
void test_upgrade_snapshot_apply(void)
@ -867,4 +1068,5 @@ void test_upgrade_snapshot_apply(void)
RUN_TEST(test_marker_apply_via_replay_checkpoints_at_marker_own_lsn);
RUN_TEST(test_marker_triggered_checkpoint_trim_preserves_standby_mid_catchup_segments);
RUN_TEST(test_marker_checkpoint_fault_hook_gates_checkpoint_call);
RUN_TEST(test_marker_checkpoint_fault_hook_crash_then_restart_resumes_materialization);
}