test(config-upgrade-snapshot): fault-inject crash/restart + leadership change (task 8.7)
Prove Requirements 5.2/5.4/5.5/6.3 via deterministic fault injection at the two critical upgrade windows, entirely at the unit level using the XCE_TESTING hooks earlier tasks built for exactly this purpose -- no live soak needed. test_marker_checkpoint_fault_hook_crash_then_restart_resumes_materialization extends the existing fault-hook seam test: after simulating a crash before the checkpoint write durably completes, it boots a genuinely fresh persist_t/cluster_t against the same on-disk WAL via persist_recover() (a real boot-replay simulation, not a same-process re-call) and confirms materialization resumes and completes correctly. test_leadership_change_mid_materialization_completes_via_same_generic_apply_drain proves a promoted member's materialization completes identically to a non-promoted control member's, by draining each through its actual production entry point and comparing checkpoints byte-for-byte -- a regression guard against any future role-conditional branch creeping into the marker-apply path.
This commit is contained in:
parent
b5fde32033
commit
d4af605fa6
3 changed files with 463 additions and 1 deletions
|
|
@ -78,6 +78,9 @@
|
|||
#include "repl.h" /* repl_add_standby / repl_on_ack -- Gap-1 WAL-trim-
|
||||
* under-marker-trigger test (task 8.1, Req 7.4),
|
||||
* mirrors test_e2e_checkpoint.c's own include */
|
||||
#include "snapverify.h" /* snapverify_path -- task 8.7 Scenario 1: verify the
|
||||
* restart-produced checkpoint is structurally
|
||||
* well-formed, not merely present */
|
||||
#include "typemap.h" /* wal_catchup_entry_t -- repl_broadcast_send_hook's
|
||||
* signature */
|
||||
#include "wal_types.h"
|
||||
|
|
@ -852,6 +855,204 @@ static void test_marker_checkpoint_fault_hook_gates_checkpoint_call(void)
|
|||
"PASS\n");
|
||||
}
|
||||
|
||||
/* ── 9: crash before checkpoint durable, restart resumes materialization
|
||||
* (task 8.7 Scenario 1, Req 5.2, 5.3, 5.5) ─────────────────────────
|
||||
*
|
||||
* Builds directly on test 8's fault-hook seam above, whose own header
|
||||
* comment is explicit that it proves ONLY the seam itself -- "not the full
|
||||
* crash/restart regression -- that is task 8.7's job, which depends on
|
||||
* this seam landing first." This test is that job.
|
||||
*
|
||||
* Phase 1 ("crash"): drive the marker's apply through the live pump with
|
||||
* the fault hook forcing "the checkpoint write never completed" -- the
|
||||
* same mechanics test 8 already proves in isolation (apply_rc ==
|
||||
* -ECANCELED, no snapshot at the marker's lsn, last_applied_lsn stuck
|
||||
* below it). This test re-confirms that baseline lightly (not exhaustively
|
||||
* -- test 8 already owns that) and then goes one step further: physically
|
||||
* tear down this persist_t/cluster_t WITHOUT deleting the on-disk tmpdir,
|
||||
* mirroring test 6's own phase-1/phase-2 split exactly (test 6 is this
|
||||
* same file's established pattern for simulating "the process exits and a
|
||||
* fresh one boots against the same on-disk state" -- persist_stop/
|
||||
* persist_fini inline, no rm -rf until the very end).
|
||||
*
|
||||
* Phase 2 ("restart succeeds this time"): clear the fault hook (the
|
||||
* transient condition that caused the crash is gone), boot a genuinely
|
||||
* FRESH persist_t + cluster_t against the SAME tmpdir/name, and call
|
||||
* persist_recover() -- boot replay, the SECOND real production call site
|
||||
* of apply_committed_entry() (test 6 above already proves this is a valid,
|
||||
* lock-free calling convention for the marker case). A bare re-call of
|
||||
* persist_apply_pump_run() on the SAME in-memory instance was deliberately
|
||||
* rejected in favor of this: last_applied_lsn never advanced past the
|
||||
* marker in phase 1, so a same-process re-call would only prove "this
|
||||
* process retries," not "a genuinely restarted process resumes from
|
||||
* durable state" -- persist_recover() against a fresh instance is the
|
||||
* faithful simulation Requirement 5.2 actually describes. Because
|
||||
* checkpoint_lsn was never advanced past 0 in phase 1 (no checkpoint ever
|
||||
* wrote), replay restarts from lsn 1 and re-drives both the noop and the
|
||||
* marker through apply_committed_entry() fresh -- this time succeeding.
|
||||
*
|
||||
* Phase 3 (idempotent re-restart, Req 5.3 in this exact crash/retry
|
||||
* shape): call persist_recover() a SECOND time on the already-recovered
|
||||
* instance (test_persist_recovery.c's own test_recover_from_empty_state
|
||||
* documents this as valid: "persist_recover() is valid to call on a
|
||||
* running instance too") -- proves a second, redundant "restart" after
|
||||
* materialization already succeeded does not duplicate or corrupt the
|
||||
* checkpoint. test_marker_apply_retry_is_idempotent (test 3 above) already
|
||||
* covers a DIFFERENT retry shape -- a bare, same-process direct
|
||||
* re-dispatch of apply_committed_entry(), never a real boot replay -- so
|
||||
* this is not a duplicate of that coverage, it closes the boot-replay-
|
||||
* specific shape that test doesn't touch.
|
||||
*
|
||||
* Discipline note: per this file's own header, every assertion-relevant
|
||||
* observable across all three phases is gathered into a local variable
|
||||
* FIRST; no TEST_ASSERT runs until every phase's teardown has already
|
||||
* happened (avoids the documented longjmp-past-teardown pathology).
|
||||
*/
|
||||
|
||||
static void
|
||||
test_marker_checkpoint_fault_hook_crash_then_restart_resumes_materialization(void)
|
||||
{
|
||||
extern int (*persist_marker_checkpoint_fault_hook)(persist_t* p,
|
||||
uint64_t marker_lsn);
|
||||
tusa_t t;
|
||||
tusa_setup(&t);
|
||||
|
||||
memset(&g_tusa_fault_obs, 0, sizeof(g_tusa_fault_obs));
|
||||
persist_marker_checkpoint_fault_hook = tusa_marker_fault_hook;
|
||||
|
||||
tusa_append_result_t ar = tusa_append_noop_then_marker(&t.p, 0x8A);
|
||||
int crash_apply_rc = persist_apply_pump_run(&t.p);
|
||||
|
||||
bool crash_snap_exists = tusa_snapshot_exists(&t.p, ar.marker_lsn);
|
||||
uint64_t crash_last_applied = atomic_load_explicit(
|
||||
&t.p.last_applied_lsn, memory_order_acquire);
|
||||
int crash_fault_calls = g_tusa_fault_obs.calls;
|
||||
|
||||
/* "Crash": tear down this instance WITHOUT deleting the on-disk tmpdir
|
||||
* -- mirrors test 6's own phase-1 teardown exactly (persist_stop/
|
||||
* persist_fini + mgr cleanup inline, no tusa_teardown/rm -rf until the
|
||||
* very end of this test, after phase 2/3 have had a chance to read the
|
||||
* same directory). */
|
||||
persist_stop(&t.p);
|
||||
persist_fini(&t.p);
|
||||
zarray_fini(&t.c.sharedinst);
|
||||
if (t.c.resmgr) dereference(t.c.resmgr);
|
||||
if (t.c.workermgr) dereference(t.c.workermgr);
|
||||
if (t.c.qmgr) dereference(t.c.qmgr);
|
||||
if (t.c.jobmgr) dereference(t.c.jobmgr);
|
||||
xce_hook_fini(&t.c.hooksys);
|
||||
|
||||
/* "Restart succeeds this time": the transient condition that caused
|
||||
* the crash is gone. */
|
||||
persist_marker_checkpoint_fault_hook = NULL;
|
||||
|
||||
/* Phase 2: fresh persist_t + fresh cluster_t, SAME tmpdir/name as phase
|
||||
* 1 -- boot replay (persist_recover -> replay_wal), mirrors test 6's
|
||||
* own phase-2 construction exactly. */
|
||||
kvmgr_params_t int_params = { .key_type = KVMGR_KEY_INT,
|
||||
.nslots = 32, .delta = 16 };
|
||||
kvmgr_params_t str_params = { .key_type = KVMGR_KEY_STRING,
|
||||
.nslots = 32, .delta = 16 };
|
||||
tusa_t t2;
|
||||
memset(&t2, 0, sizeof(t2));
|
||||
strncpy(t2.tmpdir, t.tmpdir, sizeof(t2.tmpdir) - 1);
|
||||
t2.c.myhost = &t2.c.myhost_info;
|
||||
t2.c.myhost_info.name = (char*)"ctrl-test-restart";
|
||||
t2.c.myhost_info.key = (char*)"";
|
||||
int rc_hook2 = xce_hook_init(&t2.c.hooksys, XCE_CONTROLLER_HOOK_MAX);
|
||||
t2.c.jobmgr = makeInstance(&xkvmgr_type, &int_params);
|
||||
t2.c.qmgr = makeInstance(&xkvmgr_type, &str_params);
|
||||
t2.c.workermgr = makeInstance(&xkvmgr_type, &str_params);
|
||||
t2.c.resmgr = makeInstance(&xkvmgr_type, &str_params);
|
||||
zarray_init(&t2.c.sharedinst, 16, NULL);
|
||||
|
||||
persist_config_t cfg2 = persist_config_defaults();
|
||||
cfg2.wal_sync_ms = 5000;
|
||||
cfg2.snap_keep_n = 4;
|
||||
persist_set_workdir(&t2.p, t.tmpdir, "tusa");
|
||||
int rc_init2 = persist_init(&t2.p, &cfg2, &t2.c);
|
||||
int rc_start2 = persist_start(&t2.p);
|
||||
|
||||
int rc_recover = persist_recover(&t2.p);
|
||||
|
||||
uint64_t restart_checkpoint_lsn = t2.p.checkpoint_lsn;
|
||||
bool restart_snap_exists = tusa_snapshot_exists(&t2.p, ar.marker_lsn);
|
||||
uint64_t restart_last_applied = atomic_load_explicit(
|
||||
&t2.p.last_applied_lsn, memory_order_acquire);
|
||||
int restart_fault_calls = g_tusa_fault_obs.calls;
|
||||
|
||||
char snap_path[4096];
|
||||
snprintf(snap_path, sizeof(snap_path), "%s/snap-%016" PRIx64 ".capnp",
|
||||
t2.p.snap_path, ar.marker_lsn);
|
||||
snapverify_report_t verify_rep;
|
||||
memset(&verify_rep, 0, sizeof(verify_rep));
|
||||
int verify_rc = snapverify_path(snap_path, NULL, &verify_rep);
|
||||
|
||||
/* Phase 3: a SECOND, redundant "restart" after materialization already
|
||||
* succeeded -- proves idempotency in this exact crash/retry shape. */
|
||||
int rc_recover_again = persist_recover(&t2.p);
|
||||
uint64_t checkpoint_lsn_after_second_recover = t2.p.checkpoint_lsn;
|
||||
int snap_count_after_second_recover = tusa_count_snapshots(&t2.p);
|
||||
|
||||
tusa_teardown(&t2);
|
||||
|
||||
/* ── Assertions (only now, after every phase's teardown) ───────────── */
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, ar.rc_encode);
|
||||
TEST_ASSERT_EQUAL_INT(0, ar.rc_noop);
|
||||
TEST_ASSERT_EQUAL_INT(0, ar.rc_marker);
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(1, crash_fault_calls,
|
||||
"fault hook must have fired exactly once to force the simulated "
|
||||
"crash");
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(-ECANCELED, crash_apply_rc,
|
||||
"the simulated crash must surface as the marker's apply failure");
|
||||
TEST_ASSERT_FALSE_MESSAGE(crash_snap_exists,
|
||||
"crash state: no checkpoint must exist at the marker's lsn yet");
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(ar.noop_lsn, crash_last_applied,
|
||||
"crash state: last_applied_lsn must be stuck below the marker");
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, rc_hook2);
|
||||
TEST_ASSERT_NOT_NULL(t2.c.jobmgr);
|
||||
TEST_ASSERT_NOT_NULL(t2.c.qmgr);
|
||||
TEST_ASSERT_NOT_NULL(t2.c.workermgr);
|
||||
TEST_ASSERT_NOT_NULL(t2.c.resmgr);
|
||||
TEST_ASSERT_EQUAL_INT(0, rc_init2);
|
||||
TEST_ASSERT_EQUAL_INT(0, rc_start2);
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc_recover,
|
||||
"restart's boot replay must succeed now that the fault clears");
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(1, restart_fault_calls,
|
||||
"the fault hook must NOT fire again during restart -- it was "
|
||||
"cleared, simulating the transient condition having resolved");
|
||||
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(ar.marker_lsn, restart_checkpoint_lsn,
|
||||
"Req 5.2: restart must resume materialization AT THE MARKER, not "
|
||||
"some other position");
|
||||
TEST_ASSERT_TRUE_MESSAGE(restart_snap_exists,
|
||||
"Req 5.2/5.5: restart must produce a durable checkpoint at the "
|
||||
"marker's own lsn");
|
||||
TEST_ASSERT_TRUE_MESSAGE(restart_last_applied >= ar.marker_lsn,
|
||||
"Req 5.2: restart must advance last_applied_lsn past the marker "
|
||||
"-- materialization resumed and completed, not re-stuck");
|
||||
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(0, verify_rc,
|
||||
"Req 5.5: the restart-produced checkpoint must be structurally "
|
||||
"well-formed, not a partial/corrupt write");
|
||||
TEST_ASSERT_TRUE_MESSAGE(verify_rep.ok, verify_rep.fail_reason);
|
||||
TEST_ASSERT_EQUAL_UINT64(ar.marker_lsn, verify_rep.checkpoint_lsn);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc_recover_again,
|
||||
"Req 5.3 in the crash/retry shape: a second, redundant restart "
|
||||
"after materialization already succeeded must not error");
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(ar.marker_lsn,
|
||||
checkpoint_lsn_after_second_recover,
|
||||
"a redundant restart must not relabel or move the checkpoint");
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(1, snap_count_after_second_recover,
|
||||
"a redundant restart must not duplicate the checkpoint file");
|
||||
|
||||
printf("test_marker_checkpoint_fault_hook_crash_then_restart_resumes_"
|
||||
"materialization: PASS\n");
|
||||
}
|
||||
|
||||
/* ── Dispatch ────────────────────────────────────────────────────────── */
|
||||
|
||||
void test_upgrade_snapshot_apply(void)
|
||||
|
|
@ -867,4 +1068,5 @@ void test_upgrade_snapshot_apply(void)
|
|||
RUN_TEST(test_marker_apply_via_replay_checkpoints_at_marker_own_lsn);
|
||||
RUN_TEST(test_marker_triggered_checkpoint_trim_preserves_standby_mid_catchup_segments);
|
||||
RUN_TEST(test_marker_checkpoint_fault_hook_gates_checkpoint_call);
|
||||
RUN_TEST(test_marker_checkpoint_fault_hook_crash_then_restart_resumes_materialization);
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue