test(config-upgrade-snapshot): fault-inject crash/restart + leadership change (task 8.7)
Prove Requirements 5.2/5.4/5.5/6.3 via deterministic fault injection at the two critical upgrade windows, entirely at the unit level using the XCE_TESTING hooks earlier tasks built for exactly this purpose -- no live soak needed. test_marker_checkpoint_fault_hook_crash_then_restart_resumes_materialization extends the existing fault-hook seam test: after simulating a crash before the checkpoint write durably completes, it boots a genuinely fresh persist_t/cluster_t against the same on-disk WAL via persist_recover() (a real boot-replay simulation, not a same-process re-call) and confirms materialization resumes and completes correctly. test_leadership_change_mid_materialization_completes_via_same_generic_apply_drain proves a promoted member's materialization completes identically to a non-promoted control member's, by draining each through its actual production entry point and comparing checkpoints byte-for-byte -- a regression guard against any future role-conditional branch creeping into the marker-apply path.
This commit is contained in:
parent
b5fde32033
commit
d4af605fa6
3 changed files with 463 additions and 1 deletions
|
|
@ -29,6 +29,10 @@
|
|||
#include "persist.h"
|
||||
#include "repl.h"
|
||||
#include "request.h"
|
||||
#include "snapdiff.h" /* snapdiff_paths -- task 8.7 Scenario 2: prove a
|
||||
* leadership-change-driven materialization drain
|
||||
* produces a byte-identical checkpoint to an ordinary,
|
||||
* non-promoted drain of the same marker */
|
||||
#include "typemap.h"
|
||||
#include "wal_types.h"
|
||||
#include "xce/cluster_def.h"
|
||||
|
|
@ -2487,6 +2491,257 @@ static void test_wal_config_marker_apply_activates_originating_member_config(voi
|
|||
printf("test_wal_config_marker_apply_activates_originating_member_config: PASS\n");
|
||||
}
|
||||
|
||||
/* ── config-upgrade-snapshot task 8.7 Scenario 2: leadership change mid-
|
||||
* materialization does not special-case, and completes via the SAME generic
|
||||
* apply-drain path (Req 5.4) ────────────────────────────────────────────
|
||||
*
|
||||
* design.md's own "Key simplification found during design" (persist.c's
|
||||
* WAL_CONFIG_MARKER case doc comment, apply_committed_entry): "no new
|
||||
* leader-specific 'resume' logic is needed either, since a newly-elected
|
||||
* leader satisfies this the same way, by continuing to apply its own
|
||||
* committed log." This test proves that claim structurally rather than
|
||||
* asserting it by fiat.
|
||||
*
|
||||
* The concrete mechanism (read directly out of raft_cb_become_leader,
|
||||
* ha.c:1516-1979): the promotion-time eligibility gate
|
||||
* (upgrade_materialization_fenced(c), ha.c:1763) only refuses promotion when
|
||||
* the fence has ALREADY been set by a prior upgrade_materialization_verify()
|
||||
* call. A marker that is committed but has never yet been through
|
||||
* apply_committed_entry() at all -- a genuine apply backlog, not a checkpoint
|
||||
* failure -- leaves the fence at its zero-initialized (unfenced) default,
|
||||
* because nothing has evaluated it yet. So promotion is NOT refused in this
|
||||
* case. What actually drains the backlog is ha.c:1878's
|
||||
* persist_apply_pump_run_sync(&c->persist) call, unconditionally issued right
|
||||
* after role flips to CTRL_PRIMARY -- the SAME synchronous drain function
|
||||
* bug-1043's job-backlog fix added for an entirely unrelated reason (a
|
||||
* promoted standby's un-applied committed jobs must be visible immediately).
|
||||
* The WAL_CONFIG_MARKER case (persist.c) rides that pre-existing, generic
|
||||
* drain for free: no leadership-specific config-upgrade code exists anywhere
|
||||
* in the promotion path.
|
||||
*
|
||||
* This test drives that exact mechanism directly
|
||||
* (persist_apply_pump_run_sync) rather than the full
|
||||
* raft_cb_become_leader_with_cluster_for_test() tail, deliberately: the rest
|
||||
* of that function's post-drain tail (quorum-transition logging, config push,
|
||||
* ctrlhello, persist-hook registration, heartbeat start, own-term gate arm)
|
||||
* is orthogonal to Requirement 5.4's materialization-continuation claim and
|
||||
* would require a substantially larger, novel fixture (full ha_init +
|
||||
* delaymgr_init, mirroring test_convergence_verification.c's cv_setup --
|
||||
* this file's own t42_/t43_ fixtures deliberately do NOT wire that, since no
|
||||
* other test in this file drives promotion past the fence-refusal point) to
|
||||
* exercise safely -- a scope expansion this test-only task should not take
|
||||
* on. The GATING half of Req 5.4 (a member whose fence IS already set
|
||||
* refuses promotion) is already exhaustively covered by
|
||||
* test_upgrade_materialization_fence_enforced_by_dispatch_gate_and_promotion
|
||||
* (6.2-a) and
|
||||
* test_wal_config_marker_apply_sets_fence_via_real_checkpoint_miss (6.2
|
||||
* remediation) above, both via the real raft_cb_become_leader_with_cluster_
|
||||
* for_test() gate -- this test does not repeat that proof, it closes the
|
||||
* CONTINUATION half neither of those tests reach (both stop at the refusal;
|
||||
* neither goes on to clear the condition and confirm materialization
|
||||
* completes normally afterward).
|
||||
*
|
||||
* Rigor: builds TWO independent members (A: promoted mid-backlog: role set
|
||||
* CTRL_PRIMARY the same way every other test in this file simulates
|
||||
* promotion (t42_make_leader), THEN drained via persist_apply_pump_run_sync
|
||||
* -- the same function production promotion invokes; B: never promoted,
|
||||
* drained via the ordinary persist_apply_pump_run()) starting from the
|
||||
* IDENTICAL committed marker + config bytes, and asserts their resulting
|
||||
* checkpoints are BYTE-IDENTICAL (snapdiff_paths) -- not merely that both
|
||||
* "also complete," but that they complete IDENTICALLY, mirroring bug-857's
|
||||
* own two-member convergence-proof style (test_bug857_regression.c).
|
||||
*/
|
||||
static void test_leadership_change_mid_materialization_completes_via_same_generic_apply_drain(void)
|
||||
{
|
||||
char tmp_a[256], tmp_b[256];
|
||||
strcpy(tmp_a, "/tmp/xce_cr87s2_a_XXXXXX");
|
||||
strcpy(tmp_b, "/tmp/xce_cr87s2_b_XXXXXX");
|
||||
TEST_ASSERT_NOT_NULL(mkdtemp(tmp_a));
|
||||
TEST_ASSERT_NOT_NULL(mkdtemp(tmp_b));
|
||||
|
||||
cluster_t *c_a = t42_make_cluster(tmp_a);
|
||||
cluster_t *c_b = t42_make_cluster(tmp_b);
|
||||
TEST_ASSERT_NOT_NULL(c_a);
|
||||
TEST_ASSERT_NOT_NULL(c_b);
|
||||
|
||||
myconf_t *mc_a = t43_attach_myconf(c_a, tmp_a);
|
||||
myconf_t *mc_b = t43_attach_myconf(c_b, tmp_b);
|
||||
|
||||
/* Identical config bytes on both members (t43_make_config_blob is
|
||||
* documented byte-identical across calls -- exactly what this test
|
||||
* wants, since A and B must start from the SAME upgrade content for a
|
||||
* byte-identical-outcome comparison to be meaningful). */
|
||||
size_t blen = 0;
|
||||
uint8_t *bytes = t43_make_config_blob(&blen);
|
||||
uint8_t sha[32];
|
||||
SHA256(bytes, blen, sha);
|
||||
uint64_t tag; { uint64_t tmp; memcpy(&tmp, sha, 8); tag = be64toh(tmp); }
|
||||
|
||||
int rc_save_a = conf_save_config(mc_a, bytes, blen, sha);
|
||||
int rc_save_b = conf_save_config(mc_b, bytes, blen, sha);
|
||||
free(bytes);
|
||||
|
||||
uint8_t marker_payload[WAL_CONFIG_MARKER_PAYLOAD_LEN];
|
||||
size_t out_len = 0;
|
||||
int rc_encode = config_marker_encode(tag, sha, marker_payload,
|
||||
WAL_CONFIG_MARKER_PAYLOAD_LEN,
|
||||
&out_len);
|
||||
|
||||
/* Append the SAME marker directly to each member's physical WAL
|
||||
* (persist_wal_async, bypassing config_append_marker/commit_submit's
|
||||
* commit-engine/fsync-worker machinery entirely) and advance
|
||||
* committed_lsn directly -- mirrors test_bug857_regression.c's own
|
||||
* Scenario 2 technique exactly, chosen for the identical reason: it is
|
||||
* the only DETERMINISTIC way to leave a marker genuinely committed but
|
||||
* NOT YET applied (a real apply backlog) without racing this fixture's
|
||||
* background fsync_worker thread, which a commit_submit(WAIT) round
|
||||
* trip cannot guarantee (see t42_wait_applied's own doc comment on this
|
||||
* exact race). */
|
||||
int rc_wal_a = persist_wal_async(&c_a->persist, WAL_CONFIG_MARKER,
|
||||
marker_payload,
|
||||
WAL_CONFIG_MARKER_PAYLOAD_LEN);
|
||||
uint64_t marker_lsn_a = c_a->persist.current_lsn - 1;
|
||||
uint64_t committed_a = persist_committed_lsn_advance_atomic(
|
||||
&c_a->persist, marker_lsn_a);
|
||||
|
||||
int rc_wal_b = persist_wal_async(&c_b->persist, WAL_CONFIG_MARKER,
|
||||
marker_payload,
|
||||
WAL_CONFIG_MARKER_PAYLOAD_LEN);
|
||||
uint64_t marker_lsn_b = c_b->persist.current_lsn - 1;
|
||||
uint64_t committed_b = persist_committed_lsn_advance_atomic(
|
||||
&c_b->persist, marker_lsn_b);
|
||||
|
||||
/* Precondition (validates this is genuinely the "backlog, never yet
|
||||
* evaluated" case, not the "already fenced" case 6.2-a covers): neither
|
||||
* member has applied anything yet, and neither fence has ever been
|
||||
* evaluated for this marker, so both read unfenced by construction. */
|
||||
uint64_t last_applied_before_a = persist_last_applied_lsn(&c_a->persist);
|
||||
uint64_t last_applied_before_b = persist_last_applied_lsn(&c_b->persist);
|
||||
int fenced_before_a = upgrade_materialization_fenced(c_a);
|
||||
int fenced_before_b = upgrade_materialization_fenced(c_b);
|
||||
|
||||
/* A: "leadership change" -- promoted via this fixture family's own
|
||||
* established promotion-simulation call (t42_make_leader; every test in
|
||||
* this file uses it for this purpose except the ones specifically
|
||||
* proving the real election-time GATE itself, which this test does not
|
||||
* re-prove -- see header comment). Confirm becoming primary alone has NO
|
||||
* bearing on the fence (still reads unfenced immediately after the role
|
||||
* flip -- nothing leader-specific touches it), THEN drive the exact
|
||||
* function production promotion uses to drain a backlog
|
||||
* (persist_apply_pump_run_sync, ha.c:1878). */
|
||||
t42_make_leader(c_a, /*term*/60, /*n_peers*/1);
|
||||
int32_t role_after_promotion =
|
||||
atomic_load_explicit(&c_a->ha.role, memory_order_acquire);
|
||||
int fenced_immediately_after_promotion = upgrade_materialization_fenced(c_a);
|
||||
persist_apply_pump_run_sync(&c_a->persist);
|
||||
|
||||
/* B: no leadership change at all -- drained via the ordinary,
|
||||
* always-available apply-pump entry point every non-promoted member
|
||||
* (standby or an already-serving primary catching up its own backlog)
|
||||
* uses identically. */
|
||||
int applied_b = persist_apply_pump_run(&c_b->persist);
|
||||
|
||||
uint64_t last_applied_after_a = persist_last_applied_lsn(&c_a->persist);
|
||||
uint64_t last_applied_after_b = persist_last_applied_lsn(&c_b->persist);
|
||||
int fenced_after_a = upgrade_materialization_fenced(c_a);
|
||||
int fenced_after_b = upgrade_materialization_fenced(c_b);
|
||||
|
||||
uint64_t gen_a = mc_a->cluster_config ? mc_a->cluster_config->config_generation : 0;
|
||||
uint64_t gen_b = mc_b->cluster_config ? mc_b->cluster_config->config_generation : 0;
|
||||
bool cluster_config_a_present = (mc_a->cluster_config != NULL);
|
||||
bool cluster_config_b_present = (mc_b->cluster_config != NULL);
|
||||
|
||||
char snap_a[4096], snap_b[4096];
|
||||
snprintf(snap_a, sizeof(snap_a), "%s/snap-%016" PRIx64 ".capnp",
|
||||
c_a->persist.snap_path, marker_lsn_a);
|
||||
snprintf(snap_b, sizeof(snap_b), "%s/snap-%016" PRIx64 ".capnp",
|
||||
c_b->persist.snap_path, marker_lsn_b);
|
||||
bool snap_a_exists = (access(snap_a, F_OK) == 0);
|
||||
bool snap_b_exists = (access(snap_b, F_OK) == 0);
|
||||
|
||||
snapdiff_report_t diff_rep;
|
||||
memset(&diff_rep, 0, sizeof(diff_rep));
|
||||
int diff_rc = snapdiff_paths(snap_a, snap_b, NULL, &diff_rep);
|
||||
|
||||
t43_free_myconf(c_a);
|
||||
t43_free_myconf(c_b);
|
||||
t42_teardown(c_a);
|
||||
t42_teardown(c_b);
|
||||
|
||||
/* ── Assertions ──────────────────────────────────────────────────── */
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, rc_save_a);
|
||||
TEST_ASSERT_EQUAL_INT(0, rc_save_b);
|
||||
TEST_ASSERT_EQUAL_INT(0, rc_encode);
|
||||
TEST_ASSERT_EQUAL_INT(0, rc_wal_a);
|
||||
TEST_ASSERT_EQUAL_INT(0, rc_wal_b);
|
||||
TEST_ASSERT_EQUAL_UINT64(marker_lsn_a, committed_a);
|
||||
TEST_ASSERT_EQUAL_UINT64(marker_lsn_b, committed_b);
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(marker_lsn_a, marker_lsn_b,
|
||||
"both members must start from the SAME marker lsn for a "
|
||||
"byte-identical-outcome comparison to be meaningful");
|
||||
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(0, last_applied_before_a,
|
||||
"precondition: member A must not have applied anything yet");
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(0, last_applied_before_b,
|
||||
"precondition: member B must not have applied anything yet");
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(0, fenced_before_a,
|
||||
"precondition: an unevaluated backlog must NOT already be fenced -- "
|
||||
"this is genuinely the case existing 6.2 tests don't cover, "
|
||||
"distinct from a fence already set by a prior checkpoint miss");
|
||||
TEST_ASSERT_EQUAL_INT(0, fenced_before_b);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE((int32_t)CTRL_PRIMARY, role_after_promotion,
|
||||
"the promotion simulation must actually take effect");
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(0, fenced_immediately_after_promotion,
|
||||
"becoming primary must have NO bearing on the fence by itself -- no "
|
||||
"leader-specific code sets or clears it as a side effect of the "
|
||||
"role transition");
|
||||
|
||||
TEST_ASSERT_TRUE_MESSAGE(last_applied_after_a >= marker_lsn_a,
|
||||
"Req 5.4: the SAME generic sync-drain function production "
|
||||
"promotion invokes (persist_apply_pump_run_sync) must complete "
|
||||
"materialization for the backlogged marker -- no special resume "
|
||||
"logic needed, none exists");
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(1, applied_b,
|
||||
"control: the non-promoted member's ordinary apply pump must "
|
||||
"apply exactly the one backlogged marker entry");
|
||||
TEST_ASSERT_TRUE_MESSAGE(last_applied_after_b >= marker_lsn_b,
|
||||
"control: the non-promoted member's ordinary apply pump must "
|
||||
"complete materialization identically");
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(0, fenced_after_a,
|
||||
"member A's materialization must be genuinely verified complete "
|
||||
"(fence clears), not merely attempted");
|
||||
TEST_ASSERT_EQUAL_INT(0, fenced_after_b);
|
||||
|
||||
TEST_ASSERT_TRUE_MESSAGE(cluster_config_a_present,
|
||||
"member A's config must have activated as part of the same "
|
||||
"marker-apply dispatch, exactly as it does for any non-promoted "
|
||||
"member (bug-1469)");
|
||||
TEST_ASSERT_TRUE(cluster_config_b_present);
|
||||
TEST_ASSERT_EQUAL_UINT64(1, gen_a); /* t43_make_config_blob's hardcoded generation */
|
||||
TEST_ASSERT_EQUAL_UINT64(gen_b, gen_a);
|
||||
|
||||
TEST_ASSERT_TRUE_MESSAGE(snap_a_exists,
|
||||
"member A must have produced a checkpoint at the marker's own lsn");
|
||||
TEST_ASSERT_TRUE_MESSAGE(snap_b_exists,
|
||||
"member B must have produced a checkpoint at the marker's own lsn");
|
||||
|
||||
/* THE decisive proof: materialization triggered by a leadership change
|
||||
* mid-backlog produces EXACTLY the same artifact as an ordinary,
|
||||
* never-promoted member's continued apply of the identical marker --
|
||||
* "completes identically to the non-promoted case," not merely "also
|
||||
* completes." */
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(0, diff_rc,
|
||||
"Req 5.4: a leadership-change-driven materialization drain must "
|
||||
"produce a checkpoint byte-identical to the non-promoted member's");
|
||||
TEST_ASSERT_TRUE_MESSAGE(diff_rep.identical,
|
||||
"no leader-specific divergence exists in the materialized output");
|
||||
|
||||
printf("test_leadership_change_mid_materialization_completes_via_"
|
||||
"same_generic_apply_drain: PASS\n");
|
||||
}
|
||||
|
||||
/* ── wait_all_with_timeout (XCE_TESTING shim) ───── */
|
||||
|
||||
#ifdef XCE_TESTING
|
||||
|
|
@ -5305,6 +5560,9 @@ void test_config_repl(void)
|
|||
/* bug-1469: WAL_CONFIG_MARKER apply must activate config on the
|
||||
* ORIGINATING member too, not just receiving standbys. */
|
||||
RUN_TEST(test_wal_config_marker_apply_activates_originating_member_config);
|
||||
/* config-upgrade-snapshot task 8.7 Scenario 2: leadership change
|
||||
* mid-materialization completes via the same generic apply-drain path. */
|
||||
RUN_TEST(test_leadership_change_mid_materialization_completes_via_same_generic_apply_drain);
|
||||
#ifdef XCE_TESTING
|
||||
RUN_TEST(test_load_primary_config_missing_returns_error);
|
||||
RUN_TEST(test_wait_all_with_timeout_slow_threads);
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue