lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(raft-apply-decoupling): prevent apply-divergence fence from being live-cleared (task 5.3)

Browse source 
Add explicit test coverage for the DIVERGENT fence's stay-alive and
restart-only-clear guarantees (task 5.3), which surfaced a critical,
pre-existing safety gap: the sticky recovery_unrecoverable flag is
trigger-agnostic, and two pre-existing live call sites in ha.c
(promotion_fence_timeout_fn's periodic timer, and
raft_cb_become_leader's promotion-eligibility gate) clear it based
purely on structural conditions, never checking the apply verdict.

A node whose local apply dispatch diverges from the committed log,
but whose WAL bytes are otherwise structurally healthy, would have
its fence silently released by the live timer, then be eligible for
promotion to primary on a later election -- serving genuinely
divergent state.

Fix: gate both clear-call-sites on persist_apply_verdict(&c->persist)
!= APPLY_DIVERGENT in addition to their existing structural checks.
An apply-divergence-triggered fence can now only clear via a fresh
process restart's clean persist_recover() replay.

New tests: test_promotion_fence_apply_divergent_survives_structural_catchup
(test_commit_durability.c) covers the timer site; and
test_promotion_refused_when_apply_divergent_even_if_eligible
(test_convergence_verification.c) covers the promotion-gate site,
structurally isolated so it cannot pass for the wrong reason.

Closes out task-group 5 (restart apply-correctness verification).
This commit is contained in:
Rongsong Shen 2026-07-04 13:33:03 +00:00
parent 099ee1a216
commit d3da3ca8bd
5 changed files with 616 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

118
src/controller/testsuite/test_convergence_verification.c
View file

@ -1187,6 +1187,119 @@ static void test_promotion_skips_registration_when_peer_null(void)
printf("test_promotion_skips_registration_when_peer_null: checked\n");
}
/* ── raft-apply-decoupling task 5.3 (round-3 remediation, Req 8.4) ──────────
*
* COVERAGE GAP closed here: round-2 added an `!apply_divergent` conjunct to
* BOTH the promotion_fence_timeout_fn re-alert timer (ha.c ~876) AND the
* raft_cb_become_leader pre-promotion eligibility gate (ha.c ~1130,
* `gate == RECOVER_DECISION_ELIGIBLE && !unrecoverable && !apply_divergent`).
* The round-2 reviewer proved the SECOND site had zero test coverage: reverting
* ONLY that conjunct left the entire suite (298 tests across 13 groups,
* including every test that drives raft_cb_become_leader /
* raft_cb_become_leader_with_cluster_for_test) fully green. This test targets
* exactly the scenario that conjunct alone must catch: a node that is
* structurally ELIGIBLE (ha_promotion_gate_decision returns
* RECOVER_DECISION_ELIGIBLE — not truncated, durable_floor >= the cluster
* commit high-water mark) and NOT sticky-fenced (persist_recovery_unrecoverable
* is explicitly false, i.e. the OTHER guard has nothing to contribute), whose
* apply_verdict is nonetheless APPLY_DIVERGENT — a local materialized-state
* defect that only a fresh, clean persist_recover() boot replay may clear
* (task 5.1/5.2), never a peer/HWM condition.
*
* Fixture: identical to test_promotion_registers_standbys_when_peer_set above
* (n_peers=2, n_standbys=0, appended_lsn=0) — that sibling test's own
* assertion (repl_active_standby_count == 2 after promotion) is only
* reachable if this exact fixture already drives ha_promotion_gate_decision to
* RECOVER_DECISION_ELIGIBLE with unrecoverable == false, so reusing it here
* isolates the apply_divergent conjunct as the ONLY variable changed.
*
* RED (site-2 guard reverted): the gate promotes anyway — role flips to
* CTRL_PRIMARY and the vote endpoints get registered as standbys.
* GREEN (current code): promotion is refused — role stays non-PRIMARY, the
* member is fenced (in_catch_up=1, never-self-exit — it stays alive and keeps
* retrying catch-up), and no standbys are registered.
*/
static void test_promotion_refused_when_apply_divergent_even_if_eligible(void)
{
/* n_standbys=0, appended_lsn=0: same structurally-healthy fixture as
* test_promotion_registers_standbys_when_peer_set (durable_floor=0,
* recovery_truncated=false, and — with vote_granted left at its
* calloc-zero default — ha_compute_safe_committed_lsn's only counted
* voter is self at self_lsn=0, so promotion_safe_lsn=0). durable_floor(0)
* >= promotion_safe_lsn(0) and not truncated ⇒ genuinely ELIGIBLE. */
cv_setup("applydiv", /*n_peers=*/2, /*n_standbys=*/0,
/*appended_lsn=*/0, /*n_vcpus=*/8);
/* Precondition (Req 8.4 scenario, explicit per task spec): NOT fenced by
* the sticky flag — isolates the apply_divergent conjunct from the
* already-covered `unrecoverable` conjunct. */
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(&g_cv.persist),
"fixture precondition: sticky recovery_unrecoverable must be clear so "
"this test isolates the apply_divergent guard, not the sticky-flag "
"guard (Req 8.4)");
/* Force the apply-divergent verdict directly — exactly as task 5.1's
* persist_recover() replay leaves it after detecting a materialized-state
* mismatch, and exactly as
* test_promotion_fence_apply_divergent_survives_structural_catchup
* (test_commit_durability.c) sets it for the sibling (site-1) call site. */
g_cv.persist.apply_verdict = APPLY_DIVERGENT;
ha_elect_ctx_t *ctx = calloc(1, sizeof(*ctx) + 2 * sizeof(ctx->votes[0]));
TEST_ASSERT_NOT_NULL(ctx);
ctx->n_peers = 2;
pthread_mutex_init(&ctx->lock, NULL);
sem_init(&ctx->done, 0, 0);
xce_candidate_t peer0 = {0}, peer1 = {0};
cv_set_vote(ctx, 0, &peer0, "sb0.example", 7700);
cv_set_vote(ctx, 1, &peer1, "sb1.example", 7701);
g_cv.ha.standby_endpoints = calloc(1, sizeof(zarray_t));
TEST_ASSERT_NOT_NULL(g_cv.ha.standby_endpoints);
zarray_init(g_cv.ha.standby_endpoints, 4, NULL);
atomic_store_explicit(&g_cv.ha.elect_ctx, (void *)ctx, memory_order_release);
/* Act: real promotion attempt with a live, structurally-healthy cluster. */
raft_cb_become_leader_with_cluster_for_test(&g_cv, /*term=*/1);
/* Assert (Req 8.4): refused despite genuine structural ELIGIBLE +
* unrecoverable==false — the apply_divergent conjunct alone must block
* promotion. */
TEST_ASSERT_NOT_EQUAL_MESSAGE(
(int32_t)CTRL_PRIMARY,
atomic_load_explicit(&g_cv.ha.role, memory_order_acquire),
"raft-apply-decoupling task 5.3 (round-3 coverage, Req 8.4): a "
"structurally-eligible, non-sticky-fenced node whose apply_verdict is "
"APPLY_DIVERGENT must NOT be promoted to CTRL_PRIMARY");
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, atomic_load_explicit(&g_cv.ha.in_catch_up, memory_order_acquire),
"a refused apply-divergent promotion must fence the member "
"(in_catch_up=1) and keep it alive, retrying catch-up — never "
"self-exit (Req 8.4)");
TEST_ASSERT_EQUAL_INT_MESSAGE(0,
repl_active_standby_count(&g_cv.persist.repl),
"promotion refusal must happen BEFORE the standby-registration block "
"— no vote endpoint may be registered when the node never became "
"primary");
/* Cleanup */
atomic_store_explicit(&g_cv.ha.elect_ctx, NULL, memory_order_release);
if (g_cv.ha.standby_endpoints) {
zarray_fini(g_cv.ha.standby_endpoints);
free(g_cv.ha.standby_endpoints);
g_cv.ha.standby_endpoints = NULL;
}
pthread_mutex_destroy(&ctx->lock);
sem_destroy(&ctx->done);
free(ctx);
cv_teardown();
printf("test_promotion_refused_when_apply_divergent_even_if_eligible: "
"checked\n");
}
#endif /* XCE_TESTING */
/* ── Suite entry point ──────────────────────────────────────────────────── */
@ -1223,6 +1336,11 @@ void test_convergence_verification(void)
RUN_TEST(test_promotion_registers_standbys_when_peer_set);
RUN_TEST(test_promotion_skips_registration_when_peer_null);
printf("\n>> Promotion eligibility gate site-2 coverage "
"(raft-apply-decoupling task 5.3 round-3, Req 8.4)\n\n");
RUN_TEST(test_promotion_refused_when_apply_divergent_even_if_eligible);
#else
printf("test_convergence_verification: SKIP (non-test build)\n");
#endif