lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(raft-apply-decoupling): prevent apply-divergence fence from being live-cleared (task 5.3)

Browse source 
Add explicit test coverage for the DIVERGENT fence's stay-alive and
restart-only-clear guarantees (task 5.3), which surfaced a critical,
pre-existing safety gap: the sticky recovery_unrecoverable flag is
trigger-agnostic, and two pre-existing live call sites in ha.c
(promotion_fence_timeout_fn's periodic timer, and
raft_cb_become_leader's promotion-eligibility gate) clear it based
purely on structural conditions, never checking the apply verdict.

A node whose local apply dispatch diverges from the committed log,
but whose WAL bytes are otherwise structurally healthy, would have
its fence silently released by the live timer, then be eligible for
promotion to primary on a later election -- serving genuinely
divergent state.

Fix: gate both clear-call-sites on persist_apply_verdict(&c->persist)
!= APPLY_DIVERGENT in addition to their existing structural checks.
An apply-divergence-triggered fence can now only clear via a fresh
process restart's clean persist_recover() replay.

New tests: test_promotion_fence_apply_divergent_survives_structural_catchup
(test_commit_durability.c) covers the timer site; and
test_promotion_refused_when_apply_divergent_even_if_eligible
(test_convergence_verification.c) covers the promotion-gate site,
structurally isolated so it cannot pass for the wrong reason.

Closes out task-group 5 (restart apply-correctness verification).
This commit is contained in:
Rongsong Shen 2026-07-04 13:33:03 +00:00
parent 099ee1a216
commit d3da3ca8bd
5 changed files with 616 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

65
src/controller/ha.c
View file

@ -859,10 +859,21 @@ static int promotion_fence_timeout_fn(void *arg)
? (repl_active_standby_count(ha->repl) > 0)
: false;
/* Caught up: not truncated AND durable position reached the boundary →
* release the fence + clear the sticky flag; the member is eligible on its
* next election (Req 2.6). */
if (!truncated && !below_hwm) {
/* raft-apply-decoupling task 5.3 (round-2 remediation, Req 8.4): a fence
* set for an APPLY_DIVERGENT reason must NEVER be live-cleared by this
* purely-structural re-check. apply_verdict is only ever recomputed by
* persist_recover() (a boot-time-once replay, cluster.c) — more WAL
* bytes arriving or a healthy peer appearing cannot prove a local
* apply-dispatch defect is fixed, so this periodic timer must treat an
* apply-divergent fence as untouchable until the next full restart
* replays cleanly (persist_recover_decision fences apply_divergent
* unconditionally for the same reason — see persist.c). */
bool apply_divergent = persist_apply_verdict(&c->persist) == APPLY_DIVERGENT;
/* Caught up: not truncated AND durable position reached the boundary AND
* the current apply verdict is not DIVERGENT → release the fence + clear
* the sticky flag; the member is eligible on its next election (Req 2.6). */
if (!truncated && !below_hwm && !apply_divergent) {
atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release);
persist_clear_recovery_unrecoverable(&c->persist);
INFO("HA: promotion fence RELEASED after catch-up — durable_floor=%"
@ -1077,6 +1088,16 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
uint64_t durable_floor = persist_durable_floor(&c->persist);
bool unrecoverable = persist_recovery_unrecoverable(&c->persist);
/* raft-apply-decoupling task 5.3 (round-2 remediation, Req 8.4):
* same guard as promotion_fence_timeout_fn above — a fence set for
* an APPLY_DIVERGENT reason must never be live-cleared, and this
* gate must never promote while the current apply verdict is
* divergent, REGARDLESS of what `unrecoverable` currently reads
* (defense-in-depth: it must not rely on the other call site alone
* having kept `unrecoverable` accurate). apply_verdict is only ever
* recomputed by a fresh boot-time persist_recover() replay. */
bool apply_divergent = persist_apply_verdict(&c->persist) == APPLY_DIVERGENT;
/* A healthy peer is reachable for catch-up iff some voter granted with a
* live endpoint survives this election (these become the standby
* channels promoted below). */
@ -1106,26 +1127,30 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
truncated, durable_floor, promotion_safe_lsn,
peer_reachable, /*catch_up_succeeded=*/false);
if (gate == RECOVER_DECISION_ELIGIBLE && !unrecoverable) {
/* ELIGIBLE and not sticky-unrecoverable: release the fence (a
* caught-up member regains eligibility, Req 2.6), clear the sticky
* flag if set, and cancel any pending re-alert timer. Proceed to
* authoritative promotion below. */
if (gate == RECOVER_DECISION_ELIGIBLE && !unrecoverable && !apply_divergent) {
/* ELIGIBLE, not sticky-unrecoverable, AND not apply-divergent:
* release the fence (a caught-up member regains eligibility,
* Req 2.6), clear the sticky flag if set, and cancel any pending
* re-alert timer. Proceed to authoritative promotion below. */
atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release);
persist_clear_recovery_unrecoverable(&c->persist);
ha_cancel_promotion_fence_timer(ha);
} else {
/* FENCE, UNRECOVERABLE, or sticky-unrecoverable: refuse promotion —
* do NOT become CTRL_PRIMARY. Fence the member and arm the catch-up
* re-alert timer. The member STAYS ALIVE, non-promotable, retrying
* catch-up (Req 2.2/2.4). NEVER self-exit, never fence silently. */
WARN("HA: promotion REFUSED (decision=%d unrecoverable=%d) — "
"recovery_truncated=%d durable_floor=%" PRIu64
" cluster_hwm=%" PRIu64 " peer_reachable=%d; fencing + arming "
"catch-up re-alert window, staying ALIVE "
"(ha-commit-durability Req 2.2/2.4)",
(int)gate, unrecoverable, truncated, durable_floor,
promotion_safe_lsn, peer_reachable);
/* FENCE, UNRECOVERABLE, sticky-unrecoverable, or apply-divergent:
* refuse promotion — do NOT become CTRL_PRIMARY. Fence the member
* and arm the catch-up re-alert timer. The member STAYS ALIVE,
* non-promotable, retrying catch-up (Req 2.2/2.4). NEVER
* self-exit, never fence silently. An apply-divergent member is
* refused unconditionally (Req 8.4) — no peer/HWM condition can
* clear that trigger; only a fresh, clean restart replay can. */
WARN("HA: promotion REFUSED (decision=%d unrecoverable=%d "
"apply_divergent=%d) — recovery_truncated=%d durable_floor=%"
PRIu64 " cluster_hwm=%" PRIu64 " peer_reachable=%d; fencing + "
"arming catch-up re-alert window, staying ALIVE "
"(ha-commit-durability Req 2.2/2.4; raft-apply-decoupling "
"Req 8.4)",
(int)gate, unrecoverable, apply_divergent, truncated,
durable_floor, promotion_safe_lsn, peer_reachable);
ha_arm_promotion_fence_timer(ha, c, truncated, durable_floor,
promotion_safe_lsn);
return;

367
src/controller/testsuite/test_apply_correctness_verification.c
View file

@ -814,6 +814,370 @@ static void test_apply_divergent_loud_fence_log_omits_false_peer_hwm_clause(void
"PASS\n");
}
/* ── raft-apply-decoupling task 5.3 (Req 8.4): the DIVERGENT verdict path
* stays alive and keeps retrying, and the fence clears once a subsequent
* recovery reaches a consistent state.
*
* Task 5.2's own tests already establish, implicitly, that the process
* survives a forced DIVERGENT verdict (persist_recover() and
* persist_recover_loud_fence_if_unrecoverable() both RETURN normally, and
* the whole controller_unit_test binary keeps running every later test).
* This section adds a MORE EXPLICIT, direct pin on that guarantee, plus
* NEW coverage of the fence-clearing half, which 5.2 explicitly deferred
* ("this type only classifies, it does not fence" / clearing was never
* exercised anywhere in this file). ─────────────────────────────────── */
/*
* More explicit "stays alive" pin than 5.2's implicit evidence. Forces a
* REAL, end-to-end APPLY_DIVERGENT verdict via an actual persist_recover()
* replay (the duplicate-jobid trigger, same as
* test_real_replay_divergence_sets_unrecoverable_fence), then performs
* MULTIPLE, clearly UNRELATED, real operations on the SAME now-fenced
* persist_t afterward, in the SAME test function and thread -- proving
* control genuinely returned to the caller and the process remains fully
* live and functional, not merely "hasn't crashed yet by the time one
* boolean was checked."
*
* The follow-up WAL append models a real operational fact: fencing
* (recovery_unrecoverable) only refuses PROMOTION (ha.c's promotion gate,
* ~line 1078); it does not stop the member from continuing to operate as an
* ordinary, alive standby that keeps appending/replicating entries and
* keeps retrying catch-up, per the controller-never-self-terminate
* invariant (src/controller/CLAUDE.md "Process Lifecycle").
*/
static void test_apply_divergent_process_stays_alive_and_functional_after_fence(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_tacv_alive_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
/* Phase 1: duplicate-jobid WAL -- the same genuine divergence trigger as
* test_real_replay_divergence_sets_unrecoverable_fence. */
{
persist_t* p = tacv_make_fresh_persist(tmp);
TEST_ASSERT_NOT_NULL(p);
for (int i = 0; i < 2; i++) {
xce_job_t jdef;
memset(&jdef, 0, sizeof(jdef));
jdef.jobid = 76001;
jdef.state = JOB_STAT_PEND;
job_submit_data_t submit;
memset(&submit, 0, sizeof(submit));
submit.queue = (char*)"rq1";
submit.user = (char*)"tester";
jdef.submit = &submit;
uint8_t* payload = NULL;
size_t plen = 0;
TEST_ASSERT_EQUAL_INT(0, tacv_encode_job(&jdef, &payload, &plen));
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(p, WAL_JOB_SUBMIT, payload, (uint32_t)plen));
free(payload);
}
TEST_ASSERT_EQUAL_INT(0, persist_flush(p));
tacv_teardown_persist(p);
}
/* Phase 2: recover into divergence, then prove the process stays alive
* and functional with several explicit, unrelated follow-up steps. */
{
persist_t* p2 = tacv_make_fresh_persist(tmp);
TEST_ASSERT_NOT_NULL(p2);
tacv_build_cluster();
p2->cluster = &g_tacv_c;
/* persist_recover() itself RETURNING (rc==0, not a crash / not a
* jump to an abort handler) is the first-order proof of life;
* everything below is this task's requested MORE EXPLICIT pin on
* top of that. */
TEST_ASSERT_EQUAL_INT(0, persist_recover(p2));
TEST_ASSERT_EQUAL_INT_MESSAGE(APPLY_DIVERGENT,
persist_apply_verdict(p2),
"fixture sanity: the duplicate-jobid replay must genuinely "
"diverge");
TEST_ASSERT_TRUE_MESSAGE(persist_recovery_unrecoverable(p2),
"the fence must be set before this test's stays-alive pin means "
"anything");
/* Explicit pin #1: repeated, independent reads across several
* statements -- each one a real function call plus atomic load that
* requires a live, unaborted thread to execute. Not a single
* fire-and-forget check. */
for (int i = 0; i < 5; i++) {
TEST_ASSERT_TRUE_MESSAGE(persist_recovery_unrecoverable(p2),
"the fence must remain observably set across repeated "
"reads -- each read requires the calling thread to still "
"be alive");
TEST_ASSERT_EQUAL_INT_MESSAGE(APPLY_DIVERGENT,
persist_apply_verdict(p2),
"the verdict must remain stable across repeated reads");
}
/* Explicit pin #2: a genuinely unrelated, real persist operation
* performed AFTER the fence trips -- concrete proof the process is
* fully functional (file I/O, WAL locking, fsync path all still
* operate), not merely "didn't crash by the time a boolean was
* checked." */
{
xce_job_t jdef;
memset(&jdef, 0, sizeof(jdef));
jdef.jobid = 76099;
jdef.state = JOB_STAT_PEND;
job_submit_data_t submit;
memset(&submit, 0, sizeof(submit));
submit.queue = (char*)"rq1";
submit.user = (char*)"tester";
jdef.submit = &submit;
uint8_t* payload = NULL;
size_t plen = 0;
TEST_ASSERT_EQUAL_INT(0, tacv_encode_job(&jdef, &payload, &plen));
TEST_ASSERT_EQUAL_INT_MESSAGE(0,
persist_wal_async(p2, WAL_JOB_SUBMIT, payload, (uint32_t)plen),
"a fenced-but-alive member must still be able to append WAL "
"entries -- fencing refuses PROMOTION only, never ordinary "
"operation");
free(payload);
TEST_ASSERT_EQUAL_INT_MESSAGE(0, persist_flush(p2),
"the WAL flush/fsync path must still be fully operational "
"after the fence trips");
}
/* Explicit pin #3: the fence itself is unperturbed by ordinary
* continued operation -- it stays sticky exactly as designed; only
* an explicit clear (tested separately below) releases it. */
TEST_ASSERT_TRUE_MESSAGE(persist_recovery_unrecoverable(p2),
"ordinary continued operation after the fence trips must not "
"silently clear it");
tacv_teardown_cluster();
tacv_teardown_persist(p2);
}
char cmd[512];
snprintf(cmd, sizeof(cmd), "rm -rf %s", tmp);
if (system(cmd) != 0)
fprintf(stderr, "cleanup warning: rm -rf %s failed\n", tmp);
printf("test_apply_divergent_process_stays_alive_and_functional_after_fence: "
"PASS\n");
}
/*
* "Confirm the fence clears once a subsequent recovery reaches a consistent
* state" (task 5.3 text) -- the direct, mechanical half: persist_clear_
* recovery_unrecoverable() is the SAME generic, trigger-agnostic mechanism
* the pre-existing STRUCTURAL fence already uses and already proves clears
* correctly (test_commit_durability.c test_boot_policy_row4_truncated_no_
* peer_loud_fence / row4b). Nothing in persist_recover_loud_fence_if_
* unrecoverable()'s apply_divergent branch (persist.c) creates a second,
* parallel, or harder-to-clear fence flag -- it is the exact same sticky
* `recovery_unrecoverable` atomic used everywhere else. This test is the
* apply_divergent-triggered analogue of the row4/row4b precedent, closing
* the gap that no existing test in this file ever called persist_clear_
* recovery_unrecoverable() at all.
*/
static void test_apply_divergent_fence_clears_via_existing_clear_mechanism(void)
{
persist_t p;
memset(&p, 0, sizeof(p));
recover_policy_inputs_t in = {
.verdict = RECOVER_CLEAN,
.durable_below_hwm = false,
.healthy_peer_reachable = true,
.catch_up_succeeded = true,
.durable_floor_lsn = 100,
.cluster_hwm_lsn = 100,
.apply_divergent = true,
};
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(&p),
"sanity: the sticky flag starts clear on a fresh persist_t");
recover_decision_t d = persist_recover_loud_fence_if_unrecoverable(&p, &in);
TEST_ASSERT_EQUAL_INT_MESSAGE(
RECOVER_DECISION_FENCE_UNRECOVERABLE, d,
"sanity: apply_divergent must decide the loud, non-terminating "
"fence");
TEST_ASSERT_TRUE_MESSAGE(
persist_recovery_unrecoverable(&p),
"sanity: apply_divergent must set the sticky fence before this "
"test's clear assertion means anything");
/* Req 8.4 / CLAUDE.md "clear the fenced state once a committed-
* consistent state is reached": the SAME existing, generic clear
* primitive must release an apply_divergent-triggered fence exactly as
* it releases a structural one. */
persist_clear_recovery_unrecoverable(&p);
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(&p),
"the apply_divergent-triggered fence must clear via the same "
"existing mechanism used for the structural fence -- no parallel, "
"harder-to-clear fence exists for this trigger");
printf("test_apply_divergent_fence_clears_via_existing_clear_mechanism: "
"PASS\n");
}
/*
* "Confirm the fence clears once a subsequent recovery reaches a consistent
* state" -- the end-to-end, restart-shaped half, and an important
* ARCHITECTURAL FINDING documented here rather than papered over:
*
* Unlike the pre-existing STRUCTURAL fence (recovery-truncated / behind-HWM),
* which has a LIVE self-heal path -- ha.c's promotion-fence-timeout re-check
* and promotion-gate re-evaluation periodically re-read the SAME live
* persist_t's structural getters (persist_recovery_truncated /
* persist_durable_floor) WHILE THE PROCESS STAYS UP, and call persist_clear_
* recovery_unrecoverable() the moment they observe the log has genuinely
* caught up (ha.c:865-873, ha.c:1109-1116) -- the apply_divergent trigger has
* NO analogous LIVE re-verification today. Task 5.1 established that
* persist_verify_apply_correctness() runs exactly once, at boot-time replay
* inside persist_recover(); nothing re-invokes it later in the life of the
* SAME process/persist_t (that is task 6's ordered-apply-pump territory, out
* of this task's boundary). Confirmed by inspection: persist_recover()'s
* APPLY_CLEAN branch never calls persist_clear_recovery_unrecoverable() (zero
* call sites of that function anywhere in persist.c -- grep-verified; its
* only callers are the two ha.c sites above), and this is CORRECT BY DESIGN,
* not a bug: task 5.2's own decision table fences apply_divergent
* UNCONDITIONALLY specifically because a local apply defect cannot be proven
* fixed by more WAL bytes arriving (persist_recover_decision's own comment),
* so it must never be downgraded the live way a structurally-behind log can.
*
* What IS real and available today is restart-based self-heal: the sticky
* flag lives only on a single persist_t instance (never persisted to disk),
* defaults to false on every fresh persist_init(), and persist_recover()
* only ever sets it true when it ACTUALLY diverges. This test proves that
* shape directly: one instance that genuinely diverges stays fenced, a
* separate, later instance that replays a genuinely CONSISTENT log is never
* fenced, and the first instance's own fence is untouched by the second,
* independent instance's clean outcome (ruling out a vacuously-always-false
* flag proving nothing).
*/
static void test_apply_divergent_fence_does_not_survive_a_clean_restart(void)
{
char tmp_divergent[256];
char tmp_clean[256];
strcpy(tmp_divergent, "/tmp/xce_tacv_restart_bad_XXXXXX");
strcpy(tmp_clean, "/tmp/xce_tacv_restart_ok_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp_divergent));
TEST_ASSERT_NOT_NULL(mkdtemp(tmp_clean));
tacv_build_cluster();
/* "Boot 1": a duplicate-jobid WAL genuinely diverges and fences. */
{
persist_t* p = tacv_make_fresh_persist(tmp_divergent);
TEST_ASSERT_NOT_NULL(p);
for (int i = 0; i < 2; i++) {
xce_job_t jdef;
memset(&jdef, 0, sizeof(jdef));
jdef.jobid = 77001;
jdef.state = JOB_STAT_PEND;
job_submit_data_t submit;
memset(&submit, 0, sizeof(submit));
submit.queue = (char*)"rq1";
submit.user = (char*)"tester";
jdef.submit = &submit;
uint8_t* payload = NULL;
size_t plen = 0;
TEST_ASSERT_EQUAL_INT(0, tacv_encode_job(&jdef, &payload, &plen));
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(p, WAL_JOB_SUBMIT, payload, (uint32_t)plen));
free(payload);
}
TEST_ASSERT_EQUAL_INT(0, persist_flush(p));
tacv_teardown_persist(p);
}
persist_t* p_boot1 = tacv_make_fresh_persist(tmp_divergent);
TEST_ASSERT_NOT_NULL(p_boot1);
p_boot1->cluster = &g_tacv_c;
TEST_ASSERT_EQUAL_INT(0, persist_recover(p_boot1));
TEST_ASSERT_EQUAL_INT_MESSAGE(APPLY_DIVERGENT,
persist_apply_verdict(p_boot1),
"fixture sanity: boot 1 must genuinely diverge");
TEST_ASSERT_TRUE_MESSAGE(persist_recovery_unrecoverable(p_boot1),
"boot 1 must be fenced");
/* "Boot 2": a separate, later boot -- distinct jobids, no duplication --
* replays a genuinely consistent log. A fresh persist_t always starts
* unfenced (persist_init defaults recovery_unrecoverable=false); the
* point under test is that a CONSISTENT replay leaves it that way. */
{
persist_t* p = tacv_make_fresh_persist(tmp_clean);
TEST_ASSERT_NOT_NULL(p);
for (uint64_t jobid = 78001; jobid <= 78002; jobid++) {
xce_job_t jdef;
memset(&jdef, 0, sizeof(jdef));
jdef.jobid = jobid;
jdef.state = JOB_STAT_PEND;
job_submit_data_t submit;
memset(&submit, 0, sizeof(submit));
submit.queue = (char*)"rq1";
submit.user = (char*)"tester";
jdef.submit = &submit;
uint8_t* payload = NULL;
size_t plen = 0;
TEST_ASSERT_EQUAL_INT(0, tacv_encode_job(&jdef, &payload, &plen));
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(p, WAL_JOB_SUBMIT, payload, (uint32_t)plen));
free(payload);
}
TEST_ASSERT_EQUAL_INT(0, persist_flush(p));
tacv_teardown_persist(p);
}
persist_t* p_boot2 = tacv_make_fresh_persist(tmp_clean);
TEST_ASSERT_NOT_NULL(p_boot2);
p_boot2->cluster = &g_tacv_c;
TEST_ASSERT_EQUAL_INT(0, persist_recover(p_boot2));
TEST_ASSERT_EQUAL_INT_MESSAGE(APPLY_CLEAN,
persist_apply_verdict(p_boot2),
"fixture sanity: boot 2 must genuinely replay clean");
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(p_boot2),
"a subsequent (restart) recovery that reaches a consistent state "
"must not be fenced -- the fence does not survive past a clean "
"restart (Req 8.4)");
/* The FIRST boot's own fence is untouched by the second, independent
* boot's clean outcome -- proves this is a real, per-instance sticky
* transition, not a vacuously-always-false flag. */
TEST_ASSERT_TRUE_MESSAGE(
persist_recovery_unrecoverable(p_boot1),
"an independent instance reaching a clean recovery must not "
"retroactively clear a DIFFERENT, still-divergent instance's fence");
tacv_teardown_persist(p_boot1);
tacv_teardown_persist(p_boot2);
tacv_teardown_cluster();
char cmd[512];
snprintf(cmd, sizeof(cmd), "rm -rf %s", tmp_divergent);
if (system(cmd) != 0)
fprintf(stderr, "cleanup warning: rm -rf %s failed\n", tmp_divergent);
snprintf(cmd, sizeof(cmd), "rm -rf %s", tmp_clean);
if (system(cmd) != 0)
fprintf(stderr, "cleanup warning: rm -rf %s failed\n", tmp_clean);
printf("test_apply_divergent_fence_does_not_survive_a_clean_restart: "
"PASS\n");
}
void test_apply_correctness_verification(void)
{
RUN_TEST(test_clean_replay_produces_apply_clean_alongside_crc_verdict);
@ -824,4 +1188,7 @@ void test_apply_correctness_verification(void)
RUN_TEST(test_apply_divergent_loud_fence_sets_sticky_flag_and_stays_alive);
RUN_TEST(test_real_replay_divergence_sets_unrecoverable_fence);
RUN_TEST(test_apply_divergent_loud_fence_log_omits_false_peer_hwm_clause);
RUN_TEST(test_apply_divergent_process_stays_alive_and_functional_after_fence);
RUN_TEST(test_apply_divergent_fence_clears_via_existing_clear_mechanism);
RUN_TEST(test_apply_divergent_fence_does_not_survive_a_clean_restart);
}

83
src/controller/testsuite/test_commit_durability.c
View file

@ -2315,6 +2315,85 @@ static void test_promotion_fence_cancel_stops_realert(void)
printf("test_promotion_fence_cancel_stops_realert: PASS\n");
}
/* ── raft-apply-decoupling task 5.3 (round-2 remediation, Req 8.4) ───────────
* CROSS-CONTAMINATION REGRESSION: an APPLY_DIVERGENT-triggered fence must
* NEVER be live-cleared by this purely-structural re-check timer, even when
* the structural conditions it actually inspects (truncated / durable_floor
* vs HWM) become healthy. Round-1 review found BOTH ha.c live-clear call
* sites (this timer, ha.c ~867, and the promotion-eligibility gate in
* raft_cb_become_leader, ha.c ~1115) blind to WHY the sticky flag was set —
* they only ever inspected structural getters. A node whose local apply
* dispatch is defective (materialized state diverges from the committed log
* — task 5.1's APPLY_DIVERGENT verdict) can have a perfectly healthy WAL
* (not truncated, durable_floor >= HWM) by construction, since the two
* properties are orthogonal. Before the fix, this test drives exactly that
* cross-contamination and asserts the fence does NOT clear.
*
* This mirrors test_promotion_fence_released_on_catchup's harness and
* catch-up simulation EXACTLY, differing only in also marking the fence as
* apply_divergent-triggered up front — proving the fix discriminates on the
* apply verdict, not merely on the structural fields. */
static void test_promotion_fence_apply_divergent_survives_structural_catchup(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_fence_apply_div_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
cluster_t* c = dur_make_fence_cluster(tmp, /*truncated=*/true,
/*durable_floor=*/4);
TEST_ASSERT_NOT_NULL(c);
/* Mark the fence as APPLY_DIVERGENT-triggered, exactly as persist_recover()
* does at boot (task 5.1/5.2) — the WAL-structural fields above
* (truncated/durable_floor) were already set independently above, proving
* the two conditions are orthogonal by construction. */
c->persist.apply_verdict = APPLY_DIVERGENT;
atomic_store_explicit(&c->persist.recovery_unrecoverable, true,
memory_order_release);
ha_set_promotion_fence_window_ms(120); /* generous so we can mutate first */
extern void ha_arm_promotion_fence_timer_for_test(cluster_t*, bool, uint64_t,
uint64_t);
ha_arm_promotion_fence_timer_for_test(c, /*truncated=*/true,
/*durable_floor=*/4, /*hwm=*/10);
TEST_ASSERT_EQUAL_INT(
1, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire));
/* Simulate the EXACT SAME structural catch-up as
* test_promotion_fence_released_on_catchup: log now intact and durable
* past the HWM. The apply_verdict / apply-divergent fence trigger is
* UNTOUCHED by this — it can only ever be recomputed by a fresh
* persist_recover() boot-time replay (task 5.1), never by a live
* structural mutation like this one. */
c->persist.recovery_verdict = RECOVER_CLEAN;
atomic_store_explicit(&c->persist.durable_lsn, 10, memory_order_release);
/* Wait long enough for the window to fire. */
usleep(300 * 1000);
/* THE REGRESSION ASSERTION (Req 8.4): even though the structural
* conditions the timer inspects are now fully healthy, an
* APPLY_DIVERGENT-triggered fence must NOT be released — a local apply
* defect cannot be proven fixed by more WAL bytes arriving. Before the
* fix this assertion FAILS (the timer clears the flag purely on the
* structural check, oblivious to apply_verdict). */
TEST_ASSERT_TRUE_MESSAGE(
persist_recovery_unrecoverable(&c->persist),
"5.3 round-2: an APPLY_DIVERGENT-triggered fence must survive a "
"purely-structural catch-up re-check — only a fresh, clean "
"persist_recover() restart may clear it (Req 8.4)");
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire),
"5.3 round-2: the member must remain fenced (non-promotable) while "
"apply_divergent, regardless of structural catch-up");
ha_set_promotion_fence_window_ms(0);
dur_destroy_fence_cluster(c);
printf("test_promotion_fence_apply_divergent_survives_structural_catchup: "
"PASS\n");
}
#endif /* XCE_TESTING */
/* ------------------------------------------------------------------ */
@ -2375,6 +2454,10 @@ void test_commit_durability(void)
RUN_TEST(test_promotion_fence_unrecoverable_stays_alive_loud);
RUN_TEST(test_promotion_fence_released_on_catchup);
RUN_TEST(test_promotion_fence_cancel_stops_realert);
/* task 5.3 round-2 remediation (Req 8.4): cross-contamination regression —
* an APPLY_DIVERGENT-triggered fence must not clear on structural catch-up
* alone. */
RUN_TEST(test_promotion_fence_apply_divergent_survives_structural_catchup);
#else
printf("test_commit_durability: skipped (no XCE_TESTING)\n");
#endif

118
src/controller/testsuite/test_convergence_verification.c
View file

@ -1187,6 +1187,119 @@ static void test_promotion_skips_registration_when_peer_null(void)
printf("test_promotion_skips_registration_when_peer_null: checked\n");
}
/* ── raft-apply-decoupling task 5.3 (round-3 remediation, Req 8.4) ──────────
*
* COVERAGE GAP closed here: round-2 added an `!apply_divergent` conjunct to
* BOTH the promotion_fence_timeout_fn re-alert timer (ha.c ~876) AND the
* raft_cb_become_leader pre-promotion eligibility gate (ha.c ~1130,
* `gate == RECOVER_DECISION_ELIGIBLE && !unrecoverable && !apply_divergent`).
* The round-2 reviewer proved the SECOND site had zero test coverage: reverting
* ONLY that conjunct left the entire suite (298 tests across 13 groups,
* including every test that drives raft_cb_become_leader /
* raft_cb_become_leader_with_cluster_for_test) fully green. This test targets
* exactly the scenario that conjunct alone must catch: a node that is
* structurally ELIGIBLE (ha_promotion_gate_decision returns
* RECOVER_DECISION_ELIGIBLE — not truncated, durable_floor >= the cluster
* commit high-water mark) and NOT sticky-fenced (persist_recovery_unrecoverable
* is explicitly false, i.e. the OTHER guard has nothing to contribute), whose
* apply_verdict is nonetheless APPLY_DIVERGENT — a local materialized-state
* defect that only a fresh, clean persist_recover() boot replay may clear
* (task 5.1/5.2), never a peer/HWM condition.
*
* Fixture: identical to test_promotion_registers_standbys_when_peer_set above
* (n_peers=2, n_standbys=0, appended_lsn=0) — that sibling test's own
* assertion (repl_active_standby_count == 2 after promotion) is only
* reachable if this exact fixture already drives ha_promotion_gate_decision to
* RECOVER_DECISION_ELIGIBLE with unrecoverable == false, so reusing it here
* isolates the apply_divergent conjunct as the ONLY variable changed.
*
* RED (site-2 guard reverted): the gate promotes anyway — role flips to
* CTRL_PRIMARY and the vote endpoints get registered as standbys.
* GREEN (current code): promotion is refused — role stays non-PRIMARY, the
* member is fenced (in_catch_up=1, never-self-exit — it stays alive and keeps
* retrying catch-up), and no standbys are registered.
*/
static void test_promotion_refused_when_apply_divergent_even_if_eligible(void)
{
/* n_standbys=0, appended_lsn=0: same structurally-healthy fixture as
* test_promotion_registers_standbys_when_peer_set (durable_floor=0,
* recovery_truncated=false, and — with vote_granted left at its
* calloc-zero default — ha_compute_safe_committed_lsn's only counted
* voter is self at self_lsn=0, so promotion_safe_lsn=0). durable_floor(0)
* >= promotion_safe_lsn(0) and not truncated ⇒ genuinely ELIGIBLE. */
cv_setup("applydiv", /*n_peers=*/2, /*n_standbys=*/0,
/*appended_lsn=*/0, /*n_vcpus=*/8);
/* Precondition (Req 8.4 scenario, explicit per task spec): NOT fenced by
* the sticky flag — isolates the apply_divergent conjunct from the
* already-covered `unrecoverable` conjunct. */
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(&g_cv.persist),
"fixture precondition: sticky recovery_unrecoverable must be clear so "
"this test isolates the apply_divergent guard, not the sticky-flag "
"guard (Req 8.4)");
/* Force the apply-divergent verdict directly — exactly as task 5.1's
* persist_recover() replay leaves it after detecting a materialized-state
* mismatch, and exactly as
* test_promotion_fence_apply_divergent_survives_structural_catchup
* (test_commit_durability.c) sets it for the sibling (site-1) call site. */
g_cv.persist.apply_verdict = APPLY_DIVERGENT;
ha_elect_ctx_t *ctx = calloc(1, sizeof(*ctx) + 2 * sizeof(ctx->votes[0]));
TEST_ASSERT_NOT_NULL(ctx);
ctx->n_peers = 2;
pthread_mutex_init(&ctx->lock, NULL);
sem_init(&ctx->done, 0, 0);
xce_candidate_t peer0 = {0}, peer1 = {0};
cv_set_vote(ctx, 0, &peer0, "sb0.example", 7700);
cv_set_vote(ctx, 1, &peer1, "sb1.example", 7701);
g_cv.ha.standby_endpoints = calloc(1, sizeof(zarray_t));
TEST_ASSERT_NOT_NULL(g_cv.ha.standby_endpoints);
zarray_init(g_cv.ha.standby_endpoints, 4, NULL);
atomic_store_explicit(&g_cv.ha.elect_ctx, (void *)ctx, memory_order_release);
/* Act: real promotion attempt with a live, structurally-healthy cluster. */
raft_cb_become_leader_with_cluster_for_test(&g_cv, /*term=*/1);
/* Assert (Req 8.4): refused despite genuine structural ELIGIBLE +
* unrecoverable==false — the apply_divergent conjunct alone must block
* promotion. */
TEST_ASSERT_NOT_EQUAL_MESSAGE(
(int32_t)CTRL_PRIMARY,
atomic_load_explicit(&g_cv.ha.role, memory_order_acquire),
"raft-apply-decoupling task 5.3 (round-3 coverage, Req 8.4): a "
"structurally-eligible, non-sticky-fenced node whose apply_verdict is "
"APPLY_DIVERGENT must NOT be promoted to CTRL_PRIMARY");
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, atomic_load_explicit(&g_cv.ha.in_catch_up, memory_order_acquire),
"a refused apply-divergent promotion must fence the member "
"(in_catch_up=1) and keep it alive, retrying catch-up — never "
"self-exit (Req 8.4)");
TEST_ASSERT_EQUAL_INT_MESSAGE(0,
repl_active_standby_count(&g_cv.persist.repl),
"promotion refusal must happen BEFORE the standby-registration block "
"— no vote endpoint may be registered when the node never became "
"primary");
/* Cleanup */
atomic_store_explicit(&g_cv.ha.elect_ctx, NULL, memory_order_release);
if (g_cv.ha.standby_endpoints) {
zarray_fini(g_cv.ha.standby_endpoints);
free(g_cv.ha.standby_endpoints);
g_cv.ha.standby_endpoints = NULL;
}
pthread_mutex_destroy(&ctx->lock);
sem_destroy(&ctx->done);
free(ctx);
cv_teardown();
printf("test_promotion_refused_when_apply_divergent_even_if_eligible: "
"checked\n");
}
#endif /* XCE_TESTING */
/* ── Suite entry point ──────────────────────────────────────────────────── */
@ -1223,6 +1336,11 @@ void test_convergence_verification(void)
RUN_TEST(test_promotion_registers_standbys_when_peer_set);
RUN_TEST(test_promotion_skips_registration_when_peer_null);
printf("\n>> Promotion eligibility gate site-2 coverage "
"(raft-apply-decoupling task 5.3 round-3, Req 8.4)\n\n");
RUN_TEST(test_promotion_refused_when_apply_divergent_even_if_eligible);
#else
printf("test_convergence_verification: SKIP (non-test build)\n");
#endif