wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

test(raft-apply-decoupling): durable fact retention under sustained quorum loss (task 4.2)

Browse source 
Add test_fact_state_unchanged_throughout_quorum_loss_outage to
test_commit_barrier_reply.c (30->31), no production changes. Drives
a real repl_remove_standby quorum-loss window through all 4
fact-receipt handlers, polls live state and committed_lsn at 5
time-spread points (not once), asserts a real hook-spy never sees
AFTER_EVENT fire, and confirms durable WAL presence via direct
on-disk read.

Documents an architectural asymmetry for task 6 planning: facts have
no commit-timeout concept (COMMIT_MODE_RETAIN registers no
continuation, "never honors the timeout") unlike commands'
commit_submit_async -- the apply pump must discover newly-committed
retained facts purely via committed_lsn advancement, not any
per-fact signal.

Closes out task-group 4 (fact apply-after-commit split).
This commit is contained in:
Rongsong Shen 2026-07-04 07:22:21 +00:00
parent 23cbfb48e1
commit 665b29f5b0
2 changed files with 202 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

199
src/controller/testsuite/test_commit_barrier_reply.c
View file

@ -2638,6 +2638,202 @@ static void test_receipt_facts_durably_append_without_live_mutation(void)
"PASS\n");
}
/* ════════════════════════════════════════════════════════════════════════
* Task 4.2 (Req 6.1, 6.2): durable retention of facts submitted while
* majority is unreachable, held for the SUSTAINED DURATION of the outage —
* not just a single immediate check.
*
* This closes a gap neither existing test covers on its own:
* - test_receipt_facts_durably_append_without_live_mutation (task 4.1,
* above) proves live-state-unchanged for all four fact types, but
* under ordinary (majority-reachable) conditions, and only checks
* state once, immediately after each receipt call returns.
* - test_fact_retention_across_quorum_loss (pre-existing, from
* ha-commit-barrier-rollout task 4.3, further above) drives a genuine
* quorum-loss window and proves WAL-level durability/no-duplication,
* but never inspects live job state at all — it predates task 4.1's
* receipt/apply split, when receipt-time mutation was still the norm
* for the behavior it was pinning.
*
* This test combines both: a genuine simulated quorum-loss window (same
* repl_remove_standby mechanism as test_fact_retention_across_quorum_loss)
* driving all four real fact-receipt handlers from task 4.1, with the
* job's live in-memory state polled at MULTIPLE points spread across the
* outage — not merely before/after a single call — to prove the guarantee
* holds for the outage's entire duration, sealed by one final check after
* the last poll.
*
* Per persist.c's wal_hook_append_classified (task 4.1's persist_submit_
* job_fact routes through it): COMMIT_MODE_RETAIN "never honors the
* timeout" and commit_submit() returns 0 immediately for RETAIN entries —
* facts have NO commit-timeout/failure concept at the receipt layer,
* unlike commands (task 3.2's commit_submit_async). A fact's entry simply
* sits durably appended, uncommitted, for as long as majority stays
* unreachable; nothing ever resolves it as FAILED the way a command's
* registered continuation would. This test's "outage" is therefore
* genuinely open-ended by construction (committed_lsn structurally cannot
* advance past it while both standbys are disconnected), not a race
* against a timeout that might independently fire.
*/
static void test_fact_state_unchanged_throughout_quorum_loss_outage(void)
{
static unsigned int qo_pids[1] = { 8501 };
job_pidinfo_t start;
xce_jobfinish_t fin;
job_failinfo_t failinfo;
uint64_t lsn_before, lsn_after;
uint64_t committed_at_poll;
int initial_state, poll_state, final_state;
uint32_t st_start = 0, st_fin = 0, st_fail = 0, st_upd = 0;
int cnt_start, cnt_fin, cnt_fail, cnt_upd;
int i;
enum { OUTAGE_POLL_COUNT = 5, OUTAGE_POLL_INTERVAL_MS = 20 };
cbrp_setup(5000);
memset(&g_job_submit, 0, sizeof(g_job_submit));
cbrp_make_job(85, &g_job_submit);
initial_state = cbrp_job_live_state(85);
TEST_ASSERT_EQUAL_INT_MESSAGE(JOB_STAT_PEND, initial_state,
"precondition: a freshly created job starts PEND");
g_cbrp_before_fires = 0;
g_cbrp_after_fires = 0;
TEST_ASSERT_EQUAL_INT(0,
xce_add_hook(&g_hc.hooksys, XCE_CONTROLLER_HOOK_JOB, cbrp_job_hook_spy,
NULL));
/* Act 1 — QUORUM LOST: disconnect both standby registry slots
* (repl_remove_standby mirrors the RDHUP-disconnect registry effect,
* same mechanism test_fact_retention_across_quorum_loss uses). n_peers
* stays 3 in the engine: self alone is 1 of 3, never a majority, so
* committed_lsn structurally cannot advance past this point for as
* long as both standbys remain disconnected. */
repl_remove_standby(&g_hc.persist.repl, &g_hep_a);
repl_remove_standby(&g_hc.persist.repl, &g_hep_b);
pthread_mutex_lock(&g_hc.persist.lock);
lsn_before = g_hc.persist.current_lsn;
pthread_mutex_unlock(&g_hc.persist.lock);
/* Act 2 — submit all four fact types for the SAME job while majority
* is unreachable, through the real receipt handlers from task 4.1
* (identical call sequence to test_receipt_facts_durably_append_
* without_live_mutation above, now under a genuine quorum-loss
* window instead of ordinary conditions). */
memset(&start, 0, sizeof(start));
start.jobid = 85;
start.npids = 1;
start.pids = &qo_pids[0];
CBRP_DRIVE_FACT(on_job_start(&g_hc, &start), "on_job_start(85)");
memset(&fin, 0, sizeof(fin));
fin.jobid = 85;
fin.exit_code = 0; /* -> JOB_STAT_DONE */
CBRP_DRIVE_FACT(on_job_finish(&g_hc, &fin), "on_job_finish(85)");
memset(&failinfo, 0, sizeof(failinfo));
failinfo.jobid = 85;
failinfo.stage = JOB_EXEC_STAGE_POST_EXEC;
failinfo.code = 1;
failinfo.detail = (char *)"synthetic fail during simulated quorum loss";
CBRP_DRIVE_FACT(on_job_fail(&g_hc, &failinfo), "on_job_fail(85)");
CBRP_DRIVE_FACT(on_job_postdone(&g_hc, &g_req_ep, 85),
"on_job_postdone(85)");
pthread_mutex_lock(&g_hc.persist.lock);
lsn_after = g_hc.persist.current_lsn;
pthread_mutex_unlock(&g_hc.persist.lock);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(lsn_before + 4, lsn_after,
"each of the four facts must append exactly one WAL entry while "
"majority is unreachable (Req 6.1)");
TEST_ASSERT_TRUE_MESSAGE(
atomic_load(&g_hc.persist.committed_lsn) < lsn_before,
"no fact entry may commit while majority is unreachable (Req 6.1)");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_cbrp_after_fires,
"AFTER_EVENT must never fire for an uncommitted fact — no "
"reaction (slot free, dispatch, notify) may escape the outage "
"(Req 6.1, 6.2)");
/* Act 3 — the core observable this task adds: poll live job state AND
* the commit watermark repeatedly across a simulated outage window,
* not just once immediately after receipt. Every single poll must see
* the job unchanged and the outage still genuinely in effect — proving
* the no-mutation guarantee holds for the outage's sustained duration,
* not merely at one instant. */
for (i = 0; i < OUTAGE_POLL_COUNT; i++) {
cbrp_sleep_ms(OUTAGE_POLL_INTERVAL_MS);
poll_state = cbrp_job_live_state(85);
committed_at_poll = atomic_load(&g_hc.persist.committed_lsn);
TEST_ASSERT_EQUAL_INT_MESSAGE(initial_state, poll_state,
"job state must remain unchanged at every point during the "
"quorum-loss outage (Req 6.1, 6.2) — there is no local-only "
"apply path that could mutate it while majority is "
"unreachable");
TEST_ASSERT_TRUE_MESSAGE(committed_at_poll < lsn_before,
"the outage must still be genuinely in effect at this poll — "
"committed_lsn must not have advanced while both standbys "
"remain disconnected");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_cbrp_after_fires,
"AFTER_EVENT must still not have fired at this poll point — "
"no reaction may fire mid-outage (Req 6.1, 6.2)");
}
/* Assert — nothing dropped, nothing duplicated: all four facts are on
* disk exactly once with the correct post-transition state, confirmed
* AFTER the full polled outage window (not merely right after Act 2). */
cnt_start = cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_START, 85,
&st_start);
cnt_fin = cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_FINISH, 85,
&st_fin);
cnt_fail = cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_FAIL, 85,
&st_fail);
cnt_upd = cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_UPDATE, 85,
&st_upd);
TEST_ASSERT_EQUAL_INT_MESSAGE(1, cnt_start,
"job 85 START fact must remain durably retained exactly once "
"throughout the whole outage window (Req 6.1)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, cnt_fin,
"job 85 FINISH fact must remain durably retained exactly once "
"throughout the whole outage window (Req 6.1)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, cnt_fail,
"job 85 FAIL fact must remain durably retained exactly once "
"throughout the whole outage window (Req 6.1)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, cnt_upd,
"job 85 UPDATE fact must remain durably retained exactly once "
"throughout the whole outage window (Req 6.1)");
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_RUN, st_start);
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_DONE, st_fin);
TEST_ASSERT_TRUE_MESSAGE((st_upd & JOB_STAT_PDONE) != 0,
"the recorded UPDATE fact must carry JOB_STAT_PDONE");
TEST_ASSERT_TRUE_MESSAGE((st_upd & JOB_STAT_EXIT) != 0,
"the recorded UPDATE fact must carry the backstop JOB_STAT_EXIT "
"bit postdone computes when no prior exit status was set");
(void)st_fail; /* FAIL's merge result depends on current state at the
* (never-reached) apply step; only its durable
* retention count is this test's concern. */
/* Final check: after the ENTIRE outage window, job state is STILL
* unchanged and no reaction has fired — sealing the "for the duration
* of the outage" observable end-to-end. */
final_state = cbrp_job_live_state(85);
TEST_ASSERT_EQUAL_INT_MESSAGE(initial_state, final_state,
"job state must still be unchanged at the end of the outage "
"window (Req 6.1, 6.2)");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_cbrp_after_fires,
"AFTER_EVENT must never have fired across the entire outage "
"window (Req 6.1, 6.2)");
cbrp_teardown();
printf("test_fact_state_unchanged_throughout_quorum_loss_outage: "
"PASS\n");
}
/* ── Test group entry point ──────────────────────────────────────────── */
void test_commit_barrier_reply(void)
@ -2695,5 +2891,8 @@ void test_commit_barrier_reply(void)
RUN_TEST(test_guard_fail_after_finish_preserves_perr_merge);
/* Fact receipt handlers durably append only (task 4.1, Req 5.1) */
RUN_TEST(test_receipt_facts_durably_append_without_live_mutation);
/* Durable retention held for the outage's sustained duration under
* genuine quorum loss (task 4.2, Req 6.1, 6.2) */
RUN_TEST(test_fact_state_unchanged_throughout_quorum_loss_outage);
UNITY_END();
}