lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race

Browse source 
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool,
which hardcodes VTASK_LIVENESS for every inbound peer-control packet including
vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in
persist_promotion_advance_committed) was reachable from that never-blocks task
class -- same bug family as bug-1198. Removed the force-flush; round-3
(durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat
fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL:
persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path
and does bounded pread() -- lower risk (page-cache-hot) but strictly still
blocking I/O; not safe to remove outright (reopens bug-1043-family job loss),
needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case
or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask).

bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply
raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during
the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could
regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop,
lock-free) and routed both call sites through it.

Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round
sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling
and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears
config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
This commit is contained in:
Rongsong Shen 2026-07-11 12:16:28 +00:00
parent ea0ac81c92
commit 40f510fb14
25 changed files with 1617 additions and 226 deletions
Download patch file Download diff file Expand all files Collapse all files

232
src/controller/testsuite/test_apply_pump.c
View file

@ -79,6 +79,7 @@
#include "cluster.h"
#include "commit.h"
#include "controller.capnp.h"
#include "ha.h"
#include "hook.h"
#include "jobj.h"
#include "persist.h"
@ -1058,6 +1059,235 @@ static void test_apply_pump_late_pending_apply_registration_self_heals(void)
"PASS\n");
}
/* ── 6. Standby-ingest commit-then-apply (2026-07-10 fix) ────────────────
*
* Regression coverage for ha.c's standby WAL-replication ingest path.
* Before this fix, ha_apply_single_entry / ha_apply_term_entry_via_library
* called apply_wal_entry() directly the instant a replicated entry was
* durably persisted — materializing the job into the job table regardless
* of whether the sending primary ever reached cluster-wide majority
* commit. If that primary died before reaching majority, a later
* (term,lsn) conflict truncate would remove the WAL bytes with no
* rollback of the already-materialized entity, leaving a permanent
* phantom job.
*
* The fix decouples durability from materialization: ha_apply_single_entry
* / ha_apply_term_entry_via_library now only persist + ACK. Entity
* materialization happens later, only once THIS standby's own
* persist_t.committed_lsn actually covers the entry — driven by
* ha_standby_advance_committed_and_apply (clamped to
* min(learned_committed_lsn, local durable tip)) via
* persist_apply_pump_kick -> persist_apply_pump_run -> apply_committed_entry
* (the same seam the primary's own online-apply/replay use).
*
* These tests drive the REAL standby ingest seam — ha_apply_wal_catchup_entry,
* the exact function every incoming WAL replication message routes through —
* directly against this file's real jobmgr/persist fixture, so "not yet
* visible" / "now visible" is observed via tap_find_job, the same job-table
* lookup every other test in this file uses, not a synthetic stand-in.
* g_tap_c.ha.raft_c is never set to &g_tap_c in this fixture, so
* ha_apply_wal_catchup_entry always takes the documented term-blind
* direct-persist bypass (ha_apply_single_entry) regardless of the entries'
* term — the same reachable-and-safe-by-guard path
* test_6_2_real_senders_never_produce_termblind_shape (test_ha_core_safety.c)
* documents. */
/* Build one WAL_JOB_SUBMIT sub-entry for jobid at lsn, via the same capnp
* encoding tap_encode_job already uses for the live command-entry test
* above. Caller owns *payload_out (free() after the entry is no longer
* needed) and must keep it alive across the ha_apply_wal_catchup_entry
* call — persist_wal_write_replicated copies the bytes to disk, so the
* pointer's lifetime need not outlive the call, but freeing before the
* call would read stale data mid-encode. */
static wal_catchup_entry_t tas_build_submit_entry(uint64_t lsn, uint64_t jobid,
uint32_t state,
uint8_t** payload_out)
{
memset(&g_tap_seed_submit, 0, sizeof(g_tap_seed_submit));
g_tap_seed_submit.queue = (char*)"q";
g_tap_seed_submit.user = (char*)"tester";
xce_job_t jdef;
memset(&jdef, 0, sizeof(jdef));
jdef.jobid = jobid;
jdef.state = state;
jdef.submit = &g_tap_seed_submit;
uint8_t* payload = NULL;
size_t plen = 0;
TEST_ASSERT_EQUAL_INT(0, tap_encode_job(&jdef, &payload, &plen));
wal_catchup_entry_t sub;
memset(&sub, 0, sizeof(sub));
sub.lsn = lsn;
sub.entry_type = WAL_JOB_SUBMIT;
sub.raw_payload = payload;
sub.raw_payload_len = (uint32_t)plen;
/* term == 0 deliberately: see the file-header comment above — this
* fixture always exercises the term-blind direct-persist bypass. */
*payload_out = payload;
return sub;
}
/* (a) Positive case: persisted-but-uncommitted stays invisible; a
* heartbeat-learned commit position materializes it. */
static void test_standby_apply_deferred_until_committed_then_materializes(
void)
{
tap_setup();
const uint64_t lsn = 1; /* first entry in a fresh persist_t */
const uint64_t jobid = 960;
uint8_t* payload = NULL;
wal_catchup_entry_t sub = tas_build_submit_entry(lsn, jobid, JOB_STAT_PEND,
&payload);
wal_catchup_entry_t* sub_arr[1] = { &sub };
wal_catchup_entry_t top;
memset(&top, 0, sizeof(top));
top.n_entries = 1;
top.entries = sub_arr;
/* Ingest via the REAL standby seam: persist + ACK only, no apply. */
int rc = ha_apply_wal_catchup_entry(&g_tap_c.ha, &g_tap_c.persist, &top,
NULL, &g_tap_c);
uint64_t current_lsn_after_persist = g_tap_c.persist.current_lsn;
uint64_t applied_lsn_after_persist =
atomic_load(&g_tap_c.ha.applied_lsn);
uint64_t committed_after_persist = atomic_load(&g_tap_c.persist.committed_lsn);
job_t* j_before_commit = tap_find_job(jobid);
/* Learn the commit position the way a real heartbeat does -- this is
* what drives the commit-then-apply wiring (ha_consume_heartbeat_committed
* -> ha_standby_advance_committed_and_apply -> persist_apply_pump_kick). */
atomic_store(&g_tap_c.ha.role, (int32_t)CTRL_STANDBY);
heartbeat_msg_t hb;
memset(&hb, 0, sizeof(hb));
hb.term = 0;
hb.committed_lsn = lsn;
ha_consume_heartbeat_committed(&g_tap_c, &hb);
/* See tap_wait_applied's doc comment: the walk may complete on a
* different thread than this call. */
tap_wait_applied(lsn, 2000);
uint64_t committed_after_hb = atomic_load(&g_tap_c.persist.committed_lsn);
job_t* j_after_commit = tap_find_job(jobid);
uint32_t j_state_after = j_after_commit ? j_after_commit->job.state : 0;
uint64_t last_applied_final = persist_last_applied_lsn(&g_tap_c.persist);
free(payload);
tap_teardown();
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(lsn + 1, current_lsn_after_persist,
"the entry must be durably persisted by the standby ingest seam");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(lsn, applied_lsn_after_persist,
"applied_lsn (the held/persisted watermark) advances on receipt -- "
"distinct from entity materialization");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(0, committed_after_persist,
"committed_lsn must NOT advance just because the entry was "
"persisted -- nothing has told this standby the entry reached "
"cluster majority yet");
TEST_ASSERT_NULL_MESSAGE(j_before_commit,
"REGRESSION GUARD: a replicated entry must not materialize into "
"the job table before this standby's own committed_lsn covers it "
"-- materializing here (the old eager-apply behavior) is exactly "
"the phantom-job hazard this fix closes");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(lsn, committed_after_hb,
"the heartbeat-learned commit position must advance committed_lsn");
TEST_ASSERT_NOT_NULL_MESSAGE(j_after_commit,
"once committed_lsn covers the entry, the apply pump must "
"materialize it into the job table");
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_PEND, j_state_after);
TEST_ASSERT_EQUAL_UINT64(lsn, last_applied_final);
printf("test_standby_apply_deferred_until_committed_then_materializes: "
"PASS\n");
}
/* (b) Regression guard: an entry truncated away before ever being learned
* as committed must never leave a phantom job — even when a stale/racing
* heartbeat still advertises the old (now-truncated) committed position. */
static void test_standby_truncate_before_commit_leaves_no_phantom_job(void)
{
tap_setup();
const uint64_t lsn = 1;
const uint64_t jobid = 961;
const uint64_t cutoff_lsn = lsn - 1; /* truncate away lsn and above */
uint8_t* payload = NULL;
wal_catchup_entry_t sub = tas_build_submit_entry(lsn, jobid, JOB_STAT_PEND,
&payload);
wal_catchup_entry_t* sub_arr[1] = { &sub };
wal_catchup_entry_t top;
memset(&top, 0, sizeof(top));
top.n_entries = 1;
top.entries = sub_arr;
int rc = ha_apply_wal_catchup_entry(&g_tap_c.ha, &g_tap_c.persist, &top,
NULL, &g_tap_c);
job_t* j_before_truncate = tap_find_job(jobid);
/* The sending primary died before reaching majority: a new leader's
* conflict-truncate removes the entry before this standby ever learns
* it was committed. */
int trc = persist_truncate_wal_above(&g_tap_c.persist, cutoff_lsn);
uint64_t current_lsn_after_truncate = g_tap_c.persist.current_lsn;
/* A stale/racing heartbeat still carries the old (now-truncated)
* committed position -- ha_standby_advance_committed_and_apply's clamp
* to the local durable tip (current_lsn-1) must refuse to walk past
* what is actually on disk. */
atomic_store(&g_tap_c.ha.role, (int32_t)CTRL_STANDBY);
heartbeat_msg_t hb;
memset(&hb, 0, sizeof(hb));
hb.term = 0;
hb.committed_lsn = lsn;
ha_consume_heartbeat_committed(&g_tap_c, &hb);
/* There is nothing valid left to apply, so no walk will ever reach
* `lsn` -- a bounded settle (not a wait-for) is the correct shape here. */
usleep(50000);
uint64_t committed_after = atomic_load(&g_tap_c.persist.committed_lsn);
uint64_t last_applied_final = persist_last_applied_lsn(&g_tap_c.persist);
job_t* j_after = tap_find_job(jobid);
free(payload);
tap_teardown();
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_NULL_MESSAGE(j_before_truncate,
"must not be materialized before commit (same guard as the "
"positive-case test)");
TEST_ASSERT_EQUAL_INT(0, trc);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(cutoff_lsn + 1, current_lsn_after_truncate,
"the entry must actually be gone from the WAL after truncation");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(0, committed_after,
"REGRESSION GUARD: the local-durable-tip clamp in "
"ha_standby_advance_committed_and_apply must refuse to advance "
"committed_lsn past a truncated-away entry, even when a stale "
"heartbeat still advertises it as committed");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(0, last_applied_final,
"nothing may be applied past the truncated tip");
TEST_ASSERT_NULL_MESSAGE(j_after,
"REGRESSION GUARD: no phantom job may appear -- the entry was "
"truncated before ever being committed, so it must never be "
"materialized");
printf("test_standby_truncate_before_commit_leaves_no_phantom_job: "
"PASS\n");
}
/* ── Test group entry point ───────────────────────────────────────────── */
void test_apply_pump(void)
@ -1068,5 +1298,7 @@ void test_apply_pump(void)
RUN_TEST(test_apply_pump_guard_does_not_break_job_ctrl_live_append);
RUN_TEST(test_apply_pump_fail_racing_finish_merges_against_current_state);
RUN_TEST(test_apply_pump_late_pending_apply_registration_self_heals);
RUN_TEST(test_standby_apply_deferred_until_committed_then_materializes);
RUN_TEST(test_standby_truncate_before_commit_leaves_no_phantom_job);
UNITY_END();
}