lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(raft-apply-decoupling): kill-mid-fact-stream soak + close bug-1416 boot-time TOCTOU (task 7.2)

Browse source 
Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact
traffic against a live 3-node cluster, killing the primary mid-stream and
asserting zero committed-state divergence and zero phantom fact reactions.

Building this scenario surfaced bug-1416: persist_recover()'s boot-time
apply-correctness verification can race the live apply pump and the standby
live-replication apply path, both of which mutate the same last_applied_lsn/
job-table state the verification re-reads live -- producing a false
APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with
a new persist_t.recovery_complete gate that defers only the in-memory apply
(never the durability write or replication ACK) while a recovery pass is in
flight, closed from the earliest safe point in myapp_run() so an election-
retry candidate can't apply live traffic before persist_recover() ever runs.

Round-2 adversarial review independently verified the ACK/durability paths
stay ungated, reproduced the myapp.c fix as load-bearing via revert-and-
reproduce, and confirmed real-divergence detection still fires. Live 3-node
factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover
calls. controller_unit_test: zero new failures across 23 groups.
This commit is contained in:
Rongsong Shen 2026-07-07 03:03:40 +00:00
parent 5f98d1b0ef
commit a5e29a6ffa
17 changed files with 74793 additions and 156 deletions
Download patch file Download diff file Expand all files Collapse all files

71
src/include/persist.h
View file

@ -363,6 +363,38 @@ struct persist_s {
_Atomic uint64_t last_applied_lsn;
apply_verdict_t apply_verdict;
/* bug-1416 (raft-apply-decoupling task 7.2 round 2): closes a boot-time
* TOCTOU between persist_recover()'s replay+verification pass above and
* two concurrently-reachable live mutation paths that write the SAME
* last_applied_lsn field / mutate the SAME live job table this pass
* reads: the live apply pump (persist_apply_pump_run, driven by any of
* the 6 commit.c committed_lsn-advance sites via persist_apply_pump_kick)
* and the standby live-replication apply path (ha_apply_single_entry /
* ha_apply_term_entry_via_library, ha.c). Both are reachable the moment
* the iomgr event loop starts processing connections in cluster_run()
* (myapp.c) -- which happens well BEFORE cluster_init_phase2() reaches
* its persist_recover() call (an election round-trip and the whole of
* phase2's own setup earlier in the same function separate the two).
* Without this flag a fast-electing peer's replicated traffic can mutate
* last_applied_lsn/the job table WHILE this node's own persist_recover()
* is still computing its replay-derived expectation, producing a FALSE
* APPLY_DIVERGENT verdict on a perfectly healthy boot (confirmed live,
* reproduced on a fresh 3-node cluster's ordinary two-phase bootstrap --
* committedLsn was identical across all nodes when it fired).
*
* Defaults to true (persist_init) so every existing unit-test fixture
* that calls persist_init() directly and never calls persist_recover()
* (the overwhelming majority of this spec's own test suite) sees
* IDENTICAL, ungated behavior to before this field existed. persist_
* recover() is the ONLY function that ever sets this false -- for the
* exact, single-threaded duration of its own replay+verify pass -- and
* always restores it to true (plus issues one deliberate catch-up kick)
* before returning, regardless of the resulting verdict. This does NOT
* touch the sticky recovery_unrecoverable fence or its clearing logic
* (task 5.3) -- a genuinely DIVERGENT verdict still fences exactly as
* before; this flag only prevents the comparison ITSELF from racing. */
_Atomic bool recovery_complete;
/* Paths (derived from workdir + name via persist_set_workdir) */
char wal_path[4096];
char snap_path[4096];
@ -706,6 +738,45 @@ persist_recover_loud_fence_if_unrecoverable(persist_t* p,
bool persist_recovery_unrecoverable(persist_t* p);
void persist_clear_recovery_unrecoverable(persist_t* p);
/* persist_recovery_complete (bug-1416, raft-apply-decoupling task 7.2 round
* 2) -- true unless a persist_recover() call is CURRENTLY in progress on
* this persist_t (see the recovery_complete field doc comment). Consulted
* by the standby live-replication apply path (ha_apply_single_entry /
* ha_apply_term_entry_via_library, ha.c) to defer in-memory entity mutation
* while a boot-time replay+verification pass is running, so it never races
* that pass's expectation-vs-actual comparison. Durability (WAL persist)
* and the replication ACK are NEVER gated by this -- only the in-memory
* apply; a deferred entry is picked up either by the in-progress replay
* pass itself (if it observes the entry on disk) or by the single catch-up
* kick persist_recover() issues right before returning. Safe on NULL p
* (returns true -- nothing to defer against). */
bool persist_recovery_complete(persist_t* p);
/* persist_mark_recovery_pending (bug-1416 round 2 -- live-cluster reproduction
* confirmed a gap in the original fix). persist_recover() itself already
* brackets recovery_complete false/true around its own call, but that window
* is too NARROW for the real production boot sequence: a node can be
* receiving and durably APPLYING live-replicated entries (ha_apply_single_
* entry / ha_apply_term_entry_via_library -- e.g. while stuck retrying its
* own election as a candidate that keeps losing/timing out before it
* discovers a live primary and steps down) for an arbitrarily long stretch
* BEFORE cluster_init_phase2() ever gets to call persist_recover() for the
* first time. During that whole stretch recovery_complete still holds
* persist_init()'s default (true), so the live-replication apply path runs
* UNGATED and can mutate jobmgr/last_applied_lsn well before persist_
* recover()'s own baseline capture, producing the exact same false
* APPLY_DIVERGENT this flag exists to prevent, just from an earlier entry
* point than persist_recover() alone can see.
*
* Call this ONCE, in the real boot path only (myapp_run, immediately after
* persist_init()/persist_start() succeed and BEFORE election / cluster_run()
* can process any network traffic), to close the gate from the earliest
* safe moment instead of only from persist_recover()'s own entry. Safe on
* NULL p (no-op). Never called by any test fixture -- every unit test that
* constructs a persist_t directly (bypassing myapp_run entirely) is
* unaffected and keeps persist_init()'s default (true, ungated). */
void persist_mark_recovery_pending(persist_t* p);
/* persist_verify_apply_correctness (raft-apply-decoupling task 5.1, Req
* 8.1/8.2). Pure comparison — mirrors persist_recover_decision's separation
* of decision from action: no logging, no fence side effects, no mutation of