fix(raft-apply-decoupling): kill-mid-fact-stream soak + close bug-1416 boot-time TOCTOU (task 7.2)
Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact traffic against a live 3-node cluster, killing the primary mid-stream and asserting zero committed-state divergence and zero phantom fact reactions. Building this scenario surfaced bug-1416: persist_recover()'s boot-time apply-correctness verification can race the live apply pump and the standby live-replication apply path, both of which mutate the same last_applied_lsn/ job-table state the verification re-reads live -- producing a false APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with a new persist_t.recovery_complete gate that defers only the in-memory apply (never the durability write or replication ACK) while a recovery pass is in flight, closed from the earliest safe point in myapp_run() so an election- retry candidate can't apply live traffic before persist_recover() ever runs. Round-2 adversarial review independently verified the ACK/durability paths stay ungated, reproduced the myapp.c fix as load-bearing via revert-and- reproduce, and confirmed real-divergence detection still fires. Live 3-node factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover calls. controller_unit_test: zero new failures across 23 groups.
This commit is contained in:
parent
5f98d1b0ef
commit
a5e29a6ffa
17 changed files with 74793 additions and 156 deletions
|
|
@ -618,6 +618,14 @@ int persist_init(persist_t* p, persist_config_t* cfg, cluster_t* c)
|
|||
p->apply_verdict = APPLY_CLEAN;
|
||||
atomic_store(&p->last_applied_lsn, 0);
|
||||
|
||||
/* bug-1416: defaults to true (nothing to gate) so every fixture that
|
||||
* calls persist_init() but never persist_recover() -- most of this
|
||||
* spec's own test suite -- sees identical, ungated apply-pump/live-
|
||||
* replication behavior to before this flag existed. Only persist_
|
||||
* recover() ever sets this false, for its own single-threaded
|
||||
* replay+verify duration (see the field's doc comment in persist.h). */
|
||||
atomic_store(&p->recovery_complete, true);
|
||||
|
||||
/* Durable-advance wake hook starts unregistered (ha-commit-durability 2.2):
|
||||
* the commit engine registers it in commit_engine_init. Until then every
|
||||
* flusher durable advance is a no-op wake. */
|
||||
|
|
@ -3228,6 +3236,31 @@ void persist_apply_pump_kick(persist_t* p, void* pool_arg)
|
|||
if (!p || !p->cluster)
|
||||
return;
|
||||
|
||||
/* bug-1416 (raft-apply-decoupling task 7.2 round 2): defer while this
|
||||
* node's own boot-time persist_recover() replay+verification pass is
|
||||
* still running -- last_applied_lsn is being written UNLOCKED by
|
||||
* replay_wal() during that window (single-threaded-recovery
|
||||
* assumption), and persist_verify_apply_correctness() is about to
|
||||
* compare a live re-read of it (and of the live job table) against a
|
||||
* WAL-scan-derived expectation. Letting this walk run concurrently
|
||||
* would race BOTH: it could advance last_applied_lsn out from under
|
||||
* replay_wal's own plain (non-CAS) store, and it mutates the same job
|
||||
* table persist_verify_apply_correctness re-reads as "actual" --
|
||||
* producing a false APPLY_DIVERGENT verdict on a perfectly healthy
|
||||
* boot (confirmed live). No work is lost: persist_recover() sets this
|
||||
* flag back to true and issues exactly one catch-up kick right before
|
||||
* it returns, which will walk from last_applied_lsn+1 up through
|
||||
* whatever committed_lsn has grown to by then -- a full, ordered
|
||||
* drain, not a partial one. Intentionally checked with NO lock: a
|
||||
* stale `true` read racing the flag's transition to `false` is
|
||||
* harmless (the walk itself still only advances one entry at a time
|
||||
* under apply_pump_lock, so at worst this call proceeds a few
|
||||
* instructions into a window persist_recover() is about to close, and
|
||||
* the deferred re-kick after recover() still runs); a stale `false`
|
||||
* merely defers one no-op-cost kick to the next real trigger. */
|
||||
if (!atomic_load_explicit(&p->recovery_complete, memory_order_acquire))
|
||||
return;
|
||||
|
||||
int expected = 0;
|
||||
if (!atomic_compare_exchange_strong_explicit(
|
||||
&p->apply_pump_scheduled, &expected, 1, memory_order_acq_rel,
|
||||
|
|
@ -3635,12 +3668,47 @@ uint64_t persist_last_applied_lsn(persist_t* p)
|
|||
: 0;
|
||||
}
|
||||
|
||||
/* bug-1416 (raft-apply-decoupling task 7.2 round 2) -- see persist.h's
|
||||
* doc comment on the recovery_complete field for the full contract. */
|
||||
bool persist_recovery_complete(persist_t* p)
|
||||
{
|
||||
return !p || atomic_load_explicit(&p->recovery_complete,
|
||||
memory_order_acquire);
|
||||
}
|
||||
|
||||
/* bug-1416 round 2 -- see persist.h's doc comment on this function for the
|
||||
* full rationale (closes the gap between persist_init() and the FIRST
|
||||
* persist_recover() call, which persist_recover()'s own false/true bracket
|
||||
* cannot see). */
|
||||
void persist_mark_recovery_pending(persist_t* p)
|
||||
{
|
||||
if (!p)
|
||||
return;
|
||||
atomic_store_explicit(&p->recovery_complete, false, memory_order_release);
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Public: persist_recover */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
int persist_recover(persist_t* p)
|
||||
{
|
||||
/* bug-1416 (raft-apply-decoupling task 7.2 round 2): close the boot-time
|
||||
* race window for the FULL duration of this call, not just the position
|
||||
* check inside persist_verify_apply_correctness -- see the
|
||||
* recovery_complete field's doc comment (persist.h) for the complete
|
||||
* rationale. Held false until the natural (success) return at the
|
||||
* bottom of this function, which restores it and issues one deliberate
|
||||
* catch-up kick. The two hard-I/O-error early returns below
|
||||
* (load_snapshot / replay_wal failure) deliberately do NOT restore it:
|
||||
* those are pre-existing, rare disk-I/O-level failures the caller
|
||||
* already treats as fatal to this boot attempt (cluster_init_phase2
|
||||
* tears down persist_t on a nonzero return) -- leaving the gate closed
|
||||
* is the conservative, fail-static choice (never let the live apply
|
||||
* pump or live-replication path mutate state on top of a persist_t
|
||||
* whose own recovery never finished), not a new hazard. */
|
||||
atomic_store_explicit(&p->recovery_complete, false, memory_order_release);
|
||||
|
||||
/* --- Step 1: find latest snapshot (highest LSN filename) --- */
|
||||
char snap_file[4096] = "";
|
||||
uint64_t snap_lsn = 0;
|
||||
|
|
@ -3823,6 +3891,40 @@ int persist_recover(persist_t* p)
|
|||
|
||||
/* --- Step 4: retention pass (TTL check) — no-op in P0 (no entities) --- */
|
||||
|
||||
/* bug-1416: replay + verification is fully complete -- safe to let the
|
||||
* live apply pump and the standby live-replication apply path resume
|
||||
* mutating state (see the recovery_complete field's doc comment).
|
||||
* Restored unconditionally here regardless of the verdict computed
|
||||
* above: a genuine APPLY_DIVERGENT already fenced this member via
|
||||
* persist_recover_loud_fence_if_unrecoverable (non-promotable, stays
|
||||
* alive, keeps retrying) -- that sticky fence and its clearing logic
|
||||
* (task 5.3) are untouched by this flag, which governs ONLY whether
|
||||
* in-memory apply may proceed, not whether this member may lead. Issue
|
||||
* one deliberate catch-up kick for anything that committed locally (or
|
||||
* arrived via live replication and was durably persisted but had its
|
||||
* entity-apply deferred, see ha_apply_single_entry / ha_apply_term_
|
||||
* entry_via_library) while this pass ran -- persist_apply_pump_run's
|
||||
* own walk drains fully from last_applied_lsn+1 through the current
|
||||
* committed_lsn, so this is a complete, ordered catch-up, not a partial
|
||||
* one; a harmless no-op if nothing queued. */
|
||||
atomic_store_explicit(&p->recovery_complete, true, memory_order_release);
|
||||
/* n_vcpus==0 is this codebase's own established "not yet initialized"
|
||||
* sentinel for vtaskmgr_t (mirrors cluster_init_phase2's own
|
||||
* `if (c->vtaskmgr.n_vcpus == 0) vtaskmgr_init(...)` guard) -- a real
|
||||
* cluster always has n_vcpus > 0 by the time persist_recover() runs
|
||||
* (vtaskmgr_init happens earlier in myapp_run()/phase2), while a bare
|
||||
* recovery-focused unit-test cluster_t (none of which call
|
||||
* vtaskmgr_init, confirmed by inspection) leaves it zeroed. Passing a
|
||||
* pointer to an uninitialized vtaskmgr_t into vtask_enq_classified
|
||||
* would be unsafe, so fall back to persist_apply_pump_kick's own
|
||||
* existing, already-exercised NULL-pool inline path in that case --
|
||||
* identical to how every apply-pump test fixture in this spec already
|
||||
* exercises the pump. */
|
||||
void* pool = (p->cluster && p->cluster->vtaskmgr.n_vcpus > 0)
|
||||
? &p->cluster->vtaskmgr
|
||||
: NULL;
|
||||
persist_apply_pump_kick(p, pool);
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue