wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

fix(raft-apply-decoupling): kill-mid-fact-stream soak + close bug-1416 boot-time TOCTOU (task 7.2)

Browse source 
Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact
traffic against a live 3-node cluster, killing the primary mid-stream and
asserting zero committed-state divergence and zero phantom fact reactions.

Building this scenario surfaced bug-1416: persist_recover()'s boot-time
apply-correctness verification can race the live apply pump and the standby
live-replication apply path, both of which mutate the same last_applied_lsn/
job-table state the verification re-reads live -- producing a false
APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with
a new persist_t.recovery_complete gate that defers only the in-memory apply
(never the durability write or replication ACK) while a recovery pass is in
flight, closed from the earliest safe point in myapp_run() so an election-
retry candidate can't apply live traffic before persist_recover() ever runs.

Round-2 adversarial review independently verified the ACK/durability paths
stay ungated, reproduced the myapp.c fix as load-bearing via revert-and-
reproduce, and confirmed real-divergence detection still fires. Live 3-node
factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover
calls. controller_unit_test: zero new failures across 23 groups.
This commit is contained in:
Rongsong Shen 2026-07-07 03:03:40 +00:00
parent 5f98d1b0ef
commit a5e29a6ffa
17 changed files with 74793 additions and 156 deletions
Download patch file Download diff file Expand all files Collapse all files

38
src/controller/ha.c
View file

@ -2229,6 +2229,26 @@ static int ha_apply_single_entry(persist_t *p, wal_catchup_entry_t *e)
return prc;
}
/* bug-1416 (raft-apply-decoupling task 7.2 round 2): defer the in-memory
* entity apply while this node's OWN boot-time persist_recover() replay+
* verification pass is still running. The entry is already durably
* persisted above (persist_wal_write_replicated, fsync'd) — only the
* ACK and applied-lsn advancement depend on that, per the persist-
* before-apply-before-ACK contract this function's own header comment
* documents, and bug-1043's precedent of decoupling durability/ACK from
* entity materialization. Applying here too, concurrently with
* replay_wal()'s single-threaded, unlocked bookkeeping (last_applied_lsn)
* and the live job table it reads, can produce a false APPLY_DIVERGENT
* verdict on a perfectly healthy boot (confirmed live). No entry is
* lost: this node's own in-progress replay_wal() pass will pick it up
* directly off disk if it lands within that pass's scan, and
* persist_recover()'s own deferred catch-up kick (persist_apply_pump_
* kick, issued right before it returns) covers it unconditionally
* either way, since apply_pump_read_entry() does a fresh on-disk lookup
* every call rather than trusting any earlier directory listing. */
if (!persist_recovery_complete(p))
return 0;
/* Logical entity restore only. persist_wal_write_replicated (above) already
* wrote the entry and owns the on-disk CRC chain + current_lsn — so we pass
* the entry fields directly rather than fabricating a header. (bug-778:
@ -2319,9 +2339,21 @@ static int ha_apply_term_entry_via_library(ha_state_t *ha, persist_t *p,
* find a committed job (it IS on disk; walcompare passes), looking like a
* committed-job loss. Re-apply here, mirroring the legacy path. Idempotent
* on a re-stream (restore_job upserts by jobid). WAL_NOOP / WAL_CONFIG_
* MARKER carry no entity payload — apply_wal_entry no-ops on them. */
apply_wal_entry(p, (uint32_t)e->entry_type, e->lsn,
e->raw_payload, (uint32_t)e->raw_payload_len);
* MARKER carry no entity payload — apply_wal_entry no-ops on them.
*
* bug-1416 (raft-apply-decoupling task 7.2 round 2): gated on
* persist_recovery_complete() for the identical reason documented on
* ha_apply_single_entry's own matching guard above — defer the
* in-memory apply while this node's own boot-time persist_recover()
* pass is in progress, never the ACK/applied_lsn advance below (the
* entry is already durably persisted by on_persist_entry's
* persist_wal_write_replicated, invoked via ha_consume_append_entries
* above). This node's own in-progress replay, or the deferred
* catch-up kick persist_recover() issues before returning, applies
* it instead. */
if (persist_recovery_complete(p))
apply_wal_entry(p, (uint32_t)e->entry_type, e->lsn,
e->raw_payload, (uint32_t)e->raw_payload_len);
ha_advance_applied_lsn(ha, e->lsn);
ha_send_wal_ack_reply(ha, ep, c, e->lsn);