lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race

Browse source 
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool,
which hardcodes VTASK_LIVENESS for every inbound peer-control packet including
vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in
persist_promotion_advance_committed) was reachable from that never-blocks task
class -- same bug family as bug-1198. Removed the force-flush; round-3
(durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat
fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL:
persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path
and does bounded pread() -- lower risk (page-cache-hot) but strictly still
blocking I/O; not safe to remove outright (reopens bug-1043-family job loss),
needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case
or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask).

bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply
raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during
the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could
regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop,
lock-free) and routed both call sites through it.

Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round
sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling
and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears
config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
This commit is contained in:
Rongsong Shen 2026-07-11 12:16:28 +00:00
parent ea0ac81c92
commit 40f510fb14
25 changed files with 1617 additions and 226 deletions
Download patch file Download diff file Expand all files Collapse all files

39
src/include/persist.h
View file

@ -964,6 +964,37 @@ int persist_checkpoint(persist_t* p);
* Idempotent; returns 0 on success, negative errno on I/O error. */
int persist_truncate_wal_above(persist_t* p, uint64_t cutoff_lsn);
/* persist_promotion_advance_committed (2026-07-10): called once by a newly-
* promoted leader (raft_cb_become_leader) right after persist_truncate_wal_above,
* to unconditionally advance committed_lsn toward the post-election safe commit
* point `safe_lsn`. persist_truncate_wal_above only sets committed_lsn on a
* REAL truncation; its common no-op path (this leader's own log already
* covers safe_lsn) leaves committed_lsn stale, stalling commit-gated apply
* until the normal ack-driven path catches up seconds later. Durable-capped
* (never advances committed_lsn past this node's own durable_lsn) and
* monotonic (never regresses) — uses persist_committed_lsn_advance_atomic, so
* it is also race-safe against a concurrent advance from another writer (e.g.
* ha_standby_advance_committed_and_apply, which can still run during the
* promotion window before role flips to CTRL_PRIMARY). Does NOT force a
* synchronous flush if durable_lsn lags — this function never blocks (see the
* DESIGN NOTE at its definition for why a blocking fdatasync here would
* violate the VTASK_LIVENESS thread-role contract, controller-invariant-audit
* 2026-07-10, bug-1198-class finding). Safe to call with p == NULL (no-op). */
void persist_promotion_advance_committed(persist_t* p, uint64_t safe_lsn);
/* persist_committed_lsn_advance_atomic: race-safe monotonic-max advance of
* p->committed_lsn to max(current, target) via a CAS retry loop — safe
* against ANY concurrent caller, no lock required or held. A naive
* load-then-conditional-store (the pattern this replaces) can lose an update
* under a genuine race: controller-invariant-audit (2026-07-10) found
* persist_promotion_advance_committed and ha_standby_advance_committed_and_apply
* can race during the promotion window before role flips to CTRL_PRIMARY (a
* still-in-flight WAL catch-up entry can be processed on another thread at
* the same instant). Never regresses committed_lsn. Returns the value
* observed to be current after the call (may exceed `target` if another
* writer raced ahead — this is not a failure, just the up-to-date value). */
uint64_t persist_committed_lsn_advance_atomic(persist_t* p, uint64_t target);
/* Logging helper — never logs payload content (security invariant) */
void persist_log_entry_metadata(const wal_entry_hdr_t* hdr);
@ -1193,6 +1224,14 @@ int persist_apply_pump_run(persist_t* p);
* most one walk is ever scheduled or running at a time per persist_t. */
void persist_apply_pump_kick(persist_t* p, void* pool);
/* persist_apply_pump_run_sync (2026-07-10): synchronously drain the apply
* pump on the calling thread instead of dispatching to a vtask. Only for
* callers that can tolerate bounded blocking (e.g. VTASK_LIVE_BLOCK work
* such as promotion) -- never call from iomgr or delaymgr. Same CAS-claim
* coalescing as persist_apply_pump_kick: a losing concurrent caller is a
* safe no-op. See persist.c for the promotion-race rationale. */
void persist_apply_pump_run_sync(persist_t* p);
int persist_job_hook_fun(int type, int situation, int detail,
void* cls, va_list ap);
int persist_queue_hook_fun(int type, int situation, int detail,