fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race

bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool,
which hardcodes VTASK_LIVENESS for every inbound peer-control packet including
vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in
persist_promotion_advance_committed) was reachable from that never-blocks task
class -- same bug family as bug-1198. Removed the force-flush; round-3
(durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat
fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL:
persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path
and does bounded pread() -- lower risk (page-cache-hot) but strictly still
blocking I/O; not safe to remove outright (reopens bug-1043-family job loss),
needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case
or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask).

bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply
raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during
the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could
regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop,
lock-free) and routed both call sites through it.

Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round
sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling
and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears
config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
This commit is contained in:
Rongsong Shen 2026-07-11 12:16:28 +00:00
parent ea0ac81c92
commit 40f510fb14
25 changed files with 1617 additions and 226 deletions

View file

@ -383,26 +383,44 @@ assert_no_loss() {
# Assert the cluster currently lists EXACTLY N jobs (Req 5.1/5.2), gated on the
# xce_test exit code. Used by the cold-restart-all recovery phase. Returns 0 iff
# the live count equals N.
#
# Retries up to 3 times (2s settle sleep between) before declaring a mismatch:
# bug-1396's original "extra job" observation and a fresh repro of this exact
# check both trace to the SAME client-side race as bug-1422 (see
# .wolf/buglog.json) — right after a fresh election (e.g. this phase's cold
# restart), libxce's failover retry loop can exhaust its budget without ever
# reaching the new primary and surface a stale E_XCE_STANDBY from whichever
# standby it last hit, which reads here as "xce_joblist: Unknown error 10014"
# / "could not determine job count" rather than a real count. Confirmed via a
# live repro: the aggregate expect-jobs call failed this way while the very
# next per-id assert_no_loss check (16 individual expect-job calls) found
# every job present. A count still wrong after a real settle + 2 retries is a
# genuine mismatch, not this race.
assert_job_count() {
local label="$1" expected="$2" rc
xce_test_script "$(printf 'set masters %s\nclient\nexpect-jobs %s\n' \
"$MASTERS" "$expected")"
rc=$?
if [ "$rc" -eq 0 ]; then
pass "$label: cluster lists exactly $expected jobs"
return 0
else
# bug-1396: a bare count mismatch isn't enough to diagnose an extra-job
# anomaly — dump the live joblist and the harness's own ACKED_FILE
# ledger side by side so the diverging job ID is visible in the log
# instead of only the count.
yellow " $label: job count mismatch — dumping live joblist vs ACKED_FILE for diagnosis"
xce_test_script "$(printf 'set masters %s\nclient\njoblist\n' "$MASTERS")"
echo " --- ACKED_FILE (tracked committed jobs, deduplicated) ---"
sort -u "$ACKED_FILE" | sed 's/^/ /'
fail "$label: job count != $expected after recovery (see joblist/ACKED_FILE dump above)"
return 1
fi
local label="$1" expected="$2" rc attempt
for attempt in 1 2 3; do
xce_test_script "$(printf 'set masters %s\nclient\nexpect-jobs %s\n' \
"$MASTERS" "$expected")"
rc=$?
if [ "$rc" -eq 0 ]; then
pass "$label: cluster lists exactly $expected jobs"
return 0
fi
if [ "$attempt" -lt 3 ]; then
yellow " $label attempt $attempt/3: count check failed (likely post-election client retry-storm race, bug-1422 class) — settling 2s and retrying"
sleep 2
fi
done
# bug-1396: a bare count mismatch isn't enough to diagnose an extra-job
# anomaly — dump the live joblist and the harness's own ACKED_FILE
# ledger side by side so the diverging job ID is visible in the log
# instead of only the count.
yellow " $label: job count mismatch after 3 attempts — dumping live joblist vs ACKED_FILE for diagnosis"
xce_test_script "$(printf 'set masters %s\nclient\njoblist\n' "$MASTERS")"
echo " --- ACKED_FILE (tracked committed jobs, deduplicated) ---"
sort -u "$ACKED_FILE" | sed 's/^/ /'
fail "$label: job count != $expected after recovery (see joblist/ACKED_FILE dump above)"
return 1
}
# ── soak mode (original racy reproduction) ────────────────────────────────────