forked from lei.chen/myxce
fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool, which hardcodes VTASK_LIVENESS for every inbound peer-control packet including vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in persist_promotion_advance_committed) was reachable from that never-blocks task class -- same bug family as bug-1198. Removed the force-flush; round-3 (durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL: persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path and does bounded pread() -- lower risk (page-cache-hot) but strictly still blocking I/O; not safe to remove outright (reopens bug-1043-family job loss), needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask). bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop, lock-free) and routed both call sites through it. Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
This commit is contained in:
parent
ea0ac81c92
commit
40f510fb14
25 changed files with 1617 additions and 226 deletions
|
|
@ -383,26 +383,44 @@ assert_no_loss() {
|
|||
# Assert the cluster currently lists EXACTLY N jobs (Req 5.1/5.2), gated on the
|
||||
# xce_test exit code. Used by the cold-restart-all recovery phase. Returns 0 iff
|
||||
# the live count equals N.
|
||||
#
|
||||
# Retries up to 3 times (2s settle sleep between) before declaring a mismatch:
|
||||
# bug-1396's original "extra job" observation and a fresh repro of this exact
|
||||
# check both trace to the SAME client-side race as bug-1422 (see
|
||||
# .wolf/buglog.json) — right after a fresh election (e.g. this phase's cold
|
||||
# restart), libxce's failover retry loop can exhaust its budget without ever
|
||||
# reaching the new primary and surface a stale E_XCE_STANDBY from whichever
|
||||
# standby it last hit, which reads here as "xce_joblist: Unknown error 10014"
|
||||
# / "could not determine job count" rather than a real count. Confirmed via a
|
||||
# live repro: the aggregate expect-jobs call failed this way while the very
|
||||
# next per-id assert_no_loss check (16 individual expect-job calls) found
|
||||
# every job present. A count still wrong after a real settle + 2 retries is a
|
||||
# genuine mismatch, not this race.
|
||||
assert_job_count() {
|
||||
local label="$1" expected="$2" rc
|
||||
xce_test_script "$(printf 'set masters %s\nclient\nexpect-jobs %s\n' \
|
||||
"$MASTERS" "$expected")"
|
||||
rc=$?
|
||||
if [ "$rc" -eq 0 ]; then
|
||||
pass "$label: cluster lists exactly $expected jobs"
|
||||
return 0
|
||||
else
|
||||
# bug-1396: a bare count mismatch isn't enough to diagnose an extra-job
|
||||
# anomaly — dump the live joblist and the harness's own ACKED_FILE
|
||||
# ledger side by side so the diverging job ID is visible in the log
|
||||
# instead of only the count.
|
||||
yellow " $label: job count mismatch — dumping live joblist vs ACKED_FILE for diagnosis"
|
||||
xce_test_script "$(printf 'set masters %s\nclient\njoblist\n' "$MASTERS")"
|
||||
echo " --- ACKED_FILE (tracked committed jobs, deduplicated) ---"
|
||||
sort -u "$ACKED_FILE" | sed 's/^/ /'
|
||||
fail "$label: job count != $expected after recovery (see joblist/ACKED_FILE dump above)"
|
||||
return 1
|
||||
fi
|
||||
local label="$1" expected="$2" rc attempt
|
||||
for attempt in 1 2 3; do
|
||||
xce_test_script "$(printf 'set masters %s\nclient\nexpect-jobs %s\n' \
|
||||
"$MASTERS" "$expected")"
|
||||
rc=$?
|
||||
if [ "$rc" -eq 0 ]; then
|
||||
pass "$label: cluster lists exactly $expected jobs"
|
||||
return 0
|
||||
fi
|
||||
if [ "$attempt" -lt 3 ]; then
|
||||
yellow " $label attempt $attempt/3: count check failed (likely post-election client retry-storm race, bug-1422 class) — settling 2s and retrying"
|
||||
sleep 2
|
||||
fi
|
||||
done
|
||||
# bug-1396: a bare count mismatch isn't enough to diagnose an extra-job
|
||||
# anomaly — dump the live joblist and the harness's own ACKED_FILE
|
||||
# ledger side by side so the diverging job ID is visible in the log
|
||||
# instead of only the count.
|
||||
yellow " $label: job count mismatch after 3 attempts — dumping live joblist vs ACKED_FILE for diagnosis"
|
||||
xce_test_script "$(printf 'set masters %s\nclient\njoblist\n' "$MASTERS")"
|
||||
echo " --- ACKED_FILE (tracked committed jobs, deduplicated) ---"
|
||||
sort -u "$ACKED_FILE" | sed 's/^/ /'
|
||||
fail "$label: job count != $expected after recovery (see joblist/ACKED_FILE dump above)"
|
||||
return 1
|
||||
}
|
||||
|
||||
# ── soak mode (original racy reproduction) ────────────────────────────────────
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue