lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha): close standby election-freeze (bug-1562) + flusher retry storm (bug-1563)

Browse source 
bug-1562: ha_on_primary_ctrl_fd_rdhup's 5.10b debounce branch (ha.c) returned
without re-arming the liveness watchdog when a debounced RDHUP landed after
the in-flight PreVote round had already concluded (e.g. a lone cold-booting
node whose first round loses in milliseconds with no peers up yet). That was
the last timer ever scheduled -- the node froze permanently in CTRL_STANDBY,
so a 3-node cluster with a long inter-node start stagger could never elect a
primary. Root-caused via a live local reproduction matching the reporting
user's exact timing (byte-for-byte match of their hung ctrl-1.log), fixed by
re-arming the watchdog in that branch (mirroring the function's own 5.2
branch), and confirmed end-to-end: the same scenario now reaches CTRL_PRIMARY
and full 3-node config convergence.

bug-1563: fsync_worker's retry loop (persist.c) only backs off when no work
is pending, but flusher_work_pending_locked() can't distinguish "fresh work
just landed" from "still stuck from the last failed fdatasync" -- a
sustained sync failure left durable_lsn permanently behind tip, so the loop
skipped its wait every iteration and spun fdatasync as fast as the CPU/lock
allowed (a 40-round chaotic /xce-test-ha soak hit 300k+ failed attempts on
one node in under a minute). Fixed by waiting out the normal wal_sync_ms
cadence when a flush was pending but made no progress, leaving the
legitimate idle and just-succeeded fast paths untouched.

Both fixed via TDD (RED test before the fix, GREEN after) and live-validated
by re-running 40-round sequential and chaotic HA soaks end-to-end: both pass
cleanly, and the chaotic run's excluded fdatasync-error count dropped from
304,689 to 0.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-08-17 06:35:06 +00:00
parent f8a4b1e668
commit 846713ab92
Signed by: rshen
SSH key fingerprint: SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
5 changed files with 202 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

7
src/include/persist.h
View file

@ -536,6 +536,13 @@ struct persist_s {
* must NOT advance and the failure is logged loudly. Compiled out of
* production (xce_controller is built without -DXCE_TESTING). */
atomic_int test_force_fsync_fail;
/* Flusher attempt counter (bug-1563): incremented once per
* flusher_flush_once() invocation, regardless of outcome. A sustained
* fdatasync failure must not spin the async fsync_worker thread at full
* CPU with zero backoff between retries -- this proves attempt rate
* stays bounded to roughly the wal_sync_ms cadence even while durable_lsn
* is stuck. Compiled out of production. */
atomic_uint test_flusher_flush_attempt_count;
/* fdatasync observability counter (ha-commit-durability task 2.3):
* incremented at the standby-path inline sync in
* persist_wal_write_replicated, BEFORE that function returns (and thus