lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha): close standby election-freeze (bug-1562) + flusher retry storm (bug-1563)

Browse source 
bug-1562: ha_on_primary_ctrl_fd_rdhup's 5.10b debounce branch (ha.c) returned
without re-arming the liveness watchdog when a debounced RDHUP landed after
the in-flight PreVote round had already concluded (e.g. a lone cold-booting
node whose first round loses in milliseconds with no peers up yet). That was
the last timer ever scheduled -- the node froze permanently in CTRL_STANDBY,
so a 3-node cluster with a long inter-node start stagger could never elect a
primary. Root-caused via a live local reproduction matching the reporting
user's exact timing (byte-for-byte match of their hung ctrl-1.log), fixed by
re-arming the watchdog in that branch (mirroring the function's own 5.2
branch), and confirmed end-to-end: the same scenario now reaches CTRL_PRIMARY
and full 3-node config convergence.

bug-1563: fsync_worker's retry loop (persist.c) only backs off when no work
is pending, but flusher_work_pending_locked() can't distinguish "fresh work
just landed" from "still stuck from the last failed fdatasync" -- a
sustained sync failure left durable_lsn permanently behind tip, so the loop
skipped its wait every iteration and spun fdatasync as fast as the CPU/lock
allowed (a 40-round chaotic /xce-test-ha soak hit 300k+ failed attempts on
one node in under a minute). Fixed by waiting out the normal wal_sync_ms
cadence when a flush was pending but made no progress, leaving the
legitimate idle and just-succeeded fast paths untouched.

Both fixed via TDD (RED test before the fix, GREEN after) and live-validated
by re-running 40-round sequential and chaotic HA soaks end-to-end: both pass
cleanly, and the chaotic run's excluded fdatasync-error count dropped from
304,689 to 0.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-08-17 06:35:06 +00:00
parent f8a4b1e668
commit 846713ab92
Signed by: rshen
SSH key fingerprint: SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
5 changed files with 202 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

34
src/controller/persist.c
View file

@ -354,6 +354,10 @@ static bool flusher_work_pending_locked(persist_t* p)
* durable_lsn (unchanged if there was nothing to sync). */
static uint64_t flusher_flush_once(persist_t* p)
{
#ifdef XCE_TESTING
atomic_fetch_add_explicit(&p->test_flusher_flush_attempt_count, 1,
memory_order_relaxed);
#endif
pthread_mutex_lock(&p->lock);
uint64_t tip = (p->current_lsn > 0) ? p->current_lsn - 1 : 0;
@ -491,6 +495,36 @@ static void* fsync_worker(void* arg)
}
pthread_mutex_lock(&p->fsync_mutex);
/* bug-1563: a sustained fdatasync failure leaves durable_lsn < tip
* forever (the fail-loud gate above deliberately never advances it
* on failure — that part is correct and unchanged). flusher_work_
* pending_locked() can therefore never distinguish "genuine fresh
* work just landed" from "still stuck from the last failed attempt"
* — both read as pending=true, so without this check the loop above
* would skip the wait on EVERY iteration and spin flusher_flush_once
* (and thus fdatasync) as fast as the CPU/lock allow. Live evidence:
* a 40-round chaotic /xce-test-ha soak hit 300k+ failed attempts on
* one node in well under a minute (one core pinned, log flooded).
* Only skip the wait when the flush actually made progress (pending
* was true and it succeeded) or there was nothing pending to begin
* with (the pre-existing, correct fast path); a flush that was
* pending but made no progress (after == before) still waits out the
* normal wal_sync_ms cadence before retrying — same bounded-latency
* budget the idle path already uses, no new tunable needed. */
if (pending && after == before &&
!atomic_load_explicit(&p->fsync_stop, memory_order_acquire)) {
long ms = (long)p->config.wal_sync_ms;
struct timespec dl;
clock_gettime(CLOCK_REALTIME, &dl);
dl.tv_sec += ms / 1000;
dl.tv_nsec += (ms % 1000) * 1000000L;
if (dl.tv_nsec >= 1000000000L) {
dl.tv_sec += 1;
dl.tv_nsec -= 1000000000L;
}
pthread_cond_timedwait(&p->fsync_cond, &p->fsync_mutex, &dl);
}
}
pthread_mutex_unlock(&p->fsync_mutex);
return NULL;