fix(ha): close standby election-freeze (bug-1562) + flusher retry storm (bug-1563)

bug-1562: ha_on_primary_ctrl_fd_rdhup's 5.10b debounce branch (ha.c) returned
without re-arming the liveness watchdog when a debounced RDHUP landed after
the in-flight PreVote round had already concluded (e.g. a lone cold-booting
node whose first round loses in milliseconds with no peers up yet). That was
the last timer ever scheduled -- the node froze permanently in CTRL_STANDBY,
so a 3-node cluster with a long inter-node start stagger could never elect a
primary. Root-caused via a live local reproduction matching the reporting
user's exact timing (byte-for-byte match of their hung ctrl-1.log), fixed by
re-arming the watchdog in that branch (mirroring the function's own 5.2
branch), and confirmed end-to-end: the same scenario now reaches CTRL_PRIMARY
and full 3-node config convergence.

bug-1563: fsync_worker's retry loop (persist.c) only backs off when no work
is pending, but flusher_work_pending_locked() can't distinguish "fresh work
just landed" from "still stuck from the last failed fdatasync" -- a
sustained sync failure left durable_lsn permanently behind tip, so the loop
skipped its wait every iteration and spun fdatasync as fast as the CPU/lock
allowed (a 40-round chaotic /xce-test-ha soak hit 300k+ failed attempts on
one node in under a minute). Fixed by waiting out the normal wal_sync_ms
cadence when a flush was pending but made no progress, leaving the
legitimate idle and just-succeeded fast paths untouched.

Both fixed via TDD (RED test before the fix, GREEN after) and live-validated
by re-running 40-round sequential and chaotic HA soaks end-to-end: both pass
cleanly, and the chaotic run's excluded fdatasync-error count dropped from
304,689 to 0.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-08-17 06:35:06 +00:00
parent f8a4b1e668
commit 846713ab92
Signed by: rshen
SSH key fingerprint: SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
5 changed files with 202 additions and 0 deletions

View file

@ -8585,6 +8585,18 @@ void ha_on_primary_ctrl_fd_rdhup(ha_state_t *ha, cluster_t *c)
"failover (5.10b: a sustained RDHUP cadence must not ladder the "
"term; the in-flight election's own jittered retry handles it)",
(unsigned long)(now_ms - last), (unsigned long)cooldown_ms);
/* bug-1562: unlike the 5.6 guard above (which only skips re-arming
* when it has VERIFIED an election is still in flight), this branch
* only checks elapsed time -- the earlier round may have already
* concluded (e.g. a lone node's PreVote loses in milliseconds with
* no peers up yet). Without re-arming, a debounced trigger that
* lands after that round's own conclusion drops the ONLY remaining
* retry mechanism, freezing the node in CTRL_STANDBY forever with
* no timer ever scheduled again. ha_arm_heartbeat_timeout_vtask is
* safe to call with c == NULL (unit-test mode): it still stamps
* ha->last_activity and simply cannot schedule via a live delaymgr,
* mirroring how the rest of this function already treats c == NULL. */
ha_arm_heartbeat_timeout_vtask(ha, c);
return;
}
}