fix(ha): close standby election-freeze (bug-1562) + flusher retry storm (bug-1563)
bug-1562: ha_on_primary_ctrl_fd_rdhup's 5.10b debounce branch (ha.c) returned without re-arming the liveness watchdog when a debounced RDHUP landed after the in-flight PreVote round had already concluded (e.g. a lone cold-booting node whose first round loses in milliseconds with no peers up yet). That was the last timer ever scheduled -- the node froze permanently in CTRL_STANDBY, so a 3-node cluster with a long inter-node start stagger could never elect a primary. Root-caused via a live local reproduction matching the reporting user's exact timing (byte-for-byte match of their hung ctrl-1.log), fixed by re-arming the watchdog in that branch (mirroring the function's own 5.2 branch), and confirmed end-to-end: the same scenario now reaches CTRL_PRIMARY and full 3-node config convergence. bug-1563: fsync_worker's retry loop (persist.c) only backs off when no work is pending, but flusher_work_pending_locked() can't distinguish "fresh work just landed" from "still stuck from the last failed fdatasync" -- a sustained sync failure left durable_lsn permanently behind tip, so the loop skipped its wait every iteration and spun fdatasync as fast as the CPU/lock allowed (a 40-round chaotic /xce-test-ha soak hit 300k+ failed attempts on one node in under a minute). Fixed by waiting out the normal wal_sync_ms cadence when a flush was pending but made no progress, leaving the legitimate idle and just-succeeded fast paths untouched. Both fixed via TDD (RED test before the fix, GREEN after) and live-validated by re-running 40-round sequential and chaotic HA soaks end-to-end: both pass cleanly, and the chaotic run's excluded fdatasync-error count dropped from 304,689 to 0. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
f8a4b1e668
commit
846713ab92
5 changed files with 202 additions and 0 deletions
|
|
@ -8585,6 +8585,18 @@ void ha_on_primary_ctrl_fd_rdhup(ha_state_t *ha, cluster_t *c)
|
|||
"failover (5.10b: a sustained RDHUP cadence must not ladder the "
|
||||
"term; the in-flight election's own jittered retry handles it)",
|
||||
(unsigned long)(now_ms - last), (unsigned long)cooldown_ms);
|
||||
/* bug-1562: unlike the 5.6 guard above (which only skips re-arming
|
||||
* when it has VERIFIED an election is still in flight), this branch
|
||||
* only checks elapsed time -- the earlier round may have already
|
||||
* concluded (e.g. a lone node's PreVote loses in milliseconds with
|
||||
* no peers up yet). Without re-arming, a debounced trigger that
|
||||
* lands after that round's own conclusion drops the ONLY remaining
|
||||
* retry mechanism, freezing the node in CTRL_STANDBY forever with
|
||||
* no timer ever scheduled again. ha_arm_heartbeat_timeout_vtask is
|
||||
* safe to call with c == NULL (unit-test mode): it still stamps
|
||||
* ha->last_activity and simply cannot schedule via a live delaymgr,
|
||||
* mirroring how the rest of this function already treats c == NULL. */
|
||||
ha_arm_heartbeat_timeout_vtask(ha, c);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue