fix(ha): close standby election-freeze (bug-1562) + flusher retry storm (bug-1563)

bug-1562: ha_on_primary_ctrl_fd_rdhup's 5.10b debounce branch (ha.c) returned
without re-arming the liveness watchdog when a debounced RDHUP landed after
the in-flight PreVote round had already concluded (e.g. a lone cold-booting
node whose first round loses in milliseconds with no peers up yet). That was
the last timer ever scheduled -- the node froze permanently in CTRL_STANDBY,
so a 3-node cluster with a long inter-node start stagger could never elect a
primary. Root-caused via a live local reproduction matching the reporting
user's exact timing (byte-for-byte match of their hung ctrl-1.log), fixed by
re-arming the watchdog in that branch (mirroring the function's own 5.2
branch), and confirmed end-to-end: the same scenario now reaches CTRL_PRIMARY
and full 3-node config convergence.

bug-1563: fsync_worker's retry loop (persist.c) only backs off when no work
is pending, but flusher_work_pending_locked() can't distinguish "fresh work
just landed" from "still stuck from the last failed fdatasync" -- a
sustained sync failure left durable_lsn permanently behind tip, so the loop
skipped its wait every iteration and spun fdatasync as fast as the CPU/lock
allowed (a 40-round chaotic /xce-test-ha soak hit 300k+ failed attempts on
one node in under a minute). Fixed by waiting out the normal wal_sync_ms
cadence when a flush was pending but made no progress, leaving the
legitimate idle and just-succeeded fast paths untouched.

Both fixed via TDD (RED test before the fix, GREEN after) and live-validated
by re-running 40-round sequential and chaotic HA soaks end-to-end: both pass
cleanly, and the chaotic run's excluded fdatasync-error count dropped from
304,689 to 0.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

src/controller/testsuite/test_ha_rdhup_integration.c

@@ -642,11 +642,83 @@ static void test_rdhup_within_cooldown_is_debounced(void)
     printf("test_rdhup_within_cooldown_is_debounced: PASS\n");
 }
 
+/* ═══════════════════════════════════════════════════════════════════════════
+ * Test: a debounced RDHUP must still re-arm the liveness watchdog (bug-1562)
+ *
+ * The 5.10b debounce branch (sibling test above) correctly suppresses a
+ * fresh failover attempt when an RDHUP lands within the cooldown window. But
+ * unlike the 5.6 guard right above it in the function (which only skips
+ * re-arming when it has VERIFIED an election is actually in flight,
+ * role==CTRL_ELECTING || pre_vote_in_progress), the 5.10b branch only checks
+ * elapsed time since last_failover_ms -- it never checks whether an election
+ * is still running. On a lone cold-booting node, the first PreVote round
+ * loses within milliseconds (no peers reachable yet -> connect failures
+ * count as "no" votes), which re-arms the watchdog once. When that watchdog
+ * fires again ~heartbeat_timeout_ms later, last_failover_ms is still inside
+ * the 2x-election_timeout cooldown even though no election is in flight
+ * anymore -- so pre-fix, the debounce branch returns without re-arming
+ * anything. That is the LAST timer ever scheduled in the process: a live
+ * repro (matching a real user's failed 3-node deploy exactly) confirmed
+ * every vtask worker thread idling on an empty queue and the io_worker
+ * parked in epoll_wait with zero pending connections, 9+ minutes of total
+ * silence -- the node is permanently stuck CTRL_STANDBY, and since it is the
+ * only node holding a local config, the whole cluster never elects a
+ * primary.
+ *
+ * ha_arm_heartbeat_timeout_vtask stamps ha->last_activity to "now" as its
+ * first, delaymgr-independent side effect (ha.c) -- the same observable
+ * test_ha_arm_no_crash_no_thread (test_ha_liveness.c) already uses to prove
+ * arming happened without a live delaymgr thread (c == NULL here).
+ * ═════════════════════════════════════════════════════════════════════════ */
+static void test_rdhup_debounce_still_rearms_liveness_watchdog(void)
+{
+    ha_state_t ha;
+    TEST_ASSERT_EQUAL_INT(0, ha_init(&ha, &g_cfg, 0));
+    ha_do_demote(&ha); /* role = CTRL_STANDBY */
+
+    /* Sentinel the debounce branch itself never writes -- a change proves
+     * ha_arm_heartbeat_timeout_vtask actually ran. */
+    ha.last_activity = 1;
+
+    /* A failover was triggered "just now" -> this RDHUP lands inside the
+     * cooldown window and must be debounced (mirrors the sibling test). */
+    struct timespec ts;
+    clock_gettime(CLOCK_MONOTONIC, &ts);
+    uint64_t now_ms = (uint64_t)ts.tv_sec * 1000u + (uint64_t)(ts.tv_nsec / 1000000L);
+    atomic_store_explicit(&ha.last_failover_ms, now_ms, memory_order_release);
+
+    int sv[2];
+    TEST_ASSERT_EQUAL_INT(0, socketpair(AF_UNIX, SOCK_STREAM, 0, sv));
+    ha.primary_ctrl_fd = sv[0];
+
+    ha_on_primary_ctrl_fd_rdhup(&ha, NULL);
+    close(sv[1]);
+
+    /* Still debounced -- no fresh failover, stays CTRL_STANDBY (same
+     * guarantee as the sibling test). */
+    TEST_ASSERT_EQUAL_INT_MESSAGE((int)CTRL_STANDBY,
+        (int)atomic_load_explicit(&ha.role, memory_order_acquire),
+        "a debounced RDHUP must not itself trigger a fresh failover");
+
+    /* The watchdog must have been re-armed even though this trigger was
+     * debounced -- last_activity must have moved off the sentinel. */
+    TEST_ASSERT_TRUE_MESSAGE((int)ha.last_activity != 1,
+        "a debounced RDHUP must still re-arm the liveness watchdog (bug-1562) "
+        "-- otherwise a lone node whose first PreVote loses fast then hits a "
+        "second liveness timeout inside the cooldown window freezes forever, "
+        "with no timer ever scheduled again");
+
+    ha_disarm_heartbeat_timeout_vtask(&ha);
+    ha_fini(&ha);
+    printf("test_rdhup_debounce_still_rearms_liveness_watchdog: PASS\n");
+}
+
 void test_ha_rdhup_integration(void)
 {
     UnitySetTestFile(__FILE__);
     RUN_TEST(test_rdhup_candidate_advertises_freshness_floor_not_stale_committed);
     RUN_TEST(test_rdhup_within_cooldown_is_debounced);
+    RUN_TEST(test_rdhup_debounce_still_rearms_liveness_watchdog);
     RUN_TEST(test_rdhup_full_path_reaches_primary);
     /* ack-driven promotion test deleted (ha-single-leader-role 4.1:
      * ack-driven promotion machinery removed, no shim) */