fix(controller): debounce RDHUP-triggered failover — stop the term-escalation storm (bug-1310, task 5.10b)

A dead primary's control fd RDHUPs faster than an election settles. The 5.6
guard only drops an RDHUP while the role is instantaneously CTRL_ELECTING/
prevote; an RDHUP landing in the gap after a lost election (role briefly back
to CTRL_STANDBY) re-triggers a fresh ha_begin_failover -> PreVote -> real
election (term++). One bump per RDHUP -> unbounded escalation under a sustained
cadence (observed round 4: 48 begin_failovers, terms 6->24, round overran the
harness window).

Fix: ha_state_t.last_failover_ms (atomic) stamped at ha_begin_failover;
ha_on_primary_ctrl_fd_rdhup drops an RDHUP within 2*election_timeout_ms of the
last trigger (closes the dead fd, returns) so the election's own jittered retry
(5.7) runs without the external RDHUP cadence adding term bumps. First trigger
never debounced; genuine loss still fails over, at most once per window.

Validation: unit-proven — test_rdhup_within_cooldown_is_debounced (recent
last_failover -> stays CTRL_STANDBY) vs test_rdhup_single_node_reaches_primary
(last_failover=0 -> CTRL_PRIMARY) is a differential proof. Soak: 10/10 elect,
terms 4->14 bounded, begin_failover 14/14/12 (was 48), committed_job_loss=0, no
regression. The stochastic storm did not recur this run (debounce fired 0x), so
soak-reproduction of the storm is still pending; the mechanism rests on the
unit test. Election suites green (rdhup 8/0, prevote 12/0, core_safety 24/0,
election 28/0, failover_prevote 6/0).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX

src/controller/ha.c

@@ -6725,9 +6725,22 @@ static int ha_failover_stagger_fn(void *arg)
     return 0;
 }
 
+/* CLOCK_MONOTONIC milliseconds — for the RDHUP failover debounce (5.10b). */
+static uint64_t ha_mono_ms(void)
+{
+    struct timespec ts;
+    clock_gettime(CLOCK_MONOTONIC, &ts);
+    return (uint64_t)ts.tv_sec * 1000u + (uint64_t)(ts.tv_nsec / 1000000L);
+}
+
 void ha_begin_failover(ha_state_t *ha, cluster_t *c)
 {
     if (!ha) return;
+    /* Stamp the failover-trigger time so a rapid RDHUP cadence from a dead
+     * primary's fd cannot re-trigger a fresh failover on every event (5.10b
+     * debounce in ha_on_primary_ctrl_fd_rdhup). Any trigger source (RDHUP or the
+     * deferred heartbeat-timeout stagger) records here. */
+    atomic_store_explicit(&ha->last_failover_ms, ha_mono_ms(), memory_order_release);
     INFO("HA: beginning failover via PreVote (no truncate / no term bump until "
          "PreVote wins) — staying CTRL_STANDBY");
 
@@ -6790,6 +6803,38 @@ void ha_on_primary_ctrl_fd_rdhup(ha_state_t *ha, cluster_t *c)
         }
     }
 
+    /* raft-ha-core-safety 5.10b: RDHUP failover DEBOUNCE. The 5.6 guard above
+     * only catches an RDHUP while the role is INSTANTANEOUSLY CTRL_ELECTING /
+     * pre_vote_in_progress. But a dead primary's fd RDHUPs faster than an
+     * election settles, and each RDHUP that lands in the gap AFTER a real
+     * election loses (role briefly back at CTRL_STANDDBY before the next attempt)
+     * re-triggers a fresh failover → a term bump per RDHUP (observed: 48
+     * begin_failovers, terms 6→24 in one round; research.md §"5.10b DEEPER").
+     * Suppress an RDHUP-triggered failover for up to 2× the election timeout
+     * after the last trigger — long enough to cover the jittered [T,2T] election
+     * watchdog (5.7) so the election's OWN retry runs without the external RDHUP
+     * cadence piling on extra term bumps. A genuinely-dead primary is still
+     * failed over — at most once per window, not once per RDHUP. First trigger
+     * (last_failover_ms == 0) is never debounced. */
+    {
+        uint64_t now_ms = ha_mono_ms();
+        uint64_t last   = atomic_load_explicit(&ha->last_failover_ms,
+                                               memory_order_acquire);
+        uint64_t cooldown_ms = 2u * (uint64_t)ha->config.election_timeout_ms;
+        if (last != 0 && now_ms >= last && (now_ms - last) < cooldown_ms) {
+            if (ha->primary_ctrl_fd >= 0) {
+                close(ha->primary_ctrl_fd);
+                ha->primary_ctrl_fd = -1;
+            }
+            INFO("HA: primary_ctrl_fd RDHUP within failover cooldown (%lums since "
+                 "last trigger, cooldown=%lums) — debouncing, not restarting "
+                 "failover (5.10b: a sustained RDHUP cadence must not ladder the "
+                 "term; the in-flight election's own jittered retry handles it)",
+                 (unsigned long)(now_ms - last), (unsigned long)cooldown_ms);
+            return;
+        }
+    }
+
     /* Cancel the liveness poll task first (Req 5.4): prevents a concurrent
      * poll fn from triggering a second failover call after EPOLLRDHUP fires. */
     ha_disarm_heartbeat_timeout_vtask(ha);