fix(ha-election,ha-rejoin): stop two infinite-election loops on failover and restart

Two distinct HA livelocks, both verified on a live 3-node ASAN deploy
(kill+restart of primary and standby, plus SIGSTOP wedge injection).

bug-339 — election freshness-floor asymmetry (RDHUP failover path):
After the primary dies the two survivors could never elect a leader — both
only ever became follower while terms climbed in lockstep. The RDHUP candidate
advertised c->persist.committed_lsn, which a standby never advances (only the
primary commit engine writes it), while every voter computed its freshness floor
from max(applied_lsn, last_known_committed_lsn). So each survivor advertised a
stale-low LSN against a high voter floor and the freshness gate denied on both
sides. Fix: compute the candidate LSN via ha_vote_self_freshness so the
advertised value equals the voter floor (ha_on_primary_ctrl_fd_rdhup); also route
the boot-path election (myapp_run) through the same helper for symmetry.

bug-340 — wedged standby endpoint suppressed reconnect (rejoin path):
A killed/restarted peer's outbound endpoint on the primary could wedge (epoll
re-arm ENOENT) yet stay in standby_endpoints, so primary_heartbeat_fn's reconnect
scan treated the peer as still-connected and never re-established a ctrlhello
channel — the restarted node never heard from the primary and re-elected every
heartbeat_timeout (term climbing unbounded). xce_nop_req returns success on
enqueue even when the send can't progress, so the heartbeat layer never noticed.
Fix: primary_heartbeat_fn now evicts a standby that was responsive once
(ts_last_in > 0) but has been silent past heartbeat_timeout — symmetric with the
standby's own liveness poll — and the reconnect scan re-establishes the channel.
Uses existing endpoint_t.ts_last_in (no libnetio change).

Tests (RED->GREEN):
- test_ha_rdhup_integration: candidate advertises freshness floor, not stale committed_lsn
- test_ha_liveness: evicts_silent_standby + keeps_live/keeps_unconfirmed (no false eviction)

src/controller/testsuite/test_ha_rdhup_integration.c

@@ -496,10 +496,90 @@ static void test_rdhup_configured_node_still_elects(void)
            g_guard_hook_calls);
 }
 
+/* ═══════════════════════════════════════════════════════════════════════════
+ * Test 9 (regression — infinite election loop after primary death):
+ * The RDHUP candidate must advertise its EFFECTIVE freshness floor
+ * (max(applied_lsn, last_known_committed_lsn)) as the RequestVote last_log_lsn,
+ * NOT the stale persist.committed_lsn.
+ *
+ * Root cause of the observed bug: a standby never advances
+ * persist.committed_lsn (only the primary's commit engine writes it); it
+ * advances applied_lsn (WAL catch-up) and last_known_committed_lsn (heartbeat
+ * replies).  The voter-side freshness floor (cluster.c OPCode_requestVote) is
+ * max(applied_lsn, last_known_committed_lsn).  When the RDHUP candidate
+ * advertised persist.committed_lsn (stale-low) while every voter compared
+ * against a high applied_lsn floor, the freshness gate (cand_lsn < my_lsn)
+ * tripped on BOTH surviving standbys — each denied the other's vote, no
+ * candidate ever reached quorum, and the cluster spun in an infinite
+ * RDHUP→ELECTING→STANDBY loop with terms climbing in lockstep.
+ *
+ * Captured via the ha_rdhup_election_hook my_lsn argument.
+ * Before the fix: captured == 0 (stale committed_lsn).  After: == 10.
+ * ═════════════════════════════════════════════════════════════════════════ */
+static uint64_t g_captured_my_lsn = 0;
+static uint64_t g_captured_my_tag = 0;
+
+static void capture_lsn_hook(ha_state_t *ha, cluster_t *c,
+                             int n_peers, xce_candidate_t **peers,
+                             uint64_t my_config_tag, uint64_t my_lsn,
+                             const char *my_name)
+{
+    (void)ha; (void)c; (void)n_peers; (void)peers; (void)my_name;
+    g_captured_my_tag = my_config_tag;
+    g_captured_my_lsn = my_lsn;
+}
+
+static void test_rdhup_candidate_advertises_freshness_floor_not_stale_committed(void)
+{
+    install_stubs();
+    g_captured_my_lsn = 0;
+    g_captured_my_tag = 0;
+
+    cluster_t c;
+    myconf_t  myconf;
+    make_cluster(&c, &myconf, "ctrl-a");
+
+    TEST_ASSERT_EQUAL_INT(0, ha_init(&c.ha, &g_cfg, 0));
+    ha_do_demote(&c.ha);
+    c.ha.phase2_done = 1;
+    c.ha.n_peers     = 1;
+
+    /* Standby state after applying WAL + learning commits from the (now-dead)
+     * primary: applied_lsn and last_known_committed_lsn advanced to 10, but
+     * persist.committed_lsn is still 0 — a standby never advances it. */
+    atomic_store_explicit(&c.ha.applied_lsn, 10ULL, memory_order_release);
+    atomic_store_explicit(&c.ha.last_known_committed_lsn, 10ULL,
+                          memory_order_release);
+    atomic_store_explicit(&c.persist.committed_lsn, 0ULL, memory_order_release);
+
+    ha_rdhup_election_hook = capture_lsn_hook;
+
+    int sv[2];
+    TEST_ASSERT_EQUAL_INT(0, socketpair(AF_UNIX, SOCK_STREAM, 0, sv));
+    c.ha.primary_ctrl_fd = sv[0];
+
+    ha_on_primary_ctrl_fd_rdhup(&c.ha, &c);
+
+    clear_stubs();
+    close(sv[1]);
+
+    TEST_ASSERT_EQUAL_UINT64_MESSAGE(10ULL, g_captured_my_lsn,
+        "RDHUP candidate must advertise effective freshness floor "
+        "(applied/learned-committed = 10), not stale persist.committed_lsn (0); "
+        "otherwise surviving standbys mutually deny votes -> infinite election loop");
+    TEST_ASSERT_EQUAL_UINT64_MESSAGE((uint64_t)TEST_CONFIG_TAG, g_captured_my_tag,
+        "RDHUP candidate must advertise its config_version_tag");
+
+    ha_fini(&c.ha);
+    printf("test_rdhup_candidate_advertises_freshness_floor_not_stale_committed: "
+           "PASS (my_lsn=%llu)\n", (unsigned long long)g_captured_my_lsn);
+}
+
 /* ── Suite entry point ──────────────────────────────────────────────────── */
 
 void test_ha_rdhup_integration(void)
 {
+    RUN_TEST(test_rdhup_candidate_advertises_freshness_floor_not_stale_committed);
     RUN_TEST(test_rdhup_full_path_reaches_primary);
     /* ack-driven promotion test deleted (ha-single-leader-role 4.1:
      * ack-driven promotion machinery removed, no shim) */