--- id: "bug-339" status: "fixed" status_updated: "2026-09-24" status_note: "" component: "controller" severity: "unknown" timestamp: "2026-06-09T17:30:00Z" last_seen: "2026-09-19T09:00:00Z" occurrences: 3 files: ["src/controller/ha.c", "src/controller/myapp.c", "src/controller/testsuite/test_ha_rdhup_integration.c"] tags: ["guard-clause", "c", "raft", "election-safety", "restart", "persistence", "gap-analysis", "ha", "election", "rdhup", "failover", "freshness-gate", "committed_lsn", "applied_lsn", "split-vote", "infinite-loop"] related: ["bug-338"] legacy: true --- # bug-339: After killing the primary in a 3-node controller HA cluster, the two surviving controllers enter… ## Description After killing the primary in a 3-node controller HA cluster, the two surviving controllers enter an infinite election loop: both only ever raft_cb_become_follower (never become_leader), terms climb in lockstep (4..23), RDHUP -> CTRL_ELECTING -> CTRL_STANDBY repeats forever. ## Environment _Not recorded (migrated from buglog.json)._ ## Reproduction _Not recorded (migrated from buglog.json)._ ## Root cause LSN-source asymmetry between candidate-advertised LSN and voter freshness floor, specific to the RDHUP failover election path. ha_on_primary_ctrl_fd_rdhup advertised c->persist.committed_lsn as the RequestVote last_log_lsn, but a STANDBY never advances persist.committed_lsn (only the primary's commit engine writes it at persist.c:1192). Standbys advance applied_lsn (ha_apply_wal_catchup_entry) and last_known_committed_lsn (heartbeat replies). The voter-side freshness floor (cluster.c OPCode_requestVote) is max(applied_lsn, last_known_committed_lsn) via ha_vote_self_freshness. So once a standby applied any WAL beyond its last snapshot, every surviving standby advertised a stale-low LSN (e.g. 0) while each peer computed a HIGH floor (e.g. 10); the freshness gate cand_lsn < my_lsn tripped on BOTH sides, each denied the other's vote, neither reached quorum (2 of 3), each election timed out -> become_follower -> 10s liveness timeout -> RDHUP -> re-elect at term+1. The boot path (myapp.c:217) already advertised applied_lsn, so cold-boot election worked; only the RDHUP failover path was inconsistent. ## Fix In ha_on_primary_ctrl_fd_rdhup, compute candidate freshness ONCE via ha_vote_self_freshness(ha, config_version_tag, applied_lsn, &cand_tag, &cand_lsn) right after ha->raft_c=c, and use cand_tag/cand_lsn for BOTH the elect_ctx (votes[0].my_tag/my_lsn + self_tag/self_lsn) and the XCE_TESTING ha_rdhup_election_hook args. This makes candidate-advertised LSN identical to the voter-side floor, so for any two nodes A,B: A grants B iff floor_B >= floor_A -> the freshest node is always granted by every peer; equal-floor ties are broken by raft priority delay. Regression test added: test_rdhup_candidate_advertises_freshness_floor_not_stale_committed (applied/learned=10, persist.committed=0 -> asserts advertised my_lsn==10). RED (Was 0) -> GREEN verified. Follow-up: boot election path (myapp.c) also routed through ha_vote_self_freshness (was raw applied_lsn) so both election paths are symmetric with the voter floor. ## Verification _Not recorded (migrated from buglog.json)._ ## References - Related: bug-338