--- id: "bug-1562" status: "fixed" status_updated: "2026-09-24" status_note: "" component: "controller" severity: "crash" timestamp: "2026-08-17T03:40:00.000Z" last_seen: "2026-09-19T09:00:00Z" occurrences: 1 files: ["src/controller/ha.c (ha_on_primary_ctrl_fd_rdhup", "5.10b debounce branch)"] tags: ["ha", "election", "liveness-timer", "dropped-timer", "bootstrap-hang", "raft-ha-core-safety", "5.10b"] related: ["bug-1310"] legacy: true --- # bug-1562: 3-node HA bootstrap never elects a primary: ctrl-1 (only node with local config) goes completely… ## Description 3-node HA bootstrap never elects a primary: ctrl-1 (only node with local config) goes completely silent ~22s after boot -- no crash, no log output, cluster.log shows no further activity even after ctrl-2/ctrl-3 join minutes later. Downstream symptom: worker gets 'SSL_do_handshake: Broken pipe' in a loop and ctrl-2/ctrl-3 log repeated 'incorrect worker idx -1' ERRORs from handle_endpoint_error (harmless noise -- join.c's handle_join_ stamps priv->role=HostRole_worker before the ha_no_primary_known guard can reject the join, so u.idx is still -1 when the rejected connection is torn down). ## Environment _Not recorded (migrated from buglog.json)._ ## Reproduction _Not recorded (migrated from buglog.json)._ ## Root cause Live-reproduced locally (build.dbg xce_controller, fresh 3-node deploy via deploy-ha.sh, ctrl-1 started ~45s before ctrl-2/ctrl-3 to match the user's actual timing) -- byte-for-byte matches the user's ctrl-1.log. ha_on_primary_ctrl_fd_rdhup's 5.10b debounce branch (ha.c ~8573-8589, added 2026-07-01 commit 0d70d5e4, bug-1310 task 5.10b) returns WITHOUT calling ha_arm_heartbeat_timeout_vtask() when a debounced RDHUP lands within the 2x-election-timeout cooldown window, unlike the function's other two early-return branches: the 5.6 election-in-progress guard (checks role==CTRL_ELECTING || pre_vote_in_progress, i.e. actually verifies a live timer is still running elsewhere) and the 5.2 spurious-suppression branch (explicitly re-arms via ha_arm_heartbeat_timeout_vtask before returning). The 5.10b branch only checks elapsed time since last_failover_ms -- it never checks whether an election is ACTUALLY still in flight. On a lone cold-booting node (n_peers up but none reachable yet), the first PreVote round loses within milliseconds (all connect attempts fail fast -> counted as no-votes), which re-arms the liveness watchdog once (10s). When that watchdog fires again ~10s later, last_failover_ms is still inside the 2x-election-timeout cooldown (observed 9805ms < 10000ms), so it lands in the debounce branch even though no election is in flight anymore -- and returns without re-arming anything. That is the LAST timer ever scheduled in the process: /proc//task/*/wchan confirmed every vtask worker thread idling in futex_do_wait (empty work queue, not deadlocked) and the io_worker thread parked in do_epoll_wait with zero further epoll events (lsof showed only the listen socket + epoll fd, no in-flight connections) for the remainder of the run. Node is permanently stuck CTRL_STANDBY with zero further election attempts -- other config-less nodes correctly wait forever for a primary that will never appear, so the whole cluster is permanently leaderless. Timing-dependent: reproducible only when the solo bootstrap window is long enough for a second liveness-timeout to land inside the cooldown after the first PreVote round already concluded (confirmed NOT reproducible with a tight ~0.5s inter-node start stagger, where a real peer answers before the trap window closes) -- explains why this presented as 'used to work' without any actual regression: bug has been latent since 2026-07-01, only the user's own start timing changed. ## Fix FIXED (TDD RED->GREEN). Added test_rdhup_debounce_still_rearms_liveness_watchdog (test_ha_rdhup_integration.c, next to the sibling test_rdhup_within_cooldown_is_debounced) -- asserts ha->last_activity moves off a sentinel after a debounced RDHUP, the same delaymgr-independent observable test_ha_arm_no_crash_no_thread (test_ha_liveness.c) already uses to prove ha_arm_heartbeat_timeout_vtask ran without a live delaymgr thread. RED confirmed pre-fix (assertion failed with the bug-1562 message). Fix: ha_on_primary_ctrl_fd_rdhup's 5.10b debounce branch (ha.c ~8578-8598) now calls ha_arm_heartbeat_timeout_vtask(ha, c) before returning, mirroring the function's own 5.2 spurious-suppression branch pattern -- safe with c==NULL (unit-test mode) since the arm function stamps ha->last_activity unconditionally before its internal c-null/delaymgr-running check gates only the actual timer scheduling. GREEN: controller_unit_test 1102 tests, 2 Failures (both pre-existing, unrelated overlayfs fdatasync EIO sandbox flakiness in test_config_repl.c -- test_conf_trim_keeps_only_n_versions and test_conf_get_saved_tag_no_config -- confirmed via 'persist: flusher fdatasync FAILED (errno=5 Input/output error)' immediately preceding both in the test log; identical to the cerebrum.md-documented sandbox quirk, present before this change too). LIVE END-TO-END VALIDATION (per this project's live-soak-validation-discipline convention -- unit-green alone is not proof): rebuilt xce_controller (src/xce.build), redeployed the identical 3-node scenario that hung before (ctrl-1 solo 45s, then ctrl-2, then ctrl-3 -- byte-for-byte the same timing that previously froze the process forever). Post-fix ctrl-1.log shows the debounce branch now immediately followed by 'HA: standby liveness poll armed' (previously the log went dead right after the debounce line), the watchdog continuing to retry every ~10s through several more PreVote-loses/debounce cycles while alone, and once ctrl-2/ctrl-3 came online: 'HA FSM (raft): -> CTRL_PRIMARY (term=2, first_boot=0)' -- ctrl-1 successfully elected primary, and both controller/ctrl-2/conf and controller/ctrl-3/conf received the pushed cluster.bin + current config. Full cluster convergence confirmed on the exact scenario that previously froze permanently. ## Verification _Not recorded (migrated from buglog.json)._ ## References - Related: bug-1310