---
id: "bug-340"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-06-10T00:23:21.918Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 2
files: ["src/controller/ha.c", "src/controller/testsuite/test_ha_liveness.c"]
tags: ["logic-fix", "c", "ha", "heartbeat", "rejoin", "restart", "standby-endpoints", "reconnect", "epoll", "liveness", "election-loop"]
related: []
legacy: true
---

# bug-340: After kill+RESTART of a controller (ctrl-1), the restarted node loops forever…

## Description
After kill+RESTART of a controller (ctrl-1), the restarted node loops forever: ha_on_primary_ctrl_fd_rdhup → CTRL_ELECTING → raft_cb_become_follower → CTRL_STANDBY every ~heartbeat_timeout, term climbing unbounded (473+), while ctrl-2 stays PRIMARY at term 3. On ctrl-2: every heartbeat tick logs `xce_encode op 0 (NOP)` then `WARN endpoint_update_epoll rc1=-1 < 0` (netio.c:575) — a wedged outbound standby endpoint.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
primary_heartbeat_fn never evicted a dead/wedged outbound standby endpoint from ha->standby_endpoints. When a peer is killed/restarted, its outbound ep on the primary wedges: handle_error (netio.c:420) runs iomgr_unmonitor (EPOLL_CTL_DEL, does NOT close the fd) and the err handler may leave the ep in standby_endpoints; epoll_ctl(EPOLL_CTL_MOD) on the DEL'd fd then returns ENOENT (rc1=-1) on every heartbeat enqueue. xce_nop_req still returns >=0 (enqueue ok) so the heartbeat layer never detected the wedge. Critically, primary_heartbeat_fn step 2 (reconnect) treats a peer as 'found' (connected) if ANY ep for it is in standby_endpoints — so the wedged ep SUPPRESSED reconnection. ctrl-2 therefore never opened a fresh connection to the restarted ctrl-1, never sent a ctrlhello, so ctrl-1 never set primary_ctrl_fd / refreshed last_activity and its liveness watchdog re-elected every heartbeat_timeout forever. (The earlier suspected 'become_follower doesn't reset last_activity' was NOT a bug: ha_arm_heartbeat_timeout_vtask already stamps last_activity=now, so the ~11s period is exactly one full timeout — confirming the watchdog is correct and the sole cause is the broken channel.)

## Fix
In primary_heartbeat_fn (ha.c) step 1, added liveness eviction symmetric with the standby's own liveness poll: for each standby ep, if ep->ts_last_in > 0 (was responsive at least once) AND (now_wall - ep->ts_last_in) > heartbeat_timeout_s, remove it from ha->standby_endpoints (zarray_del, under ha->lock) and collect it; after releasing ha->lock, repl_remove_standby() each evicted ep. Removal is the load-bearing action — the reconnect scan (step 2) then re-establishes a fresh ctrlhello channel to the restarted peer on this/next tick, and ctrl-1 settles via ha_discover_primary (no term gate, ha.c:3807) → become_follower → STANDBY. ts_last_in/ts_last_out are existing endpoint_t fields (wall clock, set in handle_input/handle_output), so no libnetio change. The evicted ep object itself is reclaimed by its err handler when the dead connection finally surfaces an epoll event (bounded leak, consistent with codebase). Eviction keys on ts_last_in>0 so a just-connected joiner (ts_last_in==0) gets grace. Regression tests: test_primary_hb_evicts_silent_standby (RED→GREEN), test_primary_hb_keeps_live_standby, test_primary_hb_keeps_unconfirmed_standby in test_ha_liveness.c.

## Verification
_Not recorded (migrated from buglog.json)._

## References
_Not recorded (migrated from buglog.json)._