---
id: "bug-1698"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-08-26T07:20:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/ha.c (ha_connect_finish_peer", "reconnect_lane_fn", "ha_connect_to_peer)"]
tags: ["controller", "ha", "peer-connection", "reconnect", "ha_connect_finish_peer", "live-deployment", "found-live-no-restart", "partially-root-caused", "needs-live-repro", "needs-owner", "not-a-zombie-process-artifact"]
related: ["bug-1696", "bug-854", "bug-1487"]
legacy: true
---

# bug-1698: User-reported live deployment (/tmp/xce-ha-deploy, 3-node HA cluster, build.dbg): after…

## Description
User-reported live deployment (/tmp/xce-ha-deploy, 3-node HA cluster, build.dbg): after ./scripts/start-all.sh reported 'Controller cluster healthy (primary elected)', ctrl-1's log showed it perpetually retrying ha_connect_finish_peer() against both peer[1] (localhost:9902) and peer[2] (localhost:9903) every ~3s for 8+ minutes straight (120+ attempts, INFO-level 'connecting outbound to peer[N]' logged each time -- the SUCCESS-path log line per ha_connect_finish_peer's own code, not a failure), never once completing a working peer connection. Meanwhile ctrl-2/ctrl-3 were actively, healthily replicating with each other (real AppendEntries traffic, LSN steadily advancing) -- they formed a working 2-of-3 quorum excluding ctrl-1 entirely. `xceadmin scheduler record status` failed with 'no primary available: timed out contacting localhost:9901' (ctrl-1's own client port) even though `xce_test`'s REPL joblist worked fine against the real 2-node quorum.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
PARTIALLY CONFIRMED, NOT FULLY ROOT-CAUSED. Ruled out as a test-harness/environment artifact: independently confirmed via a completely fresh, isolated 3-node deploy (~/tmp/xce-ha-fresh-repro, same build.dbg binaries, same run-ha-soak.sh topology) that the SAME symptom (impossible term jump, standby permanently failing cluster_init_phase2) reproduced ONCE due to a genuine same-session zombie-process port collision (see cerebrum.md's 2026-08-26 Do-Not-Repeat entry) -- but that specific mechanism does NOT apply to /tmp/xce-ha-deploy itself: confirmed via `ss -tlnp` that no live process holds ports 9901-9903 at investigation time, and no OTHER unrelated cluster shares those specific ports (9991-9993 was the colliding zombie's port range, unrelated). Direct on-disk evidence via `waldump` on the STOPPED cluster's data dirs CONFIRMS a real, non-artifactual divergence: ctrl-1's WAL has only 138 entries (lsn 1..138, all term=2, CRC chain intact, no gaps) while ctrl-2 has 2648 entries and ctrl-3 has 2641 entries (both showing term=3 activity in ha-term.bin, one of which -- ctrl-3 -- legitimately recorded voted_for_name='ctrl-2', a NORMAL vote-grant record, not corruption). This proves ctrl-1 was genuinely functioning (as primary, at term=2, given the job data the user's own joblist output showed) up through WAL entry 138, then something caused it to lose contact with both peers, a new election (term=3) happened among ctrl-2/ctrl-3 without it, and ctrl-1's own RECONNECT mechanism (ha_connect_finish_peer, dispatched via the SlowLane per raft-drive-async task 2.1's offload design) never once succeeded in re-establishing a working connection for the remainder of the run, despite logging its own SUCCESS path (endpoint_connect()+iomgr_monitor() both returning >=0) on every single attempt. This strongly suggests the bug is NOT in the initial connect() call itself (which appears to succeed at the socket/iomgr-registration level every time) but in whatever HIGHER-LEVEL protocol step is supposed to follow a successful low-level connect (completing a vote/ctrlhello handshake, registering the peer in standby_endpoints, or some per-peer state that should be cleared/retried differently after a genuine peer-loss) -- that step silently never completes or never gets attempted, so the SAME low-level connect()+iomgr_monitor() cycle repeats every ~3s forever without progress. Could NOT be reproduced live/interactively -- no debugger/ptrace access to the (already-stopped) real deployment's processes was available; this analysis is from static log/WAL evidence only, not a live capture of the actual failure in progress. FURTHER REPRO ATTEMPTS (2026-08-26, all failed to reproduce): (a) fresh 3-node deploy, kill+restart both standbys simultaneously while primary stays alive -- clean reconnect both times; (b) same, with continuous background jobsub load throughout -- clean reconnect (one transient ECONNREFUSED-then-retry-succeeds cycle observed, not a stuck state); (c) user-supplied exact recipe (submit a mixed array of jobs giving RUN/PEND/DONE states, then xceadmin scheduler record status) repeated 10x against a genuinely healthy full-component deploy (3x controller + worker + scheduler + xceadmin) -- succeeded cleanly every time; (d) 8x fresh-boot-only attempts (no artificial kills at all) immediately running reslist/workerlist/jobsub/joblist in sequence within the first ~5s of boot, matching the user's report that the real failure occurred within MINUTES of start-all.sh with no explicit kill -- all 8 clean, no crash, no elevated reconnect-retry counts. Total: 20+ reproduction attempts across 4 distinct scenario shapes, zero reproductions. This is a genuinely rare, timing-sensitive condition, not reliably triggerable by any tested scenario shape so far.

## Fix
NOT YET ATTEMPTED. Needs a live reproduction with process-level tracing (gdb/strace on a debug build) or targeted new logging in ha_connect_finish_peer/reconnect_lane_fn's post-connect handshake path to see exactly which step silently fails to progress after a genuine peer-loss (RDHUP) event, as distinct from initial cold-boot connection establishment (which DOES work correctly, per the fresh-repro soak's clean PASS with normal kill/restart/reconverge cycles). Next investigation step: reproduce a genuine mid-run peer-loss (not a driver-initiated kill, which already reconverges fine per the fresh-repro soak's PASS) -- e.g. a transient network blip or a peer restart landing in an unlucky window -- under gdb/strace, and trace ha_connect_finish_peer's full call chain including whatever runs after endpoint_connect()/iomgr_monitor() succeed (the ctrlhello/vote-channel handshake) to find where it silently stalls.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1696, bug-854, bug-1487