From e34e9fec23842162c5f3888c47dc820ad7a4d08f Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Wed, 26 Aug 2026 07:33:53 +0000 Subject: [PATCH] chore(wolf): checkpoint buglog.json (1039 bugs) --- .wolf/buglog.json | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/.wolf/buglog.json b/.wolf/buglog.json index d81cf1ec..f5be3ded 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -19097,6 +19097,35 @@ ], "occurrences": 1, "last_seen": "2026-08-26T04:35:00Z" + }, + { + "id": "bug-1698", + "timestamp": "2026-08-26T07:20:00Z", + "error_message": "User-reported live deployment (/tmp/xce-ha-deploy, 3-node HA cluster, build.dbg): after ./scripts/start-all.sh reported 'Controller cluster healthy (primary elected)', ctrl-1's log showed it perpetually retrying ha_connect_finish_peer() against both peer[1] (localhost:9902) and peer[2] (localhost:9903) every ~3s for 8+ minutes straight (120+ attempts, INFO-level 'connecting outbound to peer[N]' logged each time -- the SUCCESS-path log line per ha_connect_finish_peer's own code, not a failure), never once completing a working peer connection. Meanwhile ctrl-2/ctrl-3 were actively, healthily replicating with each other (real AppendEntries traffic, LSN steadily advancing) -- they formed a working 2-of-3 quorum excluding ctrl-1 entirely. `xceadmin scheduler record status` failed with 'no primary available: timed out contacting localhost:9901' (ctrl-1's own client port) even though `xce_test`'s REPL joblist worked fine against the real 2-node quorum.", + "file": "src/controller/ha.c (ha_connect_finish_peer, reconnect_lane_fn, ha_connect_to_peer)", + "root_cause": "PARTIALLY CONFIRMED, NOT FULLY ROOT-CAUSED. Ruled out as a test-harness/environment artifact: independently confirmed via a completely fresh, isolated 3-node deploy (~/tmp/xce-ha-fresh-repro, same build.dbg binaries, same run-ha-soak.sh topology) that the SAME symptom (impossible term jump, standby permanently failing cluster_init_phase2) reproduced ONCE due to a genuine same-session zombie-process port collision (see cerebrum.md's 2026-08-26 Do-Not-Repeat entry) -- but that specific mechanism does NOT apply to /tmp/xce-ha-deploy itself: confirmed via `ss -tlnp` that no live process holds ports 9901-9903 at investigation time, and no OTHER unrelated cluster shares those specific ports (9991-9993 was the colliding zombie's port range, unrelated). Direct on-disk evidence via `waldump` on the STOPPED cluster's data dirs CONFIRMS a real, non-artifactual divergence: ctrl-1's WAL has only 138 entries (lsn 1..138, all term=2, CRC chain intact, no gaps) while ctrl-2 has 2648 entries and ctrl-3 has 2641 entries (both showing term=3 activity in ha-term.bin, one of which -- ctrl-3 -- legitimately recorded voted_for_name='ctrl-2', a NORMAL vote-grant record, not corruption). This proves ctrl-1 was genuinely functioning (as primary, at term=2, given the job data the user's own joblist output showed) up through WAL entry 138, then something caused it to lose contact with both peers, a new election (term=3) happened among ctrl-2/ctrl-3 without it, and ctrl-1's own RECONNECT mechanism (ha_connect_finish_peer, dispatched via the SlowLane per raft-drive-async task 2.1's offload design) never once succeeded in re-establishing a working connection for the remainder of the run, despite logging its own SUCCESS path (endpoint_connect()+iomgr_monitor() both returning >=0) on every single attempt. This strongly suggests the bug is NOT in the initial connect() call itself (which appears to succeed at the socket/iomgr-registration level every time) but in whatever HIGHER-LEVEL protocol step is supposed to follow a successful low-level connect (completing a vote/ctrlhello handshake, registering the peer in standby_endpoints, or some per-peer state that should be cleared/retried differently after a genuine peer-loss) -- that step silently never completes or never gets attempted, so the SAME low-level connect()+iomgr_monitor() cycle repeats every ~3s forever without progress. Could NOT be reproduced live/interactively -- no debugger/ptrace access to the (already-stopped) real deployment's processes was available; this analysis is from static log/WAL evidence only, not a live capture of the actual failure in progress.", + "fix": "NOT YET ATTEMPTED. Needs a live reproduction with process-level tracing (gdb/strace on a debug build) or targeted new logging in ha_connect_finish_peer/reconnect_lane_fn's post-connect handshake path to see exactly which step silently fails to progress after a genuine peer-loss (RDHUP) event, as distinct from initial cold-boot connection establishment (which DOES work correctly, per the fresh-repro soak's clean PASS with normal kill/restart/reconverge cycles). Next investigation step: reproduce a genuine mid-run peer-loss (not a driver-initiated kill, which already reconverges fine per the fresh-repro soak's PASS) -- e.g. a transient network blip or a peer restart landing in an unlucky window -- under gdb/strace, and trace ha_connect_finish_peer's full call chain including whatever runs after endpoint_connect()/iomgr_monitor() succeed (the ctrlhello/vote-channel handshake) to find where it silently stalls.", + "tags": [ + "controller", + "ha", + "peer-connection", + "reconnect", + "ha_connect_finish_peer", + "live-deployment", + "found-live-no-restart", + "partially-root-caused", + "needs-live-repro", + "needs-owner", + "unfixed", + "not-a-zombie-process-artifact" + ], + "related_bugs": [ + "bug-1696", + "bug-854", + "bug-1487" + ], + "occurrences": 1, + "last_seen": "2026-08-26T07:20:00Z" } ] }