Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
6.5 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1698 | fixed | 2026-09-24 | controller | unknown | 2026-08-26T07:20:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true |
bug-1698: User-reported live deployment (/tmp/xce-ha-deploy, 3-node HA cluster, build.dbg): after…
Description
User-reported live deployment (/tmp/xce-ha-deploy, 3-node HA cluster, build.dbg): after ./scripts/start-all.sh reported 'Controller cluster healthy (primary elected)', ctrl-1's log showed it perpetually retrying ha_connect_finish_peer() against both peer[1] (localhost:9902) and peer[2] (localhost:9903) every ~3s for 8+ minutes straight (120+ attempts, INFO-level 'connecting outbound to peer[N]' logged each time -- the SUCCESS-path log line per ha_connect_finish_peer's own code, not a failure), never once completing a working peer connection. Meanwhile ctrl-2/ctrl-3 were actively, healthily replicating with each other (real AppendEntries traffic, LSN steadily advancing) -- they formed a working 2-of-3 quorum excluding ctrl-1 entirely. xceadmin scheduler record status failed with 'no primary available: timed out contacting localhost:9901' (ctrl-1's own client port) even though xce_test's REPL joblist worked fine against the real 2-node quorum.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
PARTIALLY CONFIRMED, NOT FULLY ROOT-CAUSED. Ruled out as a test-harness/environment artifact: independently confirmed via a completely fresh, isolated 3-node deploy (~/tmp/xce-ha-fresh-repro, same build.dbg binaries, same run-ha-soak.sh topology) that the SAME symptom (impossible term jump, standby permanently failing cluster_init_phase2) reproduced ONCE due to a genuine same-session zombie-process port collision (see cerebrum.md's 2026-08-26 Do-Not-Repeat entry) -- but that specific mechanism does NOT apply to /tmp/xce-ha-deploy itself: confirmed via ss -tlnp that no live process holds ports 9901-9903 at investigation time, and no OTHER unrelated cluster shares those specific ports (9991-9993 was the colliding zombie's port range, unrelated). Direct on-disk evidence via waldump on the STOPPED cluster's data dirs CONFIRMS a real, non-artifactual divergence: ctrl-1's WAL has only 138 entries (lsn 1..138, all term=2, CRC chain intact, no gaps) while ctrl-2 has 2648 entries and ctrl-3 has 2641 entries (both showing term=3 activity in ha-term.bin, one of which -- ctrl-3 -- legitimately recorded voted_for_name='ctrl-2', a NORMAL vote-grant record, not corruption). This proves ctrl-1 was genuinely functioning (as primary, at term=2, given the job data the user's own joblist output showed) up through WAL entry 138, then something caused it to lose contact with both peers, a new election (term=3) happened among ctrl-2/ctrl-3 without it, and ctrl-1's own RECONNECT mechanism (ha_connect_finish_peer, dispatched via the SlowLane per raft-drive-async task 2.1's offload design) never once succeeded in re-establishing a working connection for the remainder of the run, despite logging its own SUCCESS path (endpoint_connect()+iomgr_monitor() both returning >=0) on every single attempt. This strongly suggests the bug is NOT in the initial connect() call itself (which appears to succeed at the socket/iomgr-registration level every time) but in whatever HIGHER-LEVEL protocol step is supposed to follow a successful low-level connect (completing a vote/ctrlhello handshake, registering the peer in standby_endpoints, or some per-peer state that should be cleared/retried differently after a genuine peer-loss) -- that step silently never completes or never gets attempted, so the SAME low-level connect()+iomgr_monitor() cycle repeats every ~3s forever without progress. Could NOT be reproduced live/interactively -- no debugger/ptrace access to the (already-stopped) real deployment's processes was available; this analysis is from static log/WAL evidence only, not a live capture of the actual failure in progress. FURTHER REPRO ATTEMPTS (2026-08-26, all failed to reproduce): (a) fresh 3-node deploy, kill+restart both standbys simultaneously while primary stays alive -- clean reconnect both times; (b) same, with continuous background jobsub load throughout -- clean reconnect (one transient ECONNREFUSED-then-retry-succeeds cycle observed, not a stuck state); (c) user-supplied exact recipe (submit a mixed array of jobs giving RUN/PEND/DONE states, then xceadmin scheduler record status) repeated 10x against a genuinely healthy full-component deploy (3x controller + worker + scheduler + xceadmin) -- succeeded cleanly every time; (d) 8x fresh-boot-only attempts (no artificial kills at all) immediately running reslist/workerlist/jobsub/joblist in sequence within the first ~5s of boot, matching the user's report that the real failure occurred within MINUTES of start-all.sh with no explicit kill -- all 8 clean, no crash, no elevated reconnect-retry counts. Total: 20+ reproduction attempts across 4 distinct scenario shapes, zero reproductions. This is a genuinely rare, timing-sensitive condition, not reliably triggerable by any tested scenario shape so far.
Fix
NOT YET ATTEMPTED. Needs a live reproduction with process-level tracing (gdb/strace on a debug build) or targeted new logging in ha_connect_finish_peer/reconnect_lane_fn's post-connect handshake path to see exactly which step silently fails to progress after a genuine peer-loss (RDHUP) event, as distinct from initial cold-boot connection establishment (which DOES work correctly, per the fresh-repro soak's clean PASS with normal kill/restart/reconverge cycles). Next investigation step: reproduce a genuine mid-run peer-loss (not a driver-initiated kill, which already reconverges fine per the fresh-repro soak's PASS) -- e.g. a transient network blip or a peer restart landing in an unlucky window -- under gdb/strace, and trace ha_connect_finish_peer's full call chain including whatever runs after endpoint_connect()/iomgr_monitor() succeed (the ctrlhello/vote-channel handshake) to find where it silently stalls.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1696, bug-854, bug-1487