Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
6.9 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | ||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1816 | fixed | 2026-09-24 | build | unknown | 2026-09-15T13:05:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true |
bug-1816: User request, following the bug-1814/1815 mem-accounting investigation: a live 3-node HA cluster…
Description
User request, following the bug-1814/1815 mem-accounting investigation: a live 3-node HA cluster on pc13v went through a repeated PreVote-failover storm (term climbing 2->6, standbys losing primary_ctrl_fd RDHUP repeatedly, xceadmin scheduler record status failing with 'scheduler not connected'/'too many redirects') while load was confirmed low (loadavg 0.5-1.6 on 22 cores). Root cause found live by the user: ps axwww | grep control on pc13v showed SIX xce_controller processes -- two full sets of ctrl-1/ctrl-2/ctrl-3 (PIDs 294848/294869/294905 and 610530/610546/610561) all bound to the identical ports (9901-9903) and identical --workdir/--certdir/--logfile paths under /tmp/xce-deploy-ha. The user's own hand-typed cleanup command (ps axwww | grep xce-ha-deploy | ... | kill -9) had a typo -- the actual workdir is /tmp/xce-deploy-ha/... ('xce-deploy-ha'), not 'xce-ha-deploy' (words transposed) -- so it matched nothing and silently failed to kill the first deploy's processes before a second deploy started on the same ports. User asked to enhance the xce-deploy-ha skill's shutdown-all mechanism so this class of mistake becomes structurally hard to repeat.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
Two independent, compounding structural gaps let a hand-typed-command typo turn into a genuine HA failover storm instead of a clean, obvious failure: (1) deploy-ha.sh had NO pre-flight check that the target ports were actually free before starting a new cluster -- it would happily bind a second, fully independent 3-node Raft group to the exact same ports an existing, still-alive cluster already held, producing real cross-talk (mixed vote traffic, corrupted term/leader tracking between two unrelated groups) rather than a clean bind() failure with a clear error. (2) start.sh/start-all.sh already HAD a correct, working 'already running' guard (reads $TOP/run/*.pid, checks kill -0, refuses to start a duplicate) -- but the pre-existing cleanup.sh unconditionally did rm -f $TOP/run/*.pid as pure state cleanup, with NO check that the PID each file pointed to was actually dead first. Running cleanup.sh (or a fresh deploy-ha.sh invocation that regenerates it) without ALSO successfully killing the live processes first silently erased the ONLY signal that guard depends on, letting a stale process survive completely undetected by every existing safety net in the codebase. The user's specific typo was the proximate trigger, but the underlying design let a single hand-typed-command mistake cascade into two independent Raft clusters, sharing ports, cross-talking -- a much worse failure mode than the typo alone should have caused.
Fix
FIXED (2026-09-15). Three changes to src/tools/deploy-ha.sh, live-verified end-to-end against real xce_controller/xceadmin/xcecd_virtual/xce_scheduler/xce_apiserver binaries (not just syntax-checked): (1) A port pre-flight check, added immediately after PEER_ARRAY is parsed and before ANY directory/cert/cluster.bin state is created -- for each peer's port, checks via ss -tln whether something is already listening; if so, prints the offending PID/process (via ss -tlnp, best-effort) and refuses to proceed with an actionable error (the same kill-by-name commands Step 2 documents), exiting before touching disk. Soft-skips (warns, doesn't fail) if ss isn't on PATH. Live-verified: a redeploy attempt against a still-running cluster's ports was correctly refused, identifying the exact blocking PIDs, and created no directory at all under the attempted $TOP. (2) cleanup.sh's generated body now stops every live PID in $TOP/run/*.pid (kill -9, with a warning to sudo kill -9 if the real root-run worker doesn't die from a plain kill) BEFORE removing any pidfile or other state, closing the exact gap in root_cause (2) -- restoring the invariant that start.sh/start-all.sh's own guard depends on: no pidfile is ever removed while pointing at a still-alive process. stop-all.sh (new, generated alongside start.sh/cleanup.sh) is the same pidfile-based stop logic as a standalone script that leaves state untouched -- the canonical, typo-proof way to stop everything a specific deploy started, since it reads that deploy's own tracked PIDs instead of matching process names/paths by hand. (3) Found and fixed a related race while live-testing (1) and (2) together: start-all.sh launched worker/scheduler/apiserver via backgrounded setsid nohup ... & with no wait for each to actually write its own pidfile before declaring itself done -- running cleanup.sh/stop-all.sh immediately after start-all.sh returned could catch worker/scheduler BEFORE their pidfile existed, defeating the very fix in (2)/(3) for those two components specifically (reproduced live: cleanup.sh correctly stopped ctrl-1/2/3 but left worker-1/scheduler running with their pidfiles already wiped -- the identical zombie-process class this whole fix targets, via a narrower timing window instead of an operator typo). Added a small wait_for_pidfile() poll (up to 5s, 0.2s interval) after each of the three backgrounded launches; start-all.sh now only reports 'done' once every component it started has a live pidfile on disk (or warns by name if one didn't appear in time). Re-verified live after this fix: cleanup.sh run IMMEDIATELY after start-all.sh returned correctly stopped all 6 processes (3 controllers + worker + scheduler + apiserver), confirmed via ps/ss showing nothing left running and no listeners on the deployed ports. SKILL.md updated: Step 2 now prefers a known deploy's own stop-all.sh over the hand-typed blanket kill (documents exactly why, citing this incident), Clean restart section documents cleanup.sh's new stop-then-wipe behavior and stop-all.sh as the no-wipe alternative, and two new Troubleshooting rows cover the port-pre-flight error message and the 'inexplicable HA failover despite low load, multiple PIDs per role in logs' symptom pattern directly, matching this incident.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1814, bug-1815