Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
38 lines
3.6 KiB
Markdown
38 lines
3.6 KiB
Markdown
---
|
|
id: "bug-1820"
|
|
status: "fixed"
|
|
status_updated: "2026-09-24"
|
|
status_note: ""
|
|
component: "build"
|
|
severity: "unknown"
|
|
timestamp: "2026-09-17T02:45:00Z"
|
|
last_seen: "2026-09-19T09:00:00Z"
|
|
occurrences: 1
|
|
files: ["src/tools/deploy-ha.sh (generated scripts/stop-all.sh and scripts/cleanup.sh kill loops; generated scripts/start-worker.sh's sudo branch)"]
|
|
tags: ["deploy-ha", "stop-all.sh", "cleanup.sh", "start-worker.sh", "xcecd", "sudo", "orphaned-process", "real-worker", "sigkill", "live-verified"]
|
|
related: ["bug-1720"]
|
|
legacy: true
|
|
---
|
|
|
|
# bug-1820: User report: after deploying a real (root, sudo-launched) xcecd worker via /xce-deploy-ha and…
|
|
|
|
## Description
|
|
User report: after deploying a real (root, sudo-launched) xcecd worker via /xce-deploy-ha and stopping the cluster with scripts/stop-all.sh, xcecd keeps running -- ps shows a leftover xcecd process (a different PID than the one stop-all.sh reported stopping) still bound to the deploy dir's logs after stop-all.sh reported "Stopping worker-1 (PID <pid>)" with no warning.
|
|
|
|
## Environment
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Reproduction
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Root cause
|
|
start-worker.sh's real-worker branch writes the pidfile via `echo $$ > "$PIDFILE"` and then does `exec sudo "$WORKER_BIN" ...` -- `exec` replaces the pidfile-writing shell's own process image with sudo IN PLACE (same PID), so the PID recorded in the pidfile is sudo's own PID, not the actual worker daemon's. sudo then forks a SEPARATE, root-owned child process that execs and runs the real xcecd daemon; sudo itself stays alive as a thin parent/monitor. stop-all.sh's (and cleanup.sh's, identical logic) kill loop did `kill -9 "$PID"` against the recorded (sudo) PID -- SIGKILL terminates sudo instantly but is never forwarded to its child (SIGKILL cannot be caught/relayed by the dying parent), so the real xcecd process is silently orphaned (reparented to init) and keeps running. The scripts' own pre-existing "still alive after kill -9" warning branch also never fired for this exact case, because it re-checks the SAME (sudo) PID, which genuinely is dead by then -- so the orphaned child produced zero warning output, making the bug fully silent.
|
|
|
|
## Fix
|
|
FIXED (2026-09-17). In both generated scripts/stop-all.sh and scripts/cleanup.sh (src/tools/deploy-ha.sh), before killing a pidfile'd PID, check whether that PID's own /proc/$PID/cmdline starts with "sudo " (the telltale sign left by `exec sudo ...`); if so, find its child process(es) via `pgrep -P "$PID"` and kill them first via `sudo kill -9` (they run as root), before killing the sudo PID itself. The post-kill "still alive" check was also extended to additionally test for the child PID's continued existence (via `/proc/$CHILD` presence, which needs no kill permission, since `kill -0` on a root-owned PID from a non-root caller returns EPERM regardless of whether the process is alive). Live-verified end-to-end: deployed a fresh 3-node cluster with XCE_REAL_WORKER=1 (forcing the sudo xcecd path), confirmed the exact sudo-parent/root-child PID pair via `ps axwww`, ran the fixed stop-all.sh and confirmed both processes died with the new "is a sudo wrapper -- also stopping its child" line printed and zero processes remaining afterward; repeated the same reproduction against cleanup.sh with the same clean result. No unit test exists for this bash tooling (deploy-ha.sh has no meson-registered test suite, consistent with other shell tooling in this project e.g. xce-build.sh); verification is this live end-to-end reproduction.
|
|
|
|
## Verification
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## References
|
|
- Related: bug-1720
|