Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
3.5 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1575 | fixed | 2026-09-24 | unknown | unknown | 2026-08-18T03:15:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true |
bug-1575: run-resource-verify.sh's trap-based cleanup() (EXIT/INT/TERM) non-deterministically left the…
Description
run-resource-verify.sh's trap-based cleanup() (EXIT/INT/TERM) non-deterministically left the scheduler and/or apiserver process running after the script was interrupted mid-deployment via an external SIGTERM -- confirmed processes still alive (e.g. xce_scheduler --peers ... --logfile .../scheduler.log, /tmp/apiserver -d .../apiserver) seconds after the orchestrator's own wait returned with exit code 143. The SAME cleanup() code, reached via a normal fail()/exit-1 path (no external signal involved) inside the same single foreground invocation, worked perfectly every time (verified zero leftover processes).
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
cleanup() issued four SEPARATE sequential pkill -9 -f -- "<pattern>" invocations (one per component: controller/worker/scheduler/apiserver), each forking+execing its own pkill subprocess. When cleanup() ran as a normal function call/return within the script's own top-to-bottom control flow (e.g. from fail()), all four completed reliably. When cleanup() instead ran from on_exit as the EXIT trap fired by an asynchronously-delivered SIGTERM (via a separate trap 'exit 143' TERM translating the signal into an explicit exit), the sequence of pkill forks was intermittently and non-deterministically cut short partway through (confirmed via added debug instrumentation: one run stopped after the 1st pkill, another after the 3rd, another after the 2nd -- no fixed failure point, consistent with a timing race rather than a deterministic pattern-matching bug). Root cause not fully isolated at the syscall level (ruled out: pattern-matching correctness -- each individual pkill pattern was independently verified correct via direct pgrep; ruled out: cgroup pids/resource exhaustion -- process count stayed low, ~20-25 total throughout). Most likely a bash/signal-trap interaction where starting new child processes (fork/exec for each pkill) from within a trap handler invoked asynchronously (as opposed to synchronous top-level exit) has some window of unreliability in this environment.
Fix
Rewrote cleanup() to (1) combine all four per-component patterns into ONE pkill -9 -f call using extended-regex alternation ("pat1|pat2|pat3|pat4") instead of four separate invocations -- fewer fork/exec round trips through the trap-handler code path -- and (2) wrap it in a retry-until-verified loop (pgrep -f check, up to 10 attempts at 0.2s intervals) so a slow/racy single kill attempt can no longer silently leave a process behind. Verified reliable across 5+ repeated live runs deliberately interrupted via external SIGTERM at different points mid-deployment (right after controller-only confirmation, and after controller+scheduler confirmation) -- zero leftover processes every time after the fix, versus non-deterministic leftovers before it.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1436