Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
6.5 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1685 | fixed | 2026-09-24 | scheduler | crash | 2026-08-25T15:47:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
true |
bug-1685: User-reported live crash: xce_scheduler SIGABRT via glibc malloc_printerr (heap corruption…
Description
User-reported live crash: xce_scheduler SIGABRT via glibc malloc_printerr (heap corruption detected during a malloc/free bookkeeping check -- 'double free or corruption'-class abort, not a clean segfault). Core captured from a live 3-node HA deployment run (--peers ctrl-1/ctrl-2/ctrl-3, -vvv). Backtrace: malloc_printerr -> kill_limit (src/lib/xcelimit.c:181) <- limit_update_config (src/scheduler/src/plugins/limit.c:76) <- plugin_config_update (src/scheduler/src/plugin.c:246) <- sched_init_config (src/scheduler/src/scheduler.c:936) <- scheduler_req_handler (src/scheduler/src/handler.c:687) <- vtask_entry (src/lib/xcectx.c:172) <- run_task/vcpu_entry (src/lib/vtaskmgr.c). limit_update_config's own locals at the crash frame: rc=-17 (-EEXIST), nlimits=7, i=0 -- crashes on (or near) the FIRST limit in a 7-limit config update.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
FIXED (2026-08-25). Confirmed double-free: limit_update_config()'s privdata->limits is a kvmgr created with .datakiller = kill_limit (limit_priv_data_init(), limit.c). Its key_add() call (limit.c, then rc=-17/-EEXIST on the duplicate-name entry -- exactly the crash frame's own captured locals) dispatches to xkvmgr_add() (src/lib/kvmgr.c:107-129): vstore(kvm->data, value) succeeds first, THEN key_add(kvm->kvtab, key, idx) fails (EEXIST for a name already present), so xkvmgr_add rolls back via vclear(kvm->data, idx) -- which dispatches to xzarray_clear() (src/lib/zarray.c:614-637), and since the slot is in-use, fires a->killer(a->data[idx].data), i.e. kill_limit(limit), ONCE, automatically, as part of that rollback. limit_update_config()'s OLD error path then called kill_limit(limit) a SECOND time on the same already-freed pointer -- the double free glibc's malloc_printerr aborts on. Established precedent for this exact ownership-ambiguity shape: src/controller/job.c's restore_job_array() documents the identical asymmetry for a DIFFERENT kvmgr (c->jobmgr, .datakiller = NULL there) -- that kvmgr's own internal rollback is a no-op (killer is NULL), so THAT caller correctly must free the rejected value itself. Whether a caller must free a rejected key_add() value depends entirely on whether THAT specific kvmgr instance was configured with a non-NULL datakiller; privdata->limits has one (kill_limit), so limit_update_config must NOT free it itself. One accepted, documented residual gap (mirrors job.c's own accepted trade-off): if vstore() itself fails (e.g. zarray growth OOM, before ever reaching the kvtab key_add step), xkvmgr_add returns negative WITHOUT having stored or freed limit at all, and xkvmgr_add's return code doesn't distinguish this from the common EEXIST case -- removing the kill_limit() call trades a guaranteed crash on the common (duplicate-name) case for a leak in that vanishingly rare OOM-only sub-case. Leak is strictly better than crash; closing this gap cleanly would need a new kvmgr API, out of scope here.
Fix
FIXED (2026-08-25). Removed the redundant kill_limit(limit); call in limit_update_config()'s error path (src/scheduler/src/plugins/limit.c, if (rc < 0) { ... return rc; } block) -- single-line removal, replaced with a comment explaining the exact double-free mechanism above so a future reader doesn't reintroduce it 'to be safe'. Nothing else in limit.c/kvmgr.c/zarray.c touched.
Test: added src/scheduler/testsuite/test_limit_plugin.c (this plugin had ZERO test coverage before this fix -- confirmed via find src/scheduler/testsuite -iname '*limit*' returning nothing pre-fix), wired into src/scheduler/testsuite/meson.build (new sched_limit_plugin_test executable + scheduler limit plugin test meson test, suite 'scheduler', ASAN_OPTIONS=detect_leaks=0 matching every sibling test). White-box #include of ../src/plugins/limit.c (same convention as test_plugin_picker_registration.c/test_handlers.c) to reach limit_update_config()/limit_init()/limit_priv_data_init(), all static; stubs reset_sched_obj()/get_resvals()/put_resvals()/plugin_register() (scheduler/src functions outside this TU, referenced only by limit.c code paths this test never exercises) purely to satisfy the linker. test_duplicate_limit_name_no_double_free() builds a real xce_scheduler_t fixture (real groupmgr_init()'d core.ugmgr/qgmgr/pgmgr/hgmgr, real limit_init()'d plugin.data), then drives limit_update_config() with an xce_conf4sched_t containing TWO xce_limit_config_t entries sharing the SAME name ('dup_limit') -- the exact duplicate-name collision bug-1685's live crash captured -- and asserts rc == -EEXIST with no crash. test_distinct_limit_names_both_succeed() is the control case (two DISTINCT names, rc == 0), proving the fix didn't just suppress freeing on failure in general.
RED (pre-fix, verified live): built via nix develop ./src --command bash src/tools/xce-build.sh scheduler asan build-asan --incremental with the new test present but the buggy kill_limit(limit) call still in place -- sched_limit_plugin_test aborted with a genuine ASAN heap-use-after-free (not a clean assertion failure): 'AddressSanitizer: heap-use-after-free ... in kill_limit ../lib/xcelimit.c:181', READ stack kill_limit <- limit_update_config (limit.c:76, the buggy call) <- test_duplicate_limit_name_no_double_free, freed-by stack kill_limit (xcelimit.c:192) <- xzarray_clear (zarray.c:626) <- xkvmgr_add (kvmgr.c:124) <- limit_update_config (limit.c:72, the key_add call) -- an exact match to the diagnosed mechanism and to bug-1685's own captured crash signature (malloc_printerr <- kill_limit <- limit_update_config).
GREEN (post-fix, verified live): rebuilt with the single-line fix applied -- scheduler limit plugin test OK (0.05s). Full scheduler suite re-run: 52/53 OK, 1 pre-existing failure (test_recordstart_symlink_escape_replies_error, bug-1560, unrelated flake -- same failure present in the RED run too) -- zero new regressions.
Verification
Not recorded (migrated from buglog.json).
References
Not recorded (migrated from buglog.json).