Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
8.4 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | |||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1817 | fixed | 2026-09-24 | lib | crash | 2026-09-15T19:20:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true |
bug-1817: User-reported live core dump: xce_scheduler (build.dbg, HA-connected normal mode, --peers…
Description
User-reported live core dump: xce_scheduler (build.dbg, HA-connected normal mode, --peers ctrl-1/2/3) crashed with SIGSEGV in its dedicated allocator thread (sched_allocator, scheduler.c:484). gdb backtrace: zarray_get_ex (lib/zarray.c:332, data[i] = array->data[idx]) <- zarray_get <- match_job_host_limit_ (plugins/limit.c:370) <- limit_prepare_allocation <- plugin_prepare_allocation <- sched_try_alloc_on_host_ (resource.c:528) <- parjob_try_plan_ <- my_alloc <- sched_make_parjob_allocation <- sched_make_primary_alloc_ <- sched_make_primary_allocation <- sched_make_allocation <- sched_allocator <- start_thread. gdb locals at the crash frame: i=0, array->data=0x0 (NULL), while the bounds check idx (0) >= array->caps had evidently passed (i.e. array->caps was nonzero) -- a zarray_t with data==NULL but caps>0, an internally inconsistent/torn state that should be structurally impossible for a correctly-initialized array.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
zarray_fini(array) (lib/zarray.c) freed array->data and set it to NULL, but left array->caps/count/pos at whatever nonzero values they held before the call -- NOT reset to the safe (0,0,0) state zarray_init() establishes. xlist_fini() (lib/xlist.c) had the byte-for-byte identical gap (FREEUP(l->data) alone, l->caps/l->count left stale). Both functions' own bounds-checked readers (zarray_get_ex's idx >= array->caps, xlist_get's idx >= l->caps) treat a nonzero caps as a promise that array->data/l->data is a valid, sized allocation -- a promise zarray_fini()/xlist_fini() broke for any caller that reused the SAME struct after fini without an intervening re-init. schedobj.c's reset_sched_obj(xce_sched_obj_t* obj) does exactly that: xlist_fini(&(obj->glimits)); zarray_fini(&(obj->hlimits)); with no matching re-init, called from plugins/limit.c's limit_job_election() every time a job's schedobj.configver falls behind the cluster's current core.configver (i.e. whenever the cluster config has been reloaded/pushed since this job's limits were last computed) -- obj here is the job's own PERSISTENT xce_sched_obj_t (job->schedobj, allocated once at job ingest, reused for the job's entire lifetime), not a disposable scratch object. Once at least one host-limit match has grown obj->hlimits' caps beyond 0 (zarray_put() inside match_job_host_limit_(), the very next line after the crash site) and the config version later advances again while that job is still PEND, limit_job_election()'s reset_sched_obj() call tears obj->hlimits down to (data=NULL, caps=stale-nonzero) -- and the next allocation attempt for that same job (in the same or a later scheduling round, from a completely different call path: the allocator thread's plugin_prepare_allocation -> limit_prepare_allocation -> match_job_host_limit_ -> zarray_get(&(obj->hlimits), worker->idx)) dereferences the freed array, since worker->idx (0 for a single-worker deployment) is below the stale caps. This exact crash class had already caused at least one earlier, never-successfully-root-caused crash near the same call site (memory observation from 2026-08-17, 'crash occurs in driver/decision processing after limit_job_election returns successfully' -- that investigation correctly localized the area but concluded the crash was downstream in the decision pipeline, not inside reset_sched_obj()/zarray_fini() itself, for lack of as clean a gdb backtrace as this session's core dump provided). reset_sched_obj() itself was ALSO nearly lost entirely in an earlier refactor (bug-1490, 're-added reset_sched_obj() verbatim (unchanged body)' after a wholesale schedobj.c rewrite dropped it) -- its buggy body has survived unchanged through at least that refactor.
Fix
FIXED (2026-09-15). Fixed at the library-primitive level, not at reset_sched_obj()'s call site: zarray_fini() now resets array->count/array->caps/array->pos to 0 (matching zarray_init()'s own defaults) immediately after freeing/NULLing array->data; xlist_fini() now resets l->count/l->caps to 0 (matching xlist_init()) after FREEUP(l->data). This restores the invariant both functions' own doc comments now state explicitly: _fini() leaves the struct in the SAME safe, empty, immediately-reusable state _init() does, not just 'data freed, everything else however it happened to be.' schedobj.c's reset_sched_obj() and every other _fini()-without-reinit caller across the codebase are now correct by construction, with zero code changes needed there. Scoped the fix to the library primitives (not schedobj.c-side re-init calls) deliberately: audited every zarray_fini()/xlist_fini() call site outside testsuite/ (5 zarray_fini, ~25 xlist_fini, all in controller/lib) and confirmed every one either discards the struct immediately after (a stack-local scratch array/list, function returns right after) or is tearing down a struct being destroyed -- none relies on caps/count surviving fini, so this is a pure safety improvement for the whole 68+-caller zarray_init/all xlist_init blast radius, not a behavior change for any correct caller. TDD: added test_zarray_fini_resets_bounds_for_reuse (test_zarray.c) and test_xlist_fini_resets_bounds_for_reuse (test_xlist.c), both driving the raw zarray_init/zarray_put/zarray_fini/zarray_get and xlist_init/xlist_store/xlist_fini/xlist_get API directly (not the xzarray RTTI wrapper test_zarray.c's other tests use) since that's the exact API schedobj.c calls -- put an item (grows caps>0), fini WITHOUT re-init (exactly reset_sched_obj()'s pattern), assert caps/count reset to 0 and data is NULL, assert a get() for the previously-valid index now safely returns NULL instead of what used to be a NULL-pointer dereference, then assert the struct is genuinely reusable (a subsequent put/get round-trips correctly). RED confirmed by temporarily short-circuiting both fixes with if (0) { ... }: both new tests failed cleanly ('Expected 0 Was 4' / 'Expected 0 Was 2', the stale pre-fini caps value) -- Unity's assertion-failure longjmp aborts that one test function before reaching the actual NULL-deref line, so RED was captured as a clean, deterministic assertion failure rather than a process crash, letting the rest of the 300+-test suite still run and report cleanly around it. GREEN confirmed after restoring both fixes: full xcelib_test suite 314/314 individual test cases across all aggregated test files, zero regressions. Full scheduler suite also re-verified: 77/77, zero regressions (this library change is upstream of essentially all scheduler resource/allocation code). NOT investigated/fixed here, flagged only: a related, ALREADY-DOCUMENTED-BUT-UNAPPLIED bug (bug-1578, controller/sched.c's sched_worker_change()) has a DIFFERENT mechanism (a dangling ALIAS to a since-freed xlist_t backing array surviving on a separate struct field, not the same struct's own stale bookkeeping) that this fix does not touch or fix -- genuinely unrelated, noted only because it surfaced during this investigation's own zarray/xlist call-site audit.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1629, bug-1578, bug-1490