---
id: "bug-1817"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "lib"
severity: "crash"
timestamp: "2026-09-15T19:20:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/lib/zarray.c (zarray_fini() -- primary fix)", "src/lib/xlist.c (xlist_fini() -- identical fix", "same bug class)", "src/scheduler/src/schedobj.c (reset_sched_obj() -- the reachable production caller that exposed the bug", "NOT itself modified: correct by construction once the library primitives are fixed)", "src/scheduler/src/plugins/limit.c (limit_job_election()/match_job_host_limit_() -- the call chain that reaches the torn array", "not modified)", "src/lib/testsuite/test_zarray.c", "src/lib/testsuite/test_xlist.c (new regression tests)"]
tags: ["scheduler", "zarray", "xlist", "libxce", "zarray_fini", "xlist_fini", "reset_sched_obj", "schedobj", "limit-plugin", "match_job_host_limit_", "sigsegv", "null-pointer-dereference", "torn-state", "use-after-free-adjacent", "config-version", "job-election", "allocator-thread", "user-reported", "found-live", "core-dump", "gdb", "tdd", "library-primitive-fix", "recurring-bug-class", "previously-unroot-caused"]
related: ["bug-1629", "bug-1578", "bug-1490"]
legacy: true
---

# bug-1817: User-reported live core dump: xce_scheduler (build.dbg, HA-connected normal mode, --peers…

## Description
User-reported live core dump: xce_scheduler (build.dbg, HA-connected normal mode, --peers ctrl-1/2/3) crashed with SIGSEGV in its dedicated allocator thread (sched_allocator, scheduler.c:484). gdb backtrace: zarray_get_ex (lib/zarray.c:332, `data[i] = array->data[idx]`) <- zarray_get <- match_job_host_limit_ (plugins/limit.c:370) <- limit_prepare_allocation <- plugin_prepare_allocation <- sched_try_alloc_on_host_ (resource.c:528) <- parjob_try_plan_ <- my_alloc <- sched_make_parjob_allocation <- sched_make_primary_alloc_ <- sched_make_primary_allocation <- sched_make_allocation <- sched_allocator <- start_thread. gdb locals at the crash frame: i=0, array->data=0x0 (NULL), while the bounds check `idx (0) >= array->caps` had evidently passed (i.e. array->caps was nonzero) -- a zarray_t with data==NULL but caps>0, an internally inconsistent/torn state that should be structurally impossible for a correctly-initialized array.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
zarray_fini(array) (lib/zarray.c) freed array->data and set it to NULL, but left array->caps/count/pos at whatever nonzero values they held before the call -- NOT reset to the safe (0,0,0) state zarray_init() establishes. xlist_fini() (lib/xlist.c) had the byte-for-byte identical gap (FREEUP(l->data) alone, l->caps/l->count left stale). Both functions' own bounds-checked readers (zarray_get_ex's `idx >= array->caps`, xlist_get's `idx >= l->caps`) treat a nonzero caps as a promise that array->data/l->data is a valid, sized allocation -- a promise zarray_fini()/xlist_fini() broke for any caller that reused the SAME struct after fini without an intervening re-init. schedobj.c's reset_sched_obj(xce_sched_obj_t* obj) does exactly that: `xlist_fini(&(obj->glimits)); zarray_fini(&(obj->hlimits));` with no matching re-init, called from plugins/limit.c's limit_job_election() every time a job's schedobj.configver falls behind the cluster's current core.configver (i.e. whenever the cluster config has been reloaded/pushed since this job's limits were last computed) -- `obj` here is the job's own PERSISTENT xce_sched_obj_t (job->schedobj, allocated once at job ingest, reused for the job's entire lifetime), not a disposable scratch object. Once at least one host-limit match has grown obj->hlimits' caps beyond 0 (zarray_put() inside match_job_host_limit_(), the very next line after the crash site) and the config version later advances again while that job is still PEND, limit_job_election()'s reset_sched_obj() call tears obj->hlimits down to (data=NULL, caps=stale-nonzero) -- and the next allocation attempt for that same job (in the same or a later scheduling round, from a completely different call path: the allocator thread's plugin_prepare_allocation -> limit_prepare_allocation -> match_job_host_limit_ -> zarray_get(&(obj->hlimits), worker->idx)) dereferences the freed array, since worker->idx (0 for a single-worker deployment) is below the stale caps. This exact crash class had already caused at least one earlier, never-successfully-root-caused crash near the same call site (memory observation from 2026-08-17, 'crash occurs in driver/decision processing after limit_job_election returns successfully' -- that investigation correctly localized the area but concluded the crash was downstream in the decision pipeline, not inside reset_sched_obj()/zarray_fini() itself, for lack of as clean a gdb backtrace as this session's core dump provided). reset_sched_obj() itself was ALSO nearly lost entirely in an earlier refactor (bug-1490, 're-added reset_sched_obj() verbatim (unchanged body)' after a wholesale schedobj.c rewrite dropped it) -- its buggy body has survived unchanged through at least that refactor.

## Fix
FIXED (2026-09-15). Fixed at the library-primitive level, not at reset_sched_obj()'s call site: zarray_fini() now resets array->count/array->caps/array->pos to 0 (matching zarray_init()'s own defaults) immediately after freeing/NULLing array->data; xlist_fini() now resets l->count/l->caps to 0 (matching xlist_init()) after FREEUP(l->data). This restores the invariant both functions' own doc comments now state explicitly: _fini() leaves the struct in the SAME safe, empty, immediately-reusable state _init() does, not just 'data freed, everything else however it happened to be.' schedobj.c's reset_sched_obj() and every other _fini()-without-reinit caller across the codebase are now correct by construction, with zero code changes needed there. Scoped the fix to the library primitives (not schedobj.c-side re-init calls) deliberately: audited every zarray_fini()/xlist_fini() call site outside testsuite/ (5 zarray_fini, ~25 xlist_fini, all in controller/lib) and confirmed every one either discards the struct immediately after (a stack-local scratch array/list, function returns right after) or is tearing down a struct being destroyed -- none relies on caps/count surviving fini, so this is a pure safety improvement for the whole 68+-caller zarray_init/all xlist_init blast radius, not a behavior change for any correct caller. TDD: added test_zarray_fini_resets_bounds_for_reuse (test_zarray.c) and test_xlist_fini_resets_bounds_for_reuse (test_xlist.c), both driving the raw zarray_init/zarray_put/zarray_fini/zarray_get and xlist_init/xlist_store/xlist_fini/xlist_get API directly (not the xzarray RTTI wrapper test_zarray.c's other tests use) since that's the exact API schedobj.c calls -- put an item (grows caps>0), fini WITHOUT re-init (exactly reset_sched_obj()'s pattern), assert caps/count reset to 0 and data is NULL, assert a get() for the previously-valid index now safely returns NULL instead of what used to be a NULL-pointer dereference, then assert the struct is genuinely reusable (a subsequent put/get round-trips correctly). RED confirmed by temporarily short-circuiting both fixes with `if (0) { ... }`: both new tests failed cleanly ('Expected 0 Was 4' / 'Expected 0 Was 2', the stale pre-fini caps value) -- Unity's assertion-failure longjmp aborts that one test function before reaching the actual NULL-deref line, so RED was captured as a clean, deterministic assertion failure rather than a process crash, letting the rest of the 300+-test suite still run and report cleanly around it. GREEN confirmed after restoring both fixes: full xcelib_test suite 314/314 individual test cases across all aggregated test files, zero regressions. Full scheduler suite also re-verified: 77/77, zero regressions (this library change is upstream of essentially all scheduler resource/allocation code). NOT investigated/fixed here, flagged only: a related, ALREADY-DOCUMENTED-BUT-UNAPPLIED bug (bug-1578, controller/sched.c's sched_worker_change()) has a DIFFERENT mechanism (a dangling ALIAS to a since-freed xlist_t backing array surviving on a separate struct field, not the same struct's own stale bookkeeping) that this fix does not touch or fix -- genuinely unrelated, noted only because it surfaced during this investigation's own zarray/xlist call-site audit.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1629, bug-1578, bug-1490