lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(scheduler): remove double-free in limit plugin config update (bug-1685)

Browse source 
User-reported live crash: xce_scheduler SIGABRT via glibc malloc_printerr
(heap corruption) inside kill_limit(), called from limit_update_config()
during a live config reload.

limit_update_config() called kill_limit(limit) in its error path after a
failed key_add(privdata->limits, ...). But privdata->limits is a kvmgr
created with .datakiller = kill_limit, so on a kvtab-level failure (e.g.
-EEXIST for a duplicate limit name -- exactly the rc=-17 captured in the
crash's own backtrace), xkvmgr_add() already rolls back via vclear() ->
xzarray_clear(), which fires kill_limit(limit) once automatically. The
caller's explicit second call was a double free.

Removed the redundant call; the kvmgr's own configured datakiller already
owns cleanup on this failure path (mirrors the ownership asymmetry
job.c's restore_job_array() already documents for the opposite case,
where its kvmgr has no datakiller configured and the caller must free
failed items itself). One accepted, documented residual gap: a vstore()-
level OOM failure (never reaching the kvtab step) would leak `limit`
instead -- vanishingly rare, and a leak is strictly better than the
guaranteed crash on the common duplicate-name case; xkvmgr_add's return
code doesn't distinguish which sub-case occurred, so there's no clean way
to free conditionally without a kvmgr API change (out of scope).

New test_limit_plugin.c (this plugin had zero test coverage before) --
RED-phase independently reproduced a genuine ASAN heap-use-after-free
matching the live crash's exact call chain; GREEN confirms 0 failures.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
Rongsong Shen 2026-08-25 08:18:34 +00:00
parent 307f5b82e1
commit b040dfc068
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
4 changed files with 307 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

13
.wolf/buglog.json
View file

@ -18720,20 +18720,23 @@
"id": "bug-1685",
"timestamp": "2026-08-25T15:47:00Z",
"error_message": "User-reported live crash: xce_scheduler SIGABRT via glibc malloc_printerr (heap corruption detected during a malloc/free bookkeeping check -- 'double free or corruption'-class abort, not a clean segfault). Core captured from a live 3-node HA deployment run (--peers ctrl-1/ctrl-2/ctrl-3, -vvv). Backtrace: malloc_printerr -> kill_limit (src/lib/xcelimit.c:181) <- limit_update_config (src/scheduler/src/plugins/limit.c:76) <- plugin_config_update (src/scheduler/src/plugin.c:246) <- sched_init_config (src/scheduler/src/scheduler.c:936) <- scheduler_req_handler (src/scheduler/src/handler.c:687) <- vtask_entry (src/lib/xcectx.c:172) <- run_task/vcpu_entry (src/lib/vtaskmgr.c). limit_update_config's own locals at the crash frame: rc=-17 (-EEXIST), nlimits=7, i=0 -- crashes on (or near) the FIRST limit in a 7-limit config update.",
"file": "src/lib/xcelimit.c",
"root_cause": "Not yet root-caused. kill_limit() (xcelimit.c:181) is called from limit_update_config() (limit.c:76) during a live config update/reload (sched_init_config -> plugin_config_update fan-out) while nlimits=7 limits are being processed and rc=-17 (-EEXIST) is already set in the caller's frame -- suggests kill_limit is being called on a limit object that's already been freed or is being double-freed as part of an add-conflicts-with-existing (-EEXIST) error-handling/cleanup path in limit_update_config's loop over the new config's limits vs the existing limit table. Needs source read of limit_update_config()'s loop body and kill_limit()'s own free logic to confirm whether this is a double-free, a use-after-free, or a heap metadata corruption from an earlier out-of-bounds write that only surfaces later at this free call.",
"fix": "Not yet fixed -- investigating.",
"file": "src/scheduler/src/plugins/limit.c",
"root_cause": "FIXED (2026-08-25). Confirmed double-free: limit_update_config()'s privdata->limits is a kvmgr created with .datakiller = kill_limit (limit_priv_data_init(), limit.c). Its key_add() call (limit.c, then rc=-17/-EEXIST on the duplicate-name entry -- exactly the crash frame's own captured locals) dispatches to xkvmgr_add() (src/lib/kvmgr.c:107-129): vstore(kvm->data, value) succeeds first, THEN key_add(kvm->kvtab, key, idx) fails (EEXIST for a name already present), so xkvmgr_add rolls back via vclear(kvm->data, idx) -- which dispatches to xzarray_clear() (src/lib/zarray.c:614-637), and since the slot is in-use, fires a->killer(a->data[idx].data), i.e. kill_limit(limit), ONCE, automatically, as part of that rollback. limit_update_config()'s OLD error path then called kill_limit(limit) a SECOND time on the same already-freed pointer -- the double free glibc's malloc_printerr aborts on. Established precedent for this exact ownership-ambiguity shape: src/controller/job.c's restore_job_array() documents the identical asymmetry for a DIFFERENT kvmgr (c->jobmgr, .datakiller = NULL there) -- that kvmgr's own internal rollback is a no-op (killer is NULL), so THAT caller correctly must free the rejected value itself. Whether a caller must free a rejected key_add() value depends entirely on whether THAT specific kvmgr instance was configured with a non-NULL datakiller; privdata->limits has one (kill_limit), so limit_update_config must NOT free it itself. One accepted, documented residual gap (mirrors job.c's own accepted trade-off): if vstore() itself fails (e.g. zarray growth OOM, before ever reaching the kvtab key_add step), xkvmgr_add returns negative WITHOUT having stored or freed `limit` at all, and xkvmgr_add's return code doesn't distinguish this from the common EEXIST case -- removing the kill_limit() call trades a guaranteed crash on the common (duplicate-name) case for a leak in that vanishingly rare OOM-only sub-case. Leak is strictly better than crash; closing this gap cleanly would need a new kvmgr API, out of scope here.",
"fix": "FIXED (2026-08-25). Removed the redundant `kill_limit(limit);` call in limit_update_config()'s error path (src/scheduler/src/plugins/limit.c, `if (rc < 0) { ... return rc; }` block) -- single-line removal, replaced with a comment explaining the exact double-free mechanism above so a future reader doesn't reintroduce it 'to be safe'. Nothing else in limit.c/kvmgr.c/zarray.c touched.\n\nTest: added src/scheduler/testsuite/test_limit_plugin.c (this plugin had ZERO test coverage before this fix -- confirmed via `find src/scheduler/testsuite -iname '*limit*'` returning nothing pre-fix), wired into src/scheduler/testsuite/meson.build (new `sched_limit_plugin_test` executable + `scheduler limit plugin test` meson test, suite 'scheduler', ASAN_OPTIONS=detect_leaks=0 matching every sibling test). White-box #include of ../src/plugins/limit.c (same convention as test_plugin_picker_registration.c/test_handlers.c) to reach limit_update_config()/limit_init()/limit_priv_data_init(), all `static`; stubs reset_sched_obj()/get_resvals()/put_resvals()/plugin_register() (scheduler/src functions outside this TU, referenced only by limit.c code paths this test never exercises) purely to satisfy the linker. test_duplicate_limit_name_no_double_free() builds a real xce_scheduler_t fixture (real groupmgr_init()'d core.ugmgr/qgmgr/pgmgr/hgmgr, real limit_init()'d plugin.data), then drives limit_update_config() with an xce_conf4sched_t containing TWO xce_limit_config_t entries sharing the SAME name ('dup_limit') -- the exact duplicate-name collision bug-1685's live crash captured -- and asserts rc == -EEXIST with no crash. test_distinct_limit_names_both_succeed() is the control case (two DISTINCT names, rc == 0), proving the fix didn't just suppress freeing on failure in general.\n\nRED (pre-fix, verified live): built via `nix develop ./src --command bash src/tools/xce-build.sh scheduler asan build-asan --incremental` with the new test present but the buggy kill_limit(limit) call still in place -- sched_limit_plugin_test aborted with a genuine ASAN heap-use-after-free (not a clean assertion failure): 'AddressSanitizer: heap-use-after-free ... in kill_limit ../lib/xcelimit.c:181', READ stack `kill_limit <- limit_update_config (limit.c:76, the buggy call) <- test_duplicate_limit_name_no_double_free`, freed-by stack `kill_limit (xcelimit.c:192) <- xzarray_clear (zarray.c:626) <- xkvmgr_add (kvmgr.c:124) <- limit_update_config (limit.c:72, the key_add call)` -- an exact match to the diagnosed mechanism and to bug-1685's own captured crash signature (malloc_printerr <- kill_limit <- limit_update_config).\nGREEN (post-fix, verified live): rebuilt with the single-line fix applied -- `scheduler limit plugin test OK (0.05s)`. Full scheduler suite re-run: 52/53 OK, 1 pre-existing failure (`test_recordstart_symlink_escape_replies_error`, bug-1560, unrelated flake -- same failure present in the RED run too) -- zero new regressions.",
"tags": [
"scheduler",
"limit-plugin",
"heap-corruption",
"crash",
"sigabrt",
"live-deployment"
"live-deployment",
"double-free",
"kvmgr-rollback",
"fixed"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-08-25T15:47:00Z"
"last_seen": "2026-08-25T16:30:00Z"
}
]
}