myxce/.wolf/bugs/bug-1884.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related
bug-1884	open	2026-09-29	newly logged	scheduler	leak	2026-09-29T08:20:03Z	2026-09-29T08:20:03Z	1	src/scheduler/src/sim_loop.c src/scheduler/src/sim.c src/scheduler/src/plugins/limit.c src/scheduler/src/main.c	scheduler sim asan lsan leak	bug-1632 bug-1763

bug-1884: standalone ASAN xce_scheduler --sim exits 1: LeakSanitizer reports 17 MB still allocated at exit

Description

Observed: an ASAN build of xce_scheduler --sim=<file> --sim-verify replays the whole recording (0 MISMATCH lines) and then exits 1 because LeakSanitizer reports at exit:

==2234882==ERROR: LeakSanitizer: detected memory leaks
...
SUMMARY: AddressSanitizer: 17041405 byte(s) leaked in 1187 allocation(s).

Of the 17,041,405 bytes, 17,036,261 are indirect and 5,144 direct. Grouped by first scheduler/lib frame (bytes, leak groups):

• kvtab_init (lib/kv.c:106): 16,941,432 in 15 -- core kv tables, reachable only from leaked roots
• sim_process_replay_event_message (sim_loop.c:403): 49,688 in 54 -- decoded replay events (e.g. decode_JobStateChangeEvent_ptr)
• xkvmgr_add (lib/kvmgr.c:114): 21,376 in 9
• sim_locate_and_load_snapshot (sim.c:224): 14,347 in 106 -- snapshot decode (decode_Conf4Sched / decode_QueueConf_list)
• put_resvals (resource.c:99) via limit.c update_limit_res_value and sched_update_res_accounts: 2,048 in 2
• smaller: parse_resreq, core_build_job, sched_ctx_init, match_job_host_limit_

The indirect bulk fits the simulator never tearing down its core at exit (its picker/allocator threads are never joined, see bug-1632/bug-1763); the direct groups may include real per-event leaks.

Expected: a clean replay exits 0 under ASAN with LeakSanitizer enabled, or the exit-time teardown frees what the driver owns. Impact: leak; for scripts, any ASAN --sim replay must set ASAN_OPTIONS=detect_leaks=0 or its exit code is always 1, which masks real failures and hides any genuine per-event leak among the exit-time noise. Release builds are unaffected at exit; a per-event leak would still grow a long replay's memory. Found by: spec task (libxce-alloc-preason-equality 5.3, live default-settings replay, 2026-09-29)

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (src/xce.build.discrete, debug + b_sanitize=address)
• Code: scheduler-framework @ 4500ec46-dirty
• Deployment: n/a (standalone simulation mode, no cluster); the replay file was recorded by a local 3-node HA soak (run-ha-soak.sh, 2 sequential rounds, xcecd_virtual worker) at /home/claude/tmp/xce-alloceq-53

Reproduction

1. bash src/tools/xce-build.sh scheduler asan (or any ASAN build dir)
2. src/xce.build/scheduler/xce_scheduler --sim=/home/claude/tmp/bug-1884/scheduler-replay-20260929-064412.replay --sim-verify --logfile /tmp/sim.log; echo rc=$?
3. Observe rc=1 and the LeakSanitizer SUMMARY on stderr. With ASAN_OPTIONS=detect_leaks=0 the same run exits 0.

• Frequency: deterministic (2/2 runs, identical totals, 2026-09-29); also seen on every recording of /home/claude/tmp/xce-ha-dc-soak
• Evidence: /home/claude/tmp/bug-1884/lsan-report.txt (full report), /home/claude/tmp/bug-1884/scheduler-replay-20260929-064412.replay

Root cause

TODO: required for fixed/partial

Fix

TODO: required for fixed/partial

Verification

• Test: TODO: name + file; RED before the fix, GREEN after
• Suites: TODO: suite results and build profile
• Live: TODO: soak/deploy run, or n/a and why

References

• Related: bug-1632, bug-1763 (sim picker/allocator threads never joined)
• run-ha-soak.sh replay gate already runs --sim with ASAN_OPTIONS=detect_leaks=0