myxce/.wolf/bugs/bug-1928.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits
bug-1928	fixed	2026-10-09		scheduler	leak	2026-10-09T03:14:08Z	2026-10-09T03:14:08Z	1	src/scheduler/src/repl.c src/scheduler/src/repl_info.c src/scheduler/src/repl_rollup.c src/scheduler/testsuite/meson.build	repl sim leak iterator asan	bug-1884	ba6f42cd

bug-1928: xce_scheduler --sim REPL leaks a job-table iterator every continue round (LeakSanitizer at exit)

Description

Observed: a normal interactive simulator session (continue, then quit) on the ASAN build exits 1 from LeakSanitizer, and the buffered console output is lost (0 bytes on stdout):

==4149978==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 144 byte(s) in 9 object(s) allocated from:
    #0 0x79e7779215a3 in calloc (.../libasan.so.8+0x1215a3)
    #1 0x5907bfd480ae in xkvmgr_make_iter ../lib/kvmgr.c:159
    #2 0x5907bfde1eca in iter_make (.../xce_scheduler+0x652eca)
    #3 0x5907bfcf1004 in repl_count_jobs ../scheduler/src/repl.c:189
    #4 0x5907bfcf5464 in repl_query_pend_jobids ../scheduler/src/repl_rollup.c:94
    #5 0x5907bfcf64bc in repl_round_rollup_process ../scheduler/src/repl_rollup.c:295
    #6 0x5907bfcf3238 in xce_repl_post_session_check ../scheduler/src/repl.c:626
...
SUMMARY: AddressSanitizer: 160 byte(s) leaked in 10 allocation(s).

Expected: the session exits 0 with leak detection on, and its console output reaches stdout. Impact: leak of 16 bytes (one kvmgr iterator) per call of repl_count_jobs(): once at the load banner and once per completed round in continue mode (Round Rollup), so it grows linearly with the replay length; info jobs leaks one more per call. Small, but it makes every standalone ASAN REPL run fail with exit 1 and drop its stdout, and forced several REPL tests to run with detect_leaks=0 (the trace wiring test's meson comment named this leak as the reason). Found by: ASAN run of the simulator during scheduler-sim-decision-trace validation; the second site (repl_info_jobs()) by audit. A grep of every iter_make() in src/scheduler/src for a matching iter_destroy() on every path found no others.

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (xce.build), default ASAN_OPTIONS (leak detection on) for the standalone run; the affected meson tests ran with ASAN_OPTIONS=detect_leaks=0
• Code: scheduler-framework @ 5ca59dc9-dirty
• Deployment: standalone simulator only (xce_scheduler --sim ... --interactive), no controller, no workers; single dev host

Reproduction

1. Build: bash src/tools/xce-build.sh scheduler --jobs 12 --incremental at the Environment commit.
2. In a temp dir: printf 'continue\nquit\n' | TERM=dumb src/xce.build/scheduler/xce_scheduler --sim src/scheduler/testsuite/fixtures/repl_trace_baseline/contention.replay --interactive --logfile <tmp>/sched.log (paths relative to the repo root) -- exit status 1, LeakSanitizer report on stderr, stdout empty.
3. Smaller: src/xce.build/scheduler/testsuite/sched_repl_trace_baseline_test with leak detection on (the child inherits it) fails test_transcript_matches_baseline: Expected 0 Was 1. the scripted session must end with quit's exit(0); sched_repl_trace_wiring_test with leak detection on exits 1 with 5 repl_count_jobs frames.

• Frequency: deterministic (1/1 standalone runs, every leak-on test run, 2026-10-09)
• Evidence: /home/claude/tmp/bug-1928/red/ (stderr.txt, stdout.txt, per-test logs of a leak-on sweep of every sched_repl_/sched_sim_ test before the fix), /home/claude/tmp/bug-1928/green/ (same after the fix)

Root cause

repl_count_jobs() (src/scheduler/src/repl.c:189) creates an iterator over core->jobs with iter_make(), walks it, and returns the count without calling iter_destroy(). For a kvmgr container iter_make() heap-allocates the inner iterator (xkvmgr_make_iter, lib/kvmgr.c:159) and only xkvmgr_iter_destroy() frees it, so each call leaks 16 bytes. It is called from repl_print_banner() once and from repl_query_pend_jobids() (repl_rollup.c) on every completed round in continue mode, hence the per-round growth. repl_info_jobs() (src/scheduler/src/repl_info.c:172, info jobs) has the same missing iter_destroy(). The leak makes LeakSanitizer turn the exit status into 1 at exit, and the process ends before stdio flushes the redirected stdout.

Since: repl_count_jobs() since 02f7f48f (scheduler-core REPL task 12.1); repl_info_jobs() since c705cada. repl_rollup.c's own comment acknowledged the missing destroy as a "low-impact quirk", but its own per-round caller made it a growing leak.

Scope: every iter_make() in src/scheduler/src (core.c, core_session.c, plugin.c, plugins/limit.c, host_iter.c, scheduler.c, snapshot.c, sim_loop.c, sim_picker.c, repl_info.c, repl_rollup.c, repl_trace_store.c) was checked; all others destroy the iterator on every path, and their early returns are only on iter_make() failure, where kvmgr frees the iterator itself. host_iter.c keeps its iterator in the host-iterator private data and destroys it in its fini.

Fix

• src/scheduler/src/repl.c repl_count_jobs(): call iter_destroy(&it) after the walk.
• src/scheduler/src/repl_info.c repl_info_jobs(): same.
• src/scheduler/src/repl_rollup.c: replaced the stale comment that called the missing destroy a quirk.
• src/scheduler/testsuite/meson.build: scheduler repl trace baseline test and scheduler repl trace wiring test now run with leak detection on (removed ASAN_OPTIONS=detect_leaks=0 and the comment blaming this leak). The other REPL/sim tests still leak from their own fixtures (cores and replay writers never torn down), so they keep detect_leaks=0.

Every iterator the REPL makes is now destroyed, so nothing is left for LeakSanitizer at exit.

Verification

• Test: scheduler repl trace baseline test (test_repl_trace_baseline.c, leak detection now on): RED before the fix (test_transcript_matches_baseline: Expected 0 Was 1, the child exits 1 from LeakSanitizer), GREEN after (4 Tests 0 Failures). scheduler repl trace wiring test (test_repl_trace_wiring.c, leak detection now on): RED before (exit 1, LeakSanitizer with repl_count_jobs frames), GREEN after (17 Tests 0 Failures).
• Leak-on sweep of all 42 sched_repl_/sched_sim_ test binaries: repl_count_jobs/repl_info_jobs frames 0 in every log after the fix (before: present in 13).
• Suites: bash src/tools/xce-build.sh scheduler --jobs 12 --incremental: scheduler suite 118/118 OK, asan (log: /home/claude/tmp/bug-1928/gate-scheduler-suite.log).
• Live: the standalone command in Reproduction step 2 now exits 0 with leak detection on and prints its 18-line transcript (/home/claude/tmp/bug-1928/green/).

References

• Related: bug-1884 (earlier standalone --sim LeakSanitizer exit-1 fix; this is a REPL-only leak it did not cover)
• Docs: src/scheduler/CLAUDE.md (Interactive REPL)