80 lines
7.2 KiB
Markdown
80 lines
7.2 KiB
Markdown
---
|
|
id: "bug-1928"
|
|
status: "fixed"
|
|
status_updated: "2026-10-09"
|
|
status_note: ""
|
|
component: "scheduler"
|
|
severity: "leak"
|
|
timestamp: "2026-10-09T03:14:08Z"
|
|
last_seen: "2026-10-09T03:14:08Z"
|
|
occurrences: 1
|
|
files: ["src/scheduler/src/repl.c", "src/scheduler/src/repl_info.c", "src/scheduler/src/repl_rollup.c", "src/scheduler/testsuite/meson.build"]
|
|
tags: ["repl", "sim", "leak", "iterator", "asan"]
|
|
related: ["bug-1884"]
|
|
fix_commits: ["ba6f42cd"]
|
|
gitea_issue: 353
|
|
gitea_synced_status: "fixed"
|
|
gitea_synced_hash: "88fbb3b9cb25"
|
|
---
|
|
|
|
# bug-1928: xce_scheduler --sim REPL leaks a job-table iterator every continue round (LeakSanitizer at exit)
|
|
|
|
## Description
|
|
**Observed:** a normal interactive simulator session (`continue`, then `quit`) on the ASAN build exits 1 from LeakSanitizer, and the buffered console output is lost (0 bytes on stdout):
|
|
```
|
|
==4149978==ERROR: LeakSanitizer: detected memory leaks
|
|
|
|
Direct leak of 144 byte(s) in 9 object(s) allocated from:
|
|
#0 0x79e7779215a3 in calloc (.../libasan.so.8+0x1215a3)
|
|
#1 0x5907bfd480ae in xkvmgr_make_iter ../lib/kvmgr.c:159
|
|
#2 0x5907bfde1eca in iter_make (.../xce_scheduler+0x652eca)
|
|
#3 0x5907bfcf1004 in repl_count_jobs ../scheduler/src/repl.c:189
|
|
#4 0x5907bfcf5464 in repl_query_pend_jobids ../scheduler/src/repl_rollup.c:94
|
|
#5 0x5907bfcf64bc in repl_round_rollup_process ../scheduler/src/repl_rollup.c:295
|
|
#6 0x5907bfcf3238 in xce_repl_post_session_check ../scheduler/src/repl.c:626
|
|
...
|
|
SUMMARY: AddressSanitizer: 160 byte(s) leaked in 10 allocation(s).
|
|
```
|
|
**Expected:** the session exits 0 with leak detection on, and its console output reaches stdout.
|
|
**Impact:** leak of 16 bytes (one kvmgr iterator) per call of `repl_count_jobs()`: once at the load banner and once per completed round in `continue` mode (Round Rollup), so it grows linearly with the replay length; `info jobs` leaks one more per call. Small, but it makes every standalone ASAN REPL run fail with exit 1 and drop its stdout, and forced several REPL tests to run with `detect_leaks=0` (the trace wiring test's meson comment named this leak as the reason).
|
|
**Found by:** ASAN run of the simulator during scheduler-sim-decision-trace validation; the second site (`repl_info_jobs()`) by audit. A grep of every `iter_make()` in src/scheduler/src for a matching `iter_destroy()` on every path found no others.
|
|
|
|
## Environment
|
|
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
|
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: asan (xce.build), default ASAN_OPTIONS (leak detection on) for the standalone run; the affected meson tests ran with `ASAN_OPTIONS=detect_leaks=0`
|
|
- Code: scheduler-framework @ 5ca59dc9-dirty
|
|
- Deployment: standalone simulator only (`xce_scheduler --sim ... --interactive`), no controller, no workers; single dev host
|
|
|
|
## Reproduction
|
|
1. Build: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental` at the Environment commit.
|
|
2. In a temp dir: `printf 'continue\nquit\n' | TERM=dumb src/xce.build/scheduler/xce_scheduler --sim src/scheduler/testsuite/fixtures/repl_trace_baseline/contention.replay --interactive --logfile <tmp>/sched.log` (paths relative to the repo root) -- exit status 1, LeakSanitizer report on stderr, stdout empty.
|
|
3. Smaller: `src/xce.build/scheduler/testsuite/sched_repl_trace_baseline_test` with leak detection on (the child inherits it) fails `test_transcript_matches_baseline: Expected 0 Was 1. the scripted session must end with quit's exit(0)`; `sched_repl_trace_wiring_test` with leak detection on exits 1 with 5 `repl_count_jobs` frames.
|
|
- Frequency: deterministic (1/1 standalone runs, every leak-on test run, 2026-10-09)
|
|
- Evidence: /home/claude/tmp/bug-1928/red/ (stderr.txt, stdout.txt, per-test logs of a leak-on sweep of every sched_repl_*/sched_sim_* test before the fix), /home/claude/tmp/bug-1928/green/ (same after the fix)
|
|
|
|
## Root cause
|
|
`repl_count_jobs()` (src/scheduler/src/repl.c:189) creates an iterator over `core->jobs` with `iter_make()`, walks it, and returns the count without calling `iter_destroy()`. For a kvmgr container `iter_make()` heap-allocates the inner iterator (`xkvmgr_make_iter`, lib/kvmgr.c:159) and only `xkvmgr_iter_destroy()` frees it, so each call leaks 16 bytes. It is called from `repl_print_banner()` once and from `repl_query_pend_jobids()` (repl_rollup.c) on every completed round in `continue` mode, hence the per-round growth. `repl_info_jobs()` (src/scheduler/src/repl_info.c:172, `info jobs`) has the same missing `iter_destroy()`. The leak makes LeakSanitizer turn the exit status into 1 at exit, and the process ends before stdio flushes the redirected stdout.
|
|
|
|
Since: `repl_count_jobs()` since 02f7f48f (scheduler-core REPL task 12.1); `repl_info_jobs()` since c705cada. repl_rollup.c's own comment acknowledged the missing destroy as a "low-impact quirk", but its own per-round caller made it a growing leak.
|
|
|
|
Scope: every `iter_make()` in src/scheduler/src (core.c, core_session.c, plugin.c, plugins/limit.c, host_iter.c, scheduler.c, snapshot.c, sim_loop.c, sim_picker.c, repl_info.c, repl_rollup.c, repl_trace_store.c) was checked; all others destroy the iterator on every path, and their early returns are only on `iter_make()` failure, where kvmgr frees the iterator itself. `host_iter.c` keeps its iterator in the host-iterator private data and destroys it in its fini.
|
|
|
|
## Fix
|
|
- src/scheduler/src/repl.c `repl_count_jobs()`: call `iter_destroy(&it)` after the walk.
|
|
- src/scheduler/src/repl_info.c `repl_info_jobs()`: same.
|
|
- src/scheduler/src/repl_rollup.c: replaced the stale comment that called the missing destroy a quirk.
|
|
- src/scheduler/testsuite/meson.build: `scheduler repl trace baseline test` and `scheduler repl trace wiring test` now run with leak detection on (removed `ASAN_OPTIONS=detect_leaks=0` and the comment blaming this leak). The other REPL/sim tests still leak from their own fixtures (cores and replay writers never torn down), so they keep `detect_leaks=0`.
|
|
|
|
Every iterator the REPL makes is now destroyed, so nothing is left for LeakSanitizer at exit.
|
|
|
|
## Verification
|
|
- Test: `scheduler repl trace baseline test` (test_repl_trace_baseline.c, leak detection now on): RED before the fix (`test_transcript_matches_baseline: Expected 0 Was 1`, the child exits 1 from LeakSanitizer), GREEN after (4 Tests 0 Failures). `scheduler repl trace wiring test` (test_repl_trace_wiring.c, leak detection now on): RED before (exit 1, LeakSanitizer with `repl_count_jobs` frames), GREEN after (17 Tests 0 Failures).
|
|
- Leak-on sweep of all 42 sched_repl_*/sched_sim_* test binaries: `repl_count_jobs`/`repl_info_jobs` frames 0 in every log after the fix (before: present in 13).
|
|
- Suites: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental`: scheduler suite 118/118 OK, asan (log: /home/claude/tmp/bug-1928/gate-scheduler-suite.log).
|
|
- Live: the standalone command in Reproduction step 2 now exits 0 with leak detection on and prints its 18-line transcript (/home/claude/tmp/bug-1928/green/).
|
|
|
|
## References
|
|
- Related: bug-1884 (earlier standalone `--sim` LeakSanitizer exit-1 fix; this is a REPL-only leak it did not cover)
|
|
- Docs: src/scheduler/CLAUDE.md (Interactive REPL)
|