myxce/.wolf/bugs/bug-1925.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	gitea_issue	gitea_synced_status	gitea_synced_hash
bug-1925	fixed	2026-10-08		scheduler	crash	2026-10-08T15:35:55Z	2026-10-08T15:35:55Z	1	src/scheduler/src/repl.c src/scheduler/src/repl_trace.c src/scheduler/src/sched_election.c	sim repl trace use-after-free race scheduler-sim-decision-trace		fcca4c9c	350	fixed	aa034b51ed6f

bug-1925: xce_scheduler --sim REPL quit mid-session frees trace settings the allocator thread may still read

Description

Observed: found by code reading, not by a crash. repl_cmd_quit() (src/scheduler/src/repl.c) ran repl_trace_shutdown(sim) first, which stores sim->core.trace_ctl = NULL and unrefs/frees the cur/next trace settings (including the xce_sched_trace_ctl_t they wrap), while the picker and allocator threads were still running. At a breakpoint mid-session the allocator keeps popping jobs, and sched_allocator() (src/scheduler/src/sched_election.c) reads the ctl unlocked on every pop:

if (core->trace_ctl != NULL) {
    core->trace_ctl->position++;
    if (sched_trace_ctl_wants(core->trace_ctl, jobid)) {
        core->trace_rec = sched_trace_rec_new(core->trace_ctl, jobid, ...);

A pointer read before the NULL store can be dereferenced (and written, via position++) after the free. The new deterministic test shows the ordering:

test_repl_trace_wiring.c:588:test_quit_stops_threads_before_trace_shutdown:FAIL: trace settings released while threads were live
test_repl_trace_wiring.c:588:test_eof_stops_threads_before_trace_shutdown:FAIL: trace settings released while threads were live

Expected: quit (and stdin EOF, which takes the same path) must not release anything the scheduler threads can still reach; the threads are stopped before the trace settings are freed. Impact: heap use-after-free (read and write) on the allocator thread in the window just before exit(0) after quit/EOF at a mid-session REPL pause with tracing on. Simulator REPL only; live mode never publishes trace_ctl. Worst case an ASAN abort or heap corruption at exit; no effect on recorded output, which is flushed by then. Window is a few instructions, so it is rare in practice. Found by: review of scheduler-sim-decision-trace task 5.3 (code reading)

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (xce.build)
• Code: scheduler-framework @ a1cb9cba-dirty
• Deployment: offline xce_scheduler --sim --interactive (simulator REPL, no controller or workers); replay fixture src/scheduler/testsuite/fixtures/repl_trace_baseline/contention.replay; ASAN_OPTIONS=detect_leaks=0 (the pre-existing 16 B repl_count_jobs leak, noted in tasks.md 5.2)

Reproduction

1. Deterministic ordering test (on a1cb9cba, before the fix): nix develop ./src --command meson test -C src/xce.build --suite scheduler "scheduler repl trace wiring test" --print-errorlogs test_quit_stops_threads_before_trace_shutdown and test_eof_stops_threads_before_trace_shutdown fail: they fork a child with tracing on and two live stand-in picker/allocator threads parked in a cancellation point, run quit / EOF, and a -Wl,--wrap=repl_trace_shutdown probe records that the settings were released while the threads were live.
2. Live attempt at the real ASAN use-after-free: printf 'trace on all\nbreak DECISION\ncontinue\nquit\n' | ASAN_OPTIONS=detect_leaks=0 src/xce.build/scheduler/xce_scheduler --sim contention.replay --interactive --logfile sched.log (contention.replay from src/scheduler/testsuite/fixtures/repl_trace_baseline/), looped 300 times.

• Frequency: deterministic ordering test (2/2 failing cases, every run, 2026-10-08). The live ASAN use-after-free did not reproduce: 0/300 runs, 2026-10-08 (session 1 has 7 jobs; at the break the allocator is normally already parked in cbuf_put on the full 5-slot decisions buffer, outside the few-instruction window).
• Evidence: /home/claude/tmp/bug-1925/ (red-test.log, green-test.log, gate.log; live/ holds the replay, scripts, and prefix-summary.txt / postfix-summary.txt for the live loops)

Root cause

repl_cmd_quit() (repl.c), reached by the quit command and by a NULL readline() (EOF) in repl_command_loop(), called repl_trace_shutdown() (repl_trace.c) while the session's picker and allocator threads were live. The REPL pauses on the driver thread at breakpoints in the middle of a session's decision drain, and the allocator keeps working on the following jobs, reading core->trace_ctl without a lock (by design, the ctl is published only between sessions and the driver normally stops the threads on return via sched_sim_driver_run() -> sched_sim_driver_stop_threads()). quit calls exit(0) from inside the run, so that stop never happened before the free. Since: quit started calling repl_trace_shutdown in 6bac1681 (task 5.2); the race became reachable when 5.3 (a1cb9cba) began publishing trace_ctl from the sim session hooks. Scope: repl_cmd_quit is the only path that ends the process from inside a running sim session; the other repl_trace_shutdown caller is test fixture code. The other frees in quit (closing record/replay/alt-events files) are of driver-thread-only state and were not affected.

Fix

repl_cmd_quit() (repl.c) now calls sched_sim_driver_stop_threads(sim) before repl_trace_shutdown(sim). That cancels and joins the picker and allocator, so no thread can hold or load trace_ctl when it is freed. Checked: EOF goes through repl_cmd_quit too; stop_threads is a no-op when allocator_thread_live is false (quit at the pre-replay pause, before the threads start); joining cannot hang at a REPL pause because both threads block only in cbuf_get/cbuf_put, whose pthread_cond_wait is a cancellation point with an unlock cleanup handler (including the allocator parked on the full decisions buffer), and the driver thread holds no lock the allocator needs while in the REPL; nothing after quit uses the threads (quit only closes files, then exit(0)). Rejected: locking or atomically swapping trace_ctl in the allocator, which would add a cost to the live hot path the design keeps at one pointer test (Req 11.2) for a problem that only exists at process exit.

Verification

• Test: test_quit_stops_threads_before_trace_shutdown and test_eof_stops_threads_before_trace_shutdown in src/scheduler/testsuite/test_repl_trace_wiring.c (link-time wrap added in src/scheduler/testsuite/meson.build); RED before the fix ("trace settings released while threads were live"), GREEN after (17 Tests 0 Failures)
• Suites: bash src/tools/xce-build.sh scheduler --jobs 12 --incremental, scheduler suite 114/114 OK, asan
• Live: post-fix, the same trace on all / break DECISION / continue / quit script against contention.replay, 200 runs: 0 ASAN reports, 0 non-zero exits, 0 hangs (timeout 20 s); plus quit after two breakpoint hits with a trace file (exit 0, file written), quit at the pre-replay pause with no threads (exit 0), and EOF at a breakpoint (exit 0)

References

• Related: none
• Found by: scheduler-sim-decision-trace task 5.3 review