---
id: "bug-1925"
status: "fixed"
status_updated: "2026-10-08"
status_note: ""
component: "scheduler"
severity: "crash"
timestamp: "2026-10-08T15:35:55Z"
last_seen: "2026-10-08T15:35:55Z"
occurrences: 1
files: ["src/scheduler/src/repl.c", "src/scheduler/src/repl_trace.c", "src/scheduler/src/sched_election.c"]
tags: ["sim", "repl", "trace", "use-after-free", "race", "scheduler-sim-decision-trace"]
related: []
fix_commits: ["fcca4c9c"]
gitea_issue: 350
gitea_synced_status: "fixed"
gitea_synced_hash: "aa034b51ed6f"
---

# bug-1925: xce_scheduler --sim REPL quit mid-session frees trace settings the allocator thread may still read

## Description
**Observed:** found by code reading, not by a crash. `repl_cmd_quit()`
(src/scheduler/src/repl.c) ran `repl_trace_shutdown(sim)` first, which stores
`sim->core.trace_ctl = NULL` and unrefs/frees the cur/next trace settings
(including the `xce_sched_trace_ctl_t` they wrap), while the picker and
allocator threads were still running. At a breakpoint mid-session the
allocator keeps popping jobs, and `sched_allocator()`
(src/scheduler/src/sched_election.c) reads the ctl unlocked on every pop:

```c
if (core->trace_ctl != NULL) {
    core->trace_ctl->position++;
    if (sched_trace_ctl_wants(core->trace_ctl, jobid)) {
        core->trace_rec = sched_trace_rec_new(core->trace_ctl, jobid, ...);
```

A pointer read before the NULL store can be dereferenced (and written, via
`position++`) after the free. The new deterministic test shows the ordering:

```
test_repl_trace_wiring.c:588:test_quit_stops_threads_before_trace_shutdown:FAIL: trace settings released while threads were live
test_repl_trace_wiring.c:588:test_eof_stops_threads_before_trace_shutdown:FAIL: trace settings released while threads were live
```

**Expected:** `quit` (and stdin EOF, which takes the same path) must not
release anything the scheduler threads can still reach; the threads are
stopped before the trace settings are freed.
**Impact:** heap use-after-free (read and write) on the allocator thread in
the window just before `exit(0)` after `quit`/EOF at a mid-session REPL
pause with tracing on. Simulator REPL only; live mode never publishes
`trace_ctl`. Worst case an ASAN abort or heap corruption at exit; no effect
on recorded output, which is flushed by then. Window is a few instructions,
so it is rare in practice.
**Found by:** review of scheduler-sim-decision-trace task 5.3 (code reading)

## Environment
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: asan (xce.build)
- Code: scheduler-framework @ a1cb9cba-dirty
- Deployment: offline `xce_scheduler --sim --interactive` (simulator REPL, no controller or workers); replay fixture `src/scheduler/testsuite/fixtures/repl_trace_baseline/contention.replay`; `ASAN_OPTIONS=detect_leaks=0` (the pre-existing 16 B repl_count_jobs leak, noted in tasks.md 5.2)

## Reproduction
1. Deterministic ordering test (on a1cb9cba, before the fix):
   `nix develop ./src --command meson test -C src/xce.build --suite scheduler "scheduler repl trace wiring test" --print-errorlogs`
   `test_quit_stops_threads_before_trace_shutdown` and
   `test_eof_stops_threads_before_trace_shutdown` fail: they fork a child
   with tracing on and two live stand-in picker/allocator threads parked in
   a cancellation point, run `quit` / EOF, and a `-Wl,--wrap=repl_trace_shutdown`
   probe records that the settings were released while the threads were live.
2. Live attempt at the real ASAN use-after-free:
   `printf 'trace on all\nbreak DECISION\ncontinue\nquit\n' | ASAN_OPTIONS=detect_leaks=0 src/xce.build/scheduler/xce_scheduler --sim contention.replay --interactive --logfile sched.log`
   (contention.replay from src/scheduler/testsuite/fixtures/repl_trace_baseline/),
   looped 300 times.
- Frequency: deterministic ordering test (2/2 failing cases, every run, 2026-10-08). The live ASAN use-after-free did not reproduce: 0/300 runs, 2026-10-08 (session 1 has 7 jobs; at the break the allocator is normally already parked in cbuf_put on the full 5-slot decisions buffer, outside the few-instruction window).
- Evidence: /home/claude/tmp/bug-1925/ (red-test.log, green-test.log, gate.log; live/ holds the replay, scripts, and prefix-summary.txt / postfix-summary.txt for the live loops)

## Root cause
`repl_cmd_quit()` (repl.c), reached by the `quit` command and by a NULL
`readline()` (EOF) in `repl_command_loop()`, called `repl_trace_shutdown()`
(repl_trace.c) while the session's picker and allocator threads were live.
The REPL pauses on the driver thread at breakpoints in the middle of a
session's decision drain, and the allocator keeps working on the following
jobs, reading `core->trace_ctl` without a lock (by design, the ctl is
published only between sessions and the driver normally stops the threads
on return via `sched_sim_driver_run()` -> `sched_sim_driver_stop_threads()`).
`quit` calls `exit(0)` from inside the run, so that stop never happened
before the free. Since: `quit` started calling `repl_trace_shutdown` in
6bac1681 (task 5.2); the race became reachable when 5.3 (a1cb9cba) began
publishing `trace_ctl` from the sim session hooks. Scope: `repl_cmd_quit`
is the only path that ends the process from inside a running sim session;
the other `repl_trace_shutdown` caller is test fixture code. The other frees
in quit (closing record/replay/alt-events files) are of driver-thread-only
state and were not affected.

## Fix
`repl_cmd_quit()` (repl.c) now calls `sched_sim_driver_stop_threads(sim)`
before `repl_trace_shutdown(sim)`. That cancels and joins the picker and
allocator, so no thread can hold or load `trace_ctl` when it is freed.
Checked: EOF goes through `repl_cmd_quit` too; stop_threads is a no-op when
`allocator_thread_live` is false (quit at the pre-replay pause, before the
threads start); joining cannot hang at a REPL pause because both threads
block only in `cbuf_get`/`cbuf_put`, whose `pthread_cond_wait` is a
cancellation point with an unlock cleanup handler (including the allocator
parked on the full decisions buffer), and the driver thread holds no lock
the allocator needs while in the REPL; nothing after quit uses the threads
(quit only closes files, then `exit(0)`). Rejected: locking or atomically
swapping `trace_ctl` in the allocator, which would add a cost to the live
hot path the design keeps at one pointer test (Req 11.2) for a problem that
only exists at process exit.

## Verification
- Test: `test_quit_stops_threads_before_trace_shutdown` and `test_eof_stops_threads_before_trace_shutdown` in src/scheduler/testsuite/test_repl_trace_wiring.c (link-time wrap added in src/scheduler/testsuite/meson.build); RED before the fix ("trace settings released while threads were live"), GREEN after (17 Tests 0 Failures)
- Suites: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental`, scheduler suite 114/114 OK, asan
- Live: post-fix, the same `trace on all / break DECISION / continue / quit` script against contention.replay, 200 runs: 0 ASAN reports, 0 non-zero exits, 0 hangs (timeout 20 s); plus quit after two breakpoint hits with a trace file (exit 0, file written), quit at the pre-replay pause with no threads (exit 0), and EOF at a breakpoint (exit 0)

## References
- Related: none
- Found by: scheduler-sim-decision-trace task 5.3 review