121 lines
7.3 KiB
Markdown
121 lines
7.3 KiB
Markdown
---
|
|
id: "bug-1925"
|
|
status: "fixed"
|
|
status_updated: "2026-10-08"
|
|
status_note: ""
|
|
component: "scheduler"
|
|
severity: "crash"
|
|
timestamp: "2026-10-08T15:35:55Z"
|
|
last_seen: "2026-10-08T15:35:55Z"
|
|
occurrences: 1
|
|
files: ["src/scheduler/src/repl.c", "src/scheduler/src/repl_trace.c", "src/scheduler/src/sched_election.c"]
|
|
tags: ["sim", "repl", "trace", "use-after-free", "race", "scheduler-sim-decision-trace"]
|
|
related: []
|
|
fix_commits: ["fcca4c9c"]
|
|
gitea_issue: 350
|
|
gitea_synced_status: "fixed"
|
|
gitea_synced_hash: "aa034b51ed6f"
|
|
---
|
|
|
|
# bug-1925: xce_scheduler --sim REPL quit mid-session frees trace settings the allocator thread may still read
|
|
|
|
## Description
|
|
**Observed:** found by code reading, not by a crash. `repl_cmd_quit()`
|
|
(src/scheduler/src/repl.c) ran `repl_trace_shutdown(sim)` first, which stores
|
|
`sim->core.trace_ctl = NULL` and unrefs/frees the cur/next trace settings
|
|
(including the `xce_sched_trace_ctl_t` they wrap), while the picker and
|
|
allocator threads were still running. At a breakpoint mid-session the
|
|
allocator keeps popping jobs, and `sched_allocator()`
|
|
(src/scheduler/src/sched_election.c) reads the ctl unlocked on every pop:
|
|
|
|
```c
|
|
if (core->trace_ctl != NULL) {
|
|
core->trace_ctl->position++;
|
|
if (sched_trace_ctl_wants(core->trace_ctl, jobid)) {
|
|
core->trace_rec = sched_trace_rec_new(core->trace_ctl, jobid, ...);
|
|
```
|
|
|
|
A pointer read before the NULL store can be dereferenced (and written, via
|
|
`position++`) after the free. The new deterministic test shows the ordering:
|
|
|
|
```
|
|
test_repl_trace_wiring.c:588:test_quit_stops_threads_before_trace_shutdown:FAIL: trace settings released while threads were live
|
|
test_repl_trace_wiring.c:588:test_eof_stops_threads_before_trace_shutdown:FAIL: trace settings released while threads were live
|
|
```
|
|
|
|
**Expected:** `quit` (and stdin EOF, which takes the same path) must not
|
|
release anything the scheduler threads can still reach; the threads are
|
|
stopped before the trace settings are freed.
|
|
**Impact:** heap use-after-free (read and write) on the allocator thread in
|
|
the window just before `exit(0)` after `quit`/EOF at a mid-session REPL
|
|
pause with tracing on. Simulator REPL only; live mode never publishes
|
|
`trace_ctl`. Worst case an ASAN abort or heap corruption at exit; no effect
|
|
on recorded output, which is flushed by then. Window is a few instructions,
|
|
so it is rare in practice.
|
|
**Found by:** review of scheduler-sim-decision-trace task 5.3 (code reading)
|
|
|
|
## Environment
|
|
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
|
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: asan (xce.build)
|
|
- Code: scheduler-framework @ a1cb9cba-dirty
|
|
- Deployment: offline `xce_scheduler --sim --interactive` (simulator REPL, no controller or workers); replay fixture `src/scheduler/testsuite/fixtures/repl_trace_baseline/contention.replay`; `ASAN_OPTIONS=detect_leaks=0` (the pre-existing 16 B repl_count_jobs leak, noted in tasks.md 5.2)
|
|
|
|
## Reproduction
|
|
1. Deterministic ordering test (on a1cb9cba, before the fix):
|
|
`nix develop ./src --command meson test -C src/xce.build --suite scheduler "scheduler repl trace wiring test" --print-errorlogs`
|
|
`test_quit_stops_threads_before_trace_shutdown` and
|
|
`test_eof_stops_threads_before_trace_shutdown` fail: they fork a child
|
|
with tracing on and two live stand-in picker/allocator threads parked in
|
|
a cancellation point, run `quit` / EOF, and a `-Wl,--wrap=repl_trace_shutdown`
|
|
probe records that the settings were released while the threads were live.
|
|
2. Live attempt at the real ASAN use-after-free:
|
|
`printf 'trace on all\nbreak DECISION\ncontinue\nquit\n' | ASAN_OPTIONS=detect_leaks=0 src/xce.build/scheduler/xce_scheduler --sim contention.replay --interactive --logfile sched.log`
|
|
(contention.replay from src/scheduler/testsuite/fixtures/repl_trace_baseline/),
|
|
looped 300 times.
|
|
- Frequency: deterministic ordering test (2/2 failing cases, every run, 2026-10-08). The live ASAN use-after-free did not reproduce: 0/300 runs, 2026-10-08 (session 1 has 7 jobs; at the break the allocator is normally already parked in cbuf_put on the full 5-slot decisions buffer, outside the few-instruction window).
|
|
- Evidence: /home/claude/tmp/bug-1925/ (red-test.log, green-test.log, gate.log; live/ holds the replay, scripts, and prefix-summary.txt / postfix-summary.txt for the live loops)
|
|
|
|
## Root cause
|
|
`repl_cmd_quit()` (repl.c), reached by the `quit` command and by a NULL
|
|
`readline()` (EOF) in `repl_command_loop()`, called `repl_trace_shutdown()`
|
|
(repl_trace.c) while the session's picker and allocator threads were live.
|
|
The REPL pauses on the driver thread at breakpoints in the middle of a
|
|
session's decision drain, and the allocator keeps working on the following
|
|
jobs, reading `core->trace_ctl` without a lock (by design, the ctl is
|
|
published only between sessions and the driver normally stops the threads
|
|
on return via `sched_sim_driver_run()` -> `sched_sim_driver_stop_threads()`).
|
|
`quit` calls `exit(0)` from inside the run, so that stop never happened
|
|
before the free. Since: `quit` started calling `repl_trace_shutdown` in
|
|
6bac1681 (task 5.2); the race became reachable when 5.3 (a1cb9cba) began
|
|
publishing `trace_ctl` from the sim session hooks. Scope: `repl_cmd_quit`
|
|
is the only path that ends the process from inside a running sim session;
|
|
the other `repl_trace_shutdown` caller is test fixture code. The other frees
|
|
in quit (closing record/replay/alt-events files) are of driver-thread-only
|
|
state and were not affected.
|
|
|
|
## Fix
|
|
`repl_cmd_quit()` (repl.c) now calls `sched_sim_driver_stop_threads(sim)`
|
|
before `repl_trace_shutdown(sim)`. That cancels and joins the picker and
|
|
allocator, so no thread can hold or load `trace_ctl` when it is freed.
|
|
Checked: EOF goes through `repl_cmd_quit` too; stop_threads is a no-op when
|
|
`allocator_thread_live` is false (quit at the pre-replay pause, before the
|
|
threads start); joining cannot hang at a REPL pause because both threads
|
|
block only in `cbuf_get`/`cbuf_put`, whose `pthread_cond_wait` is a
|
|
cancellation point with an unlock cleanup handler (including the allocator
|
|
parked on the full decisions buffer), and the driver thread holds no lock
|
|
the allocator needs while in the REPL; nothing after quit uses the threads
|
|
(quit only closes files, then `exit(0)`). Rejected: locking or atomically
|
|
swapping `trace_ctl` in the allocator, which would add a cost to the live
|
|
hot path the design keeps at one pointer test (Req 11.2) for a problem that
|
|
only exists at process exit.
|
|
|
|
## Verification
|
|
- Test: `test_quit_stops_threads_before_trace_shutdown` and `test_eof_stops_threads_before_trace_shutdown` in src/scheduler/testsuite/test_repl_trace_wiring.c (link-time wrap added in src/scheduler/testsuite/meson.build); RED before the fix ("trace settings released while threads were live"), GREEN after (17 Tests 0 Failures)
|
|
- Suites: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental`, scheduler suite 114/114 OK, asan
|
|
- Live: post-fix, the same `trace on all / break DECISION / continue / quit` script against contention.replay, 200 runs: 0 ASAN reports, 0 non-zero exits, 0 hangs (timeout 20 s); plus quit after two breakpoint hits with a trace file (exit 0, file written), quit at the pre-replay pause with no threads (exit 0), and EOF at a breakpoint (exit 0)
|
|
|
|
## References
|
|
- Related: none
|
|
- Found by: scheduler-sim-decision-trace task 5.3 review
|