lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(buglog): bug-1925 fixed (fcca4c9c)

Browse source 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-08 15:44:53 +00:00
parent fcca4c9cfa
commit 4d8d819201
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
2 changed files with 146 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

28
.wolf/buglog.json
View file

@ -17481,6 +17481,34 @@
"record": ".wolf/bugs/bug-1924.md",
"gitea_issue": 349,
"gitea_synced_status": "fixed"
},
{
"id": "bug-1925",
"status": "fixed",
"status_updated": "2026-10-08",
"status_note": "",
"component": "scheduler",
"severity": "crash",
"timestamp": "2026-10-08T15:35:55Z",
"last_seen": "2026-10-08T15:35:55Z",
"occurrences": 1,
"error_message": "xce_scheduler --sim REPL quit mid-session frees trace settings the allocator thread may still read\n\n**Observed:** found by code reading, not by a crash. `repl_cmd_quit()`\n(src/scheduler/src/repl.c) ran `repl_trace_shutdown(sim)` first, which stores\n`sim->core.trace_ctl = NULL` and unrefs/frees the cur/next trace settings\n(including the `xce_sched_trace_ctl_t` they wrap), while the picker and\nallocator threads were still running. At a breakpoint mid-session the\nallocator keeps popping jobs, and `sched_allocator()`\n(src/scheduler/src/sched_election.c) reads the ctl unlocked on every pop:\n\n```c\nif (core->trace_ctl != NULL) {\n core->trace_ctl->position++;\n if (sched_trace_ctl_wants(core->trace_ctl, jobid)) {\n core->trace_rec = sched_trace_rec_new(core->trace_ctl, jobid, ...);\n```\n\nA pointer read before the NULL store can be dereferenced (and written, via\n`position++`) after the free. The new deterministic test shows the ordering:\n\n```\ntest_repl_trace_wiring.c:588:test_quit_stops_threads_before_trace_shutdown:FAIL: trace settings released while threads were live\ntest_repl_trace_wiring.c:588:test_eof_stops_threads_before_trace_shutdown:FAIL: trace settings released while threads were live\n```\n\n**Expected:** `quit` (and stdin EOF, which takes the same path) must not\nrelease anything the scheduler threads can still reach; the threads are\nstopped before the trace settings are freed.\n**Impact:** heap use-after-free (read and write) on the allocator thread in\nthe window just before `exit(0)` after `quit`/EOF at a mid-session REPL\npause with tracing on. Simulator REPL only; live mode never publishes\n`trace_ctl`. Worst case an ASAN abort or heap corruption at exit; no effect\non recorded output, which is flushed by then. Window is a few instructions,\nso it is rare in practice.\n**Found by:** review of scheduler-sim-decision-trace task 5.3 (code reading)",
"file": "src/scheduler/src/repl.c, src/scheduler/src/repl_trace.c, src/scheduler/src/sched_election.c",
"root_cause": "`repl_cmd_quit()` (repl.c), reached by the `quit` command and by a NULL\n`readline()` (EOF) in `repl_command_loop()`, called `repl_trace_shutdown()`\n(repl_trace.c) while the session's picker and allocator threads were live.\nThe REPL pauses on the driver thread at breakpoints in the middle of a\nsession's decision drain, and the allocator keeps working on the following\njobs, reading `core->trace_ctl` without a lock (by design, the ctl is\npublished only between sessions and the driver normally stops the threads\non return via `sched_sim_driver_run()` -> `sched_sim_driver_stop_threads()`).\n`quit` calls `exit(0)` from inside the run, so that stop never happened\nbefore the free. Since: `quit` started calling `repl_trace_shutdown` in\n6bac1681 (task 5.2); the race became reachable when 5.3 (a1cb9cba) began\npublishing `trace_ctl` from the sim session hooks. Scope: `repl_cmd_quit`\nis the only path that ends the process from inside a running sim session;\nthe other `repl_trace_shutdown` caller is test fixture code. The other frees\nin quit (closing record/replay/alt-events files) are of driver-thread-only\nstate and were not affected.",
"fix": "`repl_cmd_quit()` (repl.c) now calls `sched_sim_driver_stop_threads(sim)`\nbefore `repl_trace_shutdown(sim)`. That cancels and joins the picker and\nallocator, so no thread can hold or load `trace_ctl` when it is freed.\nChecked: EOF goes through `repl_cmd_quit` too; stop_threads is a no-op when\n`allocator_thread_live` is false (quit at the pre-replay pause, before the\nthreads start); joining cannot hang at a REPL pause because both threads\nblock only in `cbuf_get`/`cbuf_put`, whose `pthread_cond_wait` is a\ncancellation point with an unlock cleanup handler (including the allocator\nparked on the full decisions buffer), and the driver thread holds no lock\nthe allocator needs while in the REPL; nothing after quit uses the threads\n(quit only closes files, then `exit(0)`). Rejected: locking or atomically\nswapping `trace_ctl` in the allocator, which would add a cost to the live\nhot path the design keeps at one pointer test (Req 11.2) for a problem that\nonly exists at process exit.\n\nVerification:\n- Test: `test_quit_stops_threads_before_trace_shutdown` and `test_eof_stops_threads_before_trace_shutdown` in src/scheduler/testsuite/test_repl_trace_wiring.c (link-time wrap added in src/scheduler/testsuite/meson.build); RED before the fix (\"trace settings released while threads were live\"), GREEN after (17 Tests 0 Failures)\n- Suites: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental`, scheduler suite 114/114 OK, asan\n- Live: post-fix, the same `trace on all / break DECISION / continue / quit` script against contention.replay, 200 runs: 0 ASAN reports, 0 non-zero exits, 0 hangs (timeout 20 s); plus quit after two breakpoint hits with a trace file (exit 0, file written), quit at the pre-replay pause with no threads (exit 0), and EOF at a breakpoint (exit 0)",
"tags": [
"sim",
"repl",
"trace",
"use-after-free",
"race",
"scheduler-sim-decision-trace"
],
"related_bugs": [],
"fix_commits": [
"fcca4c9c"
],
"record": ".wolf/bugs/bug-1925.md"
}
]
}

118
.wolf/bugs/bug-1925.md Normal file
View file

@ -0,0 +1,118 @@
---
id: "bug-1925"
status: "fixed"
status_updated: "2026-10-08"
status_note: ""
component: "scheduler"
severity: "crash"
timestamp: "2026-10-08T15:35:55Z"
last_seen: "2026-10-08T15:35:55Z"
occurrences: 1
files: ["src/scheduler/src/repl.c", "src/scheduler/src/repl_trace.c", "src/scheduler/src/sched_election.c"]
tags: ["sim", "repl", "trace", "use-after-free", "race", "scheduler-sim-decision-trace"]
related: []
fix_commits: ["fcca4c9c"]
---
# bug-1925: xce_scheduler --sim REPL quit mid-session frees trace settings the allocator thread may still read
## Description
**Observed:** found by code reading, not by a crash. `repl_cmd_quit()`
(src/scheduler/src/repl.c) ran `repl_trace_shutdown(sim)` first, which stores
`sim->core.trace_ctl = NULL` and unrefs/frees the cur/next trace settings
(including the `xce_sched_trace_ctl_t` they wrap), while the picker and
allocator threads were still running. At a breakpoint mid-session the
allocator keeps popping jobs, and `sched_allocator()`
(src/scheduler/src/sched_election.c) reads the ctl unlocked on every pop:
```c
if (core->trace_ctl != NULL) {
core->trace_ctl->position++;
if (sched_trace_ctl_wants(core->trace_ctl, jobid)) {
core->trace_rec = sched_trace_rec_new(core->trace_ctl, jobid, ...);
```
A pointer read before the NULL store can be dereferenced (and written, via
`position++`) after the free. The new deterministic test shows the ordering:
```
test_repl_trace_wiring.c:588:test_quit_stops_threads_before_trace_shutdown:FAIL: trace settings released while threads were live
test_repl_trace_wiring.c:588:test_eof_stops_threads_before_trace_shutdown:FAIL: trace settings released while threads were live
```
**Expected:** `quit` (and stdin EOF, which takes the same path) must not
release anything the scheduler threads can still reach; the threads are
stopped before the trace settings are freed.
**Impact:** heap use-after-free (read and write) on the allocator thread in
the window just before `exit(0)` after `quit`/EOF at a mid-session REPL
pause with tracing on. Simulator REPL only; live mode never publishes
`trace_ctl`. Worst case an ASAN abort or heap corruption at exit; no effect
on recorded output, which is flushed by then. Window is a few instructions,
so it is rare in practice.
**Found by:** review of scheduler-sim-decision-trace task 5.3 (code reading)
## Environment
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: asan (xce.build)
- Code: scheduler-framework @ a1cb9cba-dirty
- Deployment: offline `xce_scheduler --sim --interactive` (simulator REPL, no controller or workers); replay fixture `src/scheduler/testsuite/fixtures/repl_trace_baseline/contention.replay`; `ASAN_OPTIONS=detect_leaks=0` (the pre-existing 16 B repl_count_jobs leak, noted in tasks.md 5.2)
## Reproduction
1. Deterministic ordering test (on a1cb9cba, before the fix):
`nix develop ./src --command meson test -C src/xce.build --suite scheduler "scheduler repl trace wiring test" --print-errorlogs`
`test_quit_stops_threads_before_trace_shutdown` and
`test_eof_stops_threads_before_trace_shutdown` fail: they fork a child
with tracing on and two live stand-in picker/allocator threads parked in
a cancellation point, run `quit` / EOF, and a `-Wl,--wrap=repl_trace_shutdown`
probe records that the settings were released while the threads were live.
2. Live attempt at the real ASAN use-after-free:
`printf 'trace on all\nbreak DECISION\ncontinue\nquit\n' | ASAN_OPTIONS=detect_leaks=0 src/xce.build/scheduler/xce_scheduler --sim contention.replay --interactive --logfile sched.log`
(contention.replay from src/scheduler/testsuite/fixtures/repl_trace_baseline/),
looped 300 times.
- Frequency: deterministic ordering test (2/2 failing cases, every run, 2026-10-08). The live ASAN use-after-free did not reproduce: 0/300 runs, 2026-10-08 (session 1 has 7 jobs; at the break the allocator is normally already parked in cbuf_put on the full 5-slot decisions buffer, outside the few-instruction window).
- Evidence: /home/claude/tmp/bug-1925/ (red-test.log, green-test.log, gate.log; live/ holds the replay, scripts, and prefix-summary.txt / postfix-summary.txt for the live loops)
## Root cause
`repl_cmd_quit()` (repl.c), reached by the `quit` command and by a NULL
`readline()` (EOF) in `repl_command_loop()`, called `repl_trace_shutdown()`
(repl_trace.c) while the session's picker and allocator threads were live.
The REPL pauses on the driver thread at breakpoints in the middle of a
session's decision drain, and the allocator keeps working on the following
jobs, reading `core->trace_ctl` without a lock (by design, the ctl is
published only between sessions and the driver normally stops the threads
on return via `sched_sim_driver_run()` -> `sched_sim_driver_stop_threads()`).
`quit` calls `exit(0)` from inside the run, so that stop never happened
before the free. Since: `quit` started calling `repl_trace_shutdown` in
6bac1681 (task 5.2); the race became reachable when 5.3 (a1cb9cba) began
publishing `trace_ctl` from the sim session hooks. Scope: `repl_cmd_quit`
is the only path that ends the process from inside a running sim session;
the other `repl_trace_shutdown` caller is test fixture code. The other frees
in quit (closing record/replay/alt-events files) are of driver-thread-only
state and were not affected.
## Fix
`repl_cmd_quit()` (repl.c) now calls `sched_sim_driver_stop_threads(sim)`
before `repl_trace_shutdown(sim)`. That cancels and joins the picker and
allocator, so no thread can hold or load `trace_ctl` when it is freed.
Checked: EOF goes through `repl_cmd_quit` too; stop_threads is a no-op when
`allocator_thread_live` is false (quit at the pre-replay pause, before the
threads start); joining cannot hang at a REPL pause because both threads
block only in `cbuf_get`/`cbuf_put`, whose `pthread_cond_wait` is a
cancellation point with an unlock cleanup handler (including the allocator
parked on the full decisions buffer), and the driver thread holds no lock
the allocator needs while in the REPL; nothing after quit uses the threads
(quit only closes files, then `exit(0)`). Rejected: locking or atomically
swapping `trace_ctl` in the allocator, which would add a cost to the live
hot path the design keeps at one pointer test (Req 11.2) for a problem that
only exists at process exit.
## Verification
- Test: `test_quit_stops_threads_before_trace_shutdown` and `test_eof_stops_threads_before_trace_shutdown` in src/scheduler/testsuite/test_repl_trace_wiring.c (link-time wrap added in src/scheduler/testsuite/meson.build); RED before the fix ("trace settings released while threads were live"), GREEN after (17 Tests 0 Failures)
- Suites: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental`, scheduler suite 114/114 OK, asan
- Live: post-fix, the same `trace on all / break DECISION / continue / quit` script against contention.replay, 200 runs: 0 ASAN reports, 0 non-zero exits, 0 hangs (timeout 20 s); plus quit after two breakpoint hits with a trace file (exit 0, file written), quit at the pre-replay pause with no threads (exit 0), and EOF at a breakpoint (exit 0)
## References
- Related: none
- Found by: scheduler-sim-decision-trace task 5.3 review