chore(buglog): bug-1632 and bug-1763 fixed (cfa0a941)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
cfa0a94166
commit
2d4b893d75
GPG key ID:
80533536B0EB651C
3 changed files with 66 additions and 38 deletions
|
|
@ -9313,18 +9313,18 @@
|
|||
},
|
||||
{
|
||||
"id": "bug-1632",
|
||||
"status": "open",
|
||||
"status_updated": "2026-09-24",
|
||||
"status_note": "test_sim_verify.c stack-local sim read by picker thread (ASAN stack-use-after-return)",
|
||||
"status": "fixed",
|
||||
"status_updated": "2026-09-30",
|
||||
"status_note": "",
|
||||
"component": "scheduler",
|
||||
"severity": "security",
|
||||
"severity": "crash",
|
||||
"timestamp": "2026-08-23T21:03:00.000Z",
|
||||
"last_seen": "2026-09-19T09:00:00Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "sched_sim_verify_test: AddressSanitizer stack-use-after-return in cbuf_get (lib/mylock.c:151)…\n\nsched_sim_verify_test: AddressSanitizer stack-use-after-return in cbuf_get (lib/mylock.c:151), thread T2 (sim_picker_thread_fn, sim_loop.c:456), reading the stack of run_sim_and_capture_stdout's `sim` (xce_sched_sim_driver_t) after that frame returned/was reused. Fired once during a full `/xce-build scheduler` run (scheduler-dispatch-commit task 5.4 regression pass): Ok:50 Fail:2 instead of the documented Ok:51 Fail:1 baseline, extra failure = 'scheduler sim verify test'. Did NOT reproduce on: 3 standalone direct re-runs of sched_sim_verify_test (16 Tests 0 Failures each time) or a second full `meson test --suite scheduler` re-run against the same build (back to Ok:51 Fail:1, only the known test_admin_request.c symlink-escape failure) -- confirmed non-deterministic/order-dependent, not caused by scheduler-dispatch-commit.",
|
||||
"file": "src/scheduler/testsuite/test_sim_verify.c",
|
||||
"root_cause": "Same root cause as bug-1598 (already diagnosed, NOT re-derived here): sched_sim_driver_run() (src/scheduler/src/sim_loop.c) spawns sim_picker_thread_fn on a pthread that is, by design, never joined/cancelled -- it loops forever on cbuf_get(&sim->core.wbuf,...) reading `sim` for the life of the process. bug-1598's fix converted 3 stack-local `xce_sched_sim_driver_t sim;` fixtures in test_sim_event_loop.c to `static` storage, but test_sim_verify.c's OWN, separately-defined run_sim_and_capture_stdout() helper (test_sim_verify.c:940-943) still declares `sim` as stack-local -- an unfixed sibling instance of the identical latent bug, never touched by bug-1598's fix. Confirmed via git diff/log that scheduler-dispatch-commit's own commits (which only touch scheduler.c/handler.c per that spec's Requirement 5.2 sim-mode-exclusion) never modified sim_loop.c or test_sim_verify.c -- this is pre-existing latent test infra, exposed non-deterministically by whatever stack-layout/MALLOC_PERTURB_ randomization a given meson test invocation happens to produce, exactly like bug-1598's own trigger mechanism.",
|
||||
"fix": "NOT FIXED -- out of scope for scheduler-dispatch-commit task 5.4 (a validation/regression task, not a scheduler-sim-infra task). Same remediation as bug-1598 would apply if picked up: change run_sim_and_capture_stdout()'s `xce_sched_sim_driver_t sim;` from stack-local to `static` storage (safe under the same reasoning bug-1598 documents: sched_sim_driver_start() memset()s the whole struct first, and this helper is not re-entered while an earlier zombie thread could still be alive within one test binary process -- verify that non-reentrancy holds for test_sim_verify.c's own call sites before applying). Logged so a future session doesn't re-diagnose from scratch.",
|
||||
"file": "src/scheduler/src/sim_loop.c (sched_sim_driver_run), src/scheduler/src/sim.c (sched_sim_driver_stop_threads, sched_sim_driver_fini), src/scheduler/testsuite/test_sim_verify.c",
|
||||
"root_cause": "sched_sim_driver_run() (sim_loop.c) starts the picker and allocator threads (sched_job_picker/sched_allocator, or a test's stand-ins) on &sim->core and returned with them still running; only sched_sim_driver_fini() (sim.c) cancelled and joined them. main.c pairs the two, but 24 scheduler test files (35 call sites) run the driver on a stack-local xce_sched_sim_driver_t and never call fini. The threads, parked in cbuf_get() on the core's cbufs, then held a pointer into a dead frame; when a later test reused that stack memory and something woke them, one read it -- the reported stack-use-after-return in cbuf_get (mylock.c). Timing-dependent, so seen only occasionally under full-suite load. Also: if the allocator thread failed to start, the already-running picker was never stopped at all.",
|
||||
"fix": "src/scheduler/src/sim_loop.c: the loop moved into static sim_driver_run_body(); sched_sim_driver_run() calls it and then sched_sim_driver_stop_threads() on every return path, so no thread outlives the run; an allocator start failure now cancels and joins the running picker. src/scheduler/src/sim.c: new sched_sim_driver_stop_threads() (cancel + join both, idempotent via allocator_thread_live), used by the run and by sched_sim_driver_fini(); it no longer pushes a jobid 0 to wake the allocator -- cbuf waits in pthread_cond_wait (a cancellation point, mutex released by a cleanup handler), and the fake end-of-session was visible to the allocator (test_sim_discrete_clock caught it as an extra session). sim.h documents the guarantee.\n\nVerification:\n- Test: test_run_returns_with_its_threads_stopped in src/scheduler/testsuite/test_sim_verify.c runs a replay through sched_sim_driver_run() on a stack-local driver and asserts no thread is left running: RED before the fix (`bug-1632: sched_sim_driver_run() returned with its picker and allocator threads still running`), GREEN after -- deterministic, unlike the original ASAN report\n- Suites: scheduler 90/90 test binaries, asan, 0 sanitizer reports (first attempt had test_sim_discrete_clock fail on the jobid-0 wake-up; fixed as above)\n- Live: n/a, test-harness lifetime bug; 30 isolated runs each of sched_sim_verify_test and sched_sim_record_output_test (ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=1) were clean both before and after the fix, matching the records' note that it never reproduced in isolation",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"asan",
|
||||
|
|
@ -9340,7 +9340,9 @@
|
|||
"related_bugs": [
|
||||
"bug-1598"
|
||||
],
|
||||
"fix_commits": [],
|
||||
"fix_commits": [
|
||||
"cfa0a941"
|
||||
],
|
||||
"record": ".wolf/bugs/bug-1632.md"
|
||||
},
|
||||
{
|
||||
|
|
@ -12327,18 +12329,18 @@
|
|||
},
|
||||
{
|
||||
"id": "bug-1763",
|
||||
"status": "open",
|
||||
"status_updated": "2026-09-24",
|
||||
"status_note": "test_sim_record_output.c stack-local sim read by unjoined picker thread (ASAN)",
|
||||
"status": "fixed",
|
||||
"status_updated": "2026-09-30",
|
||||
"status_note": "",
|
||||
"component": "scheduler",
|
||||
"severity": "unknown",
|
||||
"severity": "crash",
|
||||
"timestamp": "2026-09-02T13:00:00Z",
|
||||
"last_seen": "2026-09-19T09:00:00Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "sched_sim_record_output_test (scheduler-sim-repl-verbosity task 3.2 GREEN-phase build)…\n\nsched_sim_record_output_test (scheduler-sim-repl-verbosity task 3.2 GREEN-phase build): AddressSanitizer stack-use-after-return in cbuf_get (lib/mylock.c:151), thread T1 (sim_picker_thread_fn, sim_loop.c:456), reading the stack of test_output_file_is_decodable_with_full_stream_symmetry's stack-local `sim` (xce_sched_sim_driver_t, test_sim_record_output.c:341) after that frame returned. Fired once during a `/xce-build scheduler --incremental` run after this task's repl_break.c edits; 4/4 direct re-runs of the isolated binary afterward were clean (OK).",
|
||||
"file": "src/scheduler/testsuite/test_sim_record_output.c",
|
||||
"root_cause": "Same class already documented by bug-1598/bug-1632, not a new defect: sched_sim_driver_run() spawns sim_picker_thread_fn on an unjoined pthread that reads its xce_sched_sim_driver_t* for the rest of the process's life; any stack-local `sim` fixture is therefore a latent stack-use-after-return the moment its owning test function returns, and whether ASAN's shadow memory actually catches a given run is purely a function of incidental stack-layout/timing -- bug-1598's own root cause entry documents an unrelated new test shifting stack layout enough to make this fire deterministically for a different file. test_sim_record_output.c #includes repl_break.c white-box (test_sim_record_output.c:78), so this task's repl_break.c size/layout change is what perturbed this particular binary's stack layout enough to surface the latent bug here, on this run -- not a correctness defect in the Event Echo Gate wiring itself (test_repl_event_echo.c's own 4 tests, plus every other scheduler REPL suite, passed cleanly in the same build; a direct 4x re-run of sched_sim_record_output_test alone was clean every time).",
|
||||
"fix": "NOT FIXED -- out of scope for scheduler-sim-repl-verbosity task 3.2 (repl_break.c wiring only, no sim_loop.c/test_sim_record_output.c changes in this task's boundary). Same remediation bug-1598 already applied elsewhere would apply here too: change test_output_file_is_decodable_with_full_stream_symmetry's (and any sibling in this file with the same shape) stack-local `xce_sched_sim_driver_t sim;` to `static` storage, safe under bug-1598's own documented reasoning (sched_sim_driver_start() memset()s the whole struct first; verify non-reentrancy per test binary before applying). Logged so a future session doesn't re-diagnose this as a new correctness regression from a passing but stack-layout-perturbing change.",
|
||||
"file": "src/scheduler/src/sim_loop.c (sched_sim_driver_run), src/scheduler/src/sim.c (sched_sim_driver_stop_threads, sched_sim_driver_fini), src/scheduler/testsuite/test_sim_record_output.c",
|
||||
"root_cause": "sched_sim_driver_run() (sim_loop.c) starts the picker and allocator threads (sched_job_picker/sched_allocator, or a test's stand-ins) on &sim->core and returned with them still running; only sched_sim_driver_fini() (sim.c) cancelled and joined them. main.c pairs the two, but 24 scheduler test files (35 call sites) run the driver on a stack-local xce_sched_sim_driver_t and never call fini. The threads, parked in cbuf_get() on the core's cbufs, then held a pointer into a dead frame; when a later test reused that stack memory and something woke them, one read it -- the reported stack-use-after-return in cbuf_get (mylock.c). Timing-dependent, so seen only occasionally under full-suite load. Also: if the allocator thread failed to start, the already-running picker was never stopped at all.",
|
||||
"fix": "src/scheduler/src/sim_loop.c: the loop moved into static sim_driver_run_body(); sched_sim_driver_run() calls it and then sched_sim_driver_stop_threads() on every return path, so no thread outlives the run; an allocator start failure now cancels and joins the running picker. src/scheduler/src/sim.c: new sched_sim_driver_stop_threads() (cancel + join both, idempotent via allocator_thread_live), used by the run and by sched_sim_driver_fini(); it no longer pushes a jobid 0 to wake the allocator -- cbuf waits in pthread_cond_wait (a cancellation point, mutex released by a cleanup handler), and the fake end-of-session was visible to the allocator (test_sim_discrete_clock caught it as an extra session). sim.h documents the guarantee.\n\nVerification:\n- Test: test_run_returns_with_its_threads_stopped in src/scheduler/testsuite/test_sim_verify.c runs a replay through sched_sim_driver_run() on a stack-local driver and asserts no thread is left running: RED before the fix (`bug-1632: sched_sim_driver_run() returned with its picker and allocator threads still running`), GREEN after -- deterministic, unlike the original ASAN report\n- Suites: scheduler 90/90 test binaries, asan, 0 sanitizer reports (first attempt had test_sim_discrete_clock fail on the jobid-0 wake-up; fixed as above)\n- Live: n/a, test-harness lifetime bug; 30 isolated runs each of sched_sim_verify_test and sched_sim_record_output_test (ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=1) were clean both before and after the fix, matching the records' note that it never reproduced in isolation",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"sim",
|
||||
|
|
@ -12353,7 +12355,9 @@
|
|||
"bug-1598",
|
||||
"bug-1632"
|
||||
],
|
||||
"fix_commits": [],
|
||||
"fix_commits": [
|
||||
"cfa0a941"
|
||||
],
|
||||
"record": ".wolf/bugs/bug-1763.md"
|
||||
},
|
||||
{
|
||||
|
|
|
|||
|
|
@ -1,16 +1,17 @@
|
|||
---
|
||||
id: "bug-1632"
|
||||
status: "open"
|
||||
status_updated: "2026-09-24"
|
||||
status_note: "test_sim_verify.c stack-local sim read by picker thread (ASAN stack-use-after-return)"
|
||||
status: "fixed"
|
||||
status_updated: "2026-09-30"
|
||||
status_note: ""
|
||||
component: "scheduler"
|
||||
severity: "security"
|
||||
severity: "crash"
|
||||
timestamp: "2026-08-23T21:03:00.000Z"
|
||||
last_seen: "2026-09-19T09:00:00Z"
|
||||
occurrences: 1
|
||||
files: ["src/scheduler/testsuite/test_sim_verify.c"]
|
||||
files: ["src/scheduler/src/sim_loop.c (sched_sim_driver_run)", "src/scheduler/src/sim.c (sched_sim_driver_stop_threads, sched_sim_driver_fini)", "src/scheduler/testsuite/test_sim_verify.c"]
|
||||
tags: ["scheduler", "asan", "stack-use-after-return", "thread-lifecycle", "test-only", "sim-driver", "flaky", "non-deterministic", "pre-existing", "sibling-of-bug-1598"]
|
||||
related: ["bug-1598"]
|
||||
fix_commits: ["cfa0a941"]
|
||||
legacy: true
|
||||
---
|
||||
|
||||
|
|
@ -20,19 +21,30 @@ legacy: true
|
|||
sched_sim_verify_test: AddressSanitizer stack-use-after-return in cbuf_get (lib/mylock.c:151), thread T2 (sim_picker_thread_fn, sim_loop.c:456), reading the stack of run_sim_and_capture_stdout's `sim` (xce_sched_sim_driver_t) after that frame returned/was reused. Fired once during a full `/xce-build scheduler` run (scheduler-dispatch-commit task 5.4 regression pass): Ok:50 Fail:2 instead of the documented Ok:51 Fail:1 baseline, extra failure = 'scheduler sim verify test'. Did NOT reproduce on: 3 standalone direct re-runs of sched_sim_verify_test (16 Tests 0 Failures each time) or a second full `meson test --suite scheduler` re-run against the same build (back to Ok:51 Fail:1, only the known test_admin_request.c symlink-escape failure) -- confirmed non-deterministic/order-dependent, not caused by scheduler-dispatch-commit.
|
||||
|
||||
## Environment
|
||||
_Not recorded (migrated from buglog.json)._
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: asan; meson runs these tests with ASAN_OPTIONS=detect_leaks=0
|
||||
- Code: original sightings on scheduler-framework in Aug/Sep 2026 (commit not recorded); root-caused and fixed on 2026-09-29 at dc94da20
|
||||
- Deployment: n/a -- unit test binary, no cluster
|
||||
|
||||
## Reproduction
|
||||
_Not recorded (migrated from buglog.json)._
|
||||
1. `bash src/tools/xce-build.sh scheduler` (full suite under load); the ASAN abort shows in sched_sim_verify_test's log.
|
||||
2. Deterministic check of the mechanism: test_run_returns_with_its_threads_stopped (src/scheduler/testsuite/test_sim_verify.c) -- sched_sim_driver_run() returns with allocator_thread_live still true.
|
||||
- Frequency: original ASAN report rare (1 in several full-suite runs; 0/30 isolated runs, 2026-09-29); the mechanism check is deterministic
|
||||
- Evidence: not kept -- the original build logs were not saved; the test above reproduces the cause
|
||||
|
||||
## Root cause
|
||||
Same root cause as bug-1598 (already diagnosed, NOT re-derived here): sched_sim_driver_run() (src/scheduler/src/sim_loop.c) spawns sim_picker_thread_fn on a pthread that is, by design, never joined/cancelled -- it loops forever on cbuf_get(&sim->core.wbuf,...) reading `sim` for the life of the process. bug-1598's fix converted 3 stack-local `xce_sched_sim_driver_t sim;` fixtures in test_sim_event_loop.c to `static` storage, but test_sim_verify.c's OWN, separately-defined run_sim_and_capture_stdout() helper (test_sim_verify.c:940-943) still declares `sim` as stack-local -- an unfixed sibling instance of the identical latent bug, never touched by bug-1598's fix. Confirmed via git diff/log that scheduler-dispatch-commit's own commits (which only touch scheduler.c/handler.c per that spec's Requirement 5.2 sim-mode-exclusion) never modified sim_loop.c or test_sim_verify.c -- this is pre-existing latent test infra, exposed non-deterministically by whatever stack-layout/MALLOC_PERTURB_ randomization a given meson test invocation happens to produce, exactly like bug-1598's own trigger mechanism.
|
||||
sched_sim_driver_run() (sim_loop.c) starts the picker and allocator threads (sched_job_picker/sched_allocator, or a test's stand-ins) on &sim->core and returned with them still running; only sched_sim_driver_fini() (sim.c) cancelled and joined them. main.c pairs the two, but 24 scheduler test files (35 call sites) run the driver on a stack-local xce_sched_sim_driver_t and never call fini. The threads, parked in cbuf_get() on the core's cbufs, then held a pointer into a dead frame; when a later test reused that stack memory and something woke them, one read it -- the reported stack-use-after-return in cbuf_get (mylock.c). Timing-dependent, so seen only occasionally under full-suite load. Also: if the allocator thread failed to start, the already-running picker was never stopped at all.
|
||||
|
||||
## Fix
|
||||
NOT FIXED -- out of scope for scheduler-dispatch-commit task 5.4 (a validation/regression task, not a scheduler-sim-infra task). Same remediation as bug-1598 would apply if picked up: change run_sim_and_capture_stdout()'s `xce_sched_sim_driver_t sim;` from stack-local to `static` storage (safe under the same reasoning bug-1598 documents: sched_sim_driver_start() memset()s the whole struct first, and this helper is not re-entered while an earlier zombie thread could still be alive within one test binary process -- verify that non-reentrancy holds for test_sim_verify.c's own call sites before applying). Logged so a future session doesn't re-diagnose from scratch.
|
||||
src/scheduler/src/sim_loop.c: the loop moved into static sim_driver_run_body(); sched_sim_driver_run() calls it and then sched_sim_driver_stop_threads() on every return path, so no thread outlives the run; an allocator start failure now cancels and joins the running picker. src/scheduler/src/sim.c: new sched_sim_driver_stop_threads() (cancel + join both, idempotent via allocator_thread_live), used by the run and by sched_sim_driver_fini(); it no longer pushes a jobid 0 to wake the allocator -- cbuf waits in pthread_cond_wait (a cancellation point, mutex released by a cleanup handler), and the fake end-of-session was visible to the allocator (test_sim_discrete_clock caught it as an extra session). sim.h documents the guarantee.
|
||||
|
||||
## Verification
|
||||
_Not recorded (migrated from buglog.json)._
|
||||
- Test: test_run_returns_with_its_threads_stopped in src/scheduler/testsuite/test_sim_verify.c runs a replay through sched_sim_driver_run() on a stack-local driver and asserts no thread is left running: RED before the fix (`bug-1632: sched_sim_driver_run() returned with its picker and allocator threads still running`), GREEN after -- deterministic, unlike the original ASAN report
|
||||
- Suites: scheduler 90/90 test binaries, asan, 0 sanitizer reports (first attempt had test_sim_discrete_clock fail on the jobid-0 wake-up; fixed as above)
|
||||
- Live: n/a, test-harness lifetime bug; 30 isolated runs each of sched_sim_verify_test and sched_sim_record_output_test (ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=1) were clean both before and after the fix, matching the records' note that it never reproduced in isolation
|
||||
|
||||
## References
|
||||
- Related: bug-1598
|
||||
- Fix commit: cfa0a941
|
||||
- Related: bug-1763 (same cause, other test binary), bug-1884 (added sched_sim_driver_fini)
|
||||
|
|
|
|||
|
|
@ -1,16 +1,17 @@
|
|||
---
|
||||
id: "bug-1763"
|
||||
status: "open"
|
||||
status_updated: "2026-09-24"
|
||||
status_note: "test_sim_record_output.c stack-local sim read by unjoined picker thread (ASAN)"
|
||||
status: "fixed"
|
||||
status_updated: "2026-09-30"
|
||||
status_note: ""
|
||||
component: "scheduler"
|
||||
severity: "unknown"
|
||||
severity: "crash"
|
||||
timestamp: "2026-09-02T13:00:00Z"
|
||||
last_seen: "2026-09-19T09:00:00Z"
|
||||
occurrences: 1
|
||||
files: ["src/scheduler/testsuite/test_sim_record_output.c"]
|
||||
files: ["src/scheduler/src/sim_loop.c (sched_sim_driver_run)", "src/scheduler/src/sim.c (sched_sim_driver_stop_threads, sched_sim_driver_fini)", "src/scheduler/testsuite/test_sim_record_output.c"]
|
||||
tags: ["scheduler", "sim", "repl", "asan", "stack-use-after-return", "flaky", "thread-lifecycle", "scheduler-sim-repl-verbosity"]
|
||||
related: ["bug-1598", "bug-1632"]
|
||||
fix_commits: ["cfa0a941"]
|
||||
legacy: true
|
||||
---
|
||||
|
||||
|
|
@ -20,19 +21,30 @@ legacy: true
|
|||
sched_sim_record_output_test (scheduler-sim-repl-verbosity task 3.2 GREEN-phase build): AddressSanitizer stack-use-after-return in cbuf_get (lib/mylock.c:151), thread T1 (sim_picker_thread_fn, sim_loop.c:456), reading the stack of test_output_file_is_decodable_with_full_stream_symmetry's stack-local `sim` (xce_sched_sim_driver_t, test_sim_record_output.c:341) after that frame returned. Fired once during a `/xce-build scheduler --incremental` run after this task's repl_break.c edits; 4/4 direct re-runs of the isolated binary afterward were clean (OK).
|
||||
|
||||
## Environment
|
||||
_Not recorded (migrated from buglog.json)._
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: asan; meson runs these tests with ASAN_OPTIONS=detect_leaks=0
|
||||
- Code: original sightings on scheduler-framework in Aug/Sep 2026 (commit not recorded); root-caused and fixed on 2026-09-29 at dc94da20
|
||||
- Deployment: n/a -- unit test binary, no cluster
|
||||
|
||||
## Reproduction
|
||||
_Not recorded (migrated from buglog.json)._
|
||||
1. `bash src/tools/xce-build.sh scheduler` (full suite under load); the ASAN abort shows in sched_sim_record_output_test's log.
|
||||
2. Deterministic check of the mechanism: test_run_returns_with_its_threads_stopped (src/scheduler/testsuite/test_sim_verify.c) -- sched_sim_driver_run() returns with allocator_thread_live still true.
|
||||
- Frequency: original ASAN report rare (1 in several full-suite runs; 0/30 isolated runs, 2026-09-29); the mechanism check is deterministic
|
||||
- Evidence: not kept -- the original build logs were not saved; the test above reproduces the cause
|
||||
|
||||
## Root cause
|
||||
Same class already documented by bug-1598/bug-1632, not a new defect: sched_sim_driver_run() spawns sim_picker_thread_fn on an unjoined pthread that reads its xce_sched_sim_driver_t* for the rest of the process's life; any stack-local `sim` fixture is therefore a latent stack-use-after-return the moment its owning test function returns, and whether ASAN's shadow memory actually catches a given run is purely a function of incidental stack-layout/timing -- bug-1598's own root cause entry documents an unrelated new test shifting stack layout enough to make this fire deterministically for a different file. test_sim_record_output.c #includes repl_break.c white-box (test_sim_record_output.c:78), so this task's repl_break.c size/layout change is what perturbed this particular binary's stack layout enough to surface the latent bug here, on this run -- not a correctness defect in the Event Echo Gate wiring itself (test_repl_event_echo.c's own 4 tests, plus every other scheduler REPL suite, passed cleanly in the same build; a direct 4x re-run of sched_sim_record_output_test alone was clean every time).
|
||||
sched_sim_driver_run() (sim_loop.c) starts the picker and allocator threads (sched_job_picker/sched_allocator, or a test's stand-ins) on &sim->core and returned with them still running; only sched_sim_driver_fini() (sim.c) cancelled and joined them. main.c pairs the two, but 24 scheduler test files (35 call sites) run the driver on a stack-local xce_sched_sim_driver_t and never call fini. The threads, parked in cbuf_get() on the core's cbufs, then held a pointer into a dead frame; when a later test reused that stack memory and something woke them, one read it -- the reported stack-use-after-return in cbuf_get (mylock.c). Timing-dependent, so seen only occasionally under full-suite load. Also: if the allocator thread failed to start, the already-running picker was never stopped at all.
|
||||
|
||||
## Fix
|
||||
NOT FIXED -- out of scope for scheduler-sim-repl-verbosity task 3.2 (repl_break.c wiring only, no sim_loop.c/test_sim_record_output.c changes in this task's boundary). Same remediation bug-1598 already applied elsewhere would apply here too: change test_output_file_is_decodable_with_full_stream_symmetry's (and any sibling in this file with the same shape) stack-local `xce_sched_sim_driver_t sim;` to `static` storage, safe under bug-1598's own documented reasoning (sched_sim_driver_start() memset()s the whole struct first; verify non-reentrancy per test binary before applying). Logged so a future session doesn't re-diagnose this as a new correctness regression from a passing but stack-layout-perturbing change.
|
||||
src/scheduler/src/sim_loop.c: the loop moved into static sim_driver_run_body(); sched_sim_driver_run() calls it and then sched_sim_driver_stop_threads() on every return path, so no thread outlives the run; an allocator start failure now cancels and joins the running picker. src/scheduler/src/sim.c: new sched_sim_driver_stop_threads() (cancel + join both, idempotent via allocator_thread_live), used by the run and by sched_sim_driver_fini(); it no longer pushes a jobid 0 to wake the allocator -- cbuf waits in pthread_cond_wait (a cancellation point, mutex released by a cleanup handler), and the fake end-of-session was visible to the allocator (test_sim_discrete_clock caught it as an extra session). sim.h documents the guarantee.
|
||||
|
||||
## Verification
|
||||
_Not recorded (migrated from buglog.json)._
|
||||
- Test: test_run_returns_with_its_threads_stopped in src/scheduler/testsuite/test_sim_verify.c runs a replay through sched_sim_driver_run() on a stack-local driver and asserts no thread is left running: RED before the fix (`bug-1632: sched_sim_driver_run() returned with its picker and allocator threads still running`), GREEN after -- deterministic, unlike the original ASAN report
|
||||
- Suites: scheduler 90/90 test binaries, asan, 0 sanitizer reports (first attempt had test_sim_discrete_clock fail on the jobid-0 wake-up; fixed as above)
|
||||
- Live: n/a, test-harness lifetime bug; 30 isolated runs each of sched_sim_verify_test and sched_sim_record_output_test (ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=1) were clean both before and after the fix, matching the records' note that it never reproduced in isolation
|
||||
|
||||
## References
|
||||
- Related: bug-1598, bug-1632
|
||||
- Fix commit: cfa0a941
|
||||
- Related: bug-1632 (same cause, other test binary), bug-1884 (added sched_sim_driver_fini)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue