diff --git a/.wolf/buglog.json b/.wolf/buglog.json index c710a00c..237d5975 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -9313,18 +9313,18 @@ }, { "id": "bug-1632", - "status": "open", - "status_updated": "2026-09-24", - "status_note": "test_sim_verify.c stack-local sim read by picker thread (ASAN stack-use-after-return)", + "status": "fixed", + "status_updated": "2026-09-30", + "status_note": "", "component": "scheduler", - "severity": "security", + "severity": "crash", "timestamp": "2026-08-23T21:03:00.000Z", "last_seen": "2026-09-19T09:00:00Z", "occurrences": 1, "error_message": "sched_sim_verify_test: AddressSanitizer stack-use-after-return in cbuf_get (lib/mylock.c:151)…\n\nsched_sim_verify_test: AddressSanitizer stack-use-after-return in cbuf_get (lib/mylock.c:151), thread T2 (sim_picker_thread_fn, sim_loop.c:456), reading the stack of run_sim_and_capture_stdout's `sim` (xce_sched_sim_driver_t) after that frame returned/was reused. Fired once during a full `/xce-build scheduler` run (scheduler-dispatch-commit task 5.4 regression pass): Ok:50 Fail:2 instead of the documented Ok:51 Fail:1 baseline, extra failure = 'scheduler sim verify test'. Did NOT reproduce on: 3 standalone direct re-runs of sched_sim_verify_test (16 Tests 0 Failures each time) or a second full `meson test --suite scheduler` re-run against the same build (back to Ok:51 Fail:1, only the known test_admin_request.c symlink-escape failure) -- confirmed non-deterministic/order-dependent, not caused by scheduler-dispatch-commit.", - "file": "src/scheduler/testsuite/test_sim_verify.c", - "root_cause": "Same root cause as bug-1598 (already diagnosed, NOT re-derived here): sched_sim_driver_run() (src/scheduler/src/sim_loop.c) spawns sim_picker_thread_fn on a pthread that is, by design, never joined/cancelled -- it loops forever on cbuf_get(&sim->core.wbuf,...) reading `sim` for the life of the process. bug-1598's fix converted 3 stack-local `xce_sched_sim_driver_t sim;` fixtures in test_sim_event_loop.c to `static` storage, but test_sim_verify.c's OWN, separately-defined run_sim_and_capture_stdout() helper (test_sim_verify.c:940-943) still declares `sim` as stack-local -- an unfixed sibling instance of the identical latent bug, never touched by bug-1598's fix. Confirmed via git diff/log that scheduler-dispatch-commit's own commits (which only touch scheduler.c/handler.c per that spec's Requirement 5.2 sim-mode-exclusion) never modified sim_loop.c or test_sim_verify.c -- this is pre-existing latent test infra, exposed non-deterministically by whatever stack-layout/MALLOC_PERTURB_ randomization a given meson test invocation happens to produce, exactly like bug-1598's own trigger mechanism.", - "fix": "NOT FIXED -- out of scope for scheduler-dispatch-commit task 5.4 (a validation/regression task, not a scheduler-sim-infra task). Same remediation as bug-1598 would apply if picked up: change run_sim_and_capture_stdout()'s `xce_sched_sim_driver_t sim;` from stack-local to `static` storage (safe under the same reasoning bug-1598 documents: sched_sim_driver_start() memset()s the whole struct first, and this helper is not re-entered while an earlier zombie thread could still be alive within one test binary process -- verify that non-reentrancy holds for test_sim_verify.c's own call sites before applying). Logged so a future session doesn't re-diagnose from scratch.", + "file": "src/scheduler/src/sim_loop.c (sched_sim_driver_run), src/scheduler/src/sim.c (sched_sim_driver_stop_threads, sched_sim_driver_fini), src/scheduler/testsuite/test_sim_verify.c", + "root_cause": "sched_sim_driver_run() (sim_loop.c) starts the picker and allocator threads (sched_job_picker/sched_allocator, or a test's stand-ins) on &sim->core and returned with them still running; only sched_sim_driver_fini() (sim.c) cancelled and joined them. main.c pairs the two, but 24 scheduler test files (35 call sites) run the driver on a stack-local xce_sched_sim_driver_t and never call fini. The threads, parked in cbuf_get() on the core's cbufs, then held a pointer into a dead frame; when a later test reused that stack memory and something woke them, one read it -- the reported stack-use-after-return in cbuf_get (mylock.c). Timing-dependent, so seen only occasionally under full-suite load. Also: if the allocator thread failed to start, the already-running picker was never stopped at all.", + "fix": "src/scheduler/src/sim_loop.c: the loop moved into static sim_driver_run_body(); sched_sim_driver_run() calls it and then sched_sim_driver_stop_threads() on every return path, so no thread outlives the run; an allocator start failure now cancels and joins the running picker. src/scheduler/src/sim.c: new sched_sim_driver_stop_threads() (cancel + join both, idempotent via allocator_thread_live), used by the run and by sched_sim_driver_fini(); it no longer pushes a jobid 0 to wake the allocator -- cbuf waits in pthread_cond_wait (a cancellation point, mutex released by a cleanup handler), and the fake end-of-session was visible to the allocator (test_sim_discrete_clock caught it as an extra session). sim.h documents the guarantee.\n\nVerification:\n- Test: test_run_returns_with_its_threads_stopped in src/scheduler/testsuite/test_sim_verify.c runs a replay through sched_sim_driver_run() on a stack-local driver and asserts no thread is left running: RED before the fix (`bug-1632: sched_sim_driver_run() returned with its picker and allocator threads still running`), GREEN after -- deterministic, unlike the original ASAN report\n- Suites: scheduler 90/90 test binaries, asan, 0 sanitizer reports (first attempt had test_sim_discrete_clock fail on the jobid-0 wake-up; fixed as above)\n- Live: n/a, test-harness lifetime bug; 30 isolated runs each of sched_sim_verify_test and sched_sim_record_output_test (ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=1) were clean both before and after the fix, matching the records' note that it never reproduced in isolation", "tags": [ "scheduler", "asan", @@ -9340,7 +9340,9 @@ "related_bugs": [ "bug-1598" ], - "fix_commits": [], + "fix_commits": [ + "cfa0a941" + ], "record": ".wolf/bugs/bug-1632.md" }, { @@ -12327,18 +12329,18 @@ }, { "id": "bug-1763", - "status": "open", - "status_updated": "2026-09-24", - "status_note": "test_sim_record_output.c stack-local sim read by unjoined picker thread (ASAN)", + "status": "fixed", + "status_updated": "2026-09-30", + "status_note": "", "component": "scheduler", - "severity": "unknown", + "severity": "crash", "timestamp": "2026-09-02T13:00:00Z", "last_seen": "2026-09-19T09:00:00Z", "occurrences": 1, "error_message": "sched_sim_record_output_test (scheduler-sim-repl-verbosity task 3.2 GREEN-phase build)…\n\nsched_sim_record_output_test (scheduler-sim-repl-verbosity task 3.2 GREEN-phase build): AddressSanitizer stack-use-after-return in cbuf_get (lib/mylock.c:151), thread T1 (sim_picker_thread_fn, sim_loop.c:456), reading the stack of test_output_file_is_decodable_with_full_stream_symmetry's stack-local `sim` (xce_sched_sim_driver_t, test_sim_record_output.c:341) after that frame returned. Fired once during a `/xce-build scheduler --incremental` run after this task's repl_break.c edits; 4/4 direct re-runs of the isolated binary afterward were clean (OK).", - "file": "src/scheduler/testsuite/test_sim_record_output.c", - "root_cause": "Same class already documented by bug-1598/bug-1632, not a new defect: sched_sim_driver_run() spawns sim_picker_thread_fn on an unjoined pthread that reads its xce_sched_sim_driver_t* for the rest of the process's life; any stack-local `sim` fixture is therefore a latent stack-use-after-return the moment its owning test function returns, and whether ASAN's shadow memory actually catches a given run is purely a function of incidental stack-layout/timing -- bug-1598's own root cause entry documents an unrelated new test shifting stack layout enough to make this fire deterministically for a different file. test_sim_record_output.c #includes repl_break.c white-box (test_sim_record_output.c:78), so this task's repl_break.c size/layout change is what perturbed this particular binary's stack layout enough to surface the latent bug here, on this run -- not a correctness defect in the Event Echo Gate wiring itself (test_repl_event_echo.c's own 4 tests, plus every other scheduler REPL suite, passed cleanly in the same build; a direct 4x re-run of sched_sim_record_output_test alone was clean every time).", - "fix": "NOT FIXED -- out of scope for scheduler-sim-repl-verbosity task 3.2 (repl_break.c wiring only, no sim_loop.c/test_sim_record_output.c changes in this task's boundary). Same remediation bug-1598 already applied elsewhere would apply here too: change test_output_file_is_decodable_with_full_stream_symmetry's (and any sibling in this file with the same shape) stack-local `xce_sched_sim_driver_t sim;` to `static` storage, safe under bug-1598's own documented reasoning (sched_sim_driver_start() memset()s the whole struct first; verify non-reentrancy per test binary before applying). Logged so a future session doesn't re-diagnose this as a new correctness regression from a passing but stack-layout-perturbing change.", + "file": "src/scheduler/src/sim_loop.c (sched_sim_driver_run), src/scheduler/src/sim.c (sched_sim_driver_stop_threads, sched_sim_driver_fini), src/scheduler/testsuite/test_sim_record_output.c", + "root_cause": "sched_sim_driver_run() (sim_loop.c) starts the picker and allocator threads (sched_job_picker/sched_allocator, or a test's stand-ins) on &sim->core and returned with them still running; only sched_sim_driver_fini() (sim.c) cancelled and joined them. main.c pairs the two, but 24 scheduler test files (35 call sites) run the driver on a stack-local xce_sched_sim_driver_t and never call fini. The threads, parked in cbuf_get() on the core's cbufs, then held a pointer into a dead frame; when a later test reused that stack memory and something woke them, one read it -- the reported stack-use-after-return in cbuf_get (mylock.c). Timing-dependent, so seen only occasionally under full-suite load. Also: if the allocator thread failed to start, the already-running picker was never stopped at all.", + "fix": "src/scheduler/src/sim_loop.c: the loop moved into static sim_driver_run_body(); sched_sim_driver_run() calls it and then sched_sim_driver_stop_threads() on every return path, so no thread outlives the run; an allocator start failure now cancels and joins the running picker. src/scheduler/src/sim.c: new sched_sim_driver_stop_threads() (cancel + join both, idempotent via allocator_thread_live), used by the run and by sched_sim_driver_fini(); it no longer pushes a jobid 0 to wake the allocator -- cbuf waits in pthread_cond_wait (a cancellation point, mutex released by a cleanup handler), and the fake end-of-session was visible to the allocator (test_sim_discrete_clock caught it as an extra session). sim.h documents the guarantee.\n\nVerification:\n- Test: test_run_returns_with_its_threads_stopped in src/scheduler/testsuite/test_sim_verify.c runs a replay through sched_sim_driver_run() on a stack-local driver and asserts no thread is left running: RED before the fix (`bug-1632: sched_sim_driver_run() returned with its picker and allocator threads still running`), GREEN after -- deterministic, unlike the original ASAN report\n- Suites: scheduler 90/90 test binaries, asan, 0 sanitizer reports (first attempt had test_sim_discrete_clock fail on the jobid-0 wake-up; fixed as above)\n- Live: n/a, test-harness lifetime bug; 30 isolated runs each of sched_sim_verify_test and sched_sim_record_output_test (ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=1) were clean both before and after the fix, matching the records' note that it never reproduced in isolation", "tags": [ "scheduler", "sim", @@ -12353,7 +12355,9 @@ "bug-1598", "bug-1632" ], - "fix_commits": [], + "fix_commits": [ + "cfa0a941" + ], "record": ".wolf/bugs/bug-1763.md" }, { diff --git a/.wolf/bugs/bug-1632.md b/.wolf/bugs/bug-1632.md index 3400f4a3..c018b9db 100644 --- a/.wolf/bugs/bug-1632.md +++ b/.wolf/bugs/bug-1632.md @@ -1,16 +1,17 @@ --- id: "bug-1632" -status: "open" -status_updated: "2026-09-24" -status_note: "test_sim_verify.c stack-local sim read by picker thread (ASAN stack-use-after-return)" +status: "fixed" +status_updated: "2026-09-30" +status_note: "" component: "scheduler" -severity: "security" +severity: "crash" timestamp: "2026-08-23T21:03:00.000Z" last_seen: "2026-09-19T09:00:00Z" occurrences: 1 -files: ["src/scheduler/testsuite/test_sim_verify.c"] +files: ["src/scheduler/src/sim_loop.c (sched_sim_driver_run)", "src/scheduler/src/sim.c (sched_sim_driver_stop_threads, sched_sim_driver_fini)", "src/scheduler/testsuite/test_sim_verify.c"] tags: ["scheduler", "asan", "stack-use-after-return", "thread-lifecycle", "test-only", "sim-driver", "flaky", "non-deterministic", "pre-existing", "sibling-of-bug-1598"] related: ["bug-1598"] +fix_commits: ["cfa0a941"] legacy: true --- @@ -20,19 +21,30 @@ legacy: true sched_sim_verify_test: AddressSanitizer stack-use-after-return in cbuf_get (lib/mylock.c:151), thread T2 (sim_picker_thread_fn, sim_loop.c:456), reading the stack of run_sim_and_capture_stdout's `sim` (xce_sched_sim_driver_t) after that frame returned/was reused. Fired once during a full `/xce-build scheduler` run (scheduler-dispatch-commit task 5.4 regression pass): Ok:50 Fail:2 instead of the documented Ok:51 Fail:1 baseline, extra failure = 'scheduler sim verify test'. Did NOT reproduce on: 3 standalone direct re-runs of sched_sim_verify_test (16 Tests 0 Failures each time) or a second full `meson test --suite scheduler` re-run against the same build (back to Ok:51 Fail:1, only the known test_admin_request.c symlink-escape failure) -- confirmed non-deterministic/order-dependent, not caused by scheduler-dispatch-commit. ## Environment -_Not recorded (migrated from buglog.json)._ +- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 +- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan; meson runs these tests with ASAN_OPTIONS=detect_leaks=0 +- Code: original sightings on scheduler-framework in Aug/Sep 2026 (commit not recorded); root-caused and fixed on 2026-09-29 at dc94da20 +- Deployment: n/a -- unit test binary, no cluster ## Reproduction -_Not recorded (migrated from buglog.json)._ +1. `bash src/tools/xce-build.sh scheduler` (full suite under load); the ASAN abort shows in sched_sim_verify_test's log. +2. Deterministic check of the mechanism: test_run_returns_with_its_threads_stopped (src/scheduler/testsuite/test_sim_verify.c) -- sched_sim_driver_run() returns with allocator_thread_live still true. +- Frequency: original ASAN report rare (1 in several full-suite runs; 0/30 isolated runs, 2026-09-29); the mechanism check is deterministic +- Evidence: not kept -- the original build logs were not saved; the test above reproduces the cause ## Root cause -Same root cause as bug-1598 (already diagnosed, NOT re-derived here): sched_sim_driver_run() (src/scheduler/src/sim_loop.c) spawns sim_picker_thread_fn on a pthread that is, by design, never joined/cancelled -- it loops forever on cbuf_get(&sim->core.wbuf,...) reading `sim` for the life of the process. bug-1598's fix converted 3 stack-local `xce_sched_sim_driver_t sim;` fixtures in test_sim_event_loop.c to `static` storage, but test_sim_verify.c's OWN, separately-defined run_sim_and_capture_stdout() helper (test_sim_verify.c:940-943) still declares `sim` as stack-local -- an unfixed sibling instance of the identical latent bug, never touched by bug-1598's fix. Confirmed via git diff/log that scheduler-dispatch-commit's own commits (which only touch scheduler.c/handler.c per that spec's Requirement 5.2 sim-mode-exclusion) never modified sim_loop.c or test_sim_verify.c -- this is pre-existing latent test infra, exposed non-deterministically by whatever stack-layout/MALLOC_PERTURB_ randomization a given meson test invocation happens to produce, exactly like bug-1598's own trigger mechanism. +sched_sim_driver_run() (sim_loop.c) starts the picker and allocator threads (sched_job_picker/sched_allocator, or a test's stand-ins) on &sim->core and returned with them still running; only sched_sim_driver_fini() (sim.c) cancelled and joined them. main.c pairs the two, but 24 scheduler test files (35 call sites) run the driver on a stack-local xce_sched_sim_driver_t and never call fini. The threads, parked in cbuf_get() on the core's cbufs, then held a pointer into a dead frame; when a later test reused that stack memory and something woke them, one read it -- the reported stack-use-after-return in cbuf_get (mylock.c). Timing-dependent, so seen only occasionally under full-suite load. Also: if the allocator thread failed to start, the already-running picker was never stopped at all. ## Fix -NOT FIXED -- out of scope for scheduler-dispatch-commit task 5.4 (a validation/regression task, not a scheduler-sim-infra task). Same remediation as bug-1598 would apply if picked up: change run_sim_and_capture_stdout()'s `xce_sched_sim_driver_t sim;` from stack-local to `static` storage (safe under the same reasoning bug-1598 documents: sched_sim_driver_start() memset()s the whole struct first, and this helper is not re-entered while an earlier zombie thread could still be alive within one test binary process -- verify that non-reentrancy holds for test_sim_verify.c's own call sites before applying). Logged so a future session doesn't re-diagnose from scratch. +src/scheduler/src/sim_loop.c: the loop moved into static sim_driver_run_body(); sched_sim_driver_run() calls it and then sched_sim_driver_stop_threads() on every return path, so no thread outlives the run; an allocator start failure now cancels and joins the running picker. src/scheduler/src/sim.c: new sched_sim_driver_stop_threads() (cancel + join both, idempotent via allocator_thread_live), used by the run and by sched_sim_driver_fini(); it no longer pushes a jobid 0 to wake the allocator -- cbuf waits in pthread_cond_wait (a cancellation point, mutex released by a cleanup handler), and the fake end-of-session was visible to the allocator (test_sim_discrete_clock caught it as an extra session). sim.h documents the guarantee. ## Verification -_Not recorded (migrated from buglog.json)._ +- Test: test_run_returns_with_its_threads_stopped in src/scheduler/testsuite/test_sim_verify.c runs a replay through sched_sim_driver_run() on a stack-local driver and asserts no thread is left running: RED before the fix (`bug-1632: sched_sim_driver_run() returned with its picker and allocator threads still running`), GREEN after -- deterministic, unlike the original ASAN report +- Suites: scheduler 90/90 test binaries, asan, 0 sanitizer reports (first attempt had test_sim_discrete_clock fail on the jobid-0 wake-up; fixed as above) +- Live: n/a, test-harness lifetime bug; 30 isolated runs each of sched_sim_verify_test and sched_sim_record_output_test (ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=1) were clean both before and after the fix, matching the records' note that it never reproduced in isolation ## References -- Related: bug-1598 +- Fix commit: cfa0a941 +- Related: bug-1763 (same cause, other test binary), bug-1884 (added sched_sim_driver_fini) diff --git a/.wolf/bugs/bug-1763.md b/.wolf/bugs/bug-1763.md index 4b119bed..1fd2a8e4 100644 --- a/.wolf/bugs/bug-1763.md +++ b/.wolf/bugs/bug-1763.md @@ -1,16 +1,17 @@ --- id: "bug-1763" -status: "open" -status_updated: "2026-09-24" -status_note: "test_sim_record_output.c stack-local sim read by unjoined picker thread (ASAN)" +status: "fixed" +status_updated: "2026-09-30" +status_note: "" component: "scheduler" -severity: "unknown" +severity: "crash" timestamp: "2026-09-02T13:00:00Z" last_seen: "2026-09-19T09:00:00Z" occurrences: 1 -files: ["src/scheduler/testsuite/test_sim_record_output.c"] +files: ["src/scheduler/src/sim_loop.c (sched_sim_driver_run)", "src/scheduler/src/sim.c (sched_sim_driver_stop_threads, sched_sim_driver_fini)", "src/scheduler/testsuite/test_sim_record_output.c"] tags: ["scheduler", "sim", "repl", "asan", "stack-use-after-return", "flaky", "thread-lifecycle", "scheduler-sim-repl-verbosity"] related: ["bug-1598", "bug-1632"] +fix_commits: ["cfa0a941"] legacy: true --- @@ -20,19 +21,30 @@ legacy: true sched_sim_record_output_test (scheduler-sim-repl-verbosity task 3.2 GREEN-phase build): AddressSanitizer stack-use-after-return in cbuf_get (lib/mylock.c:151), thread T1 (sim_picker_thread_fn, sim_loop.c:456), reading the stack of test_output_file_is_decodable_with_full_stream_symmetry's stack-local `sim` (xce_sched_sim_driver_t, test_sim_record_output.c:341) after that frame returned. Fired once during a `/xce-build scheduler --incremental` run after this task's repl_break.c edits; 4/4 direct re-runs of the isolated binary afterward were clean (OK). ## Environment -_Not recorded (migrated from buglog.json)._ +- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 +- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan; meson runs these tests with ASAN_OPTIONS=detect_leaks=0 +- Code: original sightings on scheduler-framework in Aug/Sep 2026 (commit not recorded); root-caused and fixed on 2026-09-29 at dc94da20 +- Deployment: n/a -- unit test binary, no cluster ## Reproduction -_Not recorded (migrated from buglog.json)._ +1. `bash src/tools/xce-build.sh scheduler` (full suite under load); the ASAN abort shows in sched_sim_record_output_test's log. +2. Deterministic check of the mechanism: test_run_returns_with_its_threads_stopped (src/scheduler/testsuite/test_sim_verify.c) -- sched_sim_driver_run() returns with allocator_thread_live still true. +- Frequency: original ASAN report rare (1 in several full-suite runs; 0/30 isolated runs, 2026-09-29); the mechanism check is deterministic +- Evidence: not kept -- the original build logs were not saved; the test above reproduces the cause ## Root cause -Same class already documented by bug-1598/bug-1632, not a new defect: sched_sim_driver_run() spawns sim_picker_thread_fn on an unjoined pthread that reads its xce_sched_sim_driver_t* for the rest of the process's life; any stack-local `sim` fixture is therefore a latent stack-use-after-return the moment its owning test function returns, and whether ASAN's shadow memory actually catches a given run is purely a function of incidental stack-layout/timing -- bug-1598's own root cause entry documents an unrelated new test shifting stack layout enough to make this fire deterministically for a different file. test_sim_record_output.c #includes repl_break.c white-box (test_sim_record_output.c:78), so this task's repl_break.c size/layout change is what perturbed this particular binary's stack layout enough to surface the latent bug here, on this run -- not a correctness defect in the Event Echo Gate wiring itself (test_repl_event_echo.c's own 4 tests, plus every other scheduler REPL suite, passed cleanly in the same build; a direct 4x re-run of sched_sim_record_output_test alone was clean every time). +sched_sim_driver_run() (sim_loop.c) starts the picker and allocator threads (sched_job_picker/sched_allocator, or a test's stand-ins) on &sim->core and returned with them still running; only sched_sim_driver_fini() (sim.c) cancelled and joined them. main.c pairs the two, but 24 scheduler test files (35 call sites) run the driver on a stack-local xce_sched_sim_driver_t and never call fini. The threads, parked in cbuf_get() on the core's cbufs, then held a pointer into a dead frame; when a later test reused that stack memory and something woke them, one read it -- the reported stack-use-after-return in cbuf_get (mylock.c). Timing-dependent, so seen only occasionally under full-suite load. Also: if the allocator thread failed to start, the already-running picker was never stopped at all. ## Fix -NOT FIXED -- out of scope for scheduler-sim-repl-verbosity task 3.2 (repl_break.c wiring only, no sim_loop.c/test_sim_record_output.c changes in this task's boundary). Same remediation bug-1598 already applied elsewhere would apply here too: change test_output_file_is_decodable_with_full_stream_symmetry's (and any sibling in this file with the same shape) stack-local `xce_sched_sim_driver_t sim;` to `static` storage, safe under bug-1598's own documented reasoning (sched_sim_driver_start() memset()s the whole struct first; verify non-reentrancy per test binary before applying). Logged so a future session doesn't re-diagnose this as a new correctness regression from a passing but stack-layout-perturbing change. +src/scheduler/src/sim_loop.c: the loop moved into static sim_driver_run_body(); sched_sim_driver_run() calls it and then sched_sim_driver_stop_threads() on every return path, so no thread outlives the run; an allocator start failure now cancels and joins the running picker. src/scheduler/src/sim.c: new sched_sim_driver_stop_threads() (cancel + join both, idempotent via allocator_thread_live), used by the run and by sched_sim_driver_fini(); it no longer pushes a jobid 0 to wake the allocator -- cbuf waits in pthread_cond_wait (a cancellation point, mutex released by a cleanup handler), and the fake end-of-session was visible to the allocator (test_sim_discrete_clock caught it as an extra session). sim.h documents the guarantee. ## Verification -_Not recorded (migrated from buglog.json)._ +- Test: test_run_returns_with_its_threads_stopped in src/scheduler/testsuite/test_sim_verify.c runs a replay through sched_sim_driver_run() on a stack-local driver and asserts no thread is left running: RED before the fix (`bug-1632: sched_sim_driver_run() returned with its picker and allocator threads still running`), GREEN after -- deterministic, unlike the original ASAN report +- Suites: scheduler 90/90 test binaries, asan, 0 sanitizer reports (first attempt had test_sim_discrete_clock fail on the jobid-0 wake-up; fixed as above) +- Live: n/a, test-harness lifetime bug; 30 isolated runs each of sched_sim_verify_test and sched_sim_record_output_test (ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=1) were clean both before and after the fix, matching the records' note that it never reproduced in isolation ## References -- Related: bug-1598, bug-1632 +- Fix commit: cfa0a941 +- Related: bug-1632 (same cause, other test binary), bug-1884 (added sched_sim_driver_fini)