Task 2.4: outcome capture moves to sched_trace_outcome.c and the
position/host/summary/rollback half of the model to sched_trace_scan.c;
shared helpers go behind sched_trace_internal.h (sti_ prefix). Code is
moved unchanged; every target that compiles the trace sources gets all
four, and the core isolation scan covers the new files.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 1.6: the 27 tests that #include a hooked scheduler source now also
include sched_trace.c and sched_trace_capture.c.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sched_publish_retry_loop() calls sched_check_pending_rejoin() after every
successful publish -- inside the driver's decision-drain loop -- so after a
client-library reconnect the scheduler re-joins mid-round. The controller
answers every join (on_scheduler_join(), controller/sched.c) by re-pushing
config, workers and jobs. handler.c applied the config push synchronously
on the handler thread: sched_init_config() -> sched_init_queues() ->
builtin_fcfs on_queue_update(), which nulls the picker cursor when it
reinserts the queue the cursor rests on. next_job() then returned NULL and
the round ended with the rest of the queue never considered.
Evidence (scheduler-replay-20260923-085356.replay, temporary event-stream
diagnostics, reverted): 9 recorded rounds end after 11-12 decisions; every
one is followed by a workersPush + full 50-job jobsPush (the join's resync,
reporting just-dispatched jobs back as PEND), every full-length dispatching
round has none, and the config push never appears because it bypassed the
pending-update queue the recorder writes from -- which is also why replay
could not reproduce the truncation.
Fix (defer to the round edge, same direction as bug-1813): handler.c now
calls new sched_stash_config(), parking the Conf4Sched in new
xce_scheduler_t.pending_config under pending_config_lock (an older
unapplied push is freed). main.c's loop calls new
sched_apply_pending_config() before each round, running the unchanged
sched_init_config() strictly between sessions. sched_fini() frees a
never-applied push.
Tests:
- test_handlers.c: test_config_push_is_stashed_not_applied_on_handler_thread
-- RED (sched_init_config called on the handler thread), GREEN after.
- new test_config_push_deferred.c (sched_config_push_deferred_test): real
scheduler_req_handler() + real builtin_fcfs mid-session -- RED (next_job()
returned NULL right after the push), GREEN (session walks the remaining
jobs; configver unchanged until applied once between rounds).
- test_admin_request.c / test_job_array_expand.c / test_job_array_wire_e2e.c
#include handler.c directly and get a link stub for sched_stash_config().
Full scheduler suite 87/87 (asan).
Not addressed: config pushes are still not written to a recording (a
replay spanning a config change won't see it); the ~100s reconnect churn
itself (see bug-1812/1813).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
User-reported: replaying a captured session dispatched 44 single-slot jobs
onto a 22-slot host. Root cause traced through two layers:
- bug-1836: sim_reconcile_unaccounted_jobs() (the "Recorded-Decision
Reconciliation" backstop) force-applied any recorded DISPATCH decision
it found, with zero live capacity check. Added
sim_reconciled_alloc_fits_capacity(), reusing the same
sched_check_host_satisfy()/sched_check_shared_satisfy() admission checks
the live picker/allocator path already uses.
- bug-1838: even with that gate, reconciliation still preempted every
freshly-PEND job before the live picker/allocator got a turn each round
-- unnecessary now that scheduler-sim-live-allocation gave sim mode the
same real, capacity-checked picker/allocator threads normal mode uses.
Narrowed reconciliation's trigger back to JOB_IS_RUN-only (undoing a
2026-09-11 widening whose own justification no longer holds), so a
freshly-pushed job now always flows through the same live, verified
decision path as normal mode. Reconciliation now only ever fires for a
job already RUN from before the replay's own observation window --
the one case structurally invisible to the live picker.
Also logged (bug-1837, not fixed here, needs its own design decision):
sim_reconcile_unaccounted_jobs() calls a picker's on_dispatch() hook from
the driver thread instead of the allocator thread plugin.h documents it
belonging to -- currently inert since the only real picker plugin happens
to lock internally, but not contract-safe for a future lock-free one.
Fixing bug-1838 required re-examining every test exercising the old
PEND-inclusive trigger; several needed their fixtures corrected to
register a real worker (harmless before any capacity check existed), and
others needed their actual intent re-derived and their assertions
inverted or restated now that a PEND job is never eligible for
reconciliation. Dated correction notes added in place to the
scheduler-sim-eager-reconciliation spec, matching its own existing
2026-09-11 correction-note precedent.
Verified: full scheduler suite 85/85, independently re-confirmed by
running the four most affected binaries directly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sched_reconcile_job_resource_usage() was only reachable from
sched_apply_job_rusage() (scheduler.c's sched_update(), which has no live
caller), so real-usage reconciliation never actually ran outside tests.
Wire it into core_events.c's XCE_CORE_EVENT_JOB_RUSAGE handler instead,
the actual live path shared by both normal-mode (driver.c) and sim/replay
mode (sim_loop.c). Adds link-satisfying stubs to the testsuite files that
#include core_events.c directly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
test_recordstart_symlink_escape_replies_error expected "invalid record
file name" (the -EINVAL string), but sched_validate_record_filename()'s
canonical-path containment check has always, correctly, returned
-EACCES ("permission denied accessing record file") for a symlink
escaping record_dir -- confirmed against test_recording_path_validation.c's
own direct unit test for the identical scenario, which already asserts
-EACCES and has always passed.
Not environment-dependent as originally suspected (this project's
buglog carried it as a uid-1000-vs-root permission-check-ordering
theory across 7+ sightings since Aug 15) -- it fails identically
regardless of uid, since escape-containment and filesystem-permission
errors were never actually conflated in production code, only in this
test's assertion. No production code changed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
sim_loop.c never called plugin_begin_session()/plugin_end_session() at
its round boundary, unlike driver.c's normal-mode round -- a driver-
equivalence gap found by an /xce-audit-invariants scheduler pass.
Currently inert (sim mode's only reachable picker, builtin_fcfs, leaves
both hooks NULL) but structurally identical to bug-1613's class of bug.
Relocated plugin_begin_session()/plugin_end_session() from an implicit
xce_scheduler_t* to xce_sched_core_t* directly (mirroring
sched_account_jobs()'s prior relocation) -- their bodies never touched
anything beyond core->plugins, so driver.c's comment claiming they
"needed the full xce_scheduler_t*" was itself stale. sim_loop.c now
calls both, same relative position as driver.c. A full-tree grep (done
after the first build attempt failed on an untouched file) turned up a
third live call site in scheduler.c's legacy sched_begin_session()/
sched_end_session(), plus four testsuite files with their own stub
definitions of the same functions -- all updated to match the new
signature.
Extended test_sim_session_ordering.c's existing bug-1613 regression
tests with new plugin_begin_session()/plugin_end_session() spies
(call-count + session_active-window proof, same technique as the
existing sched_account_jobs spy). RED/GREEN independently verified:
temporarily disabled the two new call sites, confirmed both new
assertions fail (Expected 2 Was 0 / Expected 1 Was 0), restored,
reconfirmed green. Full scheduler suite: 51/52 green, the one failure
is the pre-existing unrelated bug-1560 sandbox flake.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Relocates sched_account_jobs()/sched_charge_account()/sched_uncharge_account()/
sched_update_account()/sched_update_res_accounts()/sched_commit_job_alloc()
from xce_scheduler_t* to xce_sched_core_t*, mirroring the sched_resolve_job_picker()
precedent, so the simulation driver can call the same accounting service the
normal-mode driver already uses. Pure signature relocation -- every body
reference was already sched->core.*, no logic change. Updates the 3 real call
sites (driver.c's normal-mode round, scheduler.c's sched_begin_session and
sched_commit_decision) and 7 existing tests' link-satisfying stub signatures.
Adds test_resource_accounting.c proving Requirement 1.4: the relocated
accounting produces identical resInUse/sharedInUse output whether called
against a core embedded in xce_scheduler_t or xce_sched_sim_driver_t.
43/44 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated recording.c sandbox-permission ordering issue).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends test_admin_request.c with 3 tests closing the gap between what
task 9.4's own suite already proved through scheduler_req_handler() and
what this task's text specifically asks for: a ".."-traversal rejection
and a real symlink(2)-escape rejection, both submitted as genuine
SchedAdminRequests (not internal calls), and a disk-full simulation
(fflush + setvbuf(_IONBF) + close(fd), matching task 9.3's technique)
proving sched_record_abort_on_error()'s cleanup is visible through a
subsequent real recordStatus request and that recording recovers cleanly
afterward.
Hard-link-aliasing coverage deliberately not re-added through the admin
path -- task 9.1/9.2 already exhaustively prove that rejection branch,
and the routing plumbing this task exists to prove is already retired by
the dotdot/symlink cases.
17/17 in sched_admin_request_test, 26/26 scheduler suite, clang-format
clean. Closes out Group 9 (Core: Recording Control).
Implements the scheduler's first request-reply opcode (OPCode_schedAdmin).
sched_handle_admin_request() (handler.c) decodes SchedAdminRequest and
dispatches by action: recordStatus answers synchronously (pure read);
recordStart/recordStop are deferred onto the existing pend_updates queue
(XCE_SCHED_OP_ADMIN_RECORD) since sched_record_start()'s snapshot call
requires the session_active==0 precondition only guaranteed on the driver
thread, not the async request-handling thread. driver.c's
xce_sched_driver_translate_pending_events() performs the deferred action
and sends the reply via a new xcecli_reply_schedadmin() (client.c/client.h),
correlated by seq copied before the original request is freed.
Adversarial review found a genuine cross-thread race: sched_record_status()
read driver-owned recording state (including a pointer the driver thread
frees) from the vtask thread with no lock -- the codebase's first such
race, missed by ASan (doesn't detect races). Fixed with a dedicated
record_lock scoped strictly to in-memory field access, never spanning file
I/O; sched_record_status() now strdup()s under the lock instead of
returning a borrowed live pointer. Verified with a real two-thread
lock-contention test (independently confirmed non-vacuous by removing the
lock and observing the test fail) plus a deterministic use-after-free
reproduction.
26/26 scheduler suite, clang-format clean.
