myxce/.kiro/specs/scheduler-core/tasks.md

Implementation Plan

• 1. Foundation: Picker Interface & Plugin Registration

• 1.1 Define the job picker plugin contract

  • Add the lifecycle hooks (init, fini), state-change hooks (on_queue_update, on_job_push, on_job_state_change, on_job_update), session hooks (begin_session, next_job, end_session), post-allocation hooks (on_dispatch, on_pend), and optional hooks (on_config, on_job_rusage MAY be NULL) to the picker plugin header
  • Document the contract: every hook signature, which thread each hook category runs in (driver/picker/allocator), cancel-safety requirement for next_job(), and session-reset requirement for begin_session()
  • Observable: the header compiles standalone and defines a complete function-pointer struct matching every hook category
  • Requirements: 1.2, 1.5, 2.6

• 1.2 Register the picker plugin category and deprecate the old election hook

  • Add a picker plugin registration category to the plugin registration table, parallel to existing plugin categories
  • Add a compatibility wrapper that adapts an old-style job_elect_fn-only plugin to the new interface, logging a deprecation warning at registration time
  • Observable: registering a plugin that implements only job_elect_fn succeeds via the wrapper and logs a deprecation warning; registering one that implements the new interface directly succeeds without a warning
  • Requirements: 1.6, 2.5
  • Depends: 1.1

• 1.3 (P) Add picker configuration fields

  • Add a global picker plugin name field to the scheduler, defaulting to builtin_fcfs when unset
  • Add a per-queue picker plugin override field, defaulting to NULL (inherit global)
  • Add a project-name field to the queue struct, and extend the project struct with a priority field (scaffolding only — no quota fields)
  • Observable: a queue config with no override inherits the global picker name; a queue config with an override reports that override, not the global default
  • Requirements: 2.1, 2.2, 2.3, 2.4
  • Boundary: SchedulerCore config fields

• 2. Core: Builtin FCFS Picker

• 2.1 Implement builtin_fcfs lifecycle and private state

  • Implement init/fini: allocate/free the picker's private ordered-queue state (queue ordering by priority desc, and one job-ordering structure per queue)
  • Never query the scheduler's own queue/job tables from inside the picker after init — all candidate state lives in the picker's private structures
  • Observable: after init(), the picker holds an empty private candidate set; after fini(), all picker-owned memory is released (verified under a leak-checked test run)
  • Requirements: 3.1, 3.3
  • Depends: 1.1, 1.2
  • Verified: 2026-08-03 (commit f7921b96) — backfilled 2026-08-07 by /kiro-validate-impl (this task and 2.2-2.6 were implemented and reviewed at the time but never got a _Verified: line). New src/scheduler/src/plugins/builtin_fcfs.c (not a fcfs.c rewrite — see design.md's BuiltinFcfsPicker notes and Implementation Notes below for why): fcfs_plugin_data_t holds ordq_t queues plus per-queue ordq_t pendJobs, allocated in init()/freed in fini(); never queries sched->queues/sched->jobs.

• 2.2 Implement builtin_fcfs state-change hooks

  • on_queue_update: upsert a queue into the picker's ordered queue list, keyed by descending priority
  • on_job_push: add a newly-pending job into its queue's candidate set, ordered by priority desc then submission/queue time asc
  • on_job_state_change: remove a job from candidates when it leaves pending state, re-add it when it returns to pending
  • on_job_update: reposition a job in its candidate set when its priority or queue assignment changes
  • Observable: feeding a sequence of these hooks directly (no live core) produces a candidate set whose order matches the documented ordering rule, verifiable by a direct hook-driven test
  • Requirements: 3.2
  • Depends: 2.1
  • Verified: 2026-08-03 (commit b7214ebe) — backfilled 2026-08-07. on_queue_update/on_job_push/on_job_state_change/on_job_update implemented on fcfs_plugin_data_t's ordq_t structures, ordering (queue priority desc, job priority desc, queue_time asc); test_picker_fcfs.c feeds the hooks directly (no live core) and asserts the resulting order.

• 2.3 Implement builtin_fcfs session and post-allocation hooks

  • begin_session: reset the iteration cursor to the head of the ordered queue list
  • next_job: walk queues then jobs in ordering, skipping not-ready queues/jobs, returning the first ready job and advancing the cursor; returns NULL when exhausted
  • end_session: no-op (iteration state already reset in begin_session)
  • on_dispatch: remove the job from its queue's candidate set permanently
  • on_pend: no-op — job remains in its candidate set for the next round in the same position
  • Observable: a full begin_session → repeated next_job → end_session cycle against a fixed candidate set returns jobs in (queue priority desc, job priority desc, queue_time asc) order and returns NULL exactly once all candidates are exhausted
  • Requirements: 3.2, 3.4, 3.6
  • Depends: 2.2
  • Verified: 2026-08-03 (commit fcb32a8b) — backfilled 2026-08-07. begin_session/next_job/end_session/on_dispatch/on_pend implemented per plan; this task's own review found the real cross-thread race later closed by 2.4 (picker thread's next_job() vs allocator thread's on_dispatch/on_pend racing unsynchronized fcfs_plugin_data_t state) — see ## Implementation Notes below, "Task 2.3's review found a real concurrency gap for task 2.4 to close."

• 2.4 Wire builtin_fcfs as the registered default and remove the hardcoded picker string

  • Auto-register builtin_fcfs at scheduler startup under the name "builtin_fcfs" and set it as the default picker plugin
  • Replace the hardcoded "xce_sched_fcfs" literal with a lookup of the configured picker plugin name through the existing plugin registration/lookup mechanism; fail scheduler startup with a clear error if the configured name isn't registered
  • Confirm the existing three-thread/cbuf pipeline (picker thread, allocator thread, driver-signaled wbuf/jbuf/decisions cbufs) is unchanged by this wiring — no new threads, no cbuf capacity changes
  • Add cancel points in next_job()'s iteration loop and confirm the picker thread's existing block-on-signal point remains cancel-safe
  • Observable: starting the scheduler with no picker config uses builtin_fcfs automatically; starting it with an unregistered picker name fails startup with a clear error instead of silently falling back
  • Requirements: 1.3, 1.4, 1.6, 2.5, 3.1, 3.5
  • Depends: 2.3, 1.3
  • Verified: 2026-08-03 (commit 7ee9a16f) — backfilled 2026-08-07. builtin_fcfs registered and defaulted (schedobj.c:364); registered "alongside the untouched legacy xce_sched_fcfs" (scheduler.c:466's own comment) rather than replacing plugins/fcfs.c in place — fcfs.c stays as task 1.2's job_elect_fn compat-wrapper exercise target. sched_job_picker() (scheduler.c) branches job_elect_fn != NULL (legacy, self-contained loop) vs NULL (native picker, generic per-session loop). Closed this task's own review-found race (see 2.3's note) with fcfs_plugin_data_t.lock (mylock_t), held by exactly 7 hooks (on_queue_update, on_job_push, on_job_state_change, on_job_update, begin_session, next_job, on_dispatch) — on_pend/end_session/init/fini correctly excluded (never touch the locked state).

• 2.5 Remove ordering logic from the core scheduling loop

  • The core loop calls only picker->next_job() to obtain the next schedulable job — no project/queue/job iteration or ordering logic remains in the loop itself
  • Observable: a code-level check confirms the scheduling loop function contains no direct queue/job traversal — all selection reads happen through the picker interface
  • Requirements: 1.1, 1.7
  • Depends: 2.4
  • Verified: 2026-08-03 (commit 9d91d470) — backfilled 2026-08-07. Confirmed via source read: sched_job_picker()'s native-picker branch calls only picker_iface->next_job() to obtain the next job — no queue/job traversal or ordering logic in the loop itself; a code-level guard test asserts this.

• 2.6 Picker regression tests and data-sharing invariant checks

  • Assert builtin_fcfs's next_job() ordering matches today's pre-change FCFS ordering exactly, given identical queue/job state
  • Confirm the existing scheduler unit test suite passes unmodified except at the specific assertions that exercised the removed hardcoded string and the decision-publishing path this feature replaces
  • Verify resreq objects are never mutated in place by any touched code path, all pend-reason allocation goes through the existing factory function (no direct calloc of the reason struct), comparisons use pointer equality as a fast path before deep comparison, and the existing per-session select-expression memoization is untouched
  • Observable: the regression suite passes with the documented ordering rule as its oracle, and a grep-based or code-review check confirms no new calloc(sizeof(...pendreason...)) call site was introduced
  • Requirements: 4.1, 4.2, 6.1, 6.2, 6.3, 6.4
  • Depends: 2.5
  • Verified: 2026-08-03 (commit 604a5649) — backfilled 2026-08-07. test_picker_pendreason_factory_audit.c and the picker regression suite confirm ordering-equivalence against legacy fcfs_elect_job and the pend-reason/resreq data-sharing invariants (Req 6.1-6.4); part of the 41/41 green suite reconfirmed by this validation pass.

• 3. Run Phase 1 HA soak regression

  • Run sequential and chaotic HA failover soak rounds against a cluster running the new picker architecture
  • Observable: zero new job loss, decision divergence, or liveness stall attributable to this phase's changes across the soak run
  • Requirements: 4.3
  • Depends: 2.6
  • Verified: 15-round sequential soak, 2026-08-03 — PASS: 15 rounds, 48 jobs submitted, zero job loss; all 3 nodes led (ctrl-1,ctrl-2,ctrl-3)

• 4. Foundation: Scheduler Core Extraction

• 4.1 Extract the scheduling core from the monolithic scheduler struct

  • Split scheduling state (jobs, queues, workers, resources, resvals, users, projects, plugin instances) into its own core structure holding zero controller I/O state (no API client context, no cluster config)
  • Confirm the core links only against the approved minimal data-structure subset and never links network/timer/API libraries — no wall-clock time call appears anywhere in the extracted core's source
  • Observable: the core structure compiles and links with zero symbols from the networking/API libraries; a build-time or link-time check confirms this
  • Requirements: 5.1, 5.2
  • Depends: 3
  • Verified: 2026-08-03 — new xce_sched_core_t (core.h/core.c) holds all scheduling state, xce_scheduler_t shrank to {core; apictx; cluster_cfg}; nm-based isolation test (test_core_isolation.c) proves zero netio/xcecli/idgen/delaymgr symbols; 14/14 scheduler suite passing; independent review APPROVED (own fresh build/test/nm run + reproduced RED)

• 4.2 Pin the determinism invariants

  • Hardcode the picker-to-allocator handoff channel's capacity to 1 inside core initialization — never expose it as a parameter
  • Audit every hash-table/key-value-store iteration reachable from the scheduling loop (jobs, queues, users, projects) and confirm each produces order deterministic in (keys, insert/delete sequence) with no process-random seeding; fix any site found unstable
  • Observable: core initialization rejects (or simply has no code path to accept) a caller-supplied handoff-channel capacity; the iteration-order audit's findings are recorded and any fix is covered by a targeted test
  • Requirements: 5.7
  • Verified: 2026-08-03 — jbuf capacity hardcoded to literal 1 (core.c), regression test test_core_jbuf_capacity_pin.c; kvmgr audit found jobs/queues/users/projects already fully order-stable (xzarray ascending-slot walk, no process-random seeding), no fix needed, findings recorded durably as a comment in core.c above xce_sched_core_init(); 15/15 scheduler suite passing; two-round review (round 1 REJECTED on missing durable record only, round 2 APPROVED after fix); core->workers flagged as an open follow-up (see Implementation Notes), not this task's scope
  • Depends: 4.1

• 4.3 Implement the core event interface and batch event push

  • Define the typed core event representation (timestamp plus type-specific payload) covering configuration push, job push, worker push, resource push, resource-value full/delta push, job state change, job update, queue update, and job resource-usage events
  • Implement batch event push: state-change picker hooks fire synchronously for each event in the batch, valid only when no scheduling session is active
  • Observable: pushing a batch of events with the picker idle fires the corresponding state-change hook exactly once per event, synchronously, before the push call returns
  • Requirements: 5.4, 5.8
  • Depends: 4.1
  • Verified: 2026-08-03 — xce_core_event_type_t (9 kinds) + xce_core_event_t in core.h; xce_core_push_events() in new core_events.c (split out of core.c to stay under the 800-line cap — design.md's file-structure sketch not updated yet); rejects -EBUSY mid-session, fires each hook exactly once synchronously (JOBS_PUSH once per job); additive only, not wired into the live path (task 5.1/5.2's job). Round 1 REJECTED: CONFIG_PUSH didn't decompose queues/resources/resvals like sched_init_config() does — fixed round 2 (core_apply_config_resources/resvals/queues + shared core_upsert_queue(), also used by QUEUE_UPDATE), round-2 review APPROVED with live RED→GREEN reproduction. 16/16 scheduler suite passing.

• 4.4 Implement the session-start signal and decision callback protocol

  • Implement the non-blocking session-start call that signals the picker thread and returns immediately, rejecting a call while a session is already active
  • Define the three decision output shapes (dispatch, pend, preempt) and the decision callback invoked once per decision as the allocator produces it, plus the explicit end-of-session marker
  • Observable: starting a session while one is already active is rejected with a distinct error; each decision reaches the registered callback in allocator-output order, followed by the end-of-session marker
  • Requirements: 5.5, 5.6
  • Depends: 4.3
  • Verified: 2026-08-03 — xce_core_go()/xce_core_end_session() (new core_session.c) implement the non-blocking session-start signal (-EINVAL on reentrant call, proven a true no-op via cbuf_tryget) and end-session cleanup; xce_core_set_decision_cb() registers the driver's on_decision callback (separate setter, not an xce_sched_core_init() param, to avoid touching its live sched_init() call site); XCE_DECISION_PREEMPT added to decision.h matching the DISPATCH/PEND/PREEMPT AC 6 table exactly. Decision-drain loop confirmed driver-owned (not a core function) via 3 independent requirements.md/design.md citations, re-derived independently by review. Additive only, sched_go/sched_begin_session/sched_end_session/sched_allocator/main.c untouched. REJECTED round 1 on a clang-format alignment regression in core.h (mid-struct insertion de-aligned 24 pre-existing lines) — fixed with clang-format -i, whitespace-only, re-verified 17/17 green.

• 4.5 Implement clock virtualization and session timing

  • Implement the virtualized clock accessor using the two-anchor-plus-speed formula, backed by atomically-updated fields; no direct wall-clock call anywhere else in the core
  • Register the core's own log-timestamp formatter, bound to the virtualized clock accessor instead of the logging library's default wall-clock read — this is the one third-party dependency inside the core that would otherwise call wall-clock time on every log line, silently reintroducing the exact dependency task 4.1 confirmed absent at link time
  • Track session start/end timestamps internally and compute session duration for diagnostics, exposed through the session summary
  • Observable: the clock accessor's output matches the documented formula for all four operating modes (normal, discrete simulation, real-time simulation, fast/slow simulation) given fixed anchor values; a log line emitted while the virtual clock is frozen at a fixed simulated timestamp shows that timestamp, not the real wall-clock time at the moment of the call
  • Requirements: 5.3, 5.12
  • Verified: 2026-08-03 — xce_core_now() (new core_clock.c, kept out of core.c so test_core_isolation.c's wall-clock scan still passes) implements the exact two-anchor+speed formula; all 4 modes proven with fixed anchors, no sleep-based flakiness. requirements.md's literal tlog_reg_format_func(fn, core_ptr) call shape doesn't exist in the real vendored tlog API (verified against tlog.h: registration takes only the callback, userptr is per-call-site) — resolved via a module-level active-core pointer read by a new core-scoped format function, not wired into mylog_init()/main.c yet. session_start/session_duration_ns stamped correctly in xce_core_go()/xce_core_end_session() (task 4.4's behavior otherwise untouched). 18/18 scheduler suite passing, clang-format clean, review APPROVED first pass. Follow-up noted: core_clock.c is the 3rd core-adjacent file (after core_events.c, core_session.c) outside test_core_isolation.c's nm-based audit — manually verified clean, not mechanically enforced.
  • Depends: 4.1

• 4.6 Core contract tests

  • Test the full push-events → start-session → drain-decisions → end-session cycle against a stub picker (no driver needed), asserting hook firing order and timing
  • Observable: the contract test suite passes, exercising both the batch-push precondition (rejects mid-session) and the double-session-start rejection
  • Requirements: 5.1, 5.2
  • Depends: 4.4
  • Verified: 2026-08-03 — new test_core_full_cycle.c drives push_events→go→drain→end_session against a real picker pthread (running the real xce_job_picker_iface_t hooks on a test-local stub picker) plus a main-thread stand-in allocator (sched_allocator() is xce_scheduler_t*-coupled, out of pure-core scope) calling real on_dispatch/on_pend/make_dispatch_decision/make_pend_decision. Both preconditions (-EINVAL reentrant core_go, -EBUSY mid-session push_events) proven against a genuinely active session, not a hand-set flag, and proven to un-latch after the session ends. Two live RED→GREEN reproductions (neutered each guard in turn), both independently reproduced by review. 19/19 scheduler suite passing, clang-format clean, 30/30 stress runs with zero flakiness, review APPROVED first pass. Zero production files touched — test-only, as scoped.

• 5. Core: Normal-Mode Driver

• 5.1 Build the normal-mode driver's event translation path

  • Translate controller capnp push/notify messages into core events, buffering them in a pending-event queue between scheduling rounds
  • Observable: a sequence of simulated controller push/notify messages, once drained, produces a core event batch whose contents match the input messages one-to-one
  • Requirements: 5.8, 5.9
  • Verified: 2026-08-03 — new driver.h/driver.c: xce_sched_driver_translate_pending_events(xce_sched_core_t*) drains pend_updates via the existing dataq-swap pattern, translates all 10 op types to xce_core_event_t (field-verified against core_events.c's actual consumption, not assumed), stamps via xce_core_now(), pushes through the real xce_core_push_events(). Deliberately scoped to xce_sched_core_t* alone (not the full design.md xce_sched_driver_t) since pend_updates already lives on core since task 4.1 and this task is translation-only — task 5.2 can wrap trivially with &driver->core. handler.c untouched. Two flagged risk items (RESVALS zarray dangling-pointer shape, zero-filled changed_fields/exit_code for fields the legacy payload can't supply) both independently verified as pre-existing/honestly-inert, not new bugs. 20/20 scheduler suite passing, real-ASan clean, clang-format clean, review APPROVED first pass.
  • Depends: 4.4

• 5.2 Wire decision publishing, the mode-selection entry point, and retire the hardcoded decision switch

  • Publish each decision to the controller immediately as it's produced (dispatch, pend-reason, preempt) via the existing client API — not batched at session end
  • Replace the scheduler process's hardcoded decision-type switch statement with registration of the driver's decision callback
  • Add the process entry point's mode-selection branch (normal-mode driver vs. simulation driver based on whether a replay file was given) and normal-mode's own CLI flags
  • Observable: submitting a synthetic session that produces a mix of dispatch and pend decisions results in each being published to the (simulated) controller in the order produced, with no batching delay; starting the process with no replay file argument runs the normal-mode driver
  • Requirements: 1.7, 5.6, 5.9
  • Depends: 5.1
  • Verified: 2026-08-04 — main.c's live loop rewired: xce_sched_driver_translate_pending_events() -> xce_core_begin_session() -> plugin_begin_session()+sched_account_jobs() -> xce_core_signal_go() -> decision-drain calling the registered xce_core_decision_cb_t -> plugin_end_session() -> xce_core_end_session(), all in the new exported xce_sched_driver_run_normal_mode_round() (driver.c). xce_core_go() (task 4.4) split into xce_core_begin_session()/xce_core_signal_go() as a pure decomposition (test_core_session.c unmodified, still passing) specifically so plugin_begin_session()/sched_account_jobs() can run in the exact safety-critical window (after resval reset, strictly before the picker is signaled) that task 4.4 deliberately excluded from pure-core scope. Empirically proven, not just documented, by test_driver_session_ordering.c (real cross-thread happens-before assertions) with a live RED->GREEN reproduction, independently reproduced by review. xcecli_preempt() confirmed not to exist anywhere in client.h; PREEMPT decisions fail loudly (-ENOSYS, logged) rather than fabricating a client API. --sim CLI flag added (parse-only, loud not-yet-implemented failure, since SimDriver is Phase 3). 22/22 scheduler suite passing, --help smoke check clean, review APPROVED after one clang-format fix. Found (not introduced, not fixed, flagged for follow-up) a genuine pre-existing plugin_end_session() guard/call mismatch bug in plugin.c predating this spec — affects the legacy xce_sched_fcfs picker's end-of-session cursor reset, not builtin_fcfs.

• 6. Run Phase 2 HA soak regression

  • Run sequential and chaotic HA soak rounds confirming decisions and controller interaction match Phase 1's behavior exactly
  • Observable: zero new job loss, divergence, or liveness stall; decision content matches Phase 1's soak baseline
  • Requirements: 4.3
  • Depends: 5.2, 4.6
  • Verified: 2026-08-04 — sequential HA soak (xce-test-ha skill), 15 rounds, 48 jobs submitted, zero job loss, all 3 nodes led as primary (ctrl-1/ctrl-2/ctrl-3), committed-prefix walcompare consistent every round. Confirms tasks 4.1-5.2's src/scheduler/ rewrite (core extraction + normal-mode driver wiring into main.c) introduced no controller-side regression — consistent with every task's independent review confirming zero src/controller/ changes throughout Group 4/5. Gates Phase 2 -> Phase 3.

• 7. Foundation: Replay Wire Format

• 7.1 Define the replay stream envelope schema

  • Add the replay header, snapshot entry, replay event, replay session, and replay decision wrapper message types to the shared wire schema, following the single-stream-of-consecutive-messages framing
  • Observable: the schema compiles and a minimal header-plus-snapshot-entry byte stream round-trips through encode/decode
  • Requirements: 5.20
  • Verified: 2026-08-04 — ReplayCluster/ReplayHeader/SnapshotEntry/ReplayEvent/SessionSummary/ReplaySession added fully; SnapshotData (empty {}) and ReplayDecision ({ts only}) added as compiler-mandated placeholders task 7.3 fills in (real fields for SnapshotData wholesale-replaced, a body union appended to ReplayDecision at new ordinals) since capnp's $C.codecgen has no per-struct opt-out from requiring the mapname'd C type to exist. ReplayEvent's union deliberately holds only the 9 members with pre-existing backing types (configPush/jobsPush/workersPush/resourcesPush/resvalsFull/resvalsDelta/queueUpdate/jobRusage/endSession); jobStateChange/jobUpdate deferred to task 7.2 as new ordinals — proven compiler-mandatory (not stylistic) by review, which independently confirmed capnp rejects both the AC 20 pseudocode's literal union syntax and any ordinal gap. Round-trip test (test_e2e_capnp.c) proven real via full clean rebuild (deleted+regenerated controller.capnp.{c,h}) both by implementer and reviewer. Full project build clean (controller/apiserver-CGO/scheduler/worker/xceadmin), 1075/1075 controller_unit_test passing, review APPROVED first pass with reviewer going beyond implementer's own verification (real capn_resolve()-omission and placeholder-removal breakage reproduced firsthand). Zero existing structs/fields touched.
  • Depends: 6

• 7.2 Define new event payload types

  • Add the two genuinely new event payload message types (job state-change delta, job update delta with embedded full record) to the wire schema; every other event payload type reuses an existing wire message already used elsewhere in the schema
  • Observable: encoding a job state-change event and a job update event and decoding them back yields field-for-field equality with the original values
  • Requirements: 5.20
  • Depends: 7.1
  • Boundary: ReplayFormat — not (P): 7.2/7.3/7.4 all edit the same schema file (controller.capnp); run sequentially to avoid concurrent edits to one file, even though each adds a logically disjoint set of structs
  • Verified: 2026-08-04 — JobStateChangeEvent/JobUpdateEvent added to controller.capnp matching design.md's spec verbatim; JobUpdateEvent.job correctly reuses the full 12-field Job/xce_job_t record (not the narrower SchedJob sharing the same mapname — verified by field list). ReplayEvent.body's union extended with jobStateChange@10/jobUpdate@11, the only two valid next ordinals per task 7.1's baseline (capnp requires no gaps) — confirmed via a from-scratch capnp regeneration. Existing @0-@9 fields byte-identical. Deep round-trip tests (non-default values throughout, including 3-level nested Job substructs) prove the Observable; JOB_F* bitmask values deliberately left undefined (later task's job). 1077/1077 controller_unit_test, full project build clean, review APPROVED first pass._

• 7.3 Define decision payload types and the snapshot schema

  • Add the dispatch, pend, and preempt decision payload message types, reusing existing allocation and pend-reason wire messages where they already carry the needed fields
  • Add the full snapshot message type covering configuration, jobs, queues, workers, resources, resource values, users, and projects, reusing existing wire messages for each domain object list; represent the two scheduler-internal-only caches and session/PRNG state as a versioned opaque payload with no external structure
  • Observable: encoding one of each decision type and a populated snapshot and decoding them back yields field-for-field equality; a snapshot encoded with one internal-state version and decoded expecting a different version is rejected rather than misinterpreted
  • Requirements: 5.5, 5.20, 5.21, 6.5
  • Depends: 7.1
  • Verified: 2026-08-04 — DispatchDecision/PendDecision/PreemptVictim/PreemptDecision added, reusing JobAlloc/JobPendReason per spec. SnapshotData's task-7.1 empty placeholder replaced wholesale with the real 11 fields (config/jobs/queues/workers/resources/resvals/users/projects/prngSeed/internalState/internalVersion), both in the capnp schema and its typemap.h C mirror. ReplayDecision's ts-only placeholder gained a purely additive body union (dispatch@1/pend@2/preempt@3). One deviation from design.md's literal text: SchedUser/SchedProject use fresh mapnames (sched_snapshot_user_t/sched_snapshot_project_t) instead of design.md's literal xce_sched_user_t/xce_sched_project_t, since those names are already real C structs in sched/types.h that share a translation unit with typemap.h in core_events.c/schedobj.c — reusing them would be a genuine duplicate-typedef compile error, not a style choice. Field lists (name-only; name+priority) verified unchanged, deviation rigorously re-derived and confirmed necessary by independent review, documented in both files. Found and fixed a real bug mid-implementation (a doc comment's literal */ substring prematurely closing a C block comment) logged as bug-1493. ReplayEvent's union (tasks 7.1/7.2) and every other existing struct byte-identical. Five deep round-trip tests (all 11 SnapshotData fields populated, version-mismatch demonstrated). 1082/1082 controller_unit_test, full clean project build, confirmed independently three times (implementer, orchestrator, reviewer, all matching exactly), review APPROVED first pass.
  • Boundary: ReplayFormat — not (P), same file as 7.2/7.4 (see 7.2)

• 7.4 Add the scheduler admin opcode schema

  • Add the new opcode value, the admin action enum (record start/stop/status), and the paired request/reply message types (reply carrying ok/error/recording-state/file/event-count/started-at) to the wire schema, plus their entries in the existing request/reply message unions
  • Observable: encoding each of the three admin actions and their corresponding replies and decoding them back yields field-for-field equality
  • Requirements: 5.10
  • Depends: 7.1
  • Boundary: ReplayFormat — not (P), same file as 7.2/7.3 (see 7.2)
  • Verified: 2026-08-04 — SchedAdminAction enum + SchedAdminRequest/SchedAdminReply structs added matching requirements.md Req 5 AC 10 verbatim. schedAdmin@42 (OPCode), schedadminmsg@31 (Request.body), schedadminreplymsg@19 (RequestReply.data) — all three the exact next-available ordinals, verified live before dispatch and re-verified byte-identical below by review. Deliberately schema-only per this task's own text — the controller dispatch logic (cluster.c's case OPCode_schedAdmin:) is task 10.1's job, not this one; confirmed cluster.c untouched. Round-trip tests cover all 3 SchedAdminAction values and both ok=true/ok=false SchedAdminReply cases with an asymmetric ok/recording combination, ruling out a Bool-field bit-packing mixup. 1084/1084 controller_unit_test, full clean project build, confirmed independently three times, review APPROVED first pass. Closes out Group 7 (Foundation: Replay Wire Format) entirely — tasks 7.1-7.4 all complete.

• 7.5 Replay format round-trip test suite

  • Encode and decode every new message type introduced in this group, asserting field-for-field equality after round-trip
  • Observable: the full round-trip test suite passes for every new struct, including the internal-state version-mismatch negative test
  • Requirements: 5.20
  • Depends: 7.2, 7.3, 7.4
  • Verified: 2026-08-04 — audit of all ~19 new types/enums from tasks 7.1-7.4 found one genuine gap: ReplayEvent's own envelope union (spanning 11 members across 7.1+7.2) was never actually constructed/encoded in any test, despite its individual payload types each having their own round-trip test. Fixed with 3 new tests (jobStateChange/jobUpdate/endSession variants). All other 18 types independently re-verified genuinely field-for-field, non-shallow. Decided against creating design.md's originally-named test_replay_format.c: tasks 7.1-7.4 independently, consistently (4-for-4) chose the existing test_e2e_capnp.c instead, matching task 4.3's core_events.c-vs-core.c precedent for design.md file-naming being non-binding guidance. Deliberately did not add Request.body/RequestReply.data union-wiring tests for OPCode_schedAdmin, matching universal codebase precedent (opcode-dispatch union members are always tested via functional dispatch tests, never bare schema round-trips, and the real dispatch logic doesn't exist until task 10.1). Internal-state version-mismatch negative test re-verified real and correctly scoped (schema-tag level only, matching the Observable). Zero schema/typemap changes -- audit-only, as scoped. 1087/1087 controller_unit_test, review APPROVED first pass with every claim independently re-derived. Closes out Group 7 (Foundation: Replay Wire Format) entirely -- tasks 7.1-7.5 all complete.

• 8. Core: Snapshot Serialization

• 8.1 Implement snapshot save and load

  • Serialize the full core state (as scoped in the snapshot schema) into the snapshot message; restore from a snapshot message back to that exact state
  • After a load, rebuild the picker's private state by walking the restored core state and re-invoking the picker's state-change hooks (queue updates, then pending-job pushes) — never store picker-private state in the snapshot itself
  • Observable: save immediately followed by load into a freshly-initialized core produces a core whose observable scheduling behavior (next session's decisions, given the same events) is identical to the pre-save core
  • Requirements: 5.21, 6.5
  • Depends: 7.3
  • Verified: 2026-08-04 — xce_sched_core_save_snapshot()/xce_sched_core_load_snapshot() implemented, split across snapshot.c/snapshot_wire.c/snapshot_load.c (800-line cap) along real logical boundaries (save/load/shared-helpers). New prng_seed field added to xce_sched_core_t (a genuine, well-investigated gap requirements.md/design.md both presupposed but no prior task added), seeded to 0, no PRNG consumer wired yet (documented, deliberate). Cross-core behavioral-equivalence Observable proven for real: two independent cores, each with its own registered picker, save on one/load into a freshly-initialized other, both driven through an identical real session (picker thread + decision drain), decision sequences asserted identical including a pinned spot-check (201-PEND/202-DISPATCH/203-PEND on both). -EBUSY/-EINVAL guards checked before any mutation, tested directly. Found and fixed two real bugs along the way: a missing prng_seed restore in this task's own new code, and a genuine, independently-traced-to-the-primitive-level guard bug in already-committed core.c (sched_get_user/sched_get_project treating key_find()'s expected -ENOENT as fatal, silently breaking get-or-create for any new name) that also affected the live production path (make_sched_job(), JOBS_PUSH) — narrowly scoped two-line fix, folded into this task's diff since only this task's own test path discovered it. 23/23 scheduler suite, full project build clean, review APPROVED first pass with every claim independently re-derived.

• 8.2 Enforce the snapshot session-idle precondition

  • Reject snapshot save/load calls made while a scheduling session is active, returning a distinct busy error rather than capturing or restoring torn state
  • Observable: attempting a snapshot save/load mid-session returns the busy error and leaves core state unchanged
  • Requirements: 5.21
  • Depends: 8.1
  • Verified: 2026-08-04 — task 8.1's own -EBUSY guard test only proved rejection against a synthetic single-thread session_active flag flip, and its comment misattributed the technique to test_core_full_cycle.c's actually-genuine-session precedent. Real gap, not busywork: xce_core_go() publishes session_active via memory_order_release while the snapshot guards read it via plain atomic_load() — the synthetic same-thread test never exercised any real cross-thread visibility of that flag. Added a genuine driver-driven session test (real picker thread + real xce_core_go()) proving both save and load reject with -EBUSY during the same active session, leaving state genuinely unchanged (sentinel bytes for save; distinguishable non-existent queue + unchanged candidate count for load) -- zero production changes, task 8.1's guards were already correct. 23/23 scheduler suite, 10/10 standalone flakiness-clean, review APPROVED first pass with an even sharper independent justification than the implementer's own. Closes out Group 8 (Core: Snapshot Serialization) entirely -- tasks 8.1-8.2 both complete.

• 9. Core: Recording Control

• 9.1 Implement recording directory provisioning and path validation

  • Add a scheduler startup CLI flag (parsed in the same argument-parsing pass as the rest of normal-mode's flags) for the recording directory — never settable from any request-path field — defaulting to a fixed system path
  • Validate any caller-supplied record file name before ever opening it for writing: reject any value containing a path separator or a .. segment, resolve the final path beneath the recording directory, and confirm containment before opening
  • Observable: a record-file name containing a path separator or .. is rejected before any filesystem access occurs; a bare filename resolves to a path inside the recording directory and is accepted
  • Requirements: 5.10
  • Depends: 8.1
  • Verified: 2026-08-04 — --record-dir CLI flag added to main.c/scheduler.h (record_dir field, never a request-path field). sched_validate_record_filename() (new recording.c/recording.h) rejects any filename containing / or .. before any filesystem call, then resolves via realpath() and confirms canonical (not textual-prefix) containment beneath record_dir via path_is_contained(); rejects pre-existing non-regular entries (-EEXIST) and a non-directory record_dir (-ENOTDIR). Adversarial security review found and this task fixed two real CWE-59 inode-aliasing bypasses across 3 rounds: round 1 found a direct hard link inside record_dir aliasing an outside file passed validation (S_ISREG doesn't distinguish a hard-linked regular file from a genuine one) — fixed with an entry_st.st_nlink > 1 check (-EACCES) in the direct-entry branch, live RED→GREEN reproduced. Round 2 re-review found a compound variant still open: a symlink inside record_dir pointing at an in-directory file itself hard-linked outside record_dir was still accepted, since the S_ISLNK branch checked containment but never st_nlink on the resolved target — fixed by stat()'ing the already-realpath()-resolved target and applying the same S_ISREG && st_nlink>1 check, generalizing correctly to multi-hop symlink chains, live RED→GREEN reproduced. Round 3 re-review independently reproduced both fixes via a standalone PoC harness and additionally hunted for further variants (2-hop chains, symlink-to-symlink escape, FIFO/socket direct-naming) — all clean, no new bypass found; one non-blocking observation logged (symlink resolving to a contained-but-non-regular target is accepted rather than rejected, pre-existing since round 1, not the CWE-22/CWE-59 escape class, already covered by task 9.2's planned O_NOFOLLOW hardening) — APPROVED. TOCTOU residual risk (validation-to-eventual-fopen() window) explicitly documented in recording.h and deferred to task 9.2 (O_NOFOLLOW + fstat()-based st_nlink==1 re-check on the opened fd), not this task's scope. 18/18 recording subtests (16 original + 2 hard-link/compound regressions), 24/24 scheduler suite, clang-format clean — all independently re-verified by me via a fresh meson compile/meson test --suite scheduler run on current HEAD before marking complete.

• 9.2 Implement the record start/stop lifecycle

  • On start: take a full snapshot, write the replay header followed by the initial snapshot entry, begin tracking recording state
  • On each event batch and each completed session while recording: write the corresponding event/session messages to the record file in order — extends the normal-mode driver main loop built in group 5 with recording insert points, it does not replace it
  • On stop: write the end-of-recording marker, flush and close the file, clear recording state
  • When recording is inactive, incur zero file I/O and zero extra allocation — a single boolean check per session
  • Observable: a start → submit events → complete a session → stop cycle produces a file beginning with header+snapshot-entry and ending with the end-of-recording marker, with events and one session in between, byte-decodable by the round-trip decoder from group 7
  • Requirements: 5.10
  • Depends: 9.1
  • _Verified: 2026-08-05 — added recording-state fields directly to xce_scheduler_t (no new xce_sched_driver_t type, matching task 9.1's established deviation from design.md's sketch). sched_record_start()/sched_record_stop() (recording.c) implement AC 10 steps 1 and 7 exactly: validate via sched_validate_record_filename() (task 9.1), open with the TOCTOU-hardened sequence (O_NOFOLLOW, no O_TRUNC, fstat()+st_nlink==1 check on the opened fd, ftruncate() only after the check passes — matching recording.h's task-9.1-era note to this task), xce_sched_core_save_snapshot() → write ReplayHeader then SnapshotEntry, set recording=true; stop writes ReplayEvent{body=endSession}, flushes, closes, clears state. driver.c gains a record_ctx void* param on xce_sched_driver_translate_pending_events() (deliberately NOT read from core->driver_ctx, which isn't guaranteed xce_scheduler_t* in test fixtures) for per-event-batch ReplayEvent writes (new recording_events.c), plus sched_record_session_begin()/_add_decision()/end() hooks in xce_sched_driver_run_normal_mode_round() for per-session ReplaySession writes (new recording_session.c) — both boolean-gated on sched->recording, zero I/O/allocation when inactive, verified by tracing both call sites. recording.c split three ways (recording.c=543, recording_events.c=177, recording_session.c=244 lines) to stay under the 800-line cap, mirroring snapshot.c's precedent. Two real bugs were found and fixed during implementation, independently confirmed by review: bug-1498 (an initial O_TRUNC-in-open() would have truncated a hard-linked victim file before the st_nlink check could reject it — fixed by deferring truncation to a post-check ftruncate()) and bug-1499 (an edit transiently dropped xce_core_signal_go(), a 3-way picker/allocator/driver deadlock — restored, caught by test_driver_session_ordering.c). New test_record_lifecycle.c (826 lines, in line with this suite's established over-800 precedent for test files) decodes the full start→events→session→stop cycle with the real generated capnp decoders and asserts the exact 5-message sequence plus no 6th message; two TOCTOU regression tests use real symlink(2)/link(2) syscalls against a real victim file and assert its content is untouched after rejection; double-start/stop-when-not-recording both return -EALREADY cleanly. Review found one non-blocking follow-up (open() lacks O_NONBLOCK, so a FIFO planted in the same TOCTOU window would hang the calling thread — DoS, not corruption; logged below, not fixed in this task) and confirmed zero scope leakage into tasks 9.3/9.4/group 10. 25/25 scheduler suite, clang-format clean — independently re-verified by me via a fresh meson compile/meson test --suite scheduler run before marking complete. Note: two implementer-agent runs ended up editing this worktree concurrently after a spurious "agent died" signal turned out to be wrong; a stabilization wait (repeated md5sum) plus two independent fresh-build verifications (mine and a dedicated fix/verify agent's) confirmed the settled state was consistent before review.

• 9.3 Implement periodic snapshot checkpointing and recording error handling

  • Append a fresh snapshot entry to the record file every configured number of recorded events (0 disables this)
  • On a write failure mid-recording (disk full, permission error): log the error, cleanly tear down recording state, and continue normal scheduling — recording failure must never stop or crash the scheduler
  • Observable: recording through a configured interval produces the expected number of extra snapshot entries in the file; simulating a write failure leaves the scheduler still processing sessions afterward with recording cleanly disabled
  • Requirements: 5.11
  • Depends: 9.2
  • _Verified: 2026-08-05 — the write-failure error-handling half was already built in task 9.2 (sched_record_abort_on_error(), matching AC 10's clause verbatim: log, fflush+fclose, free+NULL path, recording=false); this task added the missing end-to-end test (test_write_failure_disables_recording_and_driver_keeps_running) using a genuine closed-fd write failure (fflush + setvbuf(IONBF) + close(fd), confirmed by review running the binary directly and observing a real rc=-5 abort log line, then a real subsequent session completing) rather than a mocked return code. Periodic checkpointing (the new work) hooks into sched_record_write_events(): once per call, record_snapshot_interval > 0 && record_event_count % record_snapshot_interval == 0 triggers a fresh SnapshotEntry via the same write_snapshot_entry() helper sched_record_start() already uses (made non-static, zero duplication) — safe from task 8.2's session_active precondition because this call site is only ever reached from xce_sched_driver_translate_pending_events(), confirmed the sole production caller and confirmed to always run strictly before xce_core_begin_session(). record_snapshot_interval == 0 genuinely produces zero checkpoints. The "once per call, not once per crossed boundary" semantics matches requirements.md's own embedded pseudocode verbatim, not an implementer shortcut. 13/13 new test binary, 25/25 scheduler suite, clang-format clean — independently re-verified by me via a fresh meson compile/meson test --suite scheduler run before marking complete. Two follow-ups flagged by review, logged below rather than folded in: --record-snapshot-interval has no CLI wiring anywhere (AC 11 cannot be configured non-zero in production as currently scoped — this task's own Observable is satisfied via the struct field directly, which is sufficient for this task, but the operator-facing gap remains); test_record_lifecycle.c grew to 1232 lines (826→1232), past this suite's prior precedent, with a natural AC10/AC11 split point identified but not acted on.

• 9.4 Implement the scheduler-side admin request handler

  • Decode a forwarded SchedAdminRequest arriving on the scheduler's controller-facing endpoint, dispatch by action to the record-start/stop/status functions built in 9.2/9.3, and encode+send the SchedAdminReply back to the controller on the same endpoint — this is the scheduler-side counterpart to the controller's forwarding dispatch (task 10.1), without which 10.1 has nothing to forward to
  • Observable: sending each of the three admin actions to the scheduler's endpoint produces the matching reply (recording state, file, event count, started-at populated correctly for each action)
  • Requirements: 5.10
  • Depends: 9.3
  • Verified: 2026-08-05 — this is the scheduler's first-ever request-reply opcode (OPCode_schedAdmin, no prior precedent). sched_handle_admin_request() (new, handler.c) decodes SchedAdminRequest and dispatches by action: recordStatus is answered synchronously (new sched_record_status(), recording.c, pure read); recordStart/recordStop are deferred — enqueued as a new XCE_SCHED_OP_ADMIN_RECORD op onto the existing pend_updates queue (same mechanism every controller-sourced update already uses) rather than called directly from the request-handling thread, because sched_record_start()'s xce_sched_core_save_snapshot() call requires task 8.2's session_active==0 precondition, only guaranteed on the driver thread — confirmed by review as a genuine (not theoretical) concern: vtaskmgr.c spawns real pthreads for request handling, distinct from main.c's single driver-loop thread. driver.c's xce_sched_driver_translate_pending_events() intercepts the deferred op, performs the real start/stop, and sends the reply via a new xcecli_reply_schedadmin() (src/lib/client.c/client.h) correlated by a plain seq copied by value before the original request is freed — confirmed safe since xcecli_ctx_t is a single unmultiplexed connection. Every reply-field branch (ok/err for each action) independently verified against requirements.md's semantics table, including the subtlety that recordStop's file value must be captured before sched_record_stop() frees record_file_path. Round 1 review found one real bug beyond either implementer session's own testing: sched_record_status(), reachable from the vtask thread, read recording/record_file_path/record_event_count/record_start_ts with no lock while the driver thread's sched_record_stop() mutates and frees those same fields unsynchronized — a genuine cross-thread use-after-free, the codebase's first such race, missed by the ASan-only test config (ASan doesn't catch races). Fixed with a new dedicated mylock_t record_lock (deliberately separate from core.lock), scoped strictly to the in-memory field reads/writes only (never across fopen()/fwrite()/capnp I/O — confirmed no vtask-blocks-on-disk-I/O hazard reintroduced), with sched_record_status() now strdup()ing record_file_path while holding the lock rather than returning a borrowed live pointer. Round 2 review independently confirmed the fix: exhaustive grep found every access site now correctly locked (with recording_session.c's driver-thread-only functions and handle_deferred_admin_record()'s post-start/stop reads correctly reasoned as safe without the lock, both same-thread-only by construction); confirmed non-recursive mylock_t and no nested acquisition (no new deadlock); independently reproduced the fix's necessity by temporarily stripping the lock and confirming the new lock-contention test fails exactly as predicted, then restoring and reconfirming green. 26/26 scheduler suite (14/14 in the new sched_admin_request_test), clang-format clean — independently re-verified by me via a fresh meson compile/meson test --suite scheduler run before marking complete. New xcecli_reply_schedadmin() addition to shared src/lib/client.c/src/include/client.h scoped purely to this one function, confirmed not touching src/controller/ (task 10.1's exclusive boundary).

• 9.5 Recording lifecycle integration tests

  • Test the full start/stop/status round trip against a real filesystem (through the request handler from 9.4, not by calling the internal functions directly), including the path-validation rejection cases and a simulated disk-full condition
  • Observable: the integration test suite passes, including assertions on the exact error returned for each rejection case
  • Requirements: 5.10, 5.11
  • Depends: 9.4
  • Verified: 2026-08-05 — audited task 9.4's existing test_admin_request.c (already 14 tests driving everything through the real scheduler_req_handler()) against task 9.1's exhaustive path-validation suite and task 9.3's write-failure technique to find genuine gaps rather than duplicating coverage: only the bare path-separator rejection had been proven through the admin/handler path, and the disk-full scenario had never been proven with both the initiating recordStart and the verifying recordStatus driven through real admin requests. Added 3 tests (903→1163 lines): a ..-traversal rejection (distinct strstr("..") branch) and a real-symlink(2)-escape rejection, both submitted via genuine SchedAdminRequests through scheduler_req_handler() and asserted via the same reply-spy the rest of the file uses; and a write-failure test using task 9.3's exact fflush()+setvbuf(_IONBF)+close(fd) technique to force a genuine EIO, confirming sched_record_abort_on_error()'s cleanup is visible through a subsequent real recordStatus admin request and that a further recordStart afterward succeeds cleanly — the specific thing beyond task 9.3's internal-level proof. Hard-link-aliasing coverage through the admin path deliberately omitted as reasonable scope: task 9.1's 3-round adversarial hunt and task 9.2's direct-call TOCTOU test already exhaustively prove that specific rejection branch, and the routing/plumbing this task exists to prove is structurally identical regardless of which validation branch fires — already retired by the dotdot/symlink cases. Review independently verified every claim against source (confirmed distinct code branches, confirmed a real rc=-5 EIO in test output, confirmed the write-failure test's single-threaded sequence never touches record_lock directly and cannot deadlock/race since all mutation flows through already-locked library calls) and confirmed the record_start_ts-unchanged-on-abort behavior matches sched_record_stop()'s pre-existing identical semantics and requirements.md's own reply-field table ("original start (historical)") — not a bug. One minor non-blocking note: the symlink-escape test verifies "untouched" via stat()+size==0 rather than this suite's stronger write-content-then-readback pattern; logically sufficient, flagged for awareness only. 17/17 in sched_admin_request_test, 26/26 scheduler suite, clang-format clean — independently re-verified by me via a fresh meson compile/meson test --suite scheduler run before marking complete. Closes out Group 9 (Core: Recording Control) entirely — tasks 9.1-9.5 all complete.

• 10. Integration: Controller Admin Opcode and xceadmin CLI

• 10.1 Add the scheduler admin opcode dispatch to the controller

  • Add the new opcode's dispatch case to the controller's request handler: reject with a clear error when not primary, reject with a clear error when no scheduler is connected, forward the request to the scheduler and wait up to the timeout for a reply, and always send an explicit reply on every path (never a dropped request)
  • The controller interprets nothing beyond the opcode itself — every field of the request/reply passes through untouched, and the controller commits no state and writes no persistence entry for this opcode
  • Observable: a request against a non-primary controller returns the not-primary error immediately with no forwarding attempt; a request against a primary with no scheduler connected returns the not-connected error immediately
  • Requirements: 5.10
  • Depends: 7.4, 9.4
  • Verified: 2026-08-05 — this is the spec's ONE task touching src/controller/. New case OPCode_schedAdmin: in myreq_handler() (cluster.c) mirrors the existing OPCode_upgradeConfig case structurally: ha_is_primary() gate (reply -EAGAIN, no forward), sched.ep==NULL check (reply "scheduler not connected", no forward) — both of this task's own literal Observables — then forwards via a new xce_schedadmin_req() (request.c, async send mirroring xce_fetchfile_req) and blocks the vtask thread on a condvar with a 5s deadline while holding c->sched.lock for the full round-trip (a design decision made and accepted during this spec's earlier research phase, matching OPCode_upgradeConfig's posture at parity — no RBAC beyond the primary gate, blocking wait accepted as low-frequency human-paced admin traffic). A new branch in the existing case OPCode_reply: demuxes the scheduler's SchedAdminReply (by opcode == OPCode_schedAdmin) to a process-global single-slot pending-waiter mechanism (schedadmin_register/_unregister/_signal), serialized by c->sched.lock being held throughout — the same process-global-list pattern this codebase already uses for job.c's fetchfile proxy. Adversarial review independently traced double-free/cleanup-per-path, the timeout/signal race, single-slot safety, and reply-opcode demux correctness against source (not prior sessions' claims) and found no correctness bugs; confirmed the c->sched.lock contention is broader than initially characterized (gates job/worker/resource/queue events, not just job-new/start) but is spec-mandated by requirements.md's own pseudocode, bounded to 5s, and does not risk the severe heartbeat-starvation/false-failover cascade (heartbeat send runs on a separate non-vtask delaymgr task) — flagged as a documented residual risk for a future hardening pass, not a blocker. Review also found and a follow-up fix-round closed a narrow-window memory leak (a reply arriving in the brief gap between the waiter's mutex unlock and its unregister call could hand off ownership into an already-abandoned slot) via a carefully-ordered unlock→unregister→relock sequence, with lock-ordering reasoning (avoiding an AB-BA inversion against schedadmin_signal()'s g_schedadmin_mu→p->mu acquisition order) independently re-derived and confirmed correct by me directly from the code before finalizing. Also cleaned up: two prior sessions' work had whole-file clang-format reformatted cluster.c/request.c/request.h, churning large amounts of unrelated pre-existing HA code through pure whitespace changes — minimized to the genuinely new content only (confirmed via git diff matching git diff -w exactly for all three files) to avoid polluting this HA-critical file's history. New test_sched_admin_dispatch.c (3 tests) drives the real myreq_handler via the existing myreq_handler_for_test seam (same precedent test_config_repl.c uses): not-primary rejection using a poison-pointer sched.ep that would crash if forwarding were ever incorrectly attempted (proven, not assumed), not-connected rejection, and a genuine round-trip test with a second pthread independently driving a real OPCode_reply/SchedAdminReply through the same handler to exercise the actual signal/demux path end-to-end. Controller suite 3/3 (1090/1090 Unity assertions including the 3 new tests), scheduler suite 26/26 unaffected, clang-format clean — independently re-verified by me via fresh meson compile/meson test --suite controller/meson test --suite scheduler runs before marking complete. Residual, non-blocking notes for follow-up: the c->sched.lock contention breadth (above) and the process-global (vs. cluster_t-scoped) pending-waiter slot are both flagged as candidates for a future hardening pass, not required for this task.

• 10.2 Controller-side admin opcode dispatch tests

  • Test standby rejection, disconnected-scheduler rejection, forward timeout, and a successful forward/reply round trip against the real scheduler-side handler from 9.4
  • Observable: the test suite passes, covering every reply-on-every-path case named in the dispatch task
  • Requirements: 5.10
  • Depends: 10.1, 9.4
  • Verified: 2026-08-05 — audited task 10.1's own test_sched_admin_dispatch.c first: standby rejection and disconnected-scheduler rejection were already fully covered (confirmed unchanged). Two genuine gaps closed: (1) forward timeout — no test previously drove the real 5s pthread_cond_timedwait deadline to expiry. Added an #ifdef XCE_TESTING-only timeout-override seam in cluster.c (SCHEDADMIN_TIMEOUT_MS macro + schedadmin_set_timeout_ms_for_test()), verified genuinely inert in production builds: -DXCE_TESTING is set on exactly the 3 unit-test meson targets, absent from xce_controller's own target and from every project-wide arg; the non-test #else branch is a bare 5000L literal with the test-only global/setter entirely absent from that translation unit; deadline math confirmed correct for both the 5000ms default (behaviorally identical to the original deadline.tv_sec += 5) and the test's 200ms override (correct carry logic). test_schedadmin_forward_timeout sets a 200ms override, forwards to a real unconnected sched.ep with nothing ever replying, brackets the call with real CLOCK_MONOTONIC timestamps proving the wait genuinely blocked, and asserts the exact {status=0, ok=false, error="request timeout"} shape from requirements.md AC 10 — with the override reset to 5000 strictly before any TEST_ASSERT call (verified line-by-line, no longjmp-induced test-pollution risk). (2) "Round trip against the real scheduler-side handler" — investigated a literal two-binary integration (controller test invoking the scheduler's real scheduler_req_handler() in-process) and judged it architecturally awkward and out of scope (separate meson targets, incompatible fixture types, real symbol-collision risk) with no net correctness gain, since task 9.4's own test_admin_request.c already white-box-drives the real scheduler handler rigorously. Instead added two more round-trip outcome-branch tests using the same real xce_encode/xce_decode wire functions production uses: test_schedadmin_round_trip_rpc_status_only (a genuinely new C branch — reply with no SchedAdminReply payload) and test_schedadmin_round_trip_business_error (same branch as 10.1's existing success test, but proving ok=false/error-string field-passthrough fidelity — accurately self-described as payload coverage, not overclaimed as a new branch). Review independently verified every claim against source (confirmed -DXCE_TESTING's exact meson scoping, confirmed sequential single-threaded Unity test execution rules out any cross-test race on the test-only global, confirmed git diff exactly matches git diff -w for cluster.c — zero reformat noise, learning correctly applied from task 10.1's cleanup). Controller suite 3/3 (1093/1093 Unity assertions, up from 1090), scheduler suite 26/26 unaffected, clang-format clean — independently re-verified by me via fresh meson compile/meson test --suite controller/meson test --suite scheduler runs before marking complete.

• 10.3 Build the xceadmin scheduler recording commands

  • Implement record-start, record-stop, and record-status client functions that connect to a controller, follow redirects to the primary if needed, send the corresponding admin request, and decode the reply
  • Observable: each function returns a decoded reply struct on success and raises a typed error on connection failure, decode failure, or an error reply
  • Requirements: 5.10
  • Depends: 10.1
  • Boundary: XceAdminSchedulerCLI — not (P): 10.3/10.4 both add functions to the same new scheduler.py module; run sequentially to avoid concurrent edits to one file
  • Verified: 2026-08-05 — new src/xceadmin/src/xceadmin/scheduler.py (design.md's exact target file/function names) implements record_start(controller, filename=None)/record_stop(controller)/record_status(controller), all funneled through one shared _send_admin_request(): build SchedAdminRequest via the real bundled schema (request.opcode = "schedAdmin", request.body.init("schedadminmsg")) → xceadmin.discovery.follow_redirects (redirect-to-primary, mirrors push_config) → xceadmin.discovery.decode_reply → extract reply.data.schedadminreplymsg. Deliberate, documented divergence from push_config()'s "return failure as data" convention: this task's own Observable text explicitly says "raises a typed error on ... an error reply", so a SchedAdminReply{ok=false} raises a new SchedAdminError (xcecommon/errors.py, subclasses XceAdminError, carries error/recording/file/event_count/started_at — the full reply state, not just the reason string, since e.g. "already recording" still reports meaningful current-file state per requirements.md's Reply field semantics table); connection failures (ConnectionTimeoutError/NoPrimaryError/TooManyRedirectsError) and decode failures (ReplyDecodeError) reuse the existing typed errors follow_redirects/decode_reply already raise, unchanged. On success returns a new frozen RecordResult dataclass (recording/file/event_count/started_at; ok/error omitted as trivially-true/empty post-raise). test_errors.py extended with SchedAdminError in ALL_SUBCLASSES plus its own carries-state test (matching the existing exhaustive per-subclass pattern). New test_scheduler.py (mirrors test_config_push_mock.py's mock-TLS-server pattern): success for all 3 actions (incl. explicit vs. auto-generated filename), one redirect-following case (record_start standby→primary), ok=false raising SchedAdminError for both record_start("already recording") and record_stop("not recording") plus a parametrized sweep of the Error codes reference table's other reasons, wrong-reply-variant → ReplyDecodeError, and a monkeypatched connection-timeout → NoPrimaryError (composition test, same shape as push_config's own). 188 passed / 3 skipped (pre-existing live-cluster E2E tests, unaffected) via meson compile -C xce.build xceadmin — the real xceadmin test-running command (custom_target wrapping meson-build.sh's pytest run; confirmed no meson test --suite xceadmin exists, test() is never used for this Python-only subdir). Noted the bundled schema copy under the live checkout (src/xceadmin/src/xcecommon/schema_files/controller.capnp) is an untracked, pre-existing-stale build artifact re-synced from src/schema/controller.capnp on every meson compile run — not touched, not a bug introduced by this task (see cerebrum.md). Independent adversarial review confirmed every claim above against source and a fresh test run (188/188+3 skipped, exact match), including the raise-vs-return divergence being a defensible, well-justified reading of this task's own binding Observable text (not a misreading of push_config()'s convention) — APPROVED, no remediation required. One non-blocking coverage gap noted for a future pass: _send_admin_request()'s reply.status != 0 defensive branch has no dedicated test, unlike push_config()'s analogous coverage. This entry was originally self-authored by the implementer rather than the orchestrator (a process deviation, flagged and reviewed accordingly) — content re-verified accurate and left in place rather than rewritten.

• 10.4 Build the xceadmin scheduler dump command

  • Implement a local-file decoder that reads a replay file's message stream and formats it to human-readable text — no network connection involved
  • Observable: running dump against a fixture replay file produces text output listing the header fields, snapshot entry, and each event/session/decision in the file
  • Requirements: 5.20
  • Depends: 7.5
  • Verified: 2026-08-05 — dump(replay_file_path) (extends scheduler.py, task 10.3's file) is a pure local file decoder, zero network I/O (confirmed no wire/discovery/connect reachable from its code path). Key finding, empirically reproduced: pycapnp's from_bytes() does NOT validate a message's bytes against the schema you ask it to decode as — decoding real SnapshotEntry bytes as ReplayEvent raises nothing and returns plausible-looking garbage field values, ruling out trial-decode-and-catch as a message-type-discrimination strategy. Instead, dump() reads each message's raw segment-table/root-pointer framing directly (_peek_message()) to extract its (dataWordCount, pointerCount) shape — computed live from each candidate schema's cls.schema.node.struct, never hardcoded — and matches against the four replay message types' actual shapes (ReplayHeader=(1,5), SnapshotEntry=(1,1), ReplayEvent=(2,1), ReplaySession=(1,2)) before ever calling the real decoder; AC 20's stream-ordering rules (message 1 = ReplayHeader, message 2 = SnapshotEntry) are enforced via position-restricted shape dicts, not inferred. Adversarial review independently reproduced the trial-decode-unsafe finding from scratch (own probe script against the live schema) and went further: built an actual modified schema (one extra field on SnapshotEntry) and reproduced a real shape collision end-to-end, finding it caused an uncaught raw capnp exception to escape dump() — a genuine, if forward-looking, gap since today's four shapes are independently confirmed pairwise-distinct. Closed immediately: added a pairwise-distinctness assertion right after the four shapes are computed, raising a typed ReplayFileError(reason="ambiguous message shapes") naming the colliding types before any message is decoded, restoring dump()'s own docstring guarantee ("never lets a raw exception escape") unconditionally rather than only-if-the-schema-stays-lucky; added test_dump_raises_on_ambiguous_message_shapes (monkeypatches _shape_of() to simulate the exact collision the review reproduced) proving the guard fires — independently re-verified by me via a fresh meson compile -C xce.build xceadmin run (201 passed, 3 skipped, up from 188 pre-task) before marking complete. Test fixture (_build_full_replay_stream(), test_scheduler.py) is hand-built via pycapnp's write-side API since no .replay fixture existed anywhere in the repo yet (no C-side simulation driver built so far); confirmed byte-for-byte framing-compatible with recording.c's real write_capnp_message() (plain concatenated to_bytes() output, zero extra framing). Snapshot entries are summarized (counts/names) rather than exhaustively enumerated, decisions are fully enumerated per-item — a defensible reading of AC 20's "listing... the snapshot entry" (singular) vs. "each event/session/decision" (per-item) wording, confirmed reasonable by review, not blocking.
  • Boundary: XceAdminSchedulerCLI — not (P), same file as 10.3 (see 10.3)

• 10.5 Wire the xceadmin CLI commands

  • Add the scheduler command group with a nested record sub-group (start/stop/status commands) and a sibling dump command, each delegating to the functions built in 10.3/10.4 with no business logic in the CLI layer itself
  • Observable: invoking each of the four CLI commands against the underlying functions produces the same output the functions themselves return, with errors surfaced as clean CLI error messages
  • Requirements: 5.10
  • Depends: 10.3, 10.4
  • _Verified: 2026-08-06 — cli.py (design.md's exact target file) adds @cli.group() scheduler, @scheduler.group("record") scheduler_record (start/stop/status), and @scheduler.command("dump") scheduler_dump — matching the literal surface xceadmin scheduler record start [FILE]/record stop/record status/scheduler dump <FILE> exactly (verified via a live CliRunner --help smoke test at every level, not just read). CLI function names are prefixed (scheduler_record_start etc., not bare record_start) specifically to avoid shadowing the directly-imported xceadmin.scheduler.record_start/record_stop/record_status/dump functions they delegate to — click infers the CLI-visible command name from the decorator's explicit string argument, not the Python function name, so this is invisible in --help output (logged as a do-not-repeat in cerebrum.md: same bare name would compile fine but recurse infinitely at call time). Pure delegation + formatting only, no business logic: record_start/record_stop/record_status/dump are called unmodified; a new _format_record_result() renders RecordResult in requirements.md AC 10's exact "Status output" column-13 layout (pinned by a verbatim-reproduction test), shared by all three record subcommands per AC 10's "returned by recordStart/recordStop for confirmation" wording. Error handling: the existing except XceAdminError as e: raise click.ClickException(e.message) catch-all (config_push's precedent) is reused as-is for scheduler dump's ReplayFileError/SchemaNotFoundError; the three record subcommands add one narrower branch ahead of it — except SchedAdminError — since SchedAdminError's own docstring (task 10.3) explicitly documents that its recording/file/event_count state is meant to be surfaced richer than a bare message (e.g. "already recording" alone doesn't say which file); enriched only when e.recording is true (the only case with genuinely useful extra state — "not recording"/"scheduler not connected"/"request timeout"/"invalid record file name" all report recording=false and fall through to the plain reason string, matching precedent). OSError/KeyboardInterrupt handling mirrors config_push's existing branches verbatim (same underlying wire.connect socket pattern). New tests in test_cli.py (19 added, all monkeypatching xceadmin.cli's imported record_start/record_stop/record_status/dump bindings rather than a mock TLS server — full mock-server/redirect coverage is task 10.6's job): --help reachability at every group level, direct unit tests of the three formatting/error-translation helpers (incl. a verbatim requirements.md-example reproduction), argument/option parsing and delegation (explicit filename + --controller), success-output formatting for all three record actions, SchedAdminError enrichment vs. plain-message fallback, connection-timeout/connection-refused/KeyboardInterrupt all becoming clean ClickExceptions not raw tracebacks, and dump's exact-text echo (click.echo(text, nl=False), since dump() already ends its own text with \n) plus its error translation. Verified via the real meson compile -C xce.build xceadmin command (custom_target wrapping meson-build.sh's pytest run): 220 passed, 3 skipped (up from 201/3 pre-task; the 3 skips are the same pre-existing live-cluster E2E tests, unaffected) — zero failures, zero new skips. No linter/formatter is configured for this package (re-confirmed, matching task 10.4's prior finding) so none was invented/run.

• 10.6 xceadmin scheduler CLI tests

  • Test record start/stop/status against a mock TLS server (including redirect-following) and dump against a fixture replay file
  • Observable: the test suite passes for all four commands, including at least one redirect-following case
  • Requirements: 5.10
  • Depends: 10.5
  • Verified: 2026-08-06 — new test_cli_scheduler_mock.py (377 lines) closes the exact gap task 10.5 left open: real click.testing.CliRunner invocations of the actual cli entrypoint against real local TLS mock servers, zero monkeypatching of scheduler.py/cli.py bindings — distinct from test_scheduler.py's function-level mock-server tests (10.3/10.4) and test_cli.py's monkeypatched CLI tests (10.5). Covers all three record actions plus a genuine two-server standby→primary redirect handoff (test_cli_scheduler_record_start_follows_redirect_to_primary), a real SchedAdminError raise-site error path, and dump against a real on-disk .replay fixture built via pycapnp's write-side API. No production code changed (cli.py/scheduler.py untouched, confirmed via git diff). 227/227 passed, 3 skipped (baseline 220/3, +7 new, 0 failed, 0 new skips) via meson compile -C xce.build xceadmin. Review APPROVED first pass, zero findings. Closes out Group 10 (Integration: Controller Admin Opcode and xceadmin CLI) entirely — tasks 10.1-10.6 all complete.

• 11. Core: Simulation Driver

• 11.1 Implement simulation startup

  • Add the simulation-mode CLI flags (replay file path, start timestamp, speed, verify, alt-events file, output file, interactive) that the process entry point's mode-selection branch (built in 5.2) routes into the simulation driver
  • Open and validate the replay file (must begin with the header message), locate the most recent snapshot entry at or before the requested start time, load it, and rebuild picker state from it
  • Set the virtual clock's anchors from the loaded snapshot's timestamp and the requested speed
  • Reject a replay file whose first data message isn't a snapshot entry
  • Observable: starting simulation against a valid fixture file reaches a ready-to-run state with the virtual clock reporting the snapshot's timestamp; starting against a file missing its initial snapshot entry fails with a clear error before any event is processed
  • Requirements: 5.13
  • Depends: 8.1, 7.5
  • Verified: 2026-08-06 — new sim.h/sim.c implement AC 13 steps 1-4 (sched_sim_driver_start()): open+validate ReplayHeader, scan for SnapshotEntry messages tracking the most recent with ts <= --sim-start-ts, call xce_sched_core_load_snapshot(), set virtual clock anchors from core_clock.c's real field names. Step 5 (event loop) deliberately deferred to task 11.2 via an -ENOSYS placeholder in sched_sim_driver_run(). All 7 CLI flags (--sim, --sim-start-ts, --sim-speed, --sim-verify, --sim-alt-events, --sim-output, --interactive/-i) wired into main.c's existing task-5.2 mode-selection branch, replacing its -ENOSYS stub with real startup logic. Two real bugs found+fixed during TDD, both independently reproduced by review: a SnapshotEntry_get_ts() use-after-free (read through a capn_ptr after capn_free() — reviewer reverted the fix and reproduced a genuine ASan heap-UAF, then restored it) and a pend_updates[0]/[1] NULL-deref in xce_sched_core_fini() for any bare-core caller, worked around by allocating+dataq_init()ing them right after xce_sched_core_init(), matching scheduler.c's existing sched_init() pattern. Deliberately does not register a picker plugin (no xce_scheduler_t* available under AC 17 isolation) — relies on load_snapshot()'s own documented graceful degradation; a working picker becomes load-bearing only once 11.2 drives real sessions. Filenames (sim.h/sim.c, not design.md's literal sim_driver.h/sim_driver.c) and a concrete sched_sim_args_t* (not design.md's literal const void*) are documented, justified deviations preserving the mandated struct/function contract. 27/27 scheduler suite (26 baseline + 6 new sched_sim_startup_test sub-tests, individually ASan-clean), full project build clean, RED phase independently re-reproduced by review (moved sim.c aside, confirmed the same missing-file build failure). Review APPROVED with one non-blocking coverage gap noted (multi-snapshot ts-selection logic — correct by code trace and by the reviewer's own bug-reversion experiment, but not yet covered by a dedicated 2+-snapshot fixture test) and two FYI-only naming/signature deviations, both already justified in sim.h's header comment. Follow-up for a later task: --sim-start-ts/--sim-speed parse via strtoull(), which silently wraps a leading - into a huge unsigned value instead of a clear parse error.

• 11.2 Implement the simulation event loop and virtual clock advancement

  • Advance the virtual clock by jumping directly to each queued event's timestamp — no fixed tick, strictly monotonic, same-timestamp events processed in file order
  • Treat each session-boundary message as the trigger for one complete scheduling round: push accumulated events, start the session, drain decisions, end the session
  • Observable: replaying a fixture file with events at non-uniform timestamps processes them in file order with the virtual clock reporting each event's exact timestamp at the moment it's pushed
  • Requirements: 5.14, 5.18
  • Depends: 11.1
  • Verified: 2026-08-06 — new sim_loop.c (562 lines) implements sched_sim_driver_run() (replacing task 11.1's -ENOSYS placeholder): decodes .replay messages in file order, jumps the virtual clock to each ReplayEvent's exact timestamp as it's queued, and on each ReplaySession boundary runs a real core_push_events→core_go→drain decisions→core_end_session cycle before resetting the batch; mid-stream SnapshotEntry messages correctly skipped per AC 20. Discovered task 11.1 left no picker registered and no thread backing the core's cbufs, which would deadlock the decision-drain loop unconditionally the moment a real event loop existed — necessarily in-scope, not a convenience shortcut (independently confirmed by review). Fixed via new plugin_register_core() (plugin.c/plugin.h, registers a picker against a bare xce_sched_core_t*, no xce_scheduler_t* needed, precedent-matched to task 4.6's test_core_full_cycle.c) + builtin_fcfs_new_plugin() factory extracted from the existing register_builtin_fcfs() (plugins/builtin_fcfs.c) + a persistent picker-driving thread (sim_picker_thread_fn). Review independently verified the extraction is genuinely additive and behavior-preserving on the LIVE normal-mode path (32/32 whole-project tests including controller/libxce, not just the scheduler suite) — the only finding was a non-blocking doc-accuracy nit (plugin_register_core()'s doc comment overclaims parity with the legacy job_elect_fn adapter step, which it correctly omits since its only caller never uses that path). Since real resource-matching allocation is unreachable without a genuine xce_scheduler_t* (a pre-existing gap belonging to the not-yet-designed scheduler-allocation spec), sim_picker_thread_fn() conservatively PENDs every candidate rather than fabricating a DISPATCH — judged by review as honestly-scoped and sufficient for this task's narrow ordering/clock-timing Observable (AC 18's pseudocode drains "decisions" generically, not specifically dispatches), but flagged as a real gap once --sim's decisions need to be meaningful (task 11.4/scheduler-allocation's problem, not this task's). RED phase independently re-reproduced by review (reverted to -ENOSYS, confirmed real test failure, restored). 28/28 scheduler suite, 32/32 whole-project, clang-format clean. Non-blocking follow-up noted: the PEND-candidate loop itself has no direct test yet (both new tests exercise the "no resolvable picker" branch only) — worth covering when task 11.4 needs that path exercised for real.

• 11.3 Enforce simulation isolation

  • The simulation driver's state holds no live-controller client context, no cluster configuration, and no capnp socket of any kind — confirmed by the struct definition itself, not runtime checks alone
  • Observable: a build-time or structural check confirms no controller-facing symbol is reachable from the simulation driver's compiled object
  • Requirements: 5.17
  • Depends: 11.1
  • _Verified: 2026-08-06 — new test_sim_isolation.c (548 lines) enforces AC 17 across sim.h/sim.c/sim_loop.c in three layers, extending test_core_isolation.c's two-layer pattern rather than duplicating it: (1) source-text scan for forbidden #includes and forbidden-symbol call sites, reusing test_core_isolation.c's exact FORBIDDEN_SYMBOLS list; (2) a structural check test_core_isolation.c has no equivalent of — extracts only the typedef struct {...} xce_sched_sim_driver_t; body text (via a comment-stripping pass first, since sim.h's own boundary doc comment legitimately names xcecli_ctx_t/xce_cluster_config_t in prose) and scans that narrower region for forbidden field types; (3) the literal build/link-time proof — sim.c and sim_loop.c compiled together into a new isolated sched_sim_isolation_lib static_library (meson.build), inspected via nm(1)'s undefined-symbol table to prove what the compiled object actually calls, independent of transitive header declarations. Deliberately excludes the wall-clock ban (FORBIDDEN_WALLCLOCK) — out of this task's Req 5 AC 17 scope, and sim.c/sim_loop.c legitimately call clock_gettime() to seed/advance the core's virtual clock from the driver side, already documented at those call sites. 5/5 new sub-tests pass; 29/29 full scheduler suite passes; whole-project build clean. Independently reviewed and approved.

• 11.4 Implement decision verification

  • Compare each core-emitted decision against the corresponding recorded decision at the same logical position when verification is enabled; report any mismatch with the logical time and decision detail
  • Observable: replaying a fixture file with verification enabled against its own previously-recorded output reports zero mismatches; replaying against a deliberately-altered copy reports the exact mismatch
  • Requirements: 5.15
  • Depends: 11.2
  • _Verified: 2026-08-06 — new sim_verify.c (sim_decision_eq() + sim_log_mismatch()) and changes to sim_loop.c's ReplaySession branch implement AC 15/18: decode_ReplaySession_ptr() decodes the session's recorded decisions list (a deep-copied replay_session_t*, generated capnp typemap code already present from task 7.x, its first real consumer) before capn_free(), kept alive across the whole per-session drain loop and freed via free_ReplaySession_ptr() on every exit path. sim_decision_eq() is a genuine field-by-field structural comparison (xce_alloc_t/xce_host_alloc_t/xce_job_alloc_t for DISPATCH, xce_job_preason_t including nested host/resource reason lists for PEND; PREEMPT is defensively false-with-log, matching recording_session.c's own documented "not producible on any live path today" scope), not a jobid-only shortcut. Comparison runs positionally (index di) only when sim->verify is true (confirmed already fully wired end-to-end from --sim-verify through main.c→sched_sim_args_t→sim.c, no gap); both live-longer-than-recorded and recorded-longer-than-live length mismatches are surfaced, not silently dropped. New test_sim_verify.c (14 tests: 11 unit-level direct calls on sim_decision_eq/sim_log_mismatch + 3 integration-level through a real sched_sim_driver_run() fixture) covers both Observables: verify-against-self reports zero mismatches, verify-against-a-deliberately-altered copy reports the exact mismatch, verify=false reports nothing even when altered. RED phase independently reproduced by review (neutered sim_decision_eq to always-match, 8/14 tests failed including the altered-copy integration test, reverted, confirmed GREEN). 30/30 scheduler suite (was 29), whole-project build clean. Review APPROVED with two non-blocking FYI notes (a stale test-count label in a doc comment; sim_log_mismatch()'s signature intentionally drops AC 18 pseudocode's illustrative sim parameter and skips a wasteful allocate-then-discard from_replay_decision() conversion step, documented in-file, no behavioral gap against AC 15's actual contract).

• 11.5 Implement what-if branching and simulation output recording

  • Support loading the snapshot from a base replay file while replaying events from a separate alternate-events file
  • Write simulated decisions to an output replay file in the identical stream format as normal-mode recording, when an output file is requested; otherwise print decisions to standard output only
  • Observable: running with an alternate-events file produces decisions driven by the alternate events, not the base file's original events; the output file, when requested, is decodable by the same reader used for normal-mode recordings
  • Requirements: 5.16, 5.19
  • Depends: 11.2
  • _Verified: 2026-08-06 — AC 16: sim_loop.c's event/session read loop now selects sim->alt_events_file over sim->replay_file when the former is set (the initial snapshot, already loaded from the base file in task 11.1's startup, is untouched). AC 19: new sim_record.c writes a sim-run ReplayHeader (no cluster info — legitimate single-host branch write_replay_header already supports, sched_policy from sim->core.job_pick_plugin) + the loaded SnapshotEntry, then passes through each ReplayEvent and finalizes one ReplaySession per drain cycle, coexisting in the same loop with task 11.2's stdout default and task 11.4's --sim-verify. Kept the sim driver's zero-xce_scheduler_t* isolation intact (Req 5 AC 17) by extracting scheduler_t-free halves (sched_record_session_ctx_new()/_finalize()) out of recording_session.c's existing _begin()/_end() rather than giving any sim-side file a scheduler.h dependency — confirmed behavior-preserving for normal-mode's existing callers (scheduler record lifecycle test still passes) and confirmed via test_sim_isolation.c's extended 3-file (sim.c+sim_loop.c+sim_record.c) source-scan/struct-scan/nm(1) coverage. New test_sim_alt_events.c (AC 16) and test_sim_record_output.c (AC 19, decodes the sim-produced file via the same decode_* capnp typemap functions normal-mode recording round-trips against) cover both Observables. Round 1 review REJECTED solely on invalid self-reported RED-phase evidence (a git stash scope that hard-failed the build rather than reproducing the claimed 3 test failures) — round 1 explicitly confirmed the code itself was correct; round 2 fix produced genuine RED evidence via a direct in-place probe (hardcode the AC16 selector, no-op the AC19 write gates) showing 30/32 with real failures in exactly the two new tests, then reverted to 32/32. Round 2 review independently reproduced that same RED/GREEN cycle with its own probe and APPROVED. 32/32 scheduler suite (was 30/30), whole-project build clean.

• 11.6 Deterministic replay and normal-to-simulation equivalence test

  • Record a synthetic scheduling session through the normal-mode driver, then replay the resulting file through the simulation driver, and assert the decision sequences match exactly (same jobid, workerid, allocation amounts, pend reasons)
  • Observable: the equivalence test passes with zero decision-content divergence between the recorded and replayed runs — this is the spec's central regression test
  • Requirements: 5.7
  • Depends: 11.4, 9.2
  • _Verified: 2026-08-06 — new test_sim_equivalence.c (740 lines, no production code changes — this task chains two already-shipped pipelines) records a synthetic session (2 queues at different priorities, 4 jobs) through a real normal-mode xce_sched_driver_run_normal_mode_round() with recording active (task 9.2's sched_record_start()) to produce .replay file R, then replays R through sched_sim_driver_run() with --sim-output (tasks 11.1-11.5) to produce .replay file S, decodes only each file's ReplaySession.decisions via decode_ReplaySession_ptr() (deliberately never decoding ReplayHeader/SnapshotEntry/events, so the test structurally cannot assert on the timestamp/wall-clock metadata AC 7.1 explicitly excludes), and asserts R and S's decisions match in count, type, and full structural content (reusing sim_verify.c's xce_job_alloc_t/xce_job_preason_t field-by-field comparators, not a jobid-only shortcut). RED phase independently reproduced by review (corrupted a decoded jobid post-decode, confirmed FAIL, reverted, confirmed PASS). Honestly scoped to PEND-only decisions: independently confirmed (by review, reading sim_loop.c's sim_picker_thread_fn() directly, committed under task 11.2) that the simulation picker thread has no code path capable of emitting a live DISPATCH decision at all — a genuine, pre-existing structural gap predating this task (real resource-matching allocation belongs to the not-yet-designed scheduler-allocation spec), not a shortcut taken here; documented prominently rather than silently avoided. 33/33 scheduler suite (was 32/32), whole-project build clean. Review APPROVED, zero blocking findings.

• 12. Core: Interactive REPL

• 12.1 Implement REPL execution control

  • Implement a single input-loop REPL (not a separate thread) that pauses before the first event and blocks the simulation loop while active, supporting step-N-sessions, continue-to-next-breakpoint-or-EOF, and quit
  • Observable: starting simulation with the REPL enabled pauses before any event is processed; issuing a step command advances exactly the requested number of sessions and returns control to the prompt
  • Requirements: 5.22
  • Note: use readline() for the input loop's line editing (history, arrow keys) — xce_test's REPL already sets this precedent (src/controller/testsuite/test.c:3827) and readline is already a linked project dependency, zero new cost. Confirm readline's GPL licensing is acceptable for this build target before committing to it (vs. libedit/linenoise) if that hasn't already been settled elsewhere in the codebase. Resolved: switched project-wide to libedit (BSD) instead — see Implementation Notes.
  • Depends: 11.2
  • Verified: RED-phase reproduction performed during review round 2 — temporarily changed repl_should_pause() (repl.c) from return (rs->step_budget <= 0); to return false;, rebuilt, ran sched_repl_execution_control_test: 2/4 tests failed exactly as expected (test_should_pause_fires_on_exactly_the_nth_session, test_step_1_processes_exactly_one_session_then_quit_exits), the 2 unrelated tests (test_should_pause_never_fires_in_continue_mode, test_pre_replay_pause_blocks_before_any_event_is_processed) stayed green. Reverted; rebuilt; full scheduler suite 34/34 GREEN.

• 12.2 Implement REPL breakpoints and state inspection

  • Support setting/listing/deleting breakpoints on event type with optional field filters, and inspection commands reporting current simulation position, and job/queue/worker/resource/summary state
  • Observable: setting a breakpoint on a specific event type and field value pauses the REPL exactly when a matching event is about to be processed; each inspection command reports state matching what a snapshot dump of the same moment would show
  • Requirements: 5.22
  • Depends: 12.1
  • _Verified: 2026-08-07 — repl.c (task 12.1, 370 lines) split three ways under this project's 800-line cap (task 12.2's own feature surface did not fit in one file): repl.c (416 lines, execution control + dispatch, unchanged in scope), repl_break.c (breakpoint kind/filter matching, breakpoint CRUD, and the three new driver-facing hooks xce_repl_on_event()/xce_repl_on_begin_round()/xce_repl_on_decision()), repl_info.c (info job/queue/worker/resource/summary/session/break dumps) -- all sharing private state via a new repl_internal.h (the full xce_sched_repl_state_s definition, moved out of repl.c). sim_loop.c gained exactly 3 new bare call expressions (xce_repl_check_event()/xce_repl_check_begin_round()/xce_repl_check_decision(), repl.h inline wrappers gated on sim->interactive, same pattern as task 12.1's two existing hooks) at the three points design.md's pseudocode requires: per decoded ReplayEvent before it joins the accumulated batch, per ReplaySession boundary before the batch is pushed, and per drained decision after sim_print_decision() -- landed sim_loop.c at exactly 800/800 lines (trimmed its own header comment to make room). repl_should_pause()'s stale doc comment (which wrongly predicted this task would extend it) was corrected: breakpoints are a separate mechanism, never folded into the step N pause path. test_sim_isolation.c (task 11.3, already extended by 12.1) extended again to scan repl_break.c/repl_info.c/repl_internal.h (8-file precedent). New test_repl_breakpoints_inspection.c (17 tests, 4 groups: command parsing/listing, pure matcher predicates, state-inspection dumps against a real xce_sched_core_t, full-integration scripted-stdin pause proof for both an event breakpoint and a decision breakpoint). RED-phase: two REAL bugs surfaced and fixed during development (bug-1506: test fixtures calling repl_dispatch()'s break command without first calling xce_sched_core_init() left makeString()'s backing xstr_table uninitialized, silently returning NULL breakpoint fields instead of failing loud; bug-1507: repl_breakpoint_free() called a bare free() on a makeString()-produced xstring -- RTTI reference-counted, not a plain malloc() chunk -- causing a heap-corruption abort, fixed by never freeing those fields at all, matching kill_sched_job()/kill_sched_queue()'s existing "makeString()'d fields live for the process's lifetime" precedent). PLUS two targeted synthetic RED-phase probes reproduced independently after both real bugs were fixed: (1) repl_find_event_breakpoint() hardcoded to always return NULL -- rebuilt, ran the suite, exactly 3 of 17 tests failed (test_event_breakpoint_matches_jobid_filter, test_event_breakpoint_matches_state_filter, test_event_breakpoint_pauses_repl_at_matching_event), all others (including decision/begin_round matching and every info dump) stayed green; (2) repl_job_state_name() hardcoded to return a fixed bogus string -- exactly 1 of 17 failed (test_info_job_reports_state_queue_priority). Both probes reverted (grep-confirmed zero residue), rebuilt, reconfirmed green. Final: 35/35 scheduler suite (was 34/34), whole-project build clean. Remediation (2026-08-07): review found repl_info_job()'s resreq= field reported only a set/(none) presence flag instead of real content, despite format_resreq() (resreq.h) already existing and already used one file away by build_wire_job() (snapshot_wire.c) for this exact purpose; fixed by calling format_resreq() in repl_info_job() (mirroring build_wire_job()'s WARN-and-fallback error handling, freed after printing since this dump is transient), corrected the function's stale doc-comment claim that no print-form existed, and extended test_info_job_reports_state_queue_priority to assert the real formatted text ("Primary:"/"Tasks: min = 0, max = 0"/"Rusages:") appears instead of the old boolean flag. Still 35/35 scheduler suite.

• 12.3 Implement REPL snapshot capture and simulation parameter commands

  • Support writing a full snapshot to a file at the current simulation position, and adjusting speed/output-file/verify-mode from the prompt without restarting
  • Observable: a snapshot command produces a file that the simulation driver can later load via a fresh --sim invocation starting from that exact position
  • Requirements: 5.22
  • Depends: 12.1
  • Verified: 2026-08-07 — new repl_params.c (fourth sibling in the repl.c/repl_break.c/repl_info.c family, repl_internal.h's own "Why a split at all" section extended again): snapshot [FILE] and set speed|output|verify ... (repl_cmd_snapshot()/repl_cmd_set()), reusing sim_record.c's sim_record_write_header_and_snapshot() (task 11.5) unchanged for both the one-shot standalone snapshot and set output's live-stream preamble -- neither hand-rolls capnp encoding. set speed <N> mutates BOTH sim->sim_speed_num/sim_speed_den (the driver-level shadow copy) AND sim->core.speed_num/speed_den via the SAME atomic_store_explicit() pair sim_set_virtual_clock() (sim.c) uses at startup -- confirmed by reading core_clock.c's xce_core_now() that it reads the atomic pair fresh, uncached, on every call, so sim_loop.c needed zero new call sites (verified: sim->sim_speed_num/sim->sim_speed_den alone are never read by sim_loop.c or core_clock.c -- only the atomic core.speed_num/core.speed_den pair is; the shadow copy exists purely for the REPL/info config to read without reaching into core, per sim.h's own doc comment -- both fields are kept in sync by repl_set_speed() so this distinction never surfaces as a bug). info config (repl_info.c, alongside every other info dump) added as the read-only counterpart. snapshot's auto-generated filename (<basename>-snap-<event_index>.replay) uses a documented fixed basename fallback ("sim") since xce_sched_sim_driver_t retains no string form of the replay path (sim.h has only a bare FILE*) -- a deliberate, documented deviation matching repl_count_jobs()'s own AC22-pseudocode-deviation precedent, judged not worth adding a retained-path field to sim.h/sim.c for. sim_loop.c untouched, still exactly 800/800 lines. New test_repl_snapshot_params.c (7 tests): the central Observable proved end to end via a forked child (step 1/snapshot FILE/quit against a real fixture) followed by a SECOND, real sched_sim_driver_start() call in the parent against the file the child wrote, asserting success, an exact reloaded snapshot_ts (2000, frozen clock), the pre-session job present and the post-session job absent (proves the snapshot captured the position at command time, not the whole file); plus auto-filename generation, set speed/set verify state+malformed-input unit tests, set output open/preamble-readback/off round trip, info config field-format assertions, and a help listing regression check. Every other sim-family white-box test file (test_sim_equivalence.c/test_sim_event_loop.c/test_repl_breakpoints_inspection.c/test_sim_record_output.c/test_repl_execution_control.c/test_sim_alt_events.c/test_sim_verify.c) needed a one-line #include "../src/repl_params.c" addition alongside their existing repl.c include -- repl.c's repl_dispatch() now unconditionally references repl_cmd_snapshot()/repl_cmd_set(), the same "unconditional extern forces every white-box TU to add the new source file" pattern this file family already established. RED phase: initial build failed exactly as expected (meson setup error "File src/repl_params.c does not exist") after wiring the test file, repl_internal.h extern declarations, and both meson.build files but before writing the implementation; after implementation, a real bug surfaced and was self-fixed before commit (an accidental */ inside a prose doc comment -- xce_scheduler_t*/record_dir -- prematurely closed a block comment, a variant of this codebase's known recurring -Wcomment pitfall). Synthetic RED-phase probe on the finished implementation: removed both atomic_store_explicit() calls from repl_set_speed() (leaving only the shadow-copy assignment), rebuilt, ran the suite: exactly 1 of 7 tests failed (test_set_speed_updates_shadow_and_core_atomics, "Expected 7 Was 0"), all 6 others stayed green; reverted, rebuilt, reconfirmed clean. test_sim_isolation.c extended to scan repl_params.c too (9-file precedent, seventh sched_sim_isolation_lib archive member). Final: 36/36 scheduler suite (was 35/35), whole-project build clean.

• 12.4 REPL equivalence end-to-end test

  • Run a fixture file through the REPL using step/continue to completion and through batch simulation mode, and assert both produce the same decision sequence
  • Observable: both runs produce byte-identical decision sequences
  • Requirements: 5.22
  • Depends: 12.2, 12.3
  • _Verified: 2026-08-07 — TEST-ONLY task, no new production code. New test_repl_equivalence.c (683 lines) runs the SAME sched_sim_driver_run() (sim_loop.c) against ONE hand-built .replay fixture (two queues at different priorities "high"=5/"normal"=1, four jobs split across THREE ReplaySession boundaries) twice: once in batch mode (interactive=false, straight through), once REPL-driven (interactive=true, stdin freopen()'d from a scripted "step 1\ncontinue\n" -- exactly 2 lines, sufficient and safe regardless of session count since CONTINUE mode never re-reads stdin once entered, avoiding the readline()-EOF-triggers-exit(0) trap). This is a DIFFERENT comparison than task 11.6's test_sim_equivalence.c (which compares normal-mode driver vs simulation driver) -- here both sides are the identical simulation driver, differing only in sim->interactive. Both runs set sched_sim_args_t.output_path directly (confirmed unconditional on interactive by reading sim.c's args->output_path != NULL gate), so both write a full .replay-format decision stream to their own file with no set output REPL command needed. Since the fixture has THREE session boundaries (not task 11.6's one), decoding uses a new decode_all_sessions() that scans the whole output file structurally via sim.c's shared msg_root_has_shape() (ReplaySession's distinct (1,2) word/pointer-count shape) rather than a fixed message-skip count. Comparator reuses sim_verify.c's leaf-level structural comparators (sim_verify_job_alloc_eq()/sim_verify_job_preason_eq(), both static but visible via white-box #include) through a duplicated replay_decision_content_eq(), same technique test_sim_equivalence.c's own function of that name establishes. Non-tautology guards: session 1 asserted to have EXACTLY 2 decisions (an unambiguous floor -- only 2 jobs could possibly be pending that early), job 501 independently pinned as present in session 1's batch-side content, total batch decisions asserted >= 4, and every decision's type independently pinned as PEND (matching sim_picker_thread_fn()'s documented DISPATCH gap) BEFORE the cross-run equality check -- so a buggy always-return 1 comparator could not pass silently. Observed (not hand-predicted) that a PEND'd job persists as a picker candidate into later sessions (session 1: jobs 501/502 -> 2 decisions; session 2: 501/503/502 -> 3; session 3: 501/503/502/504 -> 4; total 9 on both sides), confirming the fixture is genuinely non-trivial without needing a fragile hardcoded per-session prediction. Synthetic RED-phase probe: analysis first showed that defeating repl_should_pause() (task 12.1's own already-verified probe target) would NOT turn this test red, because that only changes when stdin is read/control returns, never what gets processed -- every session still eventually completes regardless of pause timing, and this file's own CONTINUE-is-sticky script design tolerates any pause schedule. The correct probe for THIS invariant ("never alters ... what events are processed") had to touch content, and only under sim->interactive: temporarily added one line to xce_repl_on_event() (repl_break.c) -- which takes a non-const xce_core_event_t* and fires (via repl.h's sim->interactive-gated wrapper) strictly before an event joins the batch xce_core_push_events() later applies -- corrupting a JOBS_PUSH job's jobid (+= 1000000) whenever interactive. Rebuilt, ran the full scheduler suite: exactly 4/37 failed (this new test plus three pre-existing REPL tests that also exercise JOBS_PUSH under interactive mode -- sched_repl_execution_control_test, sched_repl_snapshot_params_test, sched_repl_breakpoints_inspection_test -- all correctly detecting the same real divergence), all 33 others stayed green. This test's own failure output: "REPL step/continue-to-completion must produce a byte-identical decision to batch mode at this session and position" (jobid 501 on the batch side vs 1000501 on the REPL side, directly visible in the two runs' printed PEND lines). Reverted (diff-confirmed byte-identical to the pre-probe file), rebuilt, reconfirmed 37/37 scheduler suite (was 36/36), whole-project build clean.

• 12.5 REPL live-operator-testing fixes (Requirement 5.22 amendment)

  • Fix the five gaps a live operator session surfaced (see requirements.md's AC 22 "Amendment (2026-08-19)" for the full rationale on each): (1) continue reaching EOF with no breakpoint hit re-enters the REPL instead of exiting the process; (2) step/continue issued after replay is already complete re-prompt with a message instead of being treated as "hand control to a driver loop"; (3) step/continue echo each input event (not just decisions) as it is processed, in real event/decision order; (4) info resource <name>|all reports resource definition metadata restricted to XCE_RESF_GLOBAL/XCE_RESF_SHARED resources, reading the shared value from core->sharedInst instead of aggregating host-based resources across every worker; (5) repl_split_tokens() strips one matching pair of surrounding quote characters from each token.
  • Observable: --sim --interactive against a replay file with no breakpoints set, driven by continue, stays at the (sim) prompt after "[sim] replay complete" instead of the process exiting -- info summary/info job/etc. still work at that point, and only quit (or stdin EOF) ends the process; a step while events are pending prints each event's own one-line summary before that session's decisions; info resource all lists only global/shared resources with metadata fields, never a host-based resource's fabricated total/available; info job "<jobid>" (quoted) parses identically to the unquoted form.
  • Requirements: 5.22
  • Depends: 12.4
  • Bugs: bug-1593 (info resource host-vs-shared), bug-1594 (continue/no-quit exit), bug-1595 (events not echoed in step/continue), bug-1596 (quoted REPL argument parsing)
  • _Verified: 2026-08-19 — Independently re-verified via /kiro-verify-completion (fresh evidence, not a prior self-report) across three layers. (1) Fresh build + full suite: xce-build.sh scheduler asan --incremental compiled clean; scheduler meson suite 41/42 (the one failure, test_recordstart_symlink_escape_replies_error, is the pre-existing unrelated bug-1560). Every task-12.5 unit test confirmed PASS by exact name in the fresh testlog: test_continue_with_no_breakpoints_stays_alive_after_replay_complete, test_continue_after_replay_complete_reprompts_instead_of_exiting, test_step_echoes_events_before_session_decisions, test_info_resource_all_lists_only_global_shared_resources, test_info_resource_named_lookup_reports_host_based_note, test_info_resource_reports_string_value, test_info_job_accepts_quoted_jobid_argument. (2) Live subprocess smoke test of the real xce_scheduler binary (fork/exec, real stdin/stdout, not the compiled-in test harness) for all five gaps: gaps 1/2 confirmed via a no-breakpoint replay driven by continue/step past completion -- process stays at the prompt, info summary still answers, quit exits 0; gap 3 confirmed via a fixture with QUEUE_UPDATE/JOB_PUSH/two JOB_STATE_CHANGE events -- all four event lines printed before [sim] replay complete, in file order; gap 5 confirmed via info job <id> issued unquoted, double-quoted, and single-quoted against a real pushed job -- byte-identical output all three ways; gap 4 confirmed via a hand-built .replay fixture (RESOURCES_PUSH defining a XCE_RESF_SHARED resource licenses and a host-based resource mem, RESVALS_FULL giving licenses=8.0) -- real captured output: info resource licenses → metadata + value=8.00 read from core->sharedInst; info resource mem → metadata + "host-based resource -- see 'info worker ' for per-host values", no fabricated value; info resource all → lists licenses only, never mem. (3) Direct source inspection confirmed all five fixes present and correct in current on-disk code: xce_repl_on_replay_complete()/repl_dispatch()'s replay_complete guard (repl.c), repl_print_event_summary() called before the breakpoint check in xce_repl_on_event() (repl_break.c), the GLOBAL/SHARED filter + sharedInst read in repl_info_resource_one() (repl_info.c), and the single-pair quote-strip in repl_split_tokens() (repl.c). First verification pass returned NOT_VERIFIED solely because this checkbox/line hadn't been written yet and one live check (gap 4) was still test-only; both closed by this pass.

• 13. Run full cross-phase HA soak regression

  • Run sequential and chaotic HA soak rounds against the complete feature (all three phases integrated), confirming zero job loss, divergence, or liveness stall attributable to this feature across the full run
  • Observable: the soak run completes with a clean pass across all rounds, matching the zero-regression bar set by the Phase 1 and Phase 2 soak runs
  • Requirements: 4.3
  • Depends: 12.4, 10.6
  • _Verified: 2026-08-07 — Full clean rebuild first (nix develop ./src -- bash src/tools/xce-build.sh all, profile=asan): 41/41 meson tests, 15/15 manual scheduler tests, all Go unit tests (apiserver/xcecli), all xceadmin pytest — zero failures, fresh xce_controller/xce_test binaries (mtime 13:12). Ran /xce-test-ha (.claude/skills/xce-test-ha/run-ha-soak.sh) against the complete integrated feature (all three phases — picker/core-driver/replay+REPL+simulation — live in one binary) in both required modes, 15 rounds each, fresh deploy dirs, /home/claude/tmp (non-overlay fs, dodges bug-924's overlayfs fdatasync wedge): sequential — PASS: 15 rounds, 48 jobs submitted, zero job loss; all 3 nodes led (ctrl-1,ctrl-2,ctrl-3); chaotic (overlapping kills inside election/catch-up windows, the bug-851-family stress mode) — VERDICT mode=chaotic rounds=15 jobs=48 committed_divergence=0 committed_job_loss=0 excluded_bug854_discovery=0 excluded_bug924_fdatasync=0 result=PASS. Zero job loss, zero committed-prefix divergence, zero liveness stall, zero excluded-bug incidents in either mode — matches the zero-regression bar of the sibling specs' Phase 1/Phase 2 soak runs (scheduler-allocation task 3, scheduler-preemption task 6). This closes out scheduler-core: all 45 tasks (1.1–13) complete.

Implementation Notes

• Task 1.3 added sched_set_default_job_pick_plugin() (schedobj.c) but deliberately did NOT call it from sched_init() — task 2.4 owns replacing scheduler.c:43's hardcoded "xce_sched_fcfs" literal and must also wire this defaulting call (or equivalent logic) into scheduler startup, or sched->job_pick_plugin stays NULL through a real boot.
• Background IDE diagnostics ("file not found: rtti.h", "unknown type name" for various xce_* types) have appeared consistently on every scheduler.h/plugin.h/types.h/schedobj.c change so far (tasks 1.1-1.3) and are confirmed stale clangd artifacts each time via independent real meson compile/meson test runs — not real build failures. Don't let these block a task; verify with the real build instead.
• Meson build dir is src/xce.build, not repo-root xce.build — build/test commands must run from src/ (e.g. cd src && nix develop . --command bash -c "meson test -C xce.build --suite scheduler").
• Scheduler testsuite convention: run test binaries directly with ASAN_OPTIONS=detect_leaks=0 (fixtures aren't torn down, by design — matches existing meson.build test registration).
• Task 2.3's review found a real concurrency gap for task 2.4 to close: once builtin_fcfs is wired live, next_job() (picker thread) and on_dispatch/on_pend (allocator thread) can race on the same unsynchronized fcfs_plugin_data_t private state — cbuf_get() unblocks the picker's next cbuf_put() before the allocator finishes processing the dequeued job (jbuf capacity is 1), so the picker can re-enter next_job() while the allocator is still mutating priv->queues via on_dispatch/on_pend. Today's production path never hits this because the allocator thread calls a different plugin category (limit.c's resource-limit hook), never fcfs.c's own private state. Task 2.4 must add a mutex guarding fcfs_plugin_data_t across all five picker hooks (or an equivalent handoff redesign) before registering builtin_fcfs as the live default.
• Task 2.4 closed the above race with fcfs_plugin_data_t.lock (mylock_t), taken by exactly 7 hooks (on_queue_update, on_job_push, on_job_state_change, on_job_update, begin_session, next_job, on_dispatch) — on_pend/end_session/init/fini correctly excluded (they never touch priv->queues/cursor/job_cursor), a verified refinement of this note's original "all five hooks" estimate, not a gap. sched_job_picker() now branches job_elect_fn != NULL (legacy plugins like xce_sched_fcfs, delegates unchanged to their own self-contained loop) vs NULL (native xce_job_picker_iface_t pickers, runs a new generic per-session loop) — this split exists because legacy job_elect_fn implementations (e.g. fcfs_elect_job) already contain their own internal cbuf_get(&wbuf), so a single uniform outer loop would double-consume the wbuf signal for legacy pickers. sched_allocator() now calls picker_iface->on_dispatch()/on_pend() after each allocation attempt (Req 1 AC 4), which was previously called for no picker at all — required for builtin_fcfs to actually remove dispatched jobs from its private state once live.
• Two follow-ups surfaced by task 2.4's review, not yet fixed (outside its file boundary): (1) main.c discards sched_init()'s return value, so a bad job_pick_plugin config fails sched_init() correctly but the process doesn't yet halt on it — needs a fast follow-up before relying on "fails startup" in production. (2) pthread_testcancel() in builtin_fcfs_next_job() is only at the loop entry/exit, not scattered through the queue/job walk (the walk holds fcfs_plugin_data_t.lock throughout, and this codebase has no pthread_cleanup_push precedent to safely unlock on mid-walk cancellation) — behaviorally inert today since sched_queue_is_ready/sched_job_is_ready are return 1 stubs, but becomes a real cancellation-latency gap once Phase 2 implements real filtering.
• Task 4.2's Req 5 AC 7 kvmgr iteration-order audit (scope: jobs/queues/users/projects, per design.md's bounded-scope note) found no unstable site — recorded durably as a comment in core.c above xce_sched_core_init(). One open gap surfaced but explicitly out of this task's scope: core->workers is iterated order-sensitively by sched_collect_workers() → seqjob_allocator.c's first-fit sched_try_alloc_on_hosts(), and design.md itself says the full AC 7 guarantee must close "before Phase 3 simulation work depends on it" — needs a follow-up audit/fix (or an explicit documented exemption) before Phase 3 (tasks 7+) starts relying on scheduling determinism across worker selection.
• Task 4.3 split its implementation into a NEW file core_events.c (not core.c, which design.md's File Structure Plan literally names) because implementing xce_core_push_events() directly in core.c would have pushed it to 977 lines, over this project's 800-line hard cap. core.h (the public contract) matches design.md's intent; only the .c file location differs. Worth reflecting in design.md next time it's touched. Also: test_core_isolation.c's nm(1)-based zero-controller-I/O boundary audit (Req 5 AC 3) covers only core.c, not core_events.c — manually verified clean (no netio/xceapi/client/delay/idgen usage) but not mechanically enforced; extending that test to cover core_events.c would be a reasonable low-risk follow-up before later tasks add more logic to this file.
• Same pattern compounded across tasks 4.4 (core_session.c) and 4.5 (core_clock.c, which also genuinely NEEDS to be outside the wall-clock-call scan since it's the one file allowed to call clock_gettime) — test_core_isolation.c now covers only core.c, while three sibling files (core_events.c/core_session.c/core_clock.c) sit outside its mechanical audit, each manually verified clean but not enforced by a test. Worth a dedicated follow-up task extending the forbidden-header/forbidden-symbol half of the audit (not the wall-clock half, which core_clock.c must stay exempt from) to all core_*.c files before Phase 3 adds more.
• Task 5.2's review surfaced a genuine, pre-existing bug in plugin.c's plugin_end_session() (predates this entire spec — git blame traces it to commit 638173dec, 2026-02-26; untouched by any scheduler-core task's diff): the per-plugin guard checks plugin->begin_session_fn != NULL but then calls plugin->end_session_fn(plugin) — a guard/call mismatch. Confirmed live impact: plugins/fcfs.c registers .begin_session_fn = NULL/.end_session_fn = fcfs_end_session, so fcfs_end_session() (resets the legacy job_elect_fn-style iteration cursor) is never invoked today. Only matters if a deployment's job_pick_plugin config selects the legacy xce_sched_fcfs name rather than the new default builtin_fcfs (both fields NULL on builtin_fcfs, unaffected). Out of scope to fix here — needs its own bug-log entry / small fix task, not folded into scheduler-core.
• Task 9.2's review found a non-blocking follow-up, not fixed in that task: sched_record_start()'s TOCTOU-hardened open() (recording.c, O_WRONLY|O_CREAT|O_NOFOLLOW) lacks O_NONBLOCK. A FIFO planted at the resolved path in the same race window the symlink/hard-link defenses already close (between sched_validate_record_filename() returning and open() running) would hang the calling thread indefinitely — a DoS (thread hang), not data corruption, since the existing st_nlink/S_ISREG checks still reject it once open() eventually returns. Adding O_NONBLOCK to the open flags is zero-risk for the regular-file case and closes this; worth a small follow-up before task 9.2's code sees production traffic.
• Task 9.3's review found a genuine task-breakdown gap, not fixable within 9.3's own boundary: record_snapshot_interval (the field task 9.3's periodic-checkpoint mechanism reads) has no CLI or admin-request wiring anywhere in the codebase — --record-snapshot-interval appears in AC 11's own normative text but in zero task's actual scope. Neither task 9.4 (admin request handler — dispatches SchedAdminRequest actions, which don't carry an interval field either, per controller.capnp's SchedAdminRequest{action, recordFile}) nor group 10 (controller-side dispatch) owns scheduler startup CLI flags; only main.c's argp table does, following task 9.1's own --record-dir precedent. As currently scoped, AC 11's mechanism is fully implemented and tested (via the struct field directly) but has no path to ever be configured non-zero in a real deployment. Needs a small follow-up task before this spec is considered complete: add a --record-snapshot-interval N CLI flag to main.c, wired to sched.record_snapshot_interval, mirroring --record-dir's exact pattern (task 9.1).
• Task 9.3's review also flagged test_record_lifecycle.c's size (826→1232 lines across tasks 9.2+9.3) as past this suite's established over-800 precedent (prior max test_driver_translate.c≈867) — the content itself is genuine, non-duplicative coverage (not padding), and a natural split point exists along the AC10 (lifecycle/TOCTOU)/AC11 (periodic-checkpoint/error-handling) seam by extracting shared fixture helpers into a common header. Non-blocking; worth a future cleanup pass, not urgent.
• Cross-task interaction, flagged before task 9.4 exists — resolution decided (not a fail/retry, a deferral): task 9.2's sched_record_start() calls xce_sched_core_save_snapshot() as its first action (AC 10 step 1) — the same function task 8.2 gated to return -EBUSY whenever session_active != 0. main.c's driver loop is single-threaded (while(1) { xce_sched_driver_run_normal_mode_round(&sched); ...; sleep(1); }); incoming requests (including the future SchedAdminRequest) arrive asynchronously via scheduler_req_handler (handler.c) on the I/O side, same as job/queue/worker updates. Decision: task 9.4 must NOT call sched_record_start() directly from the request-handling thread (that both risks racing session_active and violates this codebase's async-I/O rule against blocking/file-I/O in a request-handler callback). Instead, recordStart must be enqueued the same way job/queue/worker updates already are (pend_updates dataq or an equivalent dedicated pending-admin-request slot) and picked up by xce_sched_driver_translate_pending_events() — which already runs at the top of every round, strictly before begin_session(), so it is guaranteed between-sessions by the same construction argument design.md uses for core_push_events(). This means recording "takes effect" at the next scheduling round, not synchronously on request arrival. Consequence for task 9.4: since AC 10 specifies a synchronous request-reply carrying the resolved filename, the actual SchedAdminReply send must be deferred until the driver thread performs the queued start (not sent eagerly from the handler thread) — needs a completion signal back to the waiting request context (e.g. a condition variable, or replying once xce_sched_driver_translate_pending_events() reports the deferred start applied). Given this deferral makes the direct -EBUSY race effectively unreachable from the admin path, the AC 10 error-codes table doesn't need a new "busy" entry for this case — sched_record_start() keeps its -EBUSY return as a defensive/fail-loud contract check (task 9.2's job, no torn state), it just should no longer be a path task 9.4's dispatch can actually trigger by construction.
• Task 12.1's review found a licensing gap (fixed): the readline()-based REPL input loop originally linked GNU readline (GPL) directly into xce_scheduler, a shipped production daemon (unlike xce_test, a debug-only binary controller's own readline use was already confined to). Switched project-wide to libedit (BSD) — nixSupport/xce.nix, both meson.builds, repl.c, and src/controller/testsuite/test.c now use editline/readline.h (API-compatible: readline()/add_history()), no hand-written accessor changes needed anywhere.
• Task 12.1's review also surfaced bug-1504 (fixed) and bug-1505 (deferred, out of boundary) — see .wolf/buglog.json. Verifying the licensing fix required a full clean rebuild (new nix dependency), which exposed a genuine, pre-existing, deterministic SIGSEGV in write_SnapshotData unrelated to task 12.1's own repl.c: SnapshotData.internalState was a capnp Data field, and this codebase's capnpc-c backend silently no-ops Data-field encoding in generated encode_XXX() helpers without zero-initializing the caller's stack struct first — a real, reproducible UB crash, not a flaky test. Fixed (bug-1504) by switching internalState to List(UInt8) (controller.capnp, typemap.h, test_e2e_capnp.c) — this same capnpc-c backend DOES correctly auto-generate real encode/decode for List(UInt8) (proven precedent: FetchFileReply.content/bytesRead), eliminating the whole no-op class for this field with no hand-written accessor code. Re-verified with a full clean rebuild: scheduler suite 34/34, no regressions. While chasing this, also hit a second, unrelated instance of the same bug class in the controller's HA/WAL-catchup subsystem (WalCatchupEntry.payload, bug-847's own territory) — confirmed systemic (affects any :Data field in controller.capnp fed through the generic codec, not a one-off) but deliberately left unfixed (bug-1505) as out of scheduler-core's boundary; needs its own project-wide :Data-field audit as a separate task.
• bug-1505 fixed (2026-08-07, follow-up outside any specific task's boundary — controller/HA subsystem, not scheduler-core): before fixing, confirmed zero production exposure — xce_wal_batch_encode_raw()/wal_entry_to_wire() (request.c) never call the generic encode_WalCatchupEntry() at all; they memset() their own struct and call write_WalCatchupEntry() directly with every field hand-set, so the uninitialized-capn_ptr crash was 100% test-only exposure (only test_e2e_capnp.c calls the generic path). Fixed the same way as bug-1504: WalCatchupEntry.payload (@2) and batchPayloads (@10) switched from Data to List(UInt8) (new batchPayloadsLen @11 companion; payload reuses the existing wirePayloadLen @9), typemap.h's wal_catchup_entry_t field types updated, request.c's extract_data_bytes() widened to accept a raw capn_ptr (it already only used .p internally) so its 4 call sites work regardless of which Data-shaped wrapper the accessor returns now. Verified batchPayloads' single-top-level-blob shape means this does NOT reintroduce bug-847's per-element cross-segment far-pointer loss (that issue is inherent to PER-ELEMENT data regardless of Data vs List(UInt8) spelling — payload, the per-element field, was already legacy/unpopulated in production before this fix). Strengthened test_wal_catchup_append_term_fields_round_trip with real payload/batchPayloads content assertions (previously never exercised — the test's src.payload/batch_payloads were left NULL, so the crash occurred on content the test didn't even care about). Verified: controller_unit_test no longer crashes (3/3 controller suite), scheduler suite unaffected (37/37, unrelated schema area).
• Feature-level validation (/kiro-validate-impl, 2026-08-07) — GO, with 2 new non-blocking follow-ups found:
  1. Req 5 AC 3's tlog logging-integration bullet is built but never activated. core_clock.c's xce_core_register_tlog_format()/xce_core_set_active_for_logging() (task 4.5) are unit-tested but have zero call sites in main.c/sim.c/driver.c — task 4.5's own header comment explicitly deferred this ("this task does not wire it into mylog_init()/main.c; that remains a later task's job"), but no later task in this spec ever picked it up. Root-caused where it should wire in: NOT main.c's normal-mode path — normal mode seeds virtual_clock_base=0, real_clock_base=0 (core.c:139-140), so xce_core_now() there returns raw CLOCK_MONOTONIC nanoseconds-since-boot, not epoch time; activating it for normal-mode logging would make log timestamps worse, not better. The correct site is sim.c's sched_sim_driver_start(), right after sim_set_virtual_clock() (line 394) — virtual_clock_base is seeded to snapshot_ts there (a real epoch-ns value from the .replay file), so xce_core_now() becomes a meaningful simulated timestamp. Left unfixed here rather than patched under validation-gate time pressure because activation needs a matching deactivation: g_active_core_for_logging is a single process-wide pointer, sched_sim_driver_run()'s own header comment already documents "no sim-driver stop/fini function exists" (deliberate, accepted gap) with multiple early-return paths, and several test binaries (e.g. test_repl_equivalence.c) construct more than one stack-allocated xce_sched_sim_driver_t per process — activating without a paired deactivation risks a real ASAN stack-use-after-scope hazard the very next time an unrelated log call fires in that process. Needs a small dedicated follow-up task: wire activation into sched_sim_driver_start() AND deactivation into every sched_sim_driver_run() return path (or add the fini function its own comment says doesn't exist yet).
  2. plugins/fcfs.c vs plugins/builtin_fcfs.c — functionally correct, undocumented. design.md (component description) says fcfs.c is "rewritten (Phase 1) to xce_job_picker_iface_t, drops job_elect_fn." The actual implementation instead added a new parallel file, plugins/builtin_fcfs.c, and left fcfs.c untouched still registering the legacy job_elect_fn (fcfs.c:353) — confirmed deliberate, not an oversight: scheduler.c:466's own comment says the new picker is registered "alongside the untouched legacy xce_sched_fcfs", and task 1.2's compatibility-wrapper requirement needs exactly this kind of legacy job_elect_fn-only plugin to exist as its live exercise target. builtin_fcfs is confirmed the registered default (schedobj.c:364). Functionally sound; the only gap is that tasks 2.1-2.6 carry no _Verified: annotations (unlike every other task group) and design.md's literal text was never updated to record this file-layout decision — a documentation/traceability gap, not a defect. Worth a follow-up: add _Verified: notes to 2.1-2.6 and correct design.md's component description.
  • Both items are cosmetic/traceability, not correctness or determinism defects — the soak, equivalence, and integration evidence for this validation pass (15-round sequential + 15-round chaotic HA soak, both zero-loss/zero-divergence; full rebuild 41/41 meson + 15/15 manual scheduler + all Go/xceadmin tests green; independent requirements-coverage/design-alignment/cross-task-integration subagent passes) is unaffected by either.
  • Status update (2026-08-19): Follow-up 1 (tlog activation) is now FIXED — see design.md's Open Questions/Risks #6. Follow-up 2 is FULLY CLOSED: design.md's component description was corrected the same day (2026-08-07, see design.md's own Open Questions/Risks #7), and tasks 2.1-2.6 were backfilled with _Verified: lines the next day (commit d7b49690, 2026-08-08) — an earlier revision of this status-update line incorrectly claimed the backfill was still open without checking; corrected after fresh verification confirmed both the _Verified: lines and the underlying builtin_fcfs.c implementation they describe are present and accurate (60/60 picker/fcfs-related tests green on a fresh rebuild, 2026-08-19).