myxce/.kiro/specs/scheduler-core/tasks.md

Implementation Plan

• 1. Foundation: Picker Interface & Plugin Registration

• 1.1 Define the job picker plugin contract

  • Add the lifecycle hooks (init, fini), state-change hooks (on_queue_update, on_job_push, on_job_state_change, on_job_update), session hooks (begin_session, next_job, end_session), post-allocation hooks (on_dispatch, on_pend), and optional hooks (on_config, on_job_rusage MAY be NULL) to the picker plugin header
  • Document the contract: every hook signature, which thread each hook category runs in (driver/picker/allocator), cancel-safety requirement for next_job(), and session-reset requirement for begin_session()
  • Observable: the header compiles standalone and defines a complete function-pointer struct matching every hook category
  • Requirements: 1.2, 1.5, 2.6

• 1.2 Register the picker plugin category and deprecate the old election hook

  • Add a picker plugin registration category to the plugin registration table, parallel to existing plugin categories
  • Add a compatibility wrapper that adapts an old-style job_elect_fn-only plugin to the new interface, logging a deprecation warning at registration time
  • Observable: registering a plugin that implements only job_elect_fn succeeds via the wrapper and logs a deprecation warning; registering one that implements the new interface directly succeeds without a warning
  • Requirements: 1.6, 2.5
  • Depends: 1.1

• 1.3 (P) Add picker configuration fields

  • Add a global picker plugin name field to the scheduler, defaulting to builtin_fcfs when unset
  • Add a per-queue picker plugin override field, defaulting to NULL (inherit global)
  • Add a project-name field to the queue struct, and extend the project struct with a priority field (scaffolding only — no quota fields)
  • Observable: a queue config with no override inherits the global picker name; a queue config with an override reports that override, not the global default
  • Requirements: 2.1, 2.2, 2.3, 2.4
  • Boundary: SchedulerCore config fields

• 2. Core: Builtin FCFS Picker

• 2.1 Implement builtin_fcfs lifecycle and private state

  • Implement init/fini: allocate/free the picker's private ordered-queue state (queue ordering by priority desc, and one job-ordering structure per queue)
  • Never query the scheduler's own queue/job tables from inside the picker after init — all candidate state lives in the picker's private structures
  • Observable: after init(), the picker holds an empty private candidate set; after fini(), all picker-owned memory is released (verified under a leak-checked test run)
  • Requirements: 3.1, 3.3
  • Depends: 1.1, 1.2

• 2.2 Implement builtin_fcfs state-change hooks

  • on_queue_update: upsert a queue into the picker's ordered queue list, keyed by descending priority
  • on_job_push: add a newly-pending job into its queue's candidate set, ordered by priority desc then submission/queue time asc
  • on_job_state_change: remove a job from candidates when it leaves pending state, re-add it when it returns to pending
  • on_job_update: reposition a job in its candidate set when its priority or queue assignment changes
  • Observable: feeding a sequence of these hooks directly (no live core) produces a candidate set whose order matches the documented ordering rule, verifiable by a direct hook-driven test
  • Requirements: 3.2
  • Depends: 2.1

• 2.3 Implement builtin_fcfs session and post-allocation hooks

  • begin_session: reset the iteration cursor to the head of the ordered queue list
  • next_job: walk queues then jobs in ordering, skipping not-ready queues/jobs, returning the first ready job and advancing the cursor; returns NULL when exhausted
  • end_session: no-op (iteration state already reset in begin_session)
  • on_dispatch: remove the job from its queue's candidate set permanently
  • on_pend: no-op — job remains in its candidate set for the next round in the same position
  • Observable: a full begin_session → repeated next_job → end_session cycle against a fixed candidate set returns jobs in (queue priority desc, job priority desc, queue_time asc) order and returns NULL exactly once all candidates are exhausted
  • Requirements: 3.2, 3.4, 3.6
  • Depends: 2.2

• 2.4 Wire builtin_fcfs as the registered default and remove the hardcoded picker string

  • Auto-register builtin_fcfs at scheduler startup under the name "builtin_fcfs" and set it as the default picker plugin
  • Replace the hardcoded "xce_sched_fcfs" literal with a lookup of the configured picker plugin name through the existing plugin registration/lookup mechanism; fail scheduler startup with a clear error if the configured name isn't registered
  • Confirm the existing three-thread/cbuf pipeline (picker thread, allocator thread, driver-signaled wbuf/jbuf/decisions cbufs) is unchanged by this wiring — no new threads, no cbuf capacity changes
  • Add cancel points in next_job()'s iteration loop and confirm the picker thread's existing block-on-signal point remains cancel-safe
  • Observable: starting the scheduler with no picker config uses builtin_fcfs automatically; starting it with an unregistered picker name fails startup with a clear error instead of silently falling back
  • Requirements: 1.3, 1.4, 1.6, 2.5, 3.1, 3.5
  • Depends: 2.3, 1.3

• 2.5 Remove ordering logic from the core scheduling loop

  • The core loop calls only picker->next_job() to obtain the next schedulable job — no project/queue/job iteration or ordering logic remains in the loop itself
  • Observable: a code-level check confirms the scheduling loop function contains no direct queue/job traversal — all selection reads happen through the picker interface
  • Requirements: 1.1, 1.7
  • Depends: 2.4

• 2.6 Picker regression tests and data-sharing invariant checks

  • Assert builtin_fcfs's next_job() ordering matches today's pre-change FCFS ordering exactly, given identical queue/job state
  • Confirm the existing scheduler unit test suite passes unmodified except at the specific assertions that exercised the removed hardcoded string and the decision-publishing path this feature replaces
  • Verify resreq objects are never mutated in place by any touched code path, all pend-reason allocation goes through the existing factory function (no direct calloc of the reason struct), comparisons use pointer equality as a fast path before deep comparison, and the existing per-session select-expression memoization is untouched
  • Observable: the regression suite passes with the documented ordering rule as its oracle, and a grep-based or code-review check confirms no new calloc(sizeof(...pendreason...)) call site was introduced
  • Requirements: 4.1, 4.2, 6.1, 6.2, 6.3, 6.4
  • Depends: 2.5

• 3. Run Phase 1 HA soak regression

  • Run sequential and chaotic HA failover soak rounds against a cluster running the new picker architecture
  • Observable: zero new job loss, decision divergence, or liveness stall attributable to this phase's changes across the soak run
  • Requirements: 4.3
  • Depends: 2.6
  • Verified: 15-round sequential soak, 2026-08-03 — PASS: 15 rounds, 48 jobs submitted, zero job loss; all 3 nodes led (ctrl-1,ctrl-2,ctrl-3)

• 4. Foundation: Scheduler Core Extraction

• 4.1 Extract the scheduling core from the monolithic scheduler struct

  • Split scheduling state (jobs, queues, workers, resources, resvals, users, projects, plugin instances) into its own core structure holding zero controller I/O state (no API client context, no cluster config)
  • Confirm the core links only against the approved minimal data-structure subset and never links network/timer/API libraries — no wall-clock time call appears anywhere in the extracted core's source
  • Observable: the core structure compiles and links with zero symbols from the networking/API libraries; a build-time or link-time check confirms this
  • Requirements: 5.1, 5.2
  • Depends: 3
  • Verified: 2026-08-03 — new xce_sched_core_t (core.h/core.c) holds all scheduling state, xce_scheduler_t shrank to {core; apictx; cluster_cfg}; nm-based isolation test (test_core_isolation.c) proves zero netio/xcecli/idgen/delaymgr symbols; 14/14 scheduler suite passing; independent review APPROVED (own fresh build/test/nm run + reproduced RED)

• 4.2 Pin the determinism invariants

  • Hardcode the picker-to-allocator handoff channel's capacity to 1 inside core initialization — never expose it as a parameter
  • Audit every hash-table/key-value-store iteration reachable from the scheduling loop (jobs, queues, users, projects) and confirm each produces order deterministic in (keys, insert/delete sequence) with no process-random seeding; fix any site found unstable
  • Observable: core initialization rejects (or simply has no code path to accept) a caller-supplied handoff-channel capacity; the iteration-order audit's findings are recorded and any fix is covered by a targeted test
  • Requirements: 5.7
  • Verified: 2026-08-03 — jbuf capacity hardcoded to literal 1 (core.c), regression test test_core_jbuf_capacity_pin.c; kvmgr audit found jobs/queues/users/projects already fully order-stable (xzarray ascending-slot walk, no process-random seeding), no fix needed, findings recorded durably as a comment in core.c above xce_sched_core_init(); 15/15 scheduler suite passing; two-round review (round 1 REJECTED on missing durable record only, round 2 APPROVED after fix); core->workers flagged as an open follow-up (see Implementation Notes), not this task's scope
  • Depends: 4.1

• 4.3 Implement the core event interface and batch event push

  • Define the typed core event representation (timestamp plus type-specific payload) covering configuration push, job push, worker push, resource push, resource-value full/delta push, job state change, job update, queue update, and job resource-usage events
  • Implement batch event push: state-change picker hooks fire synchronously for each event in the batch, valid only when no scheduling session is active
  • Observable: pushing a batch of events with the picker idle fires the corresponding state-change hook exactly once per event, synchronously, before the push call returns
  • Requirements: 5.4, 5.8
  • Depends: 4.1
  • Verified: 2026-08-03 — xce_core_event_type_t (9 kinds) + xce_core_event_t in core.h; xce_core_push_events() in new core_events.c (split out of core.c to stay under the 800-line cap — design.md's file-structure sketch not updated yet); rejects -EBUSY mid-session, fires each hook exactly once synchronously (JOBS_PUSH once per job); additive only, not wired into the live path (task 5.1/5.2's job). Round 1 REJECTED: CONFIG_PUSH didn't decompose queues/resources/resvals like sched_init_config() does — fixed round 2 (core_apply_config_resources/resvals/queues + shared core_upsert_queue(), also used by QUEUE_UPDATE), round-2 review APPROVED with live RED→GREEN reproduction. 16/16 scheduler suite passing.

• 4.4 Implement the session-start signal and decision callback protocol

  • Implement the non-blocking session-start call that signals the picker thread and returns immediately, rejecting a call while a session is already active
  • Define the three decision output shapes (dispatch, pend, preempt) and the decision callback invoked once per decision as the allocator produces it, plus the explicit end-of-session marker
  • Observable: starting a session while one is already active is rejected with a distinct error; each decision reaches the registered callback in allocator-output order, followed by the end-of-session marker
  • Requirements: 5.5, 5.6
  • Depends: 4.3
  • Verified: 2026-08-03 — xce_core_go()/xce_core_end_session() (new core_session.c) implement the non-blocking session-start signal (-EINVAL on reentrant call, proven a true no-op via cbuf_tryget) and end-session cleanup; xce_core_set_decision_cb() registers the driver's on_decision callback (separate setter, not an xce_sched_core_init() param, to avoid touching its live sched_init() call site); XCE_DECISION_PREEMPT added to decision.h matching the DISPATCH/PEND/PREEMPT AC 6 table exactly. Decision-drain loop confirmed driver-owned (not a core function) via 3 independent requirements.md/design.md citations, re-derived independently by review. Additive only, sched_go/sched_begin_session/sched_end_session/sched_allocator/main.c untouched. REJECTED round 1 on a clang-format alignment regression in core.h (mid-struct insertion de-aligned 24 pre-existing lines) — fixed with clang-format -i, whitespace-only, re-verified 17/17 green.

• 4.5 Implement clock virtualization and session timing

  • Implement the virtualized clock accessor using the two-anchor-plus-speed formula, backed by atomically-updated fields; no direct wall-clock call anywhere else in the core
  • Register the core's own log-timestamp formatter, bound to the virtualized clock accessor instead of the logging library's default wall-clock read — this is the one third-party dependency inside the core that would otherwise call wall-clock time on every log line, silently reintroducing the exact dependency task 4.1 confirmed absent at link time
  • Track session start/end timestamps internally and compute session duration for diagnostics, exposed through the session summary
  • Observable: the clock accessor's output matches the documented formula for all four operating modes (normal, discrete simulation, real-time simulation, fast/slow simulation) given fixed anchor values; a log line emitted while the virtual clock is frozen at a fixed simulated timestamp shows that timestamp, not the real wall-clock time at the moment of the call
  • Requirements: 5.3, 5.12
  • Verified: 2026-08-03 — xce_core_now() (new core_clock.c, kept out of core.c so test_core_isolation.c's wall-clock scan still passes) implements the exact two-anchor+speed formula; all 4 modes proven with fixed anchors, no sleep-based flakiness. requirements.md's literal tlog_reg_format_func(fn, core_ptr) call shape doesn't exist in the real vendored tlog API (verified against tlog.h: registration takes only the callback, userptr is per-call-site) — resolved via a module-level active-core pointer read by a new core-scoped format function, not wired into mylog_init()/main.c yet. session_start/session_duration_ns stamped correctly in xce_core_go()/xce_core_end_session() (task 4.4's behavior otherwise untouched). 18/18 scheduler suite passing, clang-format clean, review APPROVED first pass. Follow-up noted: core_clock.c is the 3rd core-adjacent file (after core_events.c, core_session.c) outside test_core_isolation.c's nm-based audit — manually verified clean, not mechanically enforced.
  • Depends: 4.1

• 4.6 Core contract tests

  • Test the full push-events → start-session → drain-decisions → end-session cycle against a stub picker (no driver needed), asserting hook firing order and timing
  • Observable: the contract test suite passes, exercising both the batch-push precondition (rejects mid-session) and the double-session-start rejection
  • Requirements: 5.1, 5.2
  • Depends: 4.4
  • Verified: 2026-08-03 — new test_core_full_cycle.c drives push_events→go→drain→end_session against a real picker pthread (running the real xce_job_picker_iface_t hooks on a test-local stub picker) plus a main-thread stand-in allocator (sched_allocator() is xce_scheduler_t*-coupled, out of pure-core scope) calling real on_dispatch/on_pend/make_dispatch_decision/make_pend_decision. Both preconditions (-EINVAL reentrant core_go, -EBUSY mid-session push_events) proven against a genuinely active session, not a hand-set flag, and proven to un-latch after the session ends. Two live RED→GREEN reproductions (neutered each guard in turn), both independently reproduced by review. 19/19 scheduler suite passing, clang-format clean, 30/30 stress runs with zero flakiness, review APPROVED first pass. Zero production files touched — test-only, as scoped.

• 5. Core: Normal-Mode Driver

• 5.1 Build the normal-mode driver's event translation path

  • Translate controller capnp push/notify messages into core events, buffering them in a pending-event queue between scheduling rounds
  • Observable: a sequence of simulated controller push/notify messages, once drained, produces a core event batch whose contents match the input messages one-to-one
  • Requirements: 5.8, 5.9
  • Verified: 2026-08-03 — new driver.h/driver.c: xce_sched_driver_translate_pending_events(xce_sched_core_t*) drains pend_updates via the existing dataq-swap pattern, translates all 10 op types to xce_core_event_t (field-verified against core_events.c's actual consumption, not assumed), stamps via xce_core_now(), pushes through the real xce_core_push_events(). Deliberately scoped to xce_sched_core_t* alone (not the full design.md xce_sched_driver_t) since pend_updates already lives on core since task 4.1 and this task is translation-only — task 5.2 can wrap trivially with &driver->core. handler.c untouched. Two flagged risk items (RESVALS zarray dangling-pointer shape, zero-filled changed_fields/exit_code for fields the legacy payload can't supply) both independently verified as pre-existing/honestly-inert, not new bugs. 20/20 scheduler suite passing, real-ASan clean, clang-format clean, review APPROVED first pass.
  • Depends: 4.4

• 5.2 Wire decision publishing, the mode-selection entry point, and retire the hardcoded decision switch

  • Publish each decision to the controller immediately as it's produced (dispatch, pend-reason, preempt) via the existing client API — not batched at session end
  • Replace the scheduler process's hardcoded decision-type switch statement with registration of the driver's decision callback
  • Add the process entry point's mode-selection branch (normal-mode driver vs. simulation driver based on whether a replay file was given) and normal-mode's own CLI flags
  • Observable: submitting a synthetic session that produces a mix of dispatch and pend decisions results in each being published to the (simulated) controller in the order produced, with no batching delay; starting the process with no replay file argument runs the normal-mode driver
  • Requirements: 1.7, 5.6, 5.9
  • Depends: 5.1
  • Verified: 2026-08-04 — main.c's live loop rewired: xce_sched_driver_translate_pending_events() -> xce_core_begin_session() -> plugin_begin_session()+sched_account_jobs() -> xce_core_signal_go() -> decision-drain calling the registered xce_core_decision_cb_t -> plugin_end_session() -> xce_core_end_session(), all in the new exported xce_sched_driver_run_normal_mode_round() (driver.c). xce_core_go() (task 4.4) split into xce_core_begin_session()/xce_core_signal_go() as a pure decomposition (test_core_session.c unmodified, still passing) specifically so plugin_begin_session()/sched_account_jobs() can run in the exact safety-critical window (after resval reset, strictly before the picker is signaled) that task 4.4 deliberately excluded from pure-core scope. Empirically proven, not just documented, by test_driver_session_ordering.c (real cross-thread happens-before assertions) with a live RED->GREEN reproduction, independently reproduced by review. xcecli_preempt() confirmed not to exist anywhere in client.h; PREEMPT decisions fail loudly (-ENOSYS, logged) rather than fabricating a client API. --sim CLI flag added (parse-only, loud not-yet-implemented failure, since SimDriver is Phase 3). 22/22 scheduler suite passing, --help smoke check clean, review APPROVED after one clang-format fix. Found (not introduced, not fixed, flagged for follow-up) a genuine pre-existing plugin_end_session() guard/call mismatch bug in plugin.c predating this spec — affects the legacy xce_sched_fcfs picker's end-of-session cursor reset, not builtin_fcfs.

• 6. Run Phase 2 HA soak regression

  • Run sequential and chaotic HA soak rounds confirming decisions and controller interaction match Phase 1's behavior exactly
  • Observable: zero new job loss, divergence, or liveness stall; decision content matches Phase 1's soak baseline
  • Requirements: 4.3
  • Depends: 5.2, 4.6
  • Verified: 2026-08-04 — sequential HA soak (xce-test-ha skill), 15 rounds, 48 jobs submitted, zero job loss, all 3 nodes led as primary (ctrl-1/ctrl-2/ctrl-3), committed-prefix walcompare consistent every round. Confirms tasks 4.1-5.2's src/scheduler/ rewrite (core extraction + normal-mode driver wiring into main.c) introduced no controller-side regression — consistent with every task's independent review confirming zero src/controller/ changes throughout Group 4/5. Gates Phase 2 -> Phase 3.

• 7. Foundation: Replay Wire Format

• 7.1 Define the replay stream envelope schema

  • Add the replay header, snapshot entry, replay event, replay session, and replay decision wrapper message types to the shared wire schema, following the single-stream-of-consecutive-messages framing
  • Observable: the schema compiles and a minimal header-plus-snapshot-entry byte stream round-trips through encode/decode
  • Requirements: 5.20
  • Verified: 2026-08-04 — ReplayCluster/ReplayHeader/SnapshotEntry/ReplayEvent/SessionSummary/ReplaySession added fully; SnapshotData (empty {}) and ReplayDecision ({ts only}) added as compiler-mandated placeholders task 7.3 fills in (real fields for SnapshotData wholesale-replaced, a body union appended to ReplayDecision at new ordinals) since capnp's $C.codecgen has no per-struct opt-out from requiring the mapname'd C type to exist. ReplayEvent's union deliberately holds only the 9 members with pre-existing backing types (configPush/jobsPush/workersPush/resourcesPush/resvalsFull/resvalsDelta/queueUpdate/jobRusage/endSession); jobStateChange/jobUpdate deferred to task 7.2 as new ordinals — proven compiler-mandatory (not stylistic) by review, which independently confirmed capnp rejects both the AC 20 pseudocode's literal union syntax and any ordinal gap. Round-trip test (test_e2e_capnp.c) proven real via full clean rebuild (deleted+regenerated controller.capnp.{c,h}) both by implementer and reviewer. Full project build clean (controller/apiserver-CGO/scheduler/worker/xceadmin), 1075/1075 controller_unit_test passing, review APPROVED first pass with reviewer going beyond implementer's own verification (real capn_resolve()-omission and placeholder-removal breakage reproduced firsthand). Zero existing structs/fields touched.
  • Depends: 6

• 7.2 Define new event payload types

  • Add the two genuinely new event payload message types (job state-change delta, job update delta with embedded full record) to the wire schema; every other event payload type reuses an existing wire message already used elsewhere in the schema
  • Observable: encoding a job state-change event and a job update event and decoding them back yields field-for-field equality with the original values
  • Requirements: 5.20
  • Depends: 7.1
  • Boundary: ReplayFormat — not (P): 7.2/7.3/7.4 all edit the same schema file (controller.capnp); run sequentially to avoid concurrent edits to one file, even though each adds a logically disjoint set of structs
  • Verified: 2026-08-04 — JobStateChangeEvent/JobUpdateEvent added to controller.capnp matching design.md's spec verbatim; JobUpdateEvent.job correctly reuses the full 12-field Job/xce_job_t record (not the narrower SchedJob sharing the same mapname — verified by field list). ReplayEvent.body's union extended with jobStateChange@10/jobUpdate@11, the only two valid next ordinals per task 7.1's baseline (capnp requires no gaps) — confirmed via a from-scratch capnp regeneration. Existing @0-@9 fields byte-identical. Deep round-trip tests (non-default values throughout, including 3-level nested Job substructs) prove the Observable; JOB_F* bitmask values deliberately left undefined (later task's job). 1077/1077 controller_unit_test, full project build clean, review APPROVED first pass._

• 7.3 Define decision payload types and the snapshot schema

  • Add the dispatch, pend, and preempt decision payload message types, reusing existing allocation and pend-reason wire messages where they already carry the needed fields
  • Add the full snapshot message type covering configuration, jobs, queues, workers, resources, resource values, users, and projects, reusing existing wire messages for each domain object list; represent the two scheduler-internal-only caches and session/PRNG state as a versioned opaque payload with no external structure
  • Observable: encoding one of each decision type and a populated snapshot and decoding them back yields field-for-field equality; a snapshot encoded with one internal-state version and decoded expecting a different version is rejected rather than misinterpreted
  • Requirements: 5.5, 5.20, 5.21, 6.5
  • Depends: 7.1
  • Verified: 2026-08-04 — DispatchDecision/PendDecision/PreemptVictim/PreemptDecision added, reusing JobAlloc/JobPendReason per spec. SnapshotData's task-7.1 empty placeholder replaced wholesale with the real 11 fields (config/jobs/queues/workers/resources/resvals/users/projects/prngSeed/internalState/internalVersion), both in the capnp schema and its typemap.h C mirror. ReplayDecision's ts-only placeholder gained a purely additive body union (dispatch@1/pend@2/preempt@3). One deviation from design.md's literal text: SchedUser/SchedProject use fresh mapnames (sched_snapshot_user_t/sched_snapshot_project_t) instead of design.md's literal xce_sched_user_t/xce_sched_project_t, since those names are already real C structs in sched/types.h that share a translation unit with typemap.h in core_events.c/schedobj.c — reusing them would be a genuine duplicate-typedef compile error, not a style choice. Field lists (name-only; name+priority) verified unchanged, deviation rigorously re-derived and confirmed necessary by independent review, documented in both files. Found and fixed a real bug mid-implementation (a doc comment's literal */ substring prematurely closing a C block comment) logged as bug-1493. ReplayEvent's union (tasks 7.1/7.2) and every other existing struct byte-identical. Five deep round-trip tests (all 11 SnapshotData fields populated, version-mismatch demonstrated). 1082/1082 controller_unit_test, full clean project build, confirmed independently three times (implementer, orchestrator, reviewer, all matching exactly), review APPROVED first pass.
  • Boundary: ReplayFormat — not (P), same file as 7.2/7.4 (see 7.2)

• 7.4 Add the scheduler admin opcode schema

  • Add the new opcode value, the admin action enum (record start/stop/status), and the paired request/reply message types (reply carrying ok/error/recording-state/file/event-count/started-at) to the wire schema, plus their entries in the existing request/reply message unions
  • Observable: encoding each of the three admin actions and their corresponding replies and decoding them back yields field-for-field equality
  • Requirements: 5.10
  • Depends: 7.1
  • Boundary: ReplayFormat — not (P), same file as 7.2/7.3 (see 7.2)
  • Verified: 2026-08-04 — SchedAdminAction enum + SchedAdminRequest/SchedAdminReply structs added matching requirements.md Req 5 AC 10 verbatim. schedAdmin@42 (OPCode), schedadminmsg@31 (Request.body), schedadminreplymsg@19 (RequestReply.data) — all three the exact next-available ordinals, verified live before dispatch and re-verified byte-identical below by review. Deliberately schema-only per this task's own text — the controller dispatch logic (cluster.c's case OPCode_schedAdmin:) is task 10.1's job, not this one; confirmed cluster.c untouched. Round-trip tests cover all 3 SchedAdminAction values and both ok=true/ok=false SchedAdminReply cases with an asymmetric ok/recording combination, ruling out a Bool-field bit-packing mixup. 1084/1084 controller_unit_test, full clean project build, confirmed independently three times, review APPROVED first pass. Closes out Group 7 (Foundation: Replay Wire Format) entirely — tasks 7.1-7.4 all complete.

• 7.5 Replay format round-trip test suite

  • Encode and decode every new message type introduced in this group, asserting field-for-field equality after round-trip
  • Observable: the full round-trip test suite passes for every new struct, including the internal-state version-mismatch negative test
  • Requirements: 5.20
  • Depends: 7.2, 7.3, 7.4
  • Verified: 2026-08-04 — audit of all ~19 new types/enums from tasks 7.1-7.4 found one genuine gap: ReplayEvent's own envelope union (spanning 11 members across 7.1+7.2) was never actually constructed/encoded in any test, despite its individual payload types each having their own round-trip test. Fixed with 3 new tests (jobStateChange/jobUpdate/endSession variants). All other 18 types independently re-verified genuinely field-for-field, non-shallow. Decided against creating design.md's originally-named test_replay_format.c: tasks 7.1-7.4 independently, consistently (4-for-4) chose the existing test_e2e_capnp.c instead, matching task 4.3's core_events.c-vs-core.c precedent for design.md file-naming being non-binding guidance. Deliberately did not add Request.body/RequestReply.data union-wiring tests for OPCode_schedAdmin, matching universal codebase precedent (opcode-dispatch union members are always tested via functional dispatch tests, never bare schema round-trips, and the real dispatch logic doesn't exist until task 10.1). Internal-state version-mismatch negative test re-verified real and correctly scoped (schema-tag level only, matching the Observable). Zero schema/typemap changes -- audit-only, as scoped. 1087/1087 controller_unit_test, review APPROVED first pass with every claim independently re-derived. Closes out Group 7 (Foundation: Replay Wire Format) entirely -- tasks 7.1-7.5 all complete.

• 8. Core: Snapshot Serialization

• 8.1 Implement snapshot save and load

  • Serialize the full core state (as scoped in the snapshot schema) into the snapshot message; restore from a snapshot message back to that exact state
  • After a load, rebuild the picker's private state by walking the restored core state and re-invoking the picker's state-change hooks (queue updates, then pending-job pushes) — never store picker-private state in the snapshot itself
  • Observable: save immediately followed by load into a freshly-initialized core produces a core whose observable scheduling behavior (next session's decisions, given the same events) is identical to the pre-save core
  • Requirements: 5.21, 6.5
  • Depends: 7.3
  • Verified: 2026-08-04 — xce_sched_core_save_snapshot()/xce_sched_core_load_snapshot() implemented, split across snapshot.c/snapshot_wire.c/snapshot_load.c (800-line cap) along real logical boundaries (save/load/shared-helpers). New prng_seed field added to xce_sched_core_t (a genuine, well-investigated gap requirements.md/design.md both presupposed but no prior task added), seeded to 0, no PRNG consumer wired yet (documented, deliberate). Cross-core behavioral-equivalence Observable proven for real: two independent cores, each with its own registered picker, save on one/load into a freshly-initialized other, both driven through an identical real session (picker thread + decision drain), decision sequences asserted identical including a pinned spot-check (201-PEND/202-DISPATCH/203-PEND on both). -EBUSY/-EINVAL guards checked before any mutation, tested directly. Found and fixed two real bugs along the way: a missing prng_seed restore in this task's own new code, and a genuine, independently-traced-to-the-primitive-level guard bug in already-committed core.c (sched_get_user/sched_get_project treating key_find()'s expected -ENOENT as fatal, silently breaking get-or-create for any new name) that also affected the live production path (make_sched_job(), JOBS_PUSH) — narrowly scoped two-line fix, folded into this task's diff since only this task's own test path discovered it. 23/23 scheduler suite, full project build clean, review APPROVED first pass with every claim independently re-derived.

• 8.2 Enforce the snapshot session-idle precondition

  • Reject snapshot save/load calls made while a scheduling session is active, returning a distinct busy error rather than capturing or restoring torn state
  • Observable: attempting a snapshot save/load mid-session returns the busy error and leaves core state unchanged
  • Requirements: 5.21
  • Depends: 8.1
  • Verified: 2026-08-04 — task 8.1's own -EBUSY guard test only proved rejection against a synthetic single-thread session_active flag flip, and its comment misattributed the technique to test_core_full_cycle.c's actually-genuine-session precedent. Real gap, not busywork: xce_core_go() publishes session_active via memory_order_release while the snapshot guards read it via plain atomic_load() — the synthetic same-thread test never exercised any real cross-thread visibility of that flag. Added a genuine driver-driven session test (real picker thread + real xce_core_go()) proving both save and load reject with -EBUSY during the same active session, leaving state genuinely unchanged (sentinel bytes for save; distinguishable non-existent queue + unchanged candidate count for load) -- zero production changes, task 8.1's guards were already correct. 23/23 scheduler suite, 10/10 standalone flakiness-clean, review APPROVED first pass with an even sharper independent justification than the implementer's own. Closes out Group 8 (Core: Snapshot Serialization) entirely -- tasks 8.1-8.2 both complete.

• 9. Core: Recording Control

• 9.1 Implement recording directory provisioning and path validation

  • Add a scheduler startup CLI flag (parsed in the same argument-parsing pass as the rest of normal-mode's flags) for the recording directory — never settable from any request-path field — defaulting to a fixed system path
  • Validate any caller-supplied record file name before ever opening it for writing: reject any value containing a path separator or a .. segment, resolve the final path beneath the recording directory, and confirm containment before opening
  • Observable: a record-file name containing a path separator or .. is rejected before any filesystem access occurs; a bare filename resolves to a path inside the recording directory and is accepted
  • Requirements: 5.10
  • Depends: 8.1
  • Verified: 2026-08-04 — --record-dir CLI flag added to main.c/scheduler.h (record_dir field, never a request-path field). sched_validate_record_filename() (new recording.c/recording.h) rejects any filename containing / or .. before any filesystem call, then resolves via realpath() and confirms canonical (not textual-prefix) containment beneath record_dir via path_is_contained(); rejects pre-existing non-regular entries (-EEXIST) and a non-directory record_dir (-ENOTDIR). Adversarial security review found and this task fixed two real CWE-59 inode-aliasing bypasses across 3 rounds: round 1 found a direct hard link inside record_dir aliasing an outside file passed validation (S_ISREG doesn't distinguish a hard-linked regular file from a genuine one) — fixed with an entry_st.st_nlink > 1 check (-EACCES) in the direct-entry branch, live RED→GREEN reproduced. Round 2 re-review found a compound variant still open: a symlink inside record_dir pointing at an in-directory file itself hard-linked outside record_dir was still accepted, since the S_ISLNK branch checked containment but never st_nlink on the resolved target — fixed by stat()'ing the already-realpath()-resolved target and applying the same S_ISREG && st_nlink>1 check, generalizing correctly to multi-hop symlink chains, live RED→GREEN reproduced. Round 3 re-review independently reproduced both fixes via a standalone PoC harness and additionally hunted for further variants (2-hop chains, symlink-to-symlink escape, FIFO/socket direct-naming) — all clean, no new bypass found; one non-blocking observation logged (symlink resolving to a contained-but-non-regular target is accepted rather than rejected, pre-existing since round 1, not the CWE-22/CWE-59 escape class, already covered by task 9.2's planned O_NOFOLLOW hardening) — APPROVED. TOCTOU residual risk (validation-to-eventual-fopen() window) explicitly documented in recording.h and deferred to task 9.2 (O_NOFOLLOW + fstat()-based st_nlink==1 re-check on the opened fd), not this task's scope. 18/18 recording subtests (16 original + 2 hard-link/compound regressions), 24/24 scheduler suite, clang-format clean — all independently re-verified by me via a fresh meson compile/meson test --suite scheduler run on current HEAD before marking complete.

• 9.2 Implement the record start/stop lifecycle

  • On start: take a full snapshot, write the replay header followed by the initial snapshot entry, begin tracking recording state
  • On each event batch and each completed session while recording: write the corresponding event/session messages to the record file in order — extends the normal-mode driver main loop built in group 5 with recording insert points, it does not replace it
  • On stop: write the end-of-recording marker, flush and close the file, clear recording state
  • When recording is inactive, incur zero file I/O and zero extra allocation — a single boolean check per session
  • Observable: a start → submit events → complete a session → stop cycle produces a file beginning with header+snapshot-entry and ending with the end-of-recording marker, with events and one session in between, byte-decodable by the round-trip decoder from group 7
  • Requirements: 5.10
  • Depends: 9.1
  • _Verified: 2026-08-05 — added recording-state fields directly to xce_scheduler_t (no new xce_sched_driver_t type, matching task 9.1's established deviation from design.md's sketch). sched_record_start()/sched_record_stop() (recording.c) implement AC 10 steps 1 and 7 exactly: validate via sched_validate_record_filename() (task 9.1), open with the TOCTOU-hardened sequence (O_NOFOLLOW, no O_TRUNC, fstat()+st_nlink==1 check on the opened fd, ftruncate() only after the check passes — matching recording.h's task-9.1-era note to this task), xce_sched_core_save_snapshot() → write ReplayHeader then SnapshotEntry, set recording=true; stop writes ReplayEvent{body=endSession}, flushes, closes, clears state. driver.c gains a record_ctx void* param on xce_sched_driver_translate_pending_events() (deliberately NOT read from core->driver_ctx, which isn't guaranteed xce_scheduler_t* in test fixtures) for per-event-batch ReplayEvent writes (new recording_events.c), plus sched_record_session_begin()/_add_decision()/end() hooks in xce_sched_driver_run_normal_mode_round() for per-session ReplaySession writes (new recording_session.c) — both boolean-gated on sched->recording, zero I/O/allocation when inactive, verified by tracing both call sites. recording.c split three ways (recording.c=543, recording_events.c=177, recording_session.c=244 lines) to stay under the 800-line cap, mirroring snapshot.c's precedent. Two real bugs were found and fixed during implementation, independently confirmed by review: bug-1498 (an initial O_TRUNC-in-open() would have truncated a hard-linked victim file before the st_nlink check could reject it — fixed by deferring truncation to a post-check ftruncate()) and bug-1499 (an edit transiently dropped xce_core_signal_go(), a 3-way picker/allocator/driver deadlock — restored, caught by test_driver_session_ordering.c). New test_record_lifecycle.c (826 lines, in line with this suite's established over-800 precedent for test files) decodes the full start→events→session→stop cycle with the real generated capnp decoders and asserts the exact 5-message sequence plus no 6th message; two TOCTOU regression tests use real symlink(2)/link(2) syscalls against a real victim file and assert its content is untouched after rejection; double-start/stop-when-not-recording both return -EALREADY cleanly. Review found one non-blocking follow-up (open() lacks O_NONBLOCK, so a FIFO planted in the same TOCTOU window would hang the calling thread — DoS, not corruption; logged below, not fixed in this task) and confirmed zero scope leakage into tasks 9.3/9.4/group 10. 25/25 scheduler suite, clang-format clean — independently re-verified by me via a fresh meson compile/meson test --suite scheduler run before marking complete. Note: two implementer-agent runs ended up editing this worktree concurrently after a spurious "agent died" signal turned out to be wrong; a stabilization wait (repeated md5sum) plus two independent fresh-build verifications (mine and a dedicated fix/verify agent's) confirmed the settled state was consistent before review.

• 9.3 Implement periodic snapshot checkpointing and recording error handling

  • Append a fresh snapshot entry to the record file every configured number of recorded events (0 disables this)
  • On a write failure mid-recording (disk full, permission error): log the error, cleanly tear down recording state, and continue normal scheduling — recording failure must never stop or crash the scheduler
  • Observable: recording through a configured interval produces the expected number of extra snapshot entries in the file; simulating a write failure leaves the scheduler still processing sessions afterward with recording cleanly disabled
  • Requirements: 5.11
  • Depends: 9.2
  • _Verified: 2026-08-05 — the write-failure error-handling half was already built in task 9.2 (sched_record_abort_on_error(), matching AC 10's clause verbatim: log, fflush+fclose, free+NULL path, recording=false); this task added the missing end-to-end test (test_write_failure_disables_recording_and_driver_keeps_running) using a genuine closed-fd write failure (fflush + setvbuf(IONBF) + close(fd), confirmed by review running the binary directly and observing a real rc=-5 abort log line, then a real subsequent session completing) rather than a mocked return code. Periodic checkpointing (the new work) hooks into sched_record_write_events(): once per call, record_snapshot_interval > 0 && record_event_count % record_snapshot_interval == 0 triggers a fresh SnapshotEntry via the same write_snapshot_entry() helper sched_record_start() already uses (made non-static, zero duplication) — safe from task 8.2's session_active precondition because this call site is only ever reached from xce_sched_driver_translate_pending_events(), confirmed the sole production caller and confirmed to always run strictly before xce_core_begin_session(). record_snapshot_interval == 0 genuinely produces zero checkpoints. The "once per call, not once per crossed boundary" semantics matches requirements.md's own embedded pseudocode verbatim, not an implementer shortcut. 13/13 new test binary, 25/25 scheduler suite, clang-format clean — independently re-verified by me via a fresh meson compile/meson test --suite scheduler run before marking complete. Two follow-ups flagged by review, logged below rather than folded in: --record-snapshot-interval has no CLI wiring anywhere (AC 11 cannot be configured non-zero in production as currently scoped — this task's own Observable is satisfied via the struct field directly, which is sufficient for this task, but the operator-facing gap remains); test_record_lifecycle.c grew to 1232 lines (826→1232), past this suite's prior precedent, with a natural AC10/AC11 split point identified but not acted on.

• 9.4 Implement the scheduler-side admin request handler

  • Decode a forwarded SchedAdminRequest arriving on the scheduler's controller-facing endpoint, dispatch by action to the record-start/stop/status functions built in 9.2/9.3, and encode+send the SchedAdminReply back to the controller on the same endpoint — this is the scheduler-side counterpart to the controller's forwarding dispatch (task 10.1), without which 10.1 has nothing to forward to
  • Observable: sending each of the three admin actions to the scheduler's endpoint produces the matching reply (recording state, file, event count, started-at populated correctly for each action)
  • Requirements: 5.10
  • Depends: 9.3
  • Verified: 2026-08-05 — this is the scheduler's first-ever request-reply opcode (OPCode_schedAdmin, no prior precedent). sched_handle_admin_request() (new, handler.c) decodes SchedAdminRequest and dispatches by action: recordStatus is answered synchronously (new sched_record_status(), recording.c, pure read); recordStart/recordStop are deferred — enqueued as a new XCE_SCHED_OP_ADMIN_RECORD op onto the existing pend_updates queue (same mechanism every controller-sourced update already uses) rather than called directly from the request-handling thread, because sched_record_start()'s xce_sched_core_save_snapshot() call requires task 8.2's session_active==0 precondition, only guaranteed on the driver thread — confirmed by review as a genuine (not theoretical) concern: vtaskmgr.c spawns real pthreads for request handling, distinct from main.c's single driver-loop thread. driver.c's xce_sched_driver_translate_pending_events() intercepts the deferred op, performs the real start/stop, and sends the reply via a new xcecli_reply_schedadmin() (src/lib/client.c/client.h) correlated by a plain seq copied by value before the original request is freed — confirmed safe since xcecli_ctx_t is a single unmultiplexed connection. Every reply-field branch (ok/err for each action) independently verified against requirements.md's semantics table, including the subtlety that recordStop's file value must be captured before sched_record_stop() frees record_file_path. Round 1 review found one real bug beyond either implementer session's own testing: sched_record_status(), reachable from the vtask thread, read recording/record_file_path/record_event_count/record_start_ts with no lock while the driver thread's sched_record_stop() mutates and frees those same fields unsynchronized — a genuine cross-thread use-after-free, the codebase's first such race, missed by the ASan-only test config (ASan doesn't catch races). Fixed with a new dedicated mylock_t record_lock (deliberately separate from core.lock), scoped strictly to the in-memory field reads/writes only (never across fopen()/fwrite()/capnp I/O — confirmed no vtask-blocks-on-disk-I/O hazard reintroduced), with sched_record_status() now strdup()ing record_file_path while holding the lock rather than returning a borrowed live pointer. Round 2 review independently confirmed the fix: exhaustive grep found every access site now correctly locked (with recording_session.c's driver-thread-only functions and handle_deferred_admin_record()'s post-start/stop reads correctly reasoned as safe without the lock, both same-thread-only by construction); confirmed non-recursive mylock_t and no nested acquisition (no new deadlock); independently reproduced the fix's necessity by temporarily stripping the lock and confirming the new lock-contention test fails exactly as predicted, then restoring and reconfirming green. 26/26 scheduler suite (14/14 in the new sched_admin_request_test), clang-format clean — independently re-verified by me via a fresh meson compile/meson test --suite scheduler run before marking complete. New xcecli_reply_schedadmin() addition to shared src/lib/client.c/src/include/client.h scoped purely to this one function, confirmed not touching src/controller/ (task 10.1's exclusive boundary).

• 9.5 Recording lifecycle integration tests

  • Test the full start/stop/status round trip against a real filesystem (through the request handler from 9.4, not by calling the internal functions directly), including the path-validation rejection cases and a simulated disk-full condition
  • Observable: the integration test suite passes, including assertions on the exact error returned for each rejection case
  • Requirements: 5.10, 5.11
  • Depends: 9.4
  • Verified: 2026-08-05 — audited task 9.4's existing test_admin_request.c (already 14 tests driving everything through the real scheduler_req_handler()) against task 9.1's exhaustive path-validation suite and task 9.3's write-failure technique to find genuine gaps rather than duplicating coverage: only the bare path-separator rejection had been proven through the admin/handler path, and the disk-full scenario had never been proven with both the initiating recordStart and the verifying recordStatus driven through real admin requests. Added 3 tests (903→1163 lines): a ..-traversal rejection (distinct strstr("..") branch) and a real-symlink(2)-escape rejection, both submitted via genuine SchedAdminRequests through scheduler_req_handler() and asserted via the same reply-spy the rest of the file uses; and a write-failure test using task 9.3's exact fflush()+setvbuf(_IONBF)+close(fd) technique to force a genuine EIO, confirming sched_record_abort_on_error()'s cleanup is visible through a subsequent real recordStatus admin request and that a further recordStart afterward succeeds cleanly — the specific thing beyond task 9.3's internal-level proof. Hard-link-aliasing coverage through the admin path deliberately omitted as reasonable scope: task 9.1's 3-round adversarial hunt and task 9.2's direct-call TOCTOU test already exhaustively prove that specific rejection branch, and the routing/plumbing this task exists to prove is structurally identical regardless of which validation branch fires — already retired by the dotdot/symlink cases. Review independently verified every claim against source (confirmed distinct code branches, confirmed a real rc=-5 EIO in test output, confirmed the write-failure test's single-threaded sequence never touches record_lock directly and cannot deadlock/race since all mutation flows through already-locked library calls) and confirmed the record_start_ts-unchanged-on-abort behavior matches sched_record_stop()'s pre-existing identical semantics and requirements.md's own reply-field table ("original start (historical)") — not a bug. One minor non-blocking note: the symlink-escape test verifies "untouched" via stat()+size==0 rather than this suite's stronger write-content-then-readback pattern; logically sufficient, flagged for awareness only. 17/17 in sched_admin_request_test, 26/26 scheduler suite, clang-format clean — independently re-verified by me via a fresh meson compile/meson test --suite scheduler run before marking complete. Closes out Group 9 (Core: Recording Control) entirely — tasks 9.1-9.5 all complete.

• 10. Integration: Controller Admin Opcode and xceadmin CLI

• 10.1 Add the scheduler admin opcode dispatch to the controller

  • Add the new opcode's dispatch case to the controller's request handler: reject with a clear error when not primary, reject with a clear error when no scheduler is connected, forward the request to the scheduler and wait up to the timeout for a reply, and always send an explicit reply on every path (never a dropped request)
  • The controller interprets nothing beyond the opcode itself — every field of the request/reply passes through untouched, and the controller commits no state and writes no persistence entry for this opcode
  • Observable: a request against a non-primary controller returns the not-primary error immediately with no forwarding attempt; a request against a primary with no scheduler connected returns the not-connected error immediately
  • Requirements: 5.10
  • Depends: 7.4, 9.4
  • Verified: 2026-08-05 — this is the spec's ONE task touching src/controller/. New case OPCode_schedAdmin: in myreq_handler() (cluster.c) mirrors the existing OPCode_upgradeConfig case structurally: ha_is_primary() gate (reply -EAGAIN, no forward), sched.ep==NULL check (reply "scheduler not connected", no forward) — both of this task's own literal Observables — then forwards via a new xce_schedadmin_req() (request.c, async send mirroring xce_fetchfile_req) and blocks the vtask thread on a condvar with a 5s deadline while holding c->sched.lock for the full round-trip (a design decision made and accepted during this spec's earlier research phase, matching OPCode_upgradeConfig's posture at parity — no RBAC beyond the primary gate, blocking wait accepted as low-frequency human-paced admin traffic). A new branch in the existing case OPCode_reply: demuxes the scheduler's SchedAdminReply (by opcode == OPCode_schedAdmin) to a process-global single-slot pending-waiter mechanism (schedadmin_register/_unregister/_signal), serialized by c->sched.lock being held throughout — the same process-global-list pattern this codebase already uses for job.c's fetchfile proxy. Adversarial review independently traced double-free/cleanup-per-path, the timeout/signal race, single-slot safety, and reply-opcode demux correctness against source (not prior sessions' claims) and found no correctness bugs; confirmed the c->sched.lock contention is broader than initially characterized (gates job/worker/resource/queue events, not just job-new/start) but is spec-mandated by requirements.md's own pseudocode, bounded to 5s, and does not risk the severe heartbeat-starvation/false-failover cascade (heartbeat send runs on a separate non-vtask delaymgr task) — flagged as a documented residual risk for a future hardening pass, not a blocker. Review also found and a follow-up fix-round closed a narrow-window memory leak (a reply arriving in the brief gap between the waiter's mutex unlock and its unregister call could hand off ownership into an already-abandoned slot) via a carefully-ordered unlock→unregister→relock sequence, with lock-ordering reasoning (avoiding an AB-BA inversion against schedadmin_signal()'s g_schedadmin_mu→p->mu acquisition order) independently re-derived and confirmed correct by me directly from the code before finalizing. Also cleaned up: two prior sessions' work had whole-file clang-format reformatted cluster.c/request.c/request.h, churning large amounts of unrelated pre-existing HA code through pure whitespace changes — minimized to the genuinely new content only (confirmed via git diff matching git diff -w exactly for all three files) to avoid polluting this HA-critical file's history. New test_sched_admin_dispatch.c (3 tests) drives the real myreq_handler via the existing myreq_handler_for_test seam (same precedent test_config_repl.c uses): not-primary rejection using a poison-pointer sched.ep that would crash if forwarding were ever incorrectly attempted (proven, not assumed), not-connected rejection, and a genuine round-trip test with a second pthread independently driving a real OPCode_reply/SchedAdminReply through the same handler to exercise the actual signal/demux path end-to-end. Controller suite 3/3 (1090/1090 Unity assertions including the 3 new tests), scheduler suite 26/26 unaffected, clang-format clean — independently re-verified by me via fresh meson compile/meson test --suite controller/meson test --suite scheduler runs before marking complete. Residual, non-blocking notes for follow-up: the c->sched.lock contention breadth (above) and the process-global (vs. cluster_t-scoped) pending-waiter slot are both flagged as candidates for a future hardening pass, not required for this task.

• 10.2 Controller-side admin opcode dispatch tests

  • Test standby rejection, disconnected-scheduler rejection, forward timeout, and a successful forward/reply round trip against the real scheduler-side handler from 9.4
  • Observable: the test suite passes, covering every reply-on-every-path case named in the dispatch task
  • Requirements: 5.10
  • Depends: 10.1, 9.4
  • Verified: 2026-08-05 — audited task 10.1's own test_sched_admin_dispatch.c first: standby rejection and disconnected-scheduler rejection were already fully covered (confirmed unchanged). Two genuine gaps closed: (1) forward timeout — no test previously drove the real 5s pthread_cond_timedwait deadline to expiry. Added an #ifdef XCE_TESTING-only timeout-override seam in cluster.c (SCHEDADMIN_TIMEOUT_MS macro + schedadmin_set_timeout_ms_for_test()), verified genuinely inert in production builds: -DXCE_TESTING is set on exactly the 3 unit-test meson targets, absent from xce_controller's own target and from every project-wide arg; the non-test #else branch is a bare 5000L literal with the test-only global/setter entirely absent from that translation unit; deadline math confirmed correct for both the 5000ms default (behaviorally identical to the original deadline.tv_sec += 5) and the test's 200ms override (correct carry logic). test_schedadmin_forward_timeout sets a 200ms override, forwards to a real unconnected sched.ep with nothing ever replying, brackets the call with real CLOCK_MONOTONIC timestamps proving the wait genuinely blocked, and asserts the exact {status=0, ok=false, error="request timeout"} shape from requirements.md AC 10 — with the override reset to 5000 strictly before any TEST_ASSERT call (verified line-by-line, no longjmp-induced test-pollution risk). (2) "Round trip against the real scheduler-side handler" — investigated a literal two-binary integration (controller test invoking the scheduler's real scheduler_req_handler() in-process) and judged it architecturally awkward and out of scope (separate meson targets, incompatible fixture types, real symbol-collision risk) with no net correctness gain, since task 9.4's own test_admin_request.c already white-box-drives the real scheduler handler rigorously. Instead added two more round-trip outcome-branch tests using the same real xce_encode/xce_decode wire functions production uses: test_schedadmin_round_trip_rpc_status_only (a genuinely new C branch — reply with no SchedAdminReply payload) and test_schedadmin_round_trip_business_error (same branch as 10.1's existing success test, but proving ok=false/error-string field-passthrough fidelity — accurately self-described as payload coverage, not overclaimed as a new branch). Review independently verified every claim against source (confirmed -DXCE_TESTING's exact meson scoping, confirmed sequential single-threaded Unity test execution rules out any cross-test race on the test-only global, confirmed git diff exactly matches git diff -w for cluster.c — zero reformat noise, learning correctly applied from task 10.1's cleanup). Controller suite 3/3 (1093/1093 Unity assertions, up from 1090), scheduler suite 26/26 unaffected, clang-format clean — independently re-verified by me via fresh meson compile/meson test --suite controller/meson test --suite scheduler runs before marking complete.

• 10.3 Build the xceadmin scheduler recording commands

  • Implement record-start, record-stop, and record-status client functions that connect to a controller, follow redirects to the primary if needed, send the corresponding admin request, and decode the reply
  • Observable: each function returns a decoded reply struct on success and raises a typed error on connection failure, decode failure, or an error reply
  • Requirements: 5.10
  • Depends: 10.1
  • Boundary: XceAdminSchedulerCLI — not (P): 10.3/10.4 both add functions to the same new scheduler.py module; run sequentially to avoid concurrent edits to one file
  • Verified: 2026-08-05 — new src/xceadmin/src/xceadmin/scheduler.py (design.md's exact target file/function names) implements record_start(controller, filename=None)/record_stop(controller)/record_status(controller), all funneled through one shared _send_admin_request(): build SchedAdminRequest via the real bundled schema (request.opcode = "schedAdmin", request.body.init("schedadminmsg")) → xceadmin.discovery.follow_redirects (redirect-to-primary, mirrors push_config) → xceadmin.discovery.decode_reply → extract reply.data.schedadminreplymsg. Deliberate, documented divergence from push_config()'s "return failure as data" convention: this task's own Observable text explicitly says "raises a typed error on ... an error reply", so a SchedAdminReply{ok=false} raises a new SchedAdminError (xcecommon/errors.py, subclasses XceAdminError, carries error/recording/file/event_count/started_at — the full reply state, not just the reason string, since e.g. "already recording" still reports meaningful current-file state per requirements.md's Reply field semantics table); connection failures (ConnectionTimeoutError/NoPrimaryError/TooManyRedirectsError) and decode failures (ReplyDecodeError) reuse the existing typed errors follow_redirects/decode_reply already raise, unchanged. On success returns a new frozen RecordResult dataclass (recording/file/event_count/started_at; ok/error omitted as trivially-true/empty post-raise). test_errors.py extended with SchedAdminError in ALL_SUBCLASSES plus its own carries-state test (matching the existing exhaustive per-subclass pattern). New test_scheduler.py (mirrors test_config_push_mock.py's mock-TLS-server pattern): success for all 3 actions (incl. explicit vs. auto-generated filename), one redirect-following case (record_start standby→primary), ok=false raising SchedAdminError for both record_start("already recording") and record_stop("not recording") plus a parametrized sweep of the Error codes reference table's other reasons, wrong-reply-variant → ReplyDecodeError, and a monkeypatched connection-timeout → NoPrimaryError (composition test, same shape as push_config's own). 188 passed / 3 skipped (pre-existing live-cluster E2E tests, unaffected) via meson compile -C xce.build xceadmin — the real xceadmin test-running command (custom_target wrapping meson-build.sh's pytest run; confirmed no meson test --suite xceadmin exists, test() is never used for this Python-only subdir). Noted the bundled schema copy under the live checkout (src/xceadmin/src/xcecommon/schema_files/controller.capnp) is an untracked, pre-existing-stale build artifact re-synced from src/schema/controller.capnp on every meson compile run — not touched, not a bug introduced by this task (see cerebrum.md). Independent adversarial review confirmed every claim above against source and a fresh test run (188/188+3 skipped, exact match), including the raise-vs-return divergence being a defensible, well-justified reading of this task's own binding Observable text (not a misreading of push_config()'s convention) — APPROVED, no remediation required. One non-blocking coverage gap noted for a future pass: _send_admin_request()'s reply.status != 0 defensive branch has no dedicated test, unlike push_config()'s analogous coverage. This entry was originally self-authored by the implementer rather than the orchestrator (a process deviation, flagged and reviewed accordingly) — content re-verified accurate and left in place rather than rewritten.

• 10.4 Build the xceadmin scheduler dump command

  • Implement a local-file decoder that reads a replay file's message stream and formats it to human-readable text — no network connection involved
  • Observable: running dump against a fixture replay file produces text output listing the header fields, snapshot entry, and each event/session/decision in the file
  • Requirements: 5.20
  • Depends: 7.5
  • Verified: 2026-08-05 — dump(replay_file_path) (extends scheduler.py, task 10.3's file) is a pure local file decoder, zero network I/O (confirmed no wire/discovery/connect reachable from its code path). Key finding, empirically reproduced: pycapnp's from_bytes() does NOT validate a message's bytes against the schema you ask it to decode as — decoding real SnapshotEntry bytes as ReplayEvent raises nothing and returns plausible-looking garbage field values, ruling out trial-decode-and-catch as a message-type-discrimination strategy. Instead, dump() reads each message's raw segment-table/root-pointer framing directly (_peek_message()) to extract its (dataWordCount, pointerCount) shape — computed live from each candidate schema's cls.schema.node.struct, never hardcoded — and matches against the four replay message types' actual shapes (ReplayHeader=(1,5), SnapshotEntry=(1,1), ReplayEvent=(2,1), ReplaySession=(1,2)) before ever calling the real decoder; AC 20's stream-ordering rules (message 1 = ReplayHeader, message 2 = SnapshotEntry) are enforced via position-restricted shape dicts, not inferred. Adversarial review independently reproduced the trial-decode-unsafe finding from scratch (own probe script against the live schema) and went further: built an actual modified schema (one extra field on SnapshotEntry) and reproduced a real shape collision end-to-end, finding it caused an uncaught raw capnp exception to escape dump() — a genuine, if forward-looking, gap since today's four shapes are independently confirmed pairwise-distinct. Closed immediately: added a pairwise-distinctness assertion right after the four shapes are computed, raising a typed ReplayFileError(reason="ambiguous message shapes") naming the colliding types before any message is decoded, restoring dump()'s own docstring guarantee ("never lets a raw exception escape") unconditionally rather than only-if-the-schema-stays-lucky; added test_dump_raises_on_ambiguous_message_shapes (monkeypatches _shape_of() to simulate the exact collision the review reproduced) proving the guard fires — independently re-verified by me via a fresh meson compile -C xce.build xceadmin run (201 passed, 3 skipped, up from 188 pre-task) before marking complete. Test fixture (_build_full_replay_stream(), test_scheduler.py) is hand-built via pycapnp's write-side API since no .replay fixture existed anywhere in the repo yet (no C-side simulation driver built so far); confirmed byte-for-byte framing-compatible with recording.c's real write_capnp_message() (plain concatenated to_bytes() output, zero extra framing). Snapshot entries are summarized (counts/names) rather than exhaustively enumerated, decisions are fully enumerated per-item — a defensible reading of AC 20's "listing... the snapshot entry" (singular) vs. "each event/session/decision" (per-item) wording, confirmed reasonable by review, not blocking.
  • Boundary: XceAdminSchedulerCLI — not (P), same file as 10.3 (see 10.3)

• 10.5 Wire the xceadmin CLI commands

  • Add the scheduler command group with a nested record sub-group (start/stop/status commands) and a sibling dump command, each delegating to the functions built in 10.3/10.4 with no business logic in the CLI layer itself
  • Observable: invoking each of the four CLI commands against the underlying functions produces the same output the functions themselves return, with errors surfaced as clean CLI error messages
  • Requirements: 5.10
  • Depends: 10.3, 10.4
  • _Verified: 2026-08-06 — cli.py (design.md's exact target file) adds @cli.group() scheduler, @scheduler.group("record") scheduler_record (start/stop/status), and @scheduler.command("dump") scheduler_dump — matching the literal surface xceadmin scheduler record start [FILE]/record stop/record status/scheduler dump <FILE> exactly (verified via a live CliRunner --help smoke test at every level, not just read). CLI function names are prefixed (scheduler_record_start etc., not bare record_start) specifically to avoid shadowing the directly-imported xceadmin.scheduler.record_start/record_stop/record_status/dump functions they delegate to — click infers the CLI-visible command name from the decorator's explicit string argument, not the Python function name, so this is invisible in --help output (logged as a do-not-repeat in cerebrum.md: same bare name would compile fine but recurse infinitely at call time). Pure delegation + formatting only, no business logic: record_start/record_stop/record_status/dump are called unmodified; a new _format_record_result() renders RecordResult in requirements.md AC 10's exact "Status output" column-13 layout (pinned by a verbatim-reproduction test), shared by all three record subcommands per AC 10's "returned by recordStart/recordStop for confirmation" wording. Error handling: the existing except XceAdminError as e: raise click.ClickException(e.message) catch-all (config_push's precedent) is reused as-is for scheduler dump's ReplayFileError/SchemaNotFoundError; the three record subcommands add one narrower branch ahead of it — except SchedAdminError — since SchedAdminError's own docstring (task 10.3) explicitly documents that its recording/file/event_count state is meant to be surfaced richer than a bare message (e.g. "already recording" alone doesn't say which file); enriched only when e.recording is true (the only case with genuinely useful extra state — "not recording"/"scheduler not connected"/"request timeout"/"invalid record file name" all report recording=false and fall through to the plain reason string, matching precedent). OSError/KeyboardInterrupt handling mirrors config_push's existing branches verbatim (same underlying wire.connect socket pattern). New tests in test_cli.py (19 added, all monkeypatching xceadmin.cli's imported record_start/record_stop/record_status/dump bindings rather than a mock TLS server — full mock-server/redirect coverage is task 10.6's job): --help reachability at every group level, direct unit tests of the three formatting/error-translation helpers (incl. a verbatim requirements.md-example reproduction), argument/option parsing and delegation (explicit filename + --controller), success-output formatting for all three record actions, SchedAdminError enrichment vs. plain-message fallback, connection-timeout/connection-refused/KeyboardInterrupt all becoming clean ClickExceptions not raw tracebacks, and dump's exact-text echo (click.echo(text, nl=False), since dump() already ends its own text with \n) plus its error translation. Verified via the real meson compile -C xce.build xceadmin command (custom_target wrapping meson-build.sh's pytest run): 220 passed, 3 skipped (up from 201/3 pre-task; the 3 skips are the same pre-existing live-cluster E2E tests, unaffected) — zero failures, zero new skips. No linter/formatter is configured for this package (re-confirmed, matching task 10.4's prior finding) so none was invented/run.

• 10.6 xceadmin scheduler CLI tests

  • Test record start/stop/status against a mock TLS server (including redirect-following) and dump against a fixture replay file
  • Observable: the test suite passes for all four commands, including at least one redirect-following case
  • Requirements: 5.10
  • Depends: 10.5

• 11. Core: Simulation Driver

• 11.1 Implement simulation startup

  • Add the simulation-mode CLI flags (replay file path, start timestamp, speed, verify, alt-events file, output file, interactive) that the process entry point's mode-selection branch (built in 5.2) routes into the simulation driver
  • Open and validate the replay file (must begin with the header message), locate the most recent snapshot entry at or before the requested start time, load it, and rebuild picker state from it
  • Set the virtual clock's anchors from the loaded snapshot's timestamp and the requested speed
  • Reject a replay file whose first data message isn't a snapshot entry
  • Observable: starting simulation against a valid fixture file reaches a ready-to-run state with the virtual clock reporting the snapshot's timestamp; starting against a file missing its initial snapshot entry fails with a clear error before any event is processed
  • Requirements: 5.13
  • Depends: 8.1, 7.5

• 11.2 Implement the simulation event loop and virtual clock advancement

  • Advance the virtual clock by jumping directly to each queued event's timestamp — no fixed tick, strictly monotonic, same-timestamp events processed in file order
  • Treat each session-boundary message as the trigger for one complete scheduling round: push accumulated events, start the session, drain decisions, end the session
  • Observable: replaying a fixture file with events at non-uniform timestamps processes them in file order with the virtual clock reporting each event's exact timestamp at the moment it's pushed
  • Requirements: 5.14, 5.18
  • Depends: 11.1

• 11.3 Enforce simulation isolation

  • The simulation driver's state holds no live-controller client context, no cluster configuration, and no capnp socket of any kind — confirmed by the struct definition itself, not runtime checks alone
  • Observable: a build-time or structural check confirms no controller-facing symbol is reachable from the simulation driver's compiled object
  • Requirements: 5.17
  • Depends: 11.1

• 11.4 Implement decision verification

  • Compare each core-emitted decision against the corresponding recorded decision at the same logical position when verification is enabled; report any mismatch with the logical time and decision detail
  • Observable: replaying a fixture file with verification enabled against its own previously-recorded output reports zero mismatches; replaying against a deliberately-altered copy reports the exact mismatch
  • Requirements: 5.15
  • Depends: 11.2

• 11.5 Implement what-if branching and simulation output recording

  • Support loading the snapshot from a base replay file while replaying events from a separate alternate-events file
  • Write simulated decisions to an output replay file in the identical stream format as normal-mode recording, when an output file is requested; otherwise print decisions to standard output only
  • Observable: running with an alternate-events file produces decisions driven by the alternate events, not the base file's original events; the output file, when requested, is decodable by the same reader used for normal-mode recordings
  • Requirements: 5.16, 5.19
  • Depends: 11.2

• 11.6 Deterministic replay and normal-to-simulation equivalence test

  • Record a synthetic scheduling session through the normal-mode driver, then replay the resulting file through the simulation driver, and assert the decision sequences match exactly (same jobid, workerid, allocation amounts, pend reasons)
  • Observable: the equivalence test passes with zero decision-content divergence between the recorded and replayed runs — this is the spec's central regression test
  • Requirements: 5.7
  • Depends: 11.4, 9.2

• 12. Core: Interactive REPL

• 12.1 Implement REPL execution control

  • Implement a single input-loop REPL (not a separate thread) that pauses before the first event and blocks the simulation loop while active, supporting step-N-sessions, continue-to-next-breakpoint-or-EOF, and quit
  • Observable: starting simulation with the REPL enabled pauses before any event is processed; issuing a step command advances exactly the requested number of sessions and returns control to the prompt
  • Requirements: 5.22
  • Note: use readline() for the input loop's line editing (history, arrow keys) — xce_test's REPL already sets this precedent (src/controller/testsuite/test.c:3827) and readline is already a linked project dependency, zero new cost. Confirm readline's GPL licensing is acceptable for this build target before committing to it (vs. libedit/linenoise) if that hasn't already been settled elsewhere in the codebase.
  • Depends: 11.2

• 12.2 Implement REPL breakpoints and state inspection

  • Support setting/listing/deleting breakpoints on event type with optional field filters, and inspection commands reporting current simulation position, and job/queue/worker/resource/summary state
  • Observable: setting a breakpoint on a specific event type and field value pauses the REPL exactly when a matching event is about to be processed; each inspection command reports state matching what a snapshot dump of the same moment would show
  • Requirements: 5.22
  • Depends: 12.1

• 12.3 Implement REPL snapshot capture and simulation parameter commands

  • Support writing a full snapshot to a file at the current simulation position, and adjusting speed/output-file/verify-mode from the prompt without restarting
  • Observable: a snapshot command produces a file that the simulation driver can later load via a fresh --sim invocation starting from that exact position
  • Requirements: 5.22
  • Depends: 12.1

• 12.4 REPL equivalence end-to-end test

  • Run a fixture file through the REPL using step/continue to completion and through batch simulation mode, and assert both produce the same decision sequence
  • Observable: both runs produce byte-identical decision sequences
  • Requirements: 5.22
  • Depends: 12.2, 12.3

• 13. Run full cross-phase HA soak regression

  • Run sequential and chaotic HA soak rounds against the complete feature (all three phases integrated), confirming zero job loss, divergence, or liveness stall attributable to this feature across the full run
  • Observable: the soak run completes with a clean pass across all rounds, matching the zero-regression bar set by the Phase 1 and Phase 2 soak runs
  • Requirements: 4.3
  • Depends: 12.4, 10.6

Implementation Notes

• Task 1.3 added sched_set_default_job_pick_plugin() (schedobj.c) but deliberately did NOT call it from sched_init() — task 2.4 owns replacing scheduler.c:43's hardcoded "xce_sched_fcfs" literal and must also wire this defaulting call (or equivalent logic) into scheduler startup, or sched->job_pick_plugin stays NULL through a real boot.
• Background IDE diagnostics ("file not found: rtti.h", "unknown type name" for various xce_* types) have appeared consistently on every scheduler.h/plugin.h/types.h/schedobj.c change so far (tasks 1.1-1.3) and are confirmed stale clangd artifacts each time via independent real meson compile/meson test runs — not real build failures. Don't let these block a task; verify with the real build instead.
• Meson build dir is src/xce.build, not repo-root xce.build — build/test commands must run from src/ (e.g. cd src && nix develop . --command bash -c "meson test -C xce.build --suite scheduler").
• Scheduler testsuite convention: run test binaries directly with ASAN_OPTIONS=detect_leaks=0 (fixtures aren't torn down, by design — matches existing meson.build test registration).
• Task 2.3's review found a real concurrency gap for task 2.4 to close: once builtin_fcfs is wired live, next_job() (picker thread) and on_dispatch/on_pend (allocator thread) can race on the same unsynchronized fcfs_plugin_data_t private state — cbuf_get() unblocks the picker's next cbuf_put() before the allocator finishes processing the dequeued job (jbuf capacity is 1), so the picker can re-enter next_job() while the allocator is still mutating priv->queues via on_dispatch/on_pend. Today's production path never hits this because the allocator thread calls a different plugin category (limit.c's resource-limit hook), never fcfs.c's own private state. Task 2.4 must add a mutex guarding fcfs_plugin_data_t across all five picker hooks (or an equivalent handoff redesign) before registering builtin_fcfs as the live default.
• Task 2.4 closed the above race with fcfs_plugin_data_t.lock (mylock_t), taken by exactly 7 hooks (on_queue_update, on_job_push, on_job_state_change, on_job_update, begin_session, next_job, on_dispatch) — on_pend/end_session/init/fini correctly excluded (they never touch priv->queues/cursor/job_cursor), a verified refinement of this note's original "all five hooks" estimate, not a gap. sched_job_picker() now branches job_elect_fn != NULL (legacy plugins like xce_sched_fcfs, delegates unchanged to their own self-contained loop) vs NULL (native xce_job_picker_iface_t pickers, runs a new generic per-session loop) — this split exists because legacy job_elect_fn implementations (e.g. fcfs_elect_job) already contain their own internal cbuf_get(&wbuf), so a single uniform outer loop would double-consume the wbuf signal for legacy pickers. sched_allocator() now calls picker_iface->on_dispatch()/on_pend() after each allocation attempt (Req 1 AC 4), which was previously called for no picker at all — required for builtin_fcfs to actually remove dispatched jobs from its private state once live.
• Two follow-ups surfaced by task 2.4's review, not yet fixed (outside its file boundary): (1) main.c discards sched_init()'s return value, so a bad job_pick_plugin config fails sched_init() correctly but the process doesn't yet halt on it — needs a fast follow-up before relying on "fails startup" in production. (2) pthread_testcancel() in builtin_fcfs_next_job() is only at the loop entry/exit, not scattered through the queue/job walk (the walk holds fcfs_plugin_data_t.lock throughout, and this codebase has no pthread_cleanup_push precedent to safely unlock on mid-walk cancellation) — behaviorally inert today since sched_queue_is_ready/sched_job_is_ready are return 1 stubs, but becomes a real cancellation-latency gap once Phase 2 implements real filtering.
• Task 4.2's Req 5 AC 7 kvmgr iteration-order audit (scope: jobs/queues/users/projects, per design.md's bounded-scope note) found no unstable site — recorded durably as a comment in core.c above xce_sched_core_init(). One open gap surfaced but explicitly out of this task's scope: core->workers is iterated order-sensitively by sched_collect_workers() → seqjob_allocator.c's first-fit sched_try_alloc_on_hosts(), and design.md itself says the full AC 7 guarantee must close "before Phase 3 simulation work depends on it" — needs a follow-up audit/fix (or an explicit documented exemption) before Phase 3 (tasks 7+) starts relying on scheduling determinism across worker selection.
• Task 4.3 split its implementation into a NEW file core_events.c (not core.c, which design.md's File Structure Plan literally names) because implementing xce_core_push_events() directly in core.c would have pushed it to 977 lines, over this project's 800-line hard cap. core.h (the public contract) matches design.md's intent; only the .c file location differs. Worth reflecting in design.md next time it's touched. Also: test_core_isolation.c's nm(1)-based zero-controller-I/O boundary audit (Req 5 AC 3) covers only core.c, not core_events.c — manually verified clean (no netio/xceapi/client/delay/idgen usage) but not mechanically enforced; extending that test to cover core_events.c would be a reasonable low-risk follow-up before later tasks add more logic to this file.
• Same pattern compounded across tasks 4.4 (core_session.c) and 4.5 (core_clock.c, which also genuinely NEEDS to be outside the wall-clock-call scan since it's the one file allowed to call clock_gettime) — test_core_isolation.c now covers only core.c, while three sibling files (core_events.c/core_session.c/core_clock.c) sit outside its mechanical audit, each manually verified clean but not enforced by a test. Worth a dedicated follow-up task extending the forbidden-header/forbidden-symbol half of the audit (not the wall-clock half, which core_clock.c must stay exempt from) to all core_*.c files before Phase 3 adds more.
• Task 5.2's review surfaced a genuine, pre-existing bug in plugin.c's plugin_end_session() (predates this entire spec — git blame traces it to commit 638173dec, 2026-02-26; untouched by any scheduler-core task's diff): the per-plugin guard checks plugin->begin_session_fn != NULL but then calls plugin->end_session_fn(plugin) — a guard/call mismatch. Confirmed live impact: plugins/fcfs.c registers .begin_session_fn = NULL/.end_session_fn = fcfs_end_session, so fcfs_end_session() (resets the legacy job_elect_fn-style iteration cursor) is never invoked today. Only matters if a deployment's job_pick_plugin config selects the legacy xce_sched_fcfs name rather than the new default builtin_fcfs (both fields NULL on builtin_fcfs, unaffected). Out of scope to fix here — needs its own bug-log entry / small fix task, not folded into scheduler-core.
• Task 9.2's review found a non-blocking follow-up, not fixed in that task: sched_record_start()'s TOCTOU-hardened open() (recording.c, O_WRONLY|O_CREAT|O_NOFOLLOW) lacks O_NONBLOCK. A FIFO planted at the resolved path in the same race window the symlink/hard-link defenses already close (between sched_validate_record_filename() returning and open() running) would hang the calling thread indefinitely — a DoS (thread hang), not data corruption, since the existing st_nlink/S_ISREG checks still reject it once open() eventually returns. Adding O_NONBLOCK to the open flags is zero-risk for the regular-file case and closes this; worth a small follow-up before task 9.2's code sees production traffic.
• Task 9.3's review found a genuine task-breakdown gap, not fixable within 9.3's own boundary: record_snapshot_interval (the field task 9.3's periodic-checkpoint mechanism reads) has no CLI or admin-request wiring anywhere in the codebase — --record-snapshot-interval appears in AC 11's own normative text but in zero task's actual scope. Neither task 9.4 (admin request handler — dispatches SchedAdminRequest actions, which don't carry an interval field either, per controller.capnp's SchedAdminRequest{action, recordFile}) nor group 10 (controller-side dispatch) owns scheduler startup CLI flags; only main.c's argp table does, following task 9.1's own --record-dir precedent. As currently scoped, AC 11's mechanism is fully implemented and tested (via the struct field directly) but has no path to ever be configured non-zero in a real deployment. Needs a small follow-up task before this spec is considered complete: add a --record-snapshot-interval N CLI flag to main.c, wired to sched.record_snapshot_interval, mirroring --record-dir's exact pattern (task 9.1).
• Task 9.3's review also flagged test_record_lifecycle.c's size (826→1232 lines across tasks 9.2+9.3) as past this suite's established over-800 precedent (prior max test_driver_translate.c≈867) — the content itself is genuine, non-duplicative coverage (not padding), and a natural split point exists along the AC10 (lifecycle/TOCTOU)/AC11 (periodic-checkpoint/error-handling) seam by extracting shared fixture helpers into a common header. Non-blocking; worth a future cleanup pass, not urgent.
• Cross-task interaction, flagged before task 9.4 exists — resolution decided (not a fail/retry, a deferral): task 9.2's sched_record_start() calls xce_sched_core_save_snapshot() as its first action (AC 10 step 1) — the same function task 8.2 gated to return -EBUSY whenever session_active != 0. main.c's driver loop is single-threaded (while(1) { xce_sched_driver_run_normal_mode_round(&sched); ...; sleep(1); }); incoming requests (including the future SchedAdminRequest) arrive asynchronously via scheduler_req_handler (handler.c) on the I/O side, same as job/queue/worker updates. Decision: task 9.4 must NOT call sched_record_start() directly from the request-handling thread (that both risks racing session_active and violates this codebase's async-I/O rule against blocking/file-I/O in a request-handler callback). Instead, recordStart must be enqueued the same way job/queue/worker updates already are (pend_updates dataq or an equivalent dedicated pending-admin-request slot) and picked up by xce_sched_driver_translate_pending_events() — which already runs at the top of every round, strictly before begin_session(), so it is guaranteed between-sessions by the same construction argument design.md uses for core_push_events(). This means recording "takes effect" at the next scheduling round, not synchronously on request arrival. Consequence for task 9.4: since AC 10 specifies a synchronous request-reply carrying the resolved filename, the actual SchedAdminReply send must be deferred until the driver thread performs the queued start (not sent eagerly from the handler thread) — needs a completion signal back to the waiting request context (e.g. a condition variable, or replying once xce_sched_driver_translate_pending_events() reports the deferred start applied). Given this deferral makes the direct -EBUSY race effectively unreachable from the admin path, the AC 10 error-codes table doesn't need a new "busy" entry for this case — sched_record_start() keeps its -EBUSY return as a defensive/fail-loud contract check (task 9.2's job, no torn state), it just should no longer be a path task 9.4's dispatch can actually trigger by construction.