myxce/.kiro/specs/scheduler-core/tasks.md

Implementation Plan

• 1. Foundation: Picker Interface & Plugin Registration

• 1.1 Define the job picker plugin contract

  • Add the lifecycle hooks (init, fini), state-change hooks (on_queue_update, on_job_push, on_job_state_change, on_job_update), session hooks (begin_session, next_job, end_session), post-allocation hooks (on_dispatch, on_pend), and optional hooks (on_config, on_job_rusage MAY be NULL) to the picker plugin header
  • Document the contract: every hook signature, which thread each hook category runs in (driver/picker/allocator), cancel-safety requirement for next_job(), and session-reset requirement for begin_session()
  • Observable: the header compiles standalone and defines a complete function-pointer struct matching every hook category
  • Requirements: 1.2, 1.5, 2.6

• 1.2 Register the picker plugin category and deprecate the old election hook

  • Add a picker plugin registration category to the plugin registration table, parallel to existing plugin categories
  • Add a compatibility wrapper that adapts an old-style job_elect_fn-only plugin to the new interface, logging a deprecation warning at registration time
  • Observable: registering a plugin that implements only job_elect_fn succeeds via the wrapper and logs a deprecation warning; registering one that implements the new interface directly succeeds without a warning
  • Requirements: 1.6, 2.5
  • Depends: 1.1

• 1.3 (P) Add picker configuration fields

  • Add a global picker plugin name field to the scheduler, defaulting to builtin_fcfs when unset
  • Add a per-queue picker plugin override field, defaulting to NULL (inherit global)
  • Add a project-name field to the queue struct, and extend the project struct with a priority field (scaffolding only — no quota fields)
  • Observable: a queue config with no override inherits the global picker name; a queue config with an override reports that override, not the global default
  • Requirements: 2.1, 2.2, 2.3, 2.4
  • Boundary: SchedulerCore config fields

• 2. Core: Builtin FCFS Picker

• 2.1 Implement builtin_fcfs lifecycle and private state

  • Implement init/fini: allocate/free the picker's private ordered-queue state (queue ordering by priority desc, and one job-ordering structure per queue)
  • Never query the scheduler's own queue/job tables from inside the picker after init — all candidate state lives in the picker's private structures
  • Observable: after init(), the picker holds an empty private candidate set; after fini(), all picker-owned memory is released (verified under a leak-checked test run)
  • Requirements: 3.1, 3.3
  • Depends: 1.1, 1.2

• 2.2 Implement builtin_fcfs state-change hooks

  • on_queue_update: upsert a queue into the picker's ordered queue list, keyed by descending priority
  • on_job_push: add a newly-pending job into its queue's candidate set, ordered by priority desc then submission/queue time asc
  • on_job_state_change: remove a job from candidates when it leaves pending state, re-add it when it returns to pending
  • on_job_update: reposition a job in its candidate set when its priority or queue assignment changes
  • Observable: feeding a sequence of these hooks directly (no live core) produces a candidate set whose order matches the documented ordering rule, verifiable by a direct hook-driven test
  • Requirements: 3.2
  • Depends: 2.1

• 2.3 Implement builtin_fcfs session and post-allocation hooks

  • begin_session: reset the iteration cursor to the head of the ordered queue list
  • next_job: walk queues then jobs in ordering, skipping not-ready queues/jobs, returning the first ready job and advancing the cursor; returns NULL when exhausted
  • end_session: no-op (iteration state already reset in begin_session)
  • on_dispatch: remove the job from its queue's candidate set permanently
  • on_pend: no-op — job remains in its candidate set for the next round in the same position
  • Observable: a full begin_session → repeated next_job → end_session cycle against a fixed candidate set returns jobs in (queue priority desc, job priority desc, queue_time asc) order and returns NULL exactly once all candidates are exhausted
  • Requirements: 3.2, 3.4, 3.6
  • Depends: 2.2

• 2.4 Wire builtin_fcfs as the registered default and remove the hardcoded picker string

  • Auto-register builtin_fcfs at scheduler startup under the name "builtin_fcfs" and set it as the default picker plugin
  • Replace the hardcoded "xce_sched_fcfs" literal with a lookup of the configured picker plugin name through the existing plugin registration/lookup mechanism; fail scheduler startup with a clear error if the configured name isn't registered
  • Confirm the existing three-thread/cbuf pipeline (picker thread, allocator thread, driver-signaled wbuf/jbuf/decisions cbufs) is unchanged by this wiring — no new threads, no cbuf capacity changes
  • Add cancel points in next_job()'s iteration loop and confirm the picker thread's existing block-on-signal point remains cancel-safe
  • Observable: starting the scheduler with no picker config uses builtin_fcfs automatically; starting it with an unregistered picker name fails startup with a clear error instead of silently falling back
  • Requirements: 1.3, 1.4, 1.6, 2.5, 3.1, 3.5
  • Depends: 2.3, 1.3

• 2.5 Remove ordering logic from the core scheduling loop

  • The core loop calls only picker->next_job() to obtain the next schedulable job — no project/queue/job iteration or ordering logic remains in the loop itself
  • Observable: a code-level check confirms the scheduling loop function contains no direct queue/job traversal — all selection reads happen through the picker interface
  • Requirements: 1.1, 1.7
  • Depends: 2.4

• 2.6 Picker regression tests and data-sharing invariant checks

  • Assert builtin_fcfs's next_job() ordering matches today's pre-change FCFS ordering exactly, given identical queue/job state
  • Confirm the existing scheduler unit test suite passes unmodified except at the specific assertions that exercised the removed hardcoded string and the decision-publishing path this feature replaces
  • Verify resreq objects are never mutated in place by any touched code path, all pend-reason allocation goes through the existing factory function (no direct calloc of the reason struct), comparisons use pointer equality as a fast path before deep comparison, and the existing per-session select-expression memoization is untouched
  • Observable: the regression suite passes with the documented ordering rule as its oracle, and a grep-based or code-review check confirms no new calloc(sizeof(...pendreason...)) call site was introduced
  • Requirements: 4.1, 4.2, 6.1, 6.2, 6.3, 6.4
  • Depends: 2.5

• 3. Run Phase 1 HA soak regression

  • Run sequential and chaotic HA failover soak rounds against a cluster running the new picker architecture
  • Observable: zero new job loss, decision divergence, or liveness stall attributable to this phase's changes across the soak run
  • Requirements: 4.3
  • Depends: 2.6

• 4. Foundation: Scheduler Core Extraction

• 4.1 Extract the scheduling core from the monolithic scheduler struct

  • Split scheduling state (jobs, queues, workers, resources, resvals, users, projects, plugin instances) into its own core structure holding zero controller I/O state (no API client context, no cluster config)
  • Confirm the core links only against the approved minimal data-structure subset and never links network/timer/API libraries — no wall-clock time call appears anywhere in the extracted core's source
  • Observable: the core structure compiles and links with zero symbols from the networking/API libraries; a build-time or link-time check confirms this
  • Requirements: 5.1, 5.2
  • Depends: 3

• 4.2 Pin the determinism invariants

  • Hardcode the picker-to-allocator handoff channel's capacity to 1 inside core initialization — never expose it as a parameter
  • Audit every hash-table/key-value-store iteration reachable from the scheduling loop (jobs, queues, users, projects) and confirm each produces order deterministic in (keys, insert/delete sequence) with no process-random seeding; fix any site found unstable
  • Observable: core initialization rejects (or simply has no code path to accept) a caller-supplied handoff-channel capacity; the iteration-order audit's findings are recorded and any fix is covered by a targeted test
  • Requirements: 5.7
  • Depends: 4.1

• 4.3 Implement the core event interface and batch event push

  • Define the typed core event representation (timestamp plus type-specific payload) covering configuration push, job push, worker push, resource push, resource-value full/delta push, job state change, job update, queue update, and job resource-usage events
  • Implement batch event push: state-change picker hooks fire synchronously for each event in the batch, valid only when no scheduling session is active
  • Observable: pushing a batch of events with the picker idle fires the corresponding state-change hook exactly once per event, synchronously, before the push call returns
  • Requirements: 5.4, 5.8
  • Depends: 4.1

• 4.4 Implement the session-start signal and decision callback protocol

  • Implement the non-blocking session-start call that signals the picker thread and returns immediately, rejecting a call while a session is already active
  • Define the three decision output shapes (dispatch, pend, preempt) and the decision callback invoked once per decision as the allocator produces it, plus the explicit end-of-session marker
  • Observable: starting a session while one is already active is rejected with a distinct error; each decision reaches the registered callback in allocator-output order, followed by the end-of-session marker
  • Requirements: 5.5, 5.6
  • Depends: 4.3

• 4.5 Implement clock virtualization and session timing

  • Implement the virtualized clock accessor using the two-anchor-plus-speed formula, backed by atomically-updated fields; no direct wall-clock call anywhere else in the core
  • Register the core's own log-timestamp formatter, bound to the virtualized clock accessor instead of the logging library's default wall-clock read — this is the one third-party dependency inside the core that would otherwise call wall-clock time on every log line, silently reintroducing the exact dependency task 4.1 confirmed absent at link time
  • Track session start/end timestamps internally and compute session duration for diagnostics, exposed through the session summary
  • Observable: the clock accessor's output matches the documented formula for all four operating modes (normal, discrete simulation, real-time simulation, fast/slow simulation) given fixed anchor values; a log line emitted while the virtual clock is frozen at a fixed simulated timestamp shows that timestamp, not the real wall-clock time at the moment of the call
  • Requirements: 5.3, 5.12
  • Depends: 4.1

• 4.6 Core contract tests

  • Test the full push-events → start-session → drain-decisions → end-session cycle against a stub picker (no driver needed), asserting hook firing order and timing
  • Observable: the contract test suite passes, exercising both the batch-push precondition (rejects mid-session) and the double-session-start rejection
  • Requirements: 5.1, 5.2
  • Depends: 4.4

• 5. Core: Normal-Mode Driver

• 5.1 Build the normal-mode driver's event translation path

  • Translate controller capnp push/notify messages into core events, buffering them in a pending-event queue between scheduling rounds
  • Observable: a sequence of simulated controller push/notify messages, once drained, produces a core event batch whose contents match the input messages one-to-one
  • Requirements: 5.8, 5.9
  • Depends: 4.4

• 5.2 Wire decision publishing, the mode-selection entry point, and retire the hardcoded decision switch

  • Publish each decision to the controller immediately as it's produced (dispatch, pend-reason, preempt) via the existing client API — not batched at session end
  • Replace the scheduler process's hardcoded decision-type switch statement with registration of the driver's decision callback
  • Add the process entry point's mode-selection branch (normal-mode driver vs. simulation driver based on whether a replay file was given) and normal-mode's own CLI flags
  • Observable: submitting a synthetic session that produces a mix of dispatch and pend decisions results in each being published to the (simulated) controller in the order produced, with no batching delay; starting the process with no replay file argument runs the normal-mode driver
  • Requirements: 1.7, 5.6, 5.9
  • Depends: 5.1

• 6. Run Phase 2 HA soak regression

  • Run sequential and chaotic HA soak rounds confirming decisions and controller interaction match Phase 1's behavior exactly
  • Observable: zero new job loss, divergence, or liveness stall; decision content matches Phase 1's soak baseline
  • Requirements: 4.3
  • Depends: 5.2, 4.6

• 7. Foundation: Replay Wire Format

• 7.1 Define the replay stream envelope schema

  • Add the replay header, snapshot entry, replay event, replay session, and replay decision wrapper message types to the shared wire schema, following the single-stream-of-consecutive-messages framing
  • Observable: the schema compiles and a minimal header-plus-snapshot-entry byte stream round-trips through encode/decode
  • Requirements: 5.20
  • Depends: 6

• 7.2 Define new event payload types

  • Add the two genuinely new event payload message types (job state-change delta, job update delta with embedded full record) to the wire schema; every other event payload type reuses an existing wire message already used elsewhere in the schema
  • Observable: encoding a job state-change event and a job update event and decoding them back yields field-for-field equality with the original values
  • Requirements: 5.20
  • Depends: 7.1
  • Boundary: ReplayFormat — not (P): 7.2/7.3/7.4 all edit the same schema file (controller.capnp); run sequentially to avoid concurrent edits to one file, even though each adds a logically disjoint set of structs

• 7.3 Define decision payload types and the snapshot schema

  • Add the dispatch, pend, and preempt decision payload message types, reusing existing allocation and pend-reason wire messages where they already carry the needed fields
  • Add the full snapshot message type covering configuration, jobs, queues, workers, resources, resource values, users, and projects, reusing existing wire messages for each domain object list; represent the two scheduler-internal-only caches and session/PRNG state as a versioned opaque payload with no external structure
  • Observable: encoding one of each decision type and a populated snapshot and decoding them back yields field-for-field equality; a snapshot encoded with one internal-state version and decoded expecting a different version is rejected rather than misinterpreted
  • Requirements: 5.5, 5.20, 5.21, 6.5
  • Depends: 7.1
  • Boundary: ReplayFormat — not (P), same file as 7.2/7.4 (see 7.2)

• 7.4 Add the scheduler admin opcode schema

  • Add the new opcode value, the admin action enum (record start/stop/status), and the paired request/reply message types (reply carrying ok/error/recording-state/file/event-count/started-at) to the wire schema, plus their entries in the existing request/reply message unions
  • Observable: encoding each of the three admin actions and their corresponding replies and decoding them back yields field-for-field equality
  • Requirements: 5.10
  • Depends: 7.1
  • Boundary: ReplayFormat — not (P), same file as 7.2/7.3 (see 7.2)

• 7.5 Replay format round-trip test suite

  • Encode and decode every new message type introduced in this group, asserting field-for-field equality after round-trip
  • Observable: the full round-trip test suite passes for every new struct, including the internal-state version-mismatch negative test
  • Requirements: 5.20
  • Depends: 7.2, 7.3, 7.4

• 8. Core: Snapshot Serialization

• 8.1 Implement snapshot save and load

  • Serialize the full core state (as scoped in the snapshot schema) into the snapshot message; restore from a snapshot message back to that exact state
  • After a load, rebuild the picker's private state by walking the restored core state and re-invoking the picker's state-change hooks (queue updates, then pending-job pushes) — never store picker-private state in the snapshot itself
  • Observable: save immediately followed by load into a freshly-initialized core produces a core whose observable scheduling behavior (next session's decisions, given the same events) is identical to the pre-save core
  • Requirements: 5.21, 6.5
  • Depends: 7.3

• 8.2 Enforce the snapshot session-idle precondition

  • Reject snapshot save/load calls made while a scheduling session is active, returning a distinct busy error rather than capturing or restoring torn state
  • Observable: attempting a snapshot save/load mid-session returns the busy error and leaves core state unchanged
  • Requirements: 5.21
  • Depends: 8.1

• 9. Core: Recording Control

• 9.1 Implement recording directory provisioning and path validation

  • Add a scheduler startup CLI flag (parsed in the same argument-parsing pass as the rest of normal-mode's flags) for the recording directory — never settable from any request-path field — defaulting to a fixed system path
  • Validate any caller-supplied record file name before ever opening it for writing: reject any value containing a path separator or a .. segment, resolve the final path beneath the recording directory, and confirm containment before opening
  • Observable: a record-file name containing a path separator or .. is rejected before any filesystem access occurs; a bare filename resolves to a path inside the recording directory and is accepted
  • Requirements: 5.10
  • Depends: 8.1

• 9.2 Implement the record start/stop lifecycle

  • On start: take a full snapshot, write the replay header followed by the initial snapshot entry, begin tracking recording state
  • On each event batch and each completed session while recording: write the corresponding event/session messages to the record file in order — extends the normal-mode driver main loop built in group 5 with recording insert points, it does not replace it
  • On stop: write the end-of-recording marker, flush and close the file, clear recording state
  • When recording is inactive, incur zero file I/O and zero extra allocation — a single boolean check per session
  • Observable: a start → submit events → complete a session → stop cycle produces a file beginning with header+snapshot-entry and ending with the end-of-recording marker, with events and one session in between, byte-decodable by the round-trip decoder from group 7
  • Requirements: 5.10
  • Depends: 9.1

• 9.3 Implement periodic snapshot checkpointing and recording error handling

  • Append a fresh snapshot entry to the record file every configured number of recorded events (0 disables this)
  • On a write failure mid-recording (disk full, permission error): log the error, cleanly tear down recording state, and continue normal scheduling — recording failure must never stop or crash the scheduler
  • Observable: recording through a configured interval produces the expected number of extra snapshot entries in the file; simulating a write failure leaves the scheduler still processing sessions afterward with recording cleanly disabled
  • Requirements: 5.11
  • Depends: 9.2

• 9.4 Implement the scheduler-side admin request handler

  • Decode a forwarded SchedAdminRequest arriving on the scheduler's controller-facing endpoint, dispatch by action to the record-start/stop/status functions built in 9.2/9.3, and encode+send the SchedAdminReply back to the controller on the same endpoint — this is the scheduler-side counterpart to the controller's forwarding dispatch (task 10.1), without which 10.1 has nothing to forward to
  • Observable: sending each of the three admin actions to the scheduler's endpoint produces the matching reply (recording state, file, event count, started-at populated correctly for each action)
  • Requirements: 5.10
  • Depends: 9.3

• 9.5 Recording lifecycle integration tests

  • Test the full start/stop/status round trip against a real filesystem (through the request handler from 9.4, not by calling the internal functions directly), including the path-validation rejection cases and a simulated disk-full condition
  • Observable: the integration test suite passes, including assertions on the exact error returned for each rejection case
  • Requirements: 5.10, 5.11
  • Depends: 9.4

• 10. Integration: Controller Admin Opcode and xceadmin CLI

• 10.1 Add the scheduler admin opcode dispatch to the controller

  • Add the new opcode's dispatch case to the controller's request handler: reject with a clear error when not primary, reject with a clear error when no scheduler is connected, forward the request to the scheduler and wait up to the timeout for a reply, and always send an explicit reply on every path (never a dropped request)
  • The controller interprets nothing beyond the opcode itself — every field of the request/reply passes through untouched, and the controller commits no state and writes no persistence entry for this opcode
  • Observable: a request against a non-primary controller returns the not-primary error immediately with no forwarding attempt; a request against a primary with no scheduler connected returns the not-connected error immediately
  • Requirements: 5.10
  • Depends: 7.4, 9.4

• 10.2 Controller-side admin opcode dispatch tests

  • Test standby rejection, disconnected-scheduler rejection, forward timeout, and a successful forward/reply round trip against the real scheduler-side handler from 9.4
  • Observable: the test suite passes, covering every reply-on-every-path case named in the dispatch task
  • Requirements: 5.10
  • Depends: 10.1, 9.4

• 10.3 Build the xceadmin scheduler recording commands

  • Implement record-start, record-stop, and record-status client functions that connect to a controller, follow redirects to the primary if needed, send the corresponding admin request, and decode the reply
  • Observable: each function returns a decoded reply struct on success and raises a typed error on connection failure, decode failure, or an error reply
  • Requirements: 5.10
  • Depends: 10.1
  • Boundary: XceAdminSchedulerCLI — not (P): 10.3/10.4 both add functions to the same new scheduler.py module; run sequentially to avoid concurrent edits to one file

• 10.4 Build the xceadmin scheduler dump command

  • Implement a local-file decoder that reads a replay file's message stream and formats it to human-readable text — no network connection involved
  • Observable: running dump against a fixture replay file produces text output listing the header fields, snapshot entry, and each event/session/decision in the file
  • Requirements: 5.20
  • Depends: 7.5
  • Boundary: XceAdminSchedulerCLI — not (P), same file as 10.3 (see 10.3)

• 10.5 Wire the xceadmin CLI commands

  • Add the scheduler command group with a nested record sub-group (start/stop/status commands) and a sibling dump command, each delegating to the functions built in 10.3/10.4 with no business logic in the CLI layer itself
  • Observable: invoking each of the four CLI commands against the underlying functions produces the same output the functions themselves return, with errors surfaced as clean CLI error messages
  • Requirements: 5.10
  • Depends: 10.3, 10.4

• 10.6 xceadmin scheduler CLI tests

  • Test record start/stop/status against a mock TLS server (including redirect-following) and dump against a fixture replay file
  • Observable: the test suite passes for all four commands, including at least one redirect-following case
  • Requirements: 5.10
  • Depends: 10.5

• 11. Core: Simulation Driver

• 11.1 Implement simulation startup

  • Add the simulation-mode CLI flags (replay file path, start timestamp, speed, verify, alt-events file, output file, interactive) that the process entry point's mode-selection branch (built in 5.2) routes into the simulation driver
  • Open and validate the replay file (must begin with the header message), locate the most recent snapshot entry at or before the requested start time, load it, and rebuild picker state from it
  • Set the virtual clock's anchors from the loaded snapshot's timestamp and the requested speed
  • Reject a replay file whose first data message isn't a snapshot entry
  • Observable: starting simulation against a valid fixture file reaches a ready-to-run state with the virtual clock reporting the snapshot's timestamp; starting against a file missing its initial snapshot entry fails with a clear error before any event is processed
  • Requirements: 5.13
  • Depends: 8.1, 7.5

• 11.2 Implement the simulation event loop and virtual clock advancement

  • Advance the virtual clock by jumping directly to each queued event's timestamp — no fixed tick, strictly monotonic, same-timestamp events processed in file order
  • Treat each session-boundary message as the trigger for one complete scheduling round: push accumulated events, start the session, drain decisions, end the session
  • Observable: replaying a fixture file with events at non-uniform timestamps processes them in file order with the virtual clock reporting each event's exact timestamp at the moment it's pushed
  • Requirements: 5.14, 5.18
  • Depends: 11.1

• 11.3 Enforce simulation isolation

  • The simulation driver's state holds no live-controller client context, no cluster configuration, and no capnp socket of any kind — confirmed by the struct definition itself, not runtime checks alone
  • Observable: a build-time or structural check confirms no controller-facing symbol is reachable from the simulation driver's compiled object
  • Requirements: 5.17
  • Depends: 11.1

• 11.4 Implement decision verification

  • Compare each core-emitted decision against the corresponding recorded decision at the same logical position when verification is enabled; report any mismatch with the logical time and decision detail
  • Observable: replaying a fixture file with verification enabled against its own previously-recorded output reports zero mismatches; replaying against a deliberately-altered copy reports the exact mismatch
  • Requirements: 5.15
  • Depends: 11.2

• 11.5 Implement what-if branching and simulation output recording

  • Support loading the snapshot from a base replay file while replaying events from a separate alternate-events file
  • Write simulated decisions to an output replay file in the identical stream format as normal-mode recording, when an output file is requested; otherwise print decisions to standard output only
  • Observable: running with an alternate-events file produces decisions driven by the alternate events, not the base file's original events; the output file, when requested, is decodable by the same reader used for normal-mode recordings
  • Requirements: 5.16, 5.19
  • Depends: 11.2

• 11.6 Deterministic replay and normal-to-simulation equivalence test

  • Record a synthetic scheduling session through the normal-mode driver, then replay the resulting file through the simulation driver, and assert the decision sequences match exactly (same jobid, workerid, allocation amounts, pend reasons)
  • Observable: the equivalence test passes with zero decision-content divergence between the recorded and replayed runs — this is the spec's central regression test
  • Requirements: 5.7
  • Depends: 11.4, 9.2

• 12. Core: Interactive REPL

• 12.1 Implement REPL execution control

  • Implement a single input-loop REPL (not a separate thread) that pauses before the first event and blocks the simulation loop while active, supporting step-N-sessions, continue-to-next-breakpoint-or-EOF, and quit
  • Observable: starting simulation with the REPL enabled pauses before any event is processed; issuing a step command advances exactly the requested number of sessions and returns control to the prompt
  • Requirements: 5.22
  • Depends: 11.2

• 12.2 Implement REPL breakpoints and state inspection

  • Support setting/listing/deleting breakpoints on event type with optional field filters, and inspection commands reporting current simulation position, and job/queue/worker/resource/summary state
  • Observable: setting a breakpoint on a specific event type and field value pauses the REPL exactly when a matching event is about to be processed; each inspection command reports state matching what a snapshot dump of the same moment would show
  • Requirements: 5.22
  • Depends: 12.1

• 12.3 Implement REPL snapshot capture and simulation parameter commands

  • Support writing a full snapshot to a file at the current simulation position, and adjusting speed/output-file/verify-mode from the prompt without restarting
  • Observable: a snapshot command produces a file that the simulation driver can later load via a fresh --sim invocation starting from that exact position
  • Requirements: 5.22
  • Depends: 12.1

• 12.4 REPL equivalence end-to-end test

  • Run a fixture file through the REPL using step/continue to completion and through batch simulation mode, and assert both produce the same decision sequence
  • Observable: both runs produce byte-identical decision sequences
  • Requirements: 5.22
  • Depends: 12.2, 12.3

• 13. Run full cross-phase HA soak regression

  • Run sequential and chaotic HA soak rounds against the complete feature (all three phases integrated), confirming zero job loss, divergence, or liveness stall attributable to this feature across the full run
  • Observable: the soak run completes with a clean pass across all rounds, matching the zero-regression bar set by the Phase 1 and Phase 2 soak runs
  • Requirements: 4.3
  • Depends: 12.4, 10.6

Implementation Notes

• Task 1.3 added sched_set_default_job_pick_plugin() (schedobj.c) but deliberately did NOT call it from sched_init() — task 2.4 owns replacing scheduler.c:43's hardcoded "xce_sched_fcfs" literal and must also wire this defaulting call (or equivalent logic) into scheduler startup, or sched->job_pick_plugin stays NULL through a real boot.
• Background IDE diagnostics ("file not found: rtti.h", "unknown type name" for various xce_* types) have appeared consistently on every scheduler.h/plugin.h/types.h/schedobj.c change so far (tasks 1.1-1.3) and are confirmed stale clangd artifacts each time via independent real meson compile/meson test runs — not real build failures. Don't let these block a task; verify with the real build instead.
• Meson build dir is src/xce.build, not repo-root xce.build — build/test commands must run from src/ (e.g. cd src && nix develop . --command bash -c "meson test -C xce.build --suite scheduler").
• Scheduler testsuite convention: run test binaries directly with ASAN_OPTIONS=detect_leaks=0 (fixtures aren't torn down, by design — matches existing meson.build test registration).
• Task 2.3's review found a real concurrency gap for task 2.4 to close: once builtin_fcfs is wired live, next_job() (picker thread) and on_dispatch/on_pend (allocator thread) can race on the same unsynchronized fcfs_plugin_data_t private state — cbuf_get() unblocks the picker's next cbuf_put() before the allocator finishes processing the dequeued job (jbuf capacity is 1), so the picker can re-enter next_job() while the allocator is still mutating priv->queues via on_dispatch/on_pend. Today's production path never hits this because the allocator thread calls a different plugin category (limit.c's resource-limit hook), never fcfs.c's own private state. Task 2.4 must add a mutex guarding fcfs_plugin_data_t across all five picker hooks (or an equivalent handoff redesign) before registering builtin_fcfs as the live default.
• Task 2.4 closed the above race with fcfs_plugin_data_t.lock (mylock_t), taken by exactly 7 hooks (on_queue_update, on_job_push, on_job_state_change, on_job_update, begin_session, next_job, on_dispatch) — on_pend/end_session/init/fini correctly excluded (they never touch priv->queues/cursor/job_cursor), a verified refinement of this note's original "all five hooks" estimate, not a gap. sched_job_picker() now branches job_elect_fn != NULL (legacy plugins like xce_sched_fcfs, delegates unchanged to their own self-contained loop) vs NULL (native xce_job_picker_iface_t pickers, runs a new generic per-session loop) — this split exists because legacy job_elect_fn implementations (e.g. fcfs_elect_job) already contain their own internal cbuf_get(&wbuf), so a single uniform outer loop would double-consume the wbuf signal for legacy pickers. sched_allocator() now calls picker_iface->on_dispatch()/on_pend() after each allocation attempt (Req 1 AC 4), which was previously called for no picker at all — required for builtin_fcfs to actually remove dispatched jobs from its private state once live.
• Two follow-ups surfaced by task 2.4's review, not yet fixed (outside its file boundary): (1) main.c discards sched_init()'s return value, so a bad job_pick_plugin config fails sched_init() correctly but the process doesn't yet halt on it — needs a fast follow-up before relying on "fails startup" in production. (2) pthread_testcancel() in builtin_fcfs_next_job() is only at the loop entry/exit, not scattered through the queue/job walk (the walk holds fcfs_plugin_data_t.lock throughout, and this codebase has no pthread_cleanup_push precedent to safely unlock on mid-walk cancellation) — behaviorally inert today since sched_queue_is_ready/sched_job_is_ready are return 1 stubs, but becomes a real cancellation-latency gap once Phase 2 implements real filtering.