myxce/.kiro/specs/scheduler-sim-live-allocation/requirements.md

Requirements Document

Project Description (Input)

Give simulation/replay mode genuine live allocation -- today sim_picker_thread_fn() unconditionally PENDs every job it considers instead of running the real election/allocation path normal mode runs, because that path is signatured on xce_scheduler_t* (normal-mode-only) rather than the shared xce_sched_core_t*. Unify the picker/allocator thread bodies and the allocation-chain functions they call into core-safe, single implementations shared verbatim by both drivers, replacing sim_picker_thread_fn() entirely.

The concrete acceptance test for this capability is bug-1826: with real live allocation wired in, info limit's live-usage display shows real data for a replay (both global and host-scoped limits), instead of "no active usage instance yet" -- because limit matching (plugin_job_election()/match_job_global_limit_()/match_job_host_limit_()) is one of the plugin hooks that only ever fires along the now-shared election/allocation path. Limit matching is the feature that surfaced this gap and the feature this spec proves the fix against -- it is not the whole of what this spec delivers.

Background: bug-1825 (already fixed) made the limit plugin's CONFIGURED SETTING visible in sim mode by bootstrapping registration (plugin_register_core + limit_seed_config). But limit MATCHING -- the mechanism that populates live usage instances (xce_vres_t) as jobs get evaluated -- is never invoked during replay, because plugin_job_election() (plugin.c) is normal-mode-only: it casts void* arg to xce_scheduler_t* and reads sched->core.plugins, and xce_sched_sim_driver_t deliberately has no xce_scheduler_t* field.

GOVERNING PRINCIPLE (stated repeatedly by the user during discussion -- this is the design's first-class constraint, not one bullet among many)

The scheduler must use the SAME core for both normal and simulator mode, with only the driver differing. src/scheduler/CLAUDE.md already states this as "one core, two consumers": driver.c (normal mode) and sim_loop.c (simulation mode) are meant to drive the identical xce_sched_core_t state machine and decision-callback protocol, never separate reimplementations of the same logic. .kiro/steering/scheduler-driver-equivalence.md makes this an enforced DESIGN INVARIANT (state equivalence, not just decision equivalence -- identical resource-accounting state, not just a matching decision sequence, at every point in a replay).

This spec's scope is deliberately governed by that principle end to end: both global limit matching and host-scoped limit matching must be wired into sim mode by having sim mode call the literal SAME functions normal mode calls (after a mechanical core-safe migration), not by an approximation or a sim-only reimplementation. An earlier draft of this spec proposed accepting an "approximate" host-scoped match (against only the recorded winning host, since sim mode has no live per-candidate-host search) as a lesser but shippable compromise -- the user explicitly rejected that: prefer completing the core-safe migration so genuine parity is achievable for both, over shipping an approximation that violates the governing principle.

Why this matters more than a style preference (user's own framing, verbatim intent): the entire reason simulation/replay mode exists is to debug the REAL running behavior of the scheduler core without needing a live controller/worker connection. That premise only holds if sim mode reproduces the EXACT behavior of the core -- not an approximation, not a differently-shaped reimplementation that happens to usually agree. An operator debugging a production issue via xce_scheduler -i -s <replay> must be able to trust that what they see is what actually happened, including every plugin-driven side effect (limit matching among them) that live jobs went through. This is not a nice-to-have generalization of bug-1826's fix -- it is the founding premise sim mode was built to satisfy, and any place two separate implementations exist for what should be one core behavior is a latent trust gap in the debugging tool's core promise, not just a code-quality issue.

Consequently, this spec's scope extends one level higher than the limit-matching hooks alone: unify the picker/allocator thread bodies themselves, not only the functions they call. Today these are two separate hand-written implementations:

• Normal mode: sched_job_picker() (scheduler.c:80) + sched_allocator() (scheduler.c:429) -- two threads, handed off via core->jbuf.
• Sim mode: sim_picker_thread_fn() (sim_loop.c:541) -- one fused thread, jbuf unused entirely (confirmed by the audit's dimension-3 finding).

Once this spec's planned core-safe migration lands (the 7-function table below, plugin_job_election(), and the unsatisfiable_cache callback), every single thing sched_allocator() calls is core-safe. At that point there is no remaining technical reason sched_job_picker()/sched_allocator() themselves cannot be re-typed to xce_sched_core_t* and reused VERBATIM by sim mode -- one implementation, two call sites (normal mode's init spawns them against &(sched->core), sim mode's startup spawns the identical two functions against &(sim->core)) -- entirely replacing sim_picker_thread_fn() rather than leaving it as a second, hand-maintained implementation of the same loop.

Known behavioral risk to verify, not a reason to avoid this: sim mode would start genuinely using core->jbuf (a real two-thread hand-off) instead of today's single-fused-thread sequencing. The capacity-1 rendezvous invariant still holds either way (dimension 3, PASS), but the timing/interleaving of when PEND/DISPATCH decisions land on core->decisions could shift with two concurrently-running threads instead of one sequential one. test_repl_equivalence.c, test_sim_contention_parity.c, and test_sim_equivalence.c must be checked against this change before it ships -- treat any assertion that breaks as a signal to look closely (it may be asserting on an artifact of the old fused-thread sequencing rather than a real invariant), not as a reason to revert to two implementations.

Audit findings that make full parity achievable (from a completed /xce-audit-invariants scheduler run, 2026-09-18 -- do not re-derive, verify current status only)

Dimension 1 (driver state-equivalence) -- the key finding: the remaining gap is smaller than it first appeared. Tracing every function in the host-scoped allocation chain down to what it actually touches (not just its declared type) shows almost all of it is PURELY MECHANICALLY coupled to xce_scheduler_t* -- i.e. it only ever reads/writes sched->core.* or worker/ctx fields, with zero genuine dependency on anything xce_scheduler_t-specific:

Function	Location	What it actually touches
sched_meet_host_selection(xce_scheduler_t* sched, w, s)	resource.c:221	Only sched->core.sel_eval_ctx
sched_check_shared_satisfy(xce_scheduler_t* sched, ...)	resource.c:305	Only sched->core.sharedInst/sharedInUse
sched_check_host_satisfy(xce_scheduler_t* sched, w, ...)	resource.c:378	Only w->resvals/w->resInUse + delegates to the above
plugin_prepare_allocation(void* arg, ctx)	plugin.c:591	Only sched->core.plugins (same void*-cast pattern as plugin_job_election)
sched_try_alloc_on_host_(w, xce_scheduler_t* sched, ctx)	resource.c:497	Only calls the three rows above
sched_commit_decision(xce_scheduler_t* sched, decision)	scheduler.c:1607	Only sched->core.*; already-core-safe sched_commit_job_alloc(core,...) underneath
plugin_commit_decision(void* arg1, decision)	plugin.c:442	Only sched->core.plugins (same void*-cast pattern)
sched_ctx_get_uncomit_host(host, ctx, idx)	schedobj.c:522	Already core-safe -- no xce_scheduler_t* at all

This is the exact same shape plugin_begin_session, plugin_end_session, sched_resolve_job_picker, sched_get_job, sched_account_jobs, and sched_commit_job_alloc were in before their own (already-completed) migration to xce_sched_core_t* -- a pattern this codebase has proven safe six times over, not a new one to invent.

The one genuine exception: xce_scheduler_t.unsatisfiable_cache, consulted/populated only around sched_make_primary_alloc_()'s top-level dispatch chokepoint (scheduler.c, wired in by the completed scheduler-allocation spec's task 5.1). It is a pure performance optimization (skip recomputing an allocation for an already-known-unsatisfiable job fingerprint) with no correctness effect -- excise it behind a NULL-safe, driver-registered accessor mirroring the already-proven cache_clear_cb pattern (xce_sched_core_t.cache_clear_cb, set only in the driver-mode branch of main.c; that pattern exists specifically because a __attribute__((weak)) cast-based shortcut around this same field once silently corrupted xce_sched_sim_driver_t.alt_events_file at the same byte offset, invisible to ASAN and the whole test suite -- do not repeat that mistake by casting instead of using the callback). Sim mode simply always takes the slow (always-compute-fresh) path; this is correct, not an approximation, since the cache never changes what a computation yields, only whether it's skipped.

sim_picker_thread_fn()'s current behavior must also change: its own header comment documents that sim mode makes NO real allocation attempt today -- it unconditionally PENDs every job the picker offers, specifically because sched_make_allocation() needed the live xce_scheduler_t* sim mode doesn't carry. Once the chain above is core-safe, this function needs to actually attempt allocation (mirroring sched_allocator()'s structure: resolve picker -> loop: next_job() -> job election -> allocation attempt -> decide) instead of always PENDing.

Dimension 2 (core I/O isolation) -- unrelated, logged separately as bug-1827, does not block this spec. core_clock.c is missing from two of test_core_isolation.c's three enforcement checks (a test-coverage gap, not a confirmed live violation). Tracked independently; no action needed here.

Dimension 3 (jbuf capacity-1 rendezvous) -- PASS as audited; this spec deliberately supersedes that finding, not by accident. At audit time, sim mode bypassed jbuf entirely (fused picker+allocator into one thread). Once this spec's scope grew to unify sched_job_picker()/sched_allocator() themselves (Scope item 3 below; Requirement 3 AC3/AC4; Requirement 5), sim mode intentionally starts using core->jbuf for real, via the identical two-thread producer/consumer pattern normal mode already uses -- not a new, second mechanism. The invariant this dimension protects (capacity hardcoded to 1, exactly one producer and one consumer per session) still holds, because it's the same already-verified code, not a new implementation of the rendezvous. Do not read this paragraph's original "sim mode bypasses jbuf" framing as still current -- it described the state before that scope decision, not the target.

Dimension 4 (snapshot field-completeness) -- PASS, not relevant to this spec. No undocumented gaps found; no interaction with limit-matching wiring.

Scope (revised: full parity, both limit hooks)

1. Core-safe migration of the ~7 functions in the table above (mechanical, xce_scheduler_t* -> xce_sched_core_t*, following the already-proven pattern) plus the unsatisfiable_cache callback-indirection design (a new xce_sched_core_t callback slot, registered only by main.c's driver-mode branch, mirroring cache_clear_cb).
2. plugin_job_election() (plugin.c:554) made core-safe the same way, single implementation shared by both drivers (mirrors plugin_begin_session's own already-completed migration). Its one normal-mode call site (scheduler.c's sched_allocator(), called once per job right after sched_get_job() and before sched_make_allocation()) gets updated to pass &(sched->core).
3. sched_job_picker() (scheduler.c:80) and sched_allocator() (scheduler.c:429) made core-safe (xce_sched_core_t*) and reused VERBATIM by sim mode, replacing sim_picker_thread_fn() entirely -- not just the functions they call. Sim mode's startup spawns these same two thread functions against &(sim->core) instead of maintaining a separate, hand-written sim_picker_thread_fn() implementation. This means sim mode genuinely uses core->jbuf for the first time (a real two-thread hand-off, not the current single-fused-thread shortcut). sim_reconcile_unaccounted_jobs() should also call the core-safe job-election function for a reconciled job (idempotency already guarded via obj->configver >= core.configver, so calling from both places is safe). Required verification: test_repl_equivalence.c, test_sim_contention_parity.c, and test_sim_equivalence.c must be re-run and any decision-timing/ordering change investigated (see governing-principle section above) before this is considered done.
4. Mechanical gap in xce_sched_plugin_t.sched: it is a bare void*, set ONLY by plugin_register() (normal mode). plugin_register_core() (sim-mode registration, bug-1825) never sets it, so limit_job_election()/match_job_global_limit_()/match_job_host_limit_()'s existing xce_scheduler_t* sched = (xce_scheduler_t*)plugin->sched; derivation reads NULL in sim mode today. Extend the same honest-xce_sched_core_t*-parameter treatment bug-1825 already established for limit_seed_config()/limit_seed_config_impl() (config-hook path) to the job-election and prepare-allocation call chains, rather than relying on plugin->sched.
5. Both match_job_global_limit_() (global/shared limits, via job_election_hook) and match_job_host_limit_() (host-scoped limits, via prepare_allocation_hook, reached through the now-core-safe sched_try_alloc_on_host_()/plugin_prepare_allocation() chain) end up genuinely invoked from sim mode by the SAME code path normal mode uses -- no approximation, no sim-only reimplementation.
6. Ordering constraint (existing sim_loop.c comment, must be preserved): any new per-round election/allocation call must complete strictly between xce_core_begin_session()'s resval reset and xce_core_signal_go() unblocking the picker -- the same window sim_reconcile_unaccounted_jobs()/plugin_begin_session()/sched_account_jobs() already occupy.
7. Testing discipline (per scheduler-driver-equivalence.md): validate with a resource-contention fixture (multiple jobs competing for capacity insufficient for all of them, including limit-constrained capacity), not only a no-contention fixture -- a no-contention fixture cannot distinguish "accounting/limit-matching is correct" from "it's wrong but nothing currently reads it."
8. Precedent: bug-1825 (commit 487e7b94) is the direct precedent for code style (factory extraction, honest-core-typed entry points, no first-member struct cross-casting -- core.h's xce_core_cache_clear_cb_t doc comment documents the prior memory-corruption incident from that exact anti-pattern, structurally identical to the unsatisfiable_cache incident found during this investigation's audit).
9. Validation discipline, distinct from item 7's contention-fixture discipline: before this feature is considered done, run --sim-verify against a range of REAL recorded sessions (not only crafted unit fixtures) -- this is the most direct way to catch a subtle divergence the migration itself introduces, precisely because sim mode's live decision engine is new here and was never previously exercised against real production recordings. Per Requirement 3 AC5, a mismatch surfaced this way is a finding to root-cause (either a defect in this migration, or a genuine pre-existing issue the old fake-PEND behavior was hiding), never something to explain away or suppress to make the check pass.

Introduction

Simulation/replay mode (xce_scheduler -i / -s <replay>) exists so an operator can debug the scheduler's real decision-making from a recorded live session, without a connected controller or worker. Today it falls short of that promise for one whole class of behavior: job election and allocation. Simulation mode currently pends every candidate job unconditionally rather than running the same election/allocation logic normal mode runs, so any plugin-driven behavior that depends on election or allocation never happens during replay — most visibly, resource-limit matching, so info limit cannot show a replay's real limit usage even when the original live session actively enforced it. This spec makes simulation mode reproduce normal mode's genuine job-election, allocation, and limit-matching behavior, so that replaying a recording reliably reproduces what actually happened.

Boundary Context

• In scope: job election and allocation logic (including global and host-scoped resource-limit matching) running identically in simulation mode and normal mode, for both freshly-considered jobs and jobs reconstructed from a recording; simulation mode's resource-accounting state (worker and shared resource usage, per-job resource accounts, limit usage) matching normal mode's under the same input, including under resource contention; info limit showing real usage data for a replay whose original session had active limits.
• Out of scope: any change to normal mode's own observable scheduling decisions or behavior; the previously-identified, unrelated core-module-isolation test-coverage gap (tracked separately, not part of this feature); any new REPL command or info subcommand (the existing info limit command's display logic is unchanged — only the underlying data it reads becomes real); silently resolving a disagreement between the live-computed decision and the recording in either direction — such a disagreement must be surfaced, not hidden, by this feature; achieving simulation mode's allocation-decision throughput or latency parity with normal mode (correctness of decisions and accounting state is the requirement, not performance).
• Adjacent expectations: this feature depends on existing snapshot and recording formats, and existing REPL/CLI behaviors (info limit, info worker, --sim-verify, interactive vs. batch simulation), continuing to work unchanged; it does not require any change to how a live session records its decisions or how a replay file is structured. The existing recorded-decision reconciliation mechanism continues to exist as a narrower safety net for a job the live election/allocation path has not yet reached (the timing gap between an event's arrival and the live path's own asynchronous processing of it) — it is no longer expected to be the general-case source of truth for dispatch decisions once the live path can reach and decide most jobs directly. This feature does not require simulation mode to gain a clean shutdown/teardown path for its worker threads beyond what exists today.

Requirements

Requirement 1: Live Job Election and Allocation in Simulation Mode

Objective: As a scheduler operator debugging via replay, I want simulation mode to make real dispatch/pend decisions using the same job-election and allocation logic as normal mode, so that replay-based debugging reflects genuine scheduler behavior rather than a simplified stand-in.

Acceptance Criteria

1. When a job becomes a candidate for dispatch during a simulation-mode session, the Scheduler shall evaluate it using the same job-election and allocation logic normal mode uses for an equivalent job.
2. While operating in simulation mode, the Scheduler shall not unconditionally pend every candidate job regardless of actual resource availability.
3. When a job is dispatched during a simulation-mode session, whether newly decided or reconstructed from a recording, the Scheduler shall apply the identical resource-accounting effects (worker and shared resource usage) that normal mode would apply for the same dispatch.

Requirement 2: Resource-Limit Matching Parity

Objective: As a scheduler operator inspecting resource limits during replay, I want simulation mode to evaluate configured limits the same way normal mode does, so that info limit shows a limit's real, currently-accounted usage rather than reporting no usage at all.

Acceptance Criteria

1. When a job in a simulation-mode session matches a configured global (non-host-scoped) limit, the Scheduler shall record that match and its usage the same way normal mode would.
2. When a job in a simulation-mode session matches a configured host-scoped limit, the Scheduler shall record that match and its usage the same way normal mode would.
3. When an operator runs info limit <name> or info limit all during a simulation-mode session against a replay whose original live session had active limits, the Scheduler shall display the limit's live usage reflecting jobs actually dispatched in that replay, instead of reporting no active usage instance.
4. The Scheduler shall continue to report no active usage instance, via info limit, for a limit that was never matched by any job in the replay.

Requirement 3: State Equivalence Between Simulation and Normal Mode

Objective: As a scheduler operator relying on replay for debugging, I want simulation mode's resource-accounting state to match what the same events produced in the original live run, so that I can trust conclusions drawn from replay-based debugging.

Acceptance Criteria

1. When a simulation-mode session replays the exact recorded events of a normal-mode run, the Scheduler shall reproduce identical scheduling decisions and identical resource-accounting state (worker resource usage, shared resource usage, per-job resource accounts, and limit usage) at every point in the replay, not only a matching final decision sequence.
2. While multiple jobs in a replay compete for resource capacity insufficient to satisfy all of them concurrently, including capacity constrained by a matching limit, the Scheduler shall produce the same admission or pend outcome in simulation mode that normal mode produced for the same contention.
3. When the Scheduler's live job-election and allocation logic reaches and decides a job during a simulation-mode session, that live decision shall be authoritative for the job's outcome in the replay.
4. If a job's dispatch is found in the replay's recording before the live election/allocation logic has reached that job, the Scheduler shall reconstruct that job's outcome from the recording so the job is not lost or indefinitely delayed, and shall not subsequently re-decide or re-charge that same job once its outcome has been established this way.
5. If the Scheduler's live-computed decision for a job differs from that job's decision in the original recording, the Scheduler shall report this divergence to the operator rather than silently discarding either outcome or resolving the disagreement without visibility.

Requirement 4: No Regression to Existing Simulation-Mode Guarantees

Objective: As a scheduler operator using existing simulation-mode tooling, I want the introduction of live allocation to preserve today's determinism and comparison guarantees, so that existing replay-based workflows continue to work correctly.

Acceptance Criteria

1. When a simulation-mode session is run with live-vs-recorded verification enabled, the Scheduler shall continue to compare each live decision against its corresponding recorded decision and report any mismatch, using its existing comparison mechanism.
2. The Scheduler's interactive and batch simulation modes shall remain decision-equivalent for the same input.
3. While a simulation-mode session is running, the Scheduler shall preserve its existing guarantee that only one job is being selected and allocated at a time, with no job double-processed or over-committed.

Requirement 5: Single Shared Implementation, Not Independently-Verified Equivalence

Objective: As a maintainer of this codebase, I want simulation mode and normal mode to run the same job-election and allocation implementation rather than two separately-maintained implementations that are merely tested to currently agree, so that the two modes cannot silently drift apart the way this project's history shows they otherwise do.

Acceptance Criteria

1. The Scheduler shall implement job-election and allocation logic as a single implementation used by both simulation mode and normal mode, not as separate mode-specific implementations whose agreement is established only by testing.
2. Where a piece of state or behavior is genuinely mode-specific (present in normal mode but not simulation mode, or vice versa), the Scheduler shall isolate that difference behind an explicit, narrow seam rather than duplicating the surrounding shared logic to accommodate it.