lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(scheduler-sim-live-allocation): add unsatisfiable-cache callback seam (task 1.1)

Browse source 
Adds a NULL-by-default lookup/populate callback seam on xce_sched_core_t,
mirroring cache_clear_cb's proven shape, so the allocation dispatch
chokepoint can consult the unsatisfiable-resreq cache without a
normal-mode-only field access in shared code. Adapters live in
alloc_cache.c and are registered only during normal-mode startup;
simulation mode leaves both callbacks NULL and always computes fresh.

Adds the scheduler-sim-live-allocation spec (requirements/design/
research/tasks) as the planning artifact this and subsequent tasks
implement against.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
Rongsong Shen 2026-09-18 10:14:04 +00:00
parent 487e7b9419
commit 89297dc825
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
10 changed files with 1130 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

505
.kiro/specs/scheduler-sim-live-allocation/design.md Normal file
View file

@ -0,0 +1,505 @@
# Technical Design: scheduler-sim-live-allocation
## Overview
**Purpose**: This feature gives the scheduler's simulation/replay driver (`xce_scheduler -i` / `-s <replay>`) a genuine, live job-election and allocation path, sharing a single implementation with the normal-mode driver, instead of unconditionally pending every job as it does today.
**Users**: Scheduler operators and developers debugging production scheduling behavior offline via replay, and this codebase's own maintainers, who need normal-mode and simulation-mode to stay behaviorally identical without relying on two independently-verified implementations.
**Impact**: Today, `sim_picker_thread_fn()` (`sim_loop.c`) unconditionally PENDs every job it considers; every real dispatch decision in simulation mode instead comes from a separate mechanism, `sim_reconcile_unaccounted_jobs()`, that trusts a recorded decision rather than computing one. This feature replaces `sim_picker_thread_fn()` with the literal `sched_job_picker()`/`sched_allocator()` functions normal mode already uses (after migrating their remaining `xce_scheduler_t*`-coupled dependencies to the shared `xce_sched_core_t*`), makes limit matching (the concrete acceptance test, bug-1826) work as a side effect of that shared path, and narrows `sim_reconcile_unaccounted_jobs()` to a safety net for the async timing gap between an event's arrival and the live path's processing of it.
### Goals
- Job election and allocation run as one implementation, used verbatim by both drivers.
- Global and host-scoped resource-limit matching work in simulation mode with the same fidelity as normal mode.
- A divergence between a live-computed decision and its recorded counterpart is surfaced, never silently resolved.
- Zero change to normal mode's own observable behavior.
### Non-Goals
- Simulation-mode throughput/latency parity with normal mode (the `unsatisfiable_cache` optimization stays normal-mode-only).
- A new clean shutdown/teardown path for simulation-mode worker threads (out of scope; pre-existing asymmetry, not introduced by this feature).
- Any new REPL command, `info` subcommand, or replay-file format change.
- Resolving the (currently unconfirmed) `job_alloc_fn`/`register_seqjob_allocator`/`register_parjob_allocator` plugin-hook question definitively before implementation — flagged as a verify-at-implementation-time risk in `research.md`, not blocking this design.
## Boundary Commitments
### This Spec Owns
- The core-safe (`xce_sched_core_t*`) signatures of the allocation-chain functions listed in the Architecture section's migration table.
- `plugin_job_election()`, `plugin_prepare_allocation()`, and `plugin_commit_decision()`'s dispatcher signatures and their single shared implementation across both drivers.
- The limit plugin's honest-core-parameter entry points for job election and host-scoped matching (extending the `limit_seed_config()` pattern bug-1825 established for the config-hook path).
- `sched_job_picker()`/`sched_allocator()`'s core-safe signatures and their use as the sole picker/allocator implementation for both drivers, replacing `sim_picker_thread_fn()`.
- The `unsatisfiable_cache` callback seam on `xce_sched_core_t` and its NULL-safe fallback behavior.
- The authority rule between live-computed decisions and `sim_reconcile_unaccounted_jobs()`'s recorded-decision reconstruction, and the divergence-reporting behavior when they disagree.
### Out of Boundary
- The `scheduler-allocation` spec's own algorithm internals (`host_iter.c`, `parjob_allocator.c`, the unsatisfiable-resreq cache's own lookup/populate logic) — this spec only wraps that cache behind a seam, it does not change what the cache does or how it decides.
- bug-1827 (the `core_clock.c` isolation-test coverage gap) — unrelated, tracked separately.
- Any change to `.replay` file format, recording/capture (`recording.c`/`recording_events.c`), or snapshot format (`snapshot_wire.c`/`snapshot_load.c`) — confirmed field-complete and unaffected by the audit (dimension 4).
- Giving simulation mode a working `unsatisfiable_cache` implementation of its own.
- Simulation-mode thread lifecycle/teardown (`sched_fini()`-equivalent for sim mode does not exist today and this spec does not add one).
### Allowed Dependencies
- `core.c`/`core_events.c`/`core_session.c`/`core_clock.c` (the mechanically-isolated core module) — read-only from this spec's perspective; no changes needed there beyond the new optional callback field on `xce_sched_core_t`.
- The completed `scheduler-allocation` spec's `host_iter.c`/`parjob_allocator.c`/`seqjob_allocator.c` — this design depends on their *algorithm* (host iteration order, bucket/plan construction) staying unmodified; it only touches the type threaded through their existing entry points (see File Structure Plan for the specific, minimal changes each file needs).
- bug-1825's plugin-bootstrap functions (`sim_bootstrap_picker()`, `sim_bootstrap_limit_plugin()`, `limit_seed_config()`) — this spec's honest-core-parameter treatment of job-election/prepare-allocation extends the same pattern those functions already established.
- `cbuf`/`mylock` (`src/lib`) — the existing rendezvous primitive; no changes.
### Revalidation Triggers
- A future change to `xce_sched_plugin_t`'s hook signatures (`plugin_hooks_t`) must re-verify every hook dispatcher this spec touches (`job_election_hook`, `prepare_allocation_hook`, `commit_hook`) stays core-safe.
- A future change to `xce_scheduler_t.unsatisfiable_cache`'s data shape must re-verify the callback seam's contract still matches.
- A future spec giving simulation mode a working live allocator with its own cache, or a clean teardown path, must re-check this spec's Non-Goals are still accurate.
- Any change to `sched_job_picker()`/`sched_allocator()`'s signature or behavior automatically changes simulation mode too (single implementation) — normal-mode-only test coverage is no longer sufficient; both drivers' test suites must be re-run.
## Architecture
### Existing Architecture Analysis
This is an extension of the already-established "one core, two consumers" split (`scheduler-core` spec): `xce_sched_core_t` (passive state + synchronization primitives, mechanically isolated from controller I/O) is driven by two interchangeable top-level structs, `xce_scheduler_t` (normal mode) and `xce_sched_sim_driver_t` (simulation mode), each embedding `xce_sched_core_t core` as a real field. Six functions in the picker/allocator path (`plugin_begin_session`, `plugin_end_session`, `sched_resolve_job_picker`, `sched_get_job`, `sched_account_jobs`, `sched_commit_job_alloc`) already completed the `xce_scheduler_t*` → `xce_sched_core_t*` migration and are called identically by both drivers today. This design completes that migration for the remaining allocation-chain functions and the picker/allocator thread bodies themselves.
**Project invariants checked** (`/xce-audit-invariants scheduler`, run during this spec's requirements phase — see `requirements.md`'s Project Description and `research.md` for full evidence):
- **Driver state-equivalence** (`scheduler-driver-equivalence.md`) — this spec's entire purpose is closing the remaining gap in this dimension. Checked against every function in the migration table below by reading its body, not inferring from its signature.
- **Core module I/O isolation** — not touched by this spec (only `xce_sched_core_t`'s field list grows by one optional callback pointer; no I/O, no new controller/netio dependency). The unrelated coverage gap found in this dimension (bug-1827) does not block this design.
- **jbuf capacity-1 rendezvous** — this spec deliberately changes simulation mode from bypassing `jbuf` entirely to using the same already-verified two-thread producer/consumer pattern normal mode uses. The invariant (capacity hardcoded to 1, one producer, one consumer) is unaffected because it's the same code, not a new implementation.
- **Snapshot field-completeness** — not touched; no new persisted field, no change to `snapshot_wire.c`/`snapshot_load.c`.
### Architecture Pattern & Boundary Map
```mermaid
graph TB
subgraph NormalMode
NormalDriver[NormalModeRound]
end
subgraph SimMode
SimDriver[SimDriverRun]
end
subgraph SharedCore
CoreState[xce_sched_core_t]
PickerThread[SchedJobPicker]
AllocatorThread[SchedAllocator]
ElectionDispatch[PluginJobElection]
AllocationChain[AllocationChain]
CommitDispatch[PluginCommitDecision]
end
subgraph SharedPlugins
LimitPlugin[LimitPlugin GlobalAndHostMatch]
end
subgraph NormalOnly
CacheSeam[UnsatisfiableCacheOps]
end
subgraph SimOnly
Reconcile[SimReconcileUnaccountedJobs]
end
NormalDriver --> CoreState
SimDriver --> CoreState
CoreState --> PickerThread
PickerThread --> AllocatorThread
AllocatorThread --> ElectionDispatch
ElectionDispatch --> LimitPlugin
AllocatorThread --> AllocationChain
AllocationChain --> LimitPlugin
AllocationChain --> CacheSeam
AllocatorThread --> CommitDispatch
SimDriver --> Reconcile
Reconcile -.-> CoreState
```
**Architecture Integration**:
- Selected pattern: single shared implementation for election/allocation (Requirement 5), with mode-specific differences isolated behind two narrow seams — the `UnsatisfiableCacheOps` callback (normal-mode-only, NULL in sim mode) and `SimReconcileUnaccountedJobs` (sim-mode-only, narrowed to the timing-gap safety net).
- Domain/feature boundaries: `SharedCore` is driver-agnostic and callable from both; `NormalOnly`/`SimOnly` subgraphs are the only places behavior legitimately differs.
- Existing patterns preserved: `xce_sched_core_t` embedding, the `cache_clear_cb`-style optional-callback seam, the plugin-hook-dispatcher pattern (`plugin_begin_session`/`plugin_end_session` already core-safe).
- New components rationale: `UnsatisfiableCacheOps` is new (a seam that didn't need to exist while the allocation chain was normal-mode-only); everything else is an existing component moved to a core-safe signature, not a new component.
- Steering compliance: directly satisfies `scheduler-driver-equivalence.md`'s state-equivalence invariant and `src/scheduler/CLAUDE.md`'s "one core, two consumers" statement.
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Backend / Core | C (gnu17) | All changes are within the existing scheduler C codebase | No new language/runtime |
| Concurrency | POSIX pthreads, existing `cbuf`/`mylock` (`src/lib`) | Simulation mode gains a genuine two-thread producer/consumer pair, reusing the exact primitives normal mode already uses | No new synchronization primitive introduced |
| Build | Meson | New/changed files wired into `sched_sim_isolation_lib` and normal build targets | Follows existing `meson.build` conventions per `src/scheduler/CLAUDE.md` |
## File Structure Plan
### Modified Files
- `src/scheduler/include/core.h` — add `unsatisfiable_lookup_cb`/`unsatisfiable_populate_cb` (two optional function-pointer fields, NULL unless registered) to `xce_sched_core_t`, alongside the existing `cache_clear_cb` field and documented with the same `xce_core_cache_clear_cb_t`-style rationale comment.
- `src/scheduler/src/alloc_cache.c` — add the two small adapter functions implementing `unsatisfiable_lookup_cb`/`unsatisfiable_populate_cb`, alongside the existing `xce_alloc_cache_clear_cb()` adapter (same file, same reason: normal-mode-exclusive code that safely recovers `xce_scheduler_t*` from `core`, kept out of `driver.c` so the ~11 white-box test files that `#include driver.c` without linking `alloc_cache.c` are unaffected).
- `src/scheduler/src/schedobj.c` — `sched_ctx_init()` populates `xce_sched_ctx_t.sched` with `core` instead of `sched`; `sched_ctx_set_primary()`'s existing `(xce_scheduler_t*)ctx->sched` cast becomes `(xce_sched_core_t*)ctx->sched`; `sched_ctx_get_uncomit_host()` is already core-safe, unchanged.
- `src/scheduler/src/host_iter.c`, `src/scheduler/src/parjob_allocator.c`, `src/scheduler/src/seqjob_allocator.c` (`scheduler-allocation` spec's own files) — no algorithm change; their entry points (`my_alloc()`, `host_iter_init()`) accept whatever type their caller (`sched_make_primary_allocation()`, now core-safe) and `ctx->sched` (now `xce_sched_core_t*`) hand them. Confirm the exact signature ripple at implementation time (see `research.md` Risks) — this design commits to the seam (`ctx->sched`'s type), not to enumerating every internal call site in these files.
- `src/scheduler/include/plugin.h` — no signature change to `plugin_hooks_t` itself; `plugin_job_election()`, `plugin_prepare_allocation()`, `plugin_commit_decision()` declarations change from `void*` to `xce_sched_core_t*`; `xce_sched_plugin_t` gains a new `xce_sched_core_t* core` field (alongside the existing, untouched `sched` field — see Limit Plugin Core-Safe Matching for why they stay separate).
- `src/scheduler/src/plugin.c` — re-type `plugin_job_election()`, `plugin_prepare_allocation()`, `plugin_commit_decision()` to take `xce_sched_core_t* core` and iterate `core->plugins` directly (mirrors `plugin_begin_session()`'s existing body); `plugin_register()` additionally sets `plugin->core = &(sched->core);`; `plugin_register_core()` additionally sets `plugin->core = core;`.
- `src/scheduler/src/resource.c` — re-type `sched_meet_host_selection()`, `sched_check_shared_satisfy()`/`sched_check_shared_satisfy_()`, `sched_check_host_satisfy()`/`sched_check_host_satisfy_()`, `sched_try_alloc_on_host_()` to take `xce_sched_core_t* core`.
- `src/scheduler/src/scheduler.c` — re-type `sched_commit_decision()` and the full allocation-dispatch switch (`sched_make_allocation()`, `sched_make_primary_allocation()`, `sched_make_compound_allocation()`, `sched_make_alter_allocation()`, and whichever internal function turns out to hold the direct `sched->unsatisfiable_cache` touch — believed to be `sched_make_primary_alloc_()`, confirm at implementation time) to take `xce_sched_core_t* core`; update their bodies' internal `sched->core.X` references to `core->X`; replace the one direct `sched->unsatisfiable_cache` touch with NULL-checked calls to `core->unsatisfiable_lookup_cb`/`core->unsatisfiable_populate_cb`; **remove** `sched_job_picker()`/`sched_allocator()` (moved out, see new file below); update `sched_init()` to spawn the moved functions and pass `&(sched->core)`.
- `src/scheduler/src/sched_election.c` (**new file**) — houses the core-safe `sched_job_picker()`/`sched_allocator()`, extracted verbatim (signature migrated) from `scheduler.c`. Two reasons, not one: (1) it is the single implementation both drivers spawn, so it should not live inside a file (`scheduler.c`) that is otherwise entirely normal-mode-only and controller-coupled; (2) `scheduler.c` is already 1382 lines, over this project's own 800-line cap, and `src/scheduler/CLAUDE.md` explicitly asks for opportunistic splitting rather than further growth — this migration is exactly such an opportunity, not scope creep invented for its own sake. Declared in a small new header or an existing shared one (`scheduler.h`), whichever this codebase's own convention for a two-function extraction favors — confirm at implementation time.
- `src/scheduler/src/plugins/limit.c` — add `limit_job_election_core()`/`limit_prepare_allocation_core()` (or equivalently-named honest-core entry points) alongside the existing hook-registered `limit_job_election()`/`limit_prepare_allocation()`, following the `limit_seed_config()`/`limit_seed_config_impl()` split bug-1825 established; `match_job_global_limit_()`/`match_job_host_limit_()` take `xce_sched_core_t* core` directly instead of deriving `sched` from `plugin->sched`.
- `src/scheduler/src/sim_loop.c` — replace the `pthread_create(&picker_tid, NULL, sim_picker_thread_fn, &(sim->core))` call in `sched_sim_driver_run()` with two `pthread_create()` calls against `sched_job_picker`/`sched_allocator`, passing `&(sim->core)`; remove `sim_picker_thread_fn()` entirely; `sim_reconcile_unaccounted_jobs()` gains a call to the core-safe job-election entry point before its FOUND-branch decision reconstruction (idempotency already guarded by `obj->configver >= core.configver`).
- `src/scheduler/src/main.c` — register `unsatisfiable_lookup_cb`/`unsatisfiable_populate_cb` only in the normal-mode (controller-connected) startup branch, mirroring the existing `cache_clear_cb` registration's placement.
- `src/scheduler/meson.build` — add `sched_election.c` to the normal build's `scheduler_srcs` (both drivers link it, since it lives in the single `xce_scheduler` binary alongside `scheduler.c`/`sim_loop.c`); confirm whether it also needs adding to `sched_sim_isolation_lib`'s source set (Req 5 AC 17 coverage), mirroring how `sim_limit.c` was added there for the analogous reason during `scheduler-sim-diagnostics` task 6.
- White-box test files (`test_sim_*.c`, `test_repl_*.c`, `test_scheduler_*.c`, per this codebase's established "duplicate, don't share" stub convention) — any file that `#include`s `plugin.c`/`resource.c`/`scheduler.c` without linking its full dependency set will need its link-satisfying stubs updated for the new `xce_sched_core_t*` signatures, following the exact ripple-fix pattern already used for `limit_get_configured_limits()`/`limit_get_usage_instances()` (scheduler-sim-diagnostics task 2.1's Implementation Note) and `limit_seed_config()` (bug-1825's own remediation). Extracting `sched_job_picker()`/`sched_allocator()` into `sched_election.c` specifically limits this ripple for `sim_loop.c`'s own white-box tests: they now only need to link the small new file to exercise real election/allocation, not the whole of `scheduler.c`.
> Exactly one new file (`sched_election.c`) is introduced, extracting two existing functions rather than adding new behavior — justified by the isolation-boundary and file-size reasons given above, not by any new component or abstraction layer. Every other change is a signature/call-site migration of an existing function, or a small addition inside an existing file.
## System Flows
```mermaid
sequenceDiagram
participant Driver as DriverPerMode
participant Core as SchedCore
participant Picker as SchedJobPicker
participant Allocator as SchedAllocator
participant Election as PluginJobElection
participant Limit as LimitPlugin
participant Alloc as AllocationChain
participant Commit as PluginCommitDecision
Driver->>Core: PushEvents
Driver->>Core: BeginSession
Driver->>Core: SignalGo
Core->>Picker: wbuf go signal
Picker->>Core: next_job via jbuf
Core->>Allocator: jbuf handoff
Allocator->>Election: job election
Election->>Limit: match global limit
Allocator->>Alloc: attempt allocation
Alloc->>Limit: match host scoped limit
Alloc-->>Allocator: dispatch or pend
Allocator->>Commit: commit decision
Commit-->>Core: decisions cbuf
Core-->>Driver: drain decisions
```
This flow is identical for both drivers — the only per-mode difference is what `Driver` does before pushing events (capnp translation for normal mode, replay decode for simulation mode) and after draining decisions (publish/commit for normal mode, print/record/`--sim-verify`-compare for simulation mode).
```mermaid
flowchart TD
Start[Job accounts is null this round] --> LiveReached{Live election allocation already reached this job}
LiveReached -->|Yes| LiveWins[Live decision is authoritative]
LiveReached -->|No| RecordingFound{Recorded dispatch found for this job}
RecordingFound -->|Yes| Reconstruct[Reconstruct outcome from recording]
RecordingFound -->|No| Wait[Leave pending re-check next round]
LiveWins --> Compare{Verify mode enabled}
Reconstruct --> Compare
Compare -->|Yes and outcomes differ| Report[Report divergence to operator]
Compare -->|Yes and outcomes match| Continue[Continue]
Compare -->|No| Continue
```
Key decision: `Reconstruct` never re-fires for a job the `LiveWins` path already handled — both branches set `job->accounts`, and the existing `job->accounts == NULL` trigger guard on `sim_reconcile_unaccounted_jobs()` naturally excludes an already-decided job from being reconsidered (Requirement 3 AC4).
## Requirements Traceability
| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|------------|------------|-------|
| 1.1 | Job election/allocation uses same logic in both modes | Core-Safe Allocation Chain, Unified Picker/Allocator Threads | `sched_allocator(core)`, `plugin_job_election(core, job)` | Round flow (sequence) |
| 1.2 | Sim mode no longer always pends | Unified Picker/Allocator Threads | `sched_allocator(core)` | Round flow |
| 1.3 | Dispatch applies identical accounting effects | Core-Safe Allocation Chain | `sched_commit_decision(core, decision)`, `sched_commit_job_alloc(core, job)` | Round flow |
| 2.1 | Global limit match recorded same way | Limit Plugin Core-Safe Matching | `limit_job_election_core(core, job)` | Round flow (Election → Limit) |
| 2.2 | Host-scoped limit match recorded same way | Limit Plugin Core-Safe Matching, Core-Safe Allocation Chain | `limit_prepare_allocation_core(core, ctx)` | Round flow (Alloc → Limit) |
| 2.3 | `info limit` shows real usage | Limit Plugin Core-Safe Matching (no REPL change; reads existing accessors) | `limit_get_usage_instances()` (unchanged, from scheduler-sim-diagnostics) | — |
| 2.4 | Unmatched limit still reports no instance | Limit Plugin Core-Safe Matching | (unchanged accessor behavior) | — |
| 3.1 | Identical decisions/state at every replay point | Core-Safe Allocation Chain, Unified Picker/Allocator Threads | full chain | Round flow |
| 3.2 | Same contention outcome | Core-Safe Allocation Chain | `sched_check_host_satisfy`, `sched_check_shared_satisfy` | Round flow |
| 3.3 | Live decision authoritative when reached | Live/Recorded Authority Rule | — | Decision-authority flowchart |
| 3.4 | Recording safety net for timing gap | Sim-Mode Reconciliation Narrowing | `sim_reconcile_unaccounted_jobs` (unchanged guard) | Decision-authority flowchart |
| 3.5 | Divergence reported, not resolved silently | Live/Recorded Authority Rule | existing `--sim-verify` comparison/`sim_log_mismatch` | Decision-authority flowchart |
| 4.1 | `--sim-verify` continues comparing/reporting | Live/Recorded Authority Rule | unchanged comparison mechanism | Decision-authority flowchart |
| 4.2 | Interactive/batch remain decision-equivalent | Unified Picker/Allocator Threads | unchanged (`sim->interactive` gate) | — |
| 4.3 | One-job-at-a-time preserved | Unified Picker/Allocator Threads | `jbuf` (capacity 1, unchanged) | Round flow |
| 5.1 | Single shared implementation | Unified Picker/Allocator Threads, Core-Safe Allocation Chain | all core-safe signatures | — |
| 5.2 | Mode-specific differences isolated behind a seam | Unsatisfiable-Cache Callback Seam, Sim-Mode Reconciliation Narrowing | `unsatisfiable_lookup_cb`, `unsatisfiable_populate_cb` | — |
## Components and Interfaces
| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|--------------|--------|--------------|--------------------------|-----------|
| Core-Safe Allocation Chain | Scheduler Core | Migrate the 7 audit-identified functions plus the 4-function `sched_make_allocation()` dispatch switch from `xce_scheduler_t*` to `xce_sched_core_t*` | 1.1, 1.3, 3.1, 3.2 | Existing `resource.c`/`scheduler.c` internals (P0) | Service |
| Unsatisfiable-Cache Callback Seam | Scheduler Core | Isolate the one genuinely mode-specific piece of state behind a NULL-safe seam | 5.2 | `main.c` registration (P0) | Service |
| Plugin Hook Dispatchers | Plugin Framework | Core-safe `plugin_job_election`/`plugin_prepare_allocation`/`plugin_commit_decision` | 1.1, 2.1, 2.2 | `core->plugins` registry (P0) | Service |
| Limit Plugin Core-Safe Matching | Limit Plugin | Honest-core-parameter entry points for global/host-scoped matching | 2.1, 2.2, 2.3, 2.4 | Plugin Hook Dispatchers (P0), bug-1825's `limit_seed_config` pattern (P1) | Service |
| Unified Picker/Allocator Threads | Driver Threading | Single `sched_job_picker`/`sched_allocator` implementation, spawned by both drivers | 1.1, 1.2, 4.2, 4.3, 5.1 | Core-Safe Allocation Chain (P0), `jbuf`/`decisions` cbufs (P0) | Service |
| Live/Recorded Authority Rule | Sim Driver | Live decision wins when reached; divergence from recording is reported | 3.3, 3.5, 4.1 | Unified Picker/Allocator Threads (P0), existing `--sim-verify` comparison (P0) | Service |
| Sim-Mode Reconciliation Narrowing | Sim Driver | `sim_reconcile_unaccounted_jobs()` continues, scoped to the async timing gap | 3.4 | Unified Picker/Allocator Threads (P1) | Service |
### Scheduler Core
#### Core-Safe Allocation Chain
| Field | Detail |
|-------|--------|
| Intent | Every function between "a job is picked" and "a decision is committed" operates on `xce_sched_core_t*`, callable from either driver |
| Requirements | 1.1, 1.3, 3.1, 3.2 |
**Responsibilities & Constraints**
- Owns the per-host/shared capacity check and commit-decision state transition (`job->allocs`/`accounts`/`state`).
- Must not read or write anything outside `core->*` and the `xce_sched_worker_t*`/`xce_sched_ctx_t*` passed to it — this is the boundary that makes verbatim reuse safe.
- Must not introduce a first-member struct cast to reach normal-mode-only state (the documented anti-pattern, `core.h`'s `xce_core_cache_clear_cb_t` comment) — the one case that needs mode-specific state (the cache) goes through the seam below instead.
**Dependencies**
- Inbound: Unified Picker/Allocator Threads — the sole caller (P0)
- Outbound: Unsatisfiable-Cache Callback Seam — consulted at the primary-dispatch chokepoint (P0)
- Outbound: Plugin Hook Dispatchers — `plugin_prepare_allocation`/`plugin_commit_decision` (P0)
**Contracts**: Service [x]
##### Service Interface
```c
/* resource.c */
int sched_meet_host_selection(xce_sched_core_t* core, xce_sched_worker_t* w, xce_select_t* s);
int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, xvec_idval_t* wanted, xvec_idval_t* unmeet);
int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, xvec_t* uncomitted, xvec_idval_t* wanted, xvec_idval_t* unmeet);
int sched_try_alloc_on_host_(xce_sched_worker_t* w, xce_sched_core_t* core, xce_sched_ctx_t* ctx);
/* scheduler.c -- the full dispatch switch, not just the leaf commit, must migrate
together: sched_allocator() (core-safe, see Unified Picker/Allocator Threads)
calls sched_make_allocation() directly, so every function between that call
and the resource-check leaves above must share the same xce_sched_core_t*
the caller now has -- there is no xce_scheduler_t* left in that call chain
to pass down otherwise. */
int sched_make_allocation(xce_sched_core_t* core, xce_sched_obj_t* obj);
int sched_make_primary_allocation(xce_sched_core_t* core, xce_sched_obj_t* obj);
int sched_make_compound_allocation(xce_sched_core_t* core, xce_sched_obj_t* obj);
int sched_make_alter_allocation(xce_sched_core_t* core, xce_sched_obj_t* obj);
int sched_commit_decision(xce_sched_core_t* core, xce_decision_t* decision);
```
- Preconditions: `core` non-NULL; `w`/`ctx`/`decision`/`obj` non-NULL where used (matches existing NULL-guard behavior of the pre-migration functions).
- Postconditions: identical to today's `xce_scheduler_t*`-typed behavior — this is a pure signature migration, not a behavior change.
- Invariants: no function in this chain reads any field outside `core->*`/`w->*`/`ctx->*`, **except** the one primary-resreq dispatch chokepoint inside `sched_make_primary_allocation()`'s own call chain (believed to be `sched_make_primary_alloc_()`, per `scheduler-allocation`'s task 5.1 notes — not yet directly read in full; verify at implementation time) that currently reads `sched->unsatisfiable_cache` directly. That one call site routes through the Unsatisfiable-Cache Callback Seam instead of a direct field read once migrated — it is the seam's entire reason to exist.
**Implementation Notes**
- Integration: this migration is wider than just the 7 functions originally identified during the audit (the table in `requirements.md`) — it necessarily also includes the whole `sched_make_allocation()` dispatch switch and its three primary/compound/alter sub-functions, since `sched_allocator()` calls into this switch directly and there is no `xce_scheduler_t*` available to hand it after `sched_allocator()` itself becomes core-safe. Confirm during implementation that `sched_make_primary_allocation()`/`sched_make_compound_allocation()`/`sched_make_alter_allocation()` (not yet read in full at design time) touch nothing beyond `core->*`, worker/ctx fields, and the one cache touch point — if any of them reads other `xce_scheduler_t`-only state, that state needs its own seam, following the same NULL-safe-callback pattern as the cache.
- Validation: contention-fixture tests (per `scheduler-driver-equivalence.md`'s testing discipline) exercising this chain from both drivers.
- Risks: (1) the open question about `job_alloc_fn`/`register_seqjob_allocator`/`register_parjob_allocator` (see `research.md`) — verify before assuming this chain is fully closed; (2) `sched_make_primary_allocation`/`sched_make_compound_allocation`/`sched_make_alter_allocation`'s full bodies were not read in full during design — treat their migration as believed-mechanical based on the pattern established by every sibling function in this chain, but confirm by reading each body before migrating it, not by assumption.
#### Unsatisfiable-Cache Callback Seam
| Field | Detail |
|-------|--------|
| Intent | Isolate the one genuinely `xce_scheduler_t`-only piece of state behind a NULL-safe seam, so the surrounding allocation code never needs a mode-specific branch |
| Requirements | 5.2 |
**Responsibilities & Constraints**
- Owns nothing itself — a thin, optional indirection over `xce_scheduler_t.unsatisfiable_cache`'s existing lookup/populate behavior.
- A NULL callback must mean "always compute fresh" — never an error, never a degraded-correctness path, only a degraded-performance one (Non-Goals).
- Must never be bridged via a struct-prefix cast **inside code shared by both drivers** — but the small adapter function that implements each callback is itself normal-mode-exclusive (only ever registered by `main.c`'s normal-mode branch, only ever invoked with a `core` pointer that is genuinely embedded in an `xce_scheduler_t`, since sim mode never registers it), so it may safely recover `xce_scheduler_t*` via `(xce_scheduler_t*)core` — the identical, already-reviewed-safe pattern `alloc_cache.c`'s existing `xce_alloc_cache_clear_cb()` adapter uses for `cache_clear_cb`. No separate opaque `impl`/user-data pointer is needed; simplification over an earlier draft of this design that added one unnecessarily (single registrant, no foreseeable second implementation).
**Dependencies**
- Inbound: Core-Safe Allocation Chain, at `sched_make_primary_alloc_()`'s existing chokepoint (P0)
- Outbound: `xce_scheduler_t.unsatisfiable_cache`'s existing lookup/populate implementation, reached via a small adapter in `alloc_cache.c` (P0), the same file the existing `xce_alloc_cache_clear_cb()` adapter already lives in and for the same reason (white-box tests that `#include driver.c` without linking `alloc_cache.c`)
**Contracts**: Service [x]
##### Service Interface
```c
/* core.h -- mirrors cache_clear_cb's existing shape exactly: a bare function
pointer taking xce_sched_core_t*, NULL unless registered, no wrapper struct */
typedef int (*xce_unsatisfiable_lookup_cb_t)(xce_sched_core_t* core, xce_sched_obj_t* obj, int* out_reason_category);
typedef void (*xce_unsatisfiable_populate_cb_t)(xce_sched_core_t* core, xce_sched_obj_t* obj, int reason_category);
/* two new fields on xce_sched_core_t, alongside the existing cache_clear_cb field */
xce_unsatisfiable_lookup_cb_t unsatisfiable_lookup_cb; /* NULL unless registered */
xce_unsatisfiable_populate_cb_t unsatisfiable_populate_cb; /* NULL unless registered */
```
Corrected during task 1.1's implementation: the parameter list is `xce_sched_obj_t* obj` + `int*`/`int` reason-category, matching the REAL `cache_lookup()`/`cache_insert()` API (`alloc_cache.c`) exactly, not the `uint64_t fingerprint`/`int category` shape drafted above at design time — the real cache already keys on the full `xce_sched_obj_t*` (its embedded `xce_resreq_fingerprint_t` plus user/queue/project identity), so this seam needed no invented fingerprint abstraction on top of it.
- Preconditions: registered (non-NULL) only by normal mode's `main.c`, exactly once, before `sched_init()` spawns any thread — identical registration discipline to `cache_clear_cb`.
- Postconditions: NULL callbacks mean every call to `sched_make_primary_alloc_()` computes fully, bypassing the cache fast path — behaviorally correct, matching what happens today whenever the cache has no entry for a fingerprint.
- Invariants: the allocation chain (shared code) only ever NULL-checks and calls these — it never interprets `core`'s contents itself; only the mode-exclusive adapter functions in `alloc_cache.c` do, and only when they are guaranteed (by registration discipline, not by a runtime check) to have a normal-mode `core`.
**Implementation Notes**
- Integration: registered in `main.c`'s normal-mode branch only, alongside the existing `cache_clear_cb` registration — same placement pattern, same rationale, same file for the adapter implementation (`alloc_cache.c`).
- Validation: a unit test confirming NULL callbacks never crash and always take the compute-fresh path; a normal-mode test confirming the cache is still consulted/populated exactly as before this change.
- Risks: none identified beyond the general struct-cast anti-pattern this design explicitly avoids in shared code.
### Plugin Framework
#### Plugin Hook Dispatchers
| Field | Detail |
|-------|--------|
| Intent | `plugin_job_election`, `plugin_prepare_allocation`, `plugin_commit_decision` iterate `core->plugins` directly, callable from either driver |
| Requirements | 1.1, 2.1, 2.2 |
**Responsibilities & Constraints**
- Mirrors `plugin_begin_session()`/`plugin_end_session()`'s already-completed migration exactly — same iteration pattern, same NULL-guard shape.
- Does not change `plugin_hooks_t`'s function-pointer signatures (`job_election_hook`, `prepare_allocation_hook`, `commit_hook` keep their existing `(void* arg, ...)` shape, since `arg` there is the plugin instance, not the driver struct — only the dispatcher functions that iterate plugins change).
**Dependencies**
- Inbound: Unified Picker/Allocator Threads, Core-Safe Allocation Chain (P0)
- Outbound: Limit Plugin Core-Safe Matching, via the hook function pointers (P0)
**Contracts**: Service [x]
##### Service Interface
```c
int plugin_job_election(xce_sched_core_t* core, xce_sched_job_t* job);
int plugin_prepare_allocation(xce_sched_core_t* core, xce_sched_ctx_t* ctx);
int plugin_commit_decision(xce_sched_core_t* core, xce_decision_t* decision);
```
- Preconditions: `core` non-NULL.
- Postconditions: every registered plugin's corresponding hook is invoked exactly once per call, in registration order — unchanged from today's behavior.
- Invariants: iterates `core->plugins` (a field both `xce_scheduler_t.core` and `xce_sched_sim_driver_t.core` genuinely have), never a driver-specific field.
**Implementation Notes**
- Integration: single normal-mode call site each (`sched_allocator()`, `sched_try_alloc_on_host_()`, `sched_commit_decision()`) gets updated to pass `&(sched->core)` / `core` (already core-typed after that function's own migration); simulation mode's `sched_allocator()` (now the same function) passes `&(sim->core)`.
- Validation: existing plugin-hook unit tests re-run unchanged (behavior-preserving refactor); white-box test files that `#include plugin.c` need their link-satisfying stubs' signatures updated.
- Risks: none beyond the standard ripple-to-white-box-stubs risk already seen in bug-1825/scheduler-sim-diagnostics. Separately, **bug-1771** (an unfixed ASAN SEGV in `fcfs_del_job()`, reached via `plugin_commit_decision()`'s `commit_hook` dispatch to `fcfs_commit`) is a pre-existing, unrelated defect in exactly this dispatch path, discovered during `scheduler-allocation`'s own end-to-end testing. This migration is a pure signature change with no behavior change, so it neither fixes nor worsens bug-1771 — noted here only so an implementer who hits this crash during testing attributes it correctly instead of assuming their own migration caused it.
### Limit Plugin
#### Limit Plugin Core-Safe Matching
| Field | Detail |
|-------|--------|
| Intent | `match_job_global_limit_()`/`match_job_host_limit_()` derive `xce_sched_core_t*` honestly instead of via `plugin->sched` (a bare `void*`, NULL for sim-registered plugins) |
| Requirements | 2.1, 2.2, 2.3, 2.4 |
**Responsibilities & Constraints**
- Extends the exact split bug-1825 established for the config-hook path (`limit_update_config()` / `sim_bootstrap_limit_plugin()` both delegating to a shared `limit_seed_config_impl()`) to the job-election and prepare-allocation call chains.
- **Concrete mechanism (corrected during task planning — the hook signature itself cannot carry `core`):** `job_election_hook`/`prepare_allocation_hook` keep their existing `(void* arg, ...)` shape where `arg` is the *plugin instance*, not `core` — so `plugin_job_election(core, job)`'s own `core` parameter does not automatically reach the leaf hook function it calls. `xce_sched_plugin_t` (`plugin.h`) gains a **new, honestly-typed field**, `xce_sched_core_t* core`, set by BOTH `plugin_register()` (`plugin->core = &(sched->core);`) and `plugin_register_core()` (`plugin->core = core;` — closing the gap bug-1825 left, since that function never sets anything today). `limit_job_election()`/`limit_prepare_allocation()`/`match_job_global_limit_()`/`match_job_host_limit_()` read `plugin->core` (never NULL after either registration path), not `plugin->sched`.
- **The existing `plugin->sched` field is deliberately left untouched** — still `xce_scheduler_t*`-flavored, still set only by `plugin_register()`, still read by `plugins/fcfs.c`'s legacy `job_elect_fn` path exactly as today. Repurposing `plugin->sched` itself (an earlier draft of this design considered this) was rejected: it is a *shared* field across every plugin category, and `fcfs.c`'s legacy picker also reads it expecting `xce_scheduler_t*` — changing its meaning would silently break that unrelated, normal-mode-only consumer. Adding a new field costs one pointer per plugin and avoids that regression entirely, consistent with this codebase's own preference (the `limit_seed_config()` precedent) for an honest new entry point over reinterpreting an existing ambiguous one.
- The plugin-hook-registered entry points (`limit_job_election()`, `limit_prepare_allocation()`, unchanged `(void* arg, ...)` signature) become thin wrappers that read `plugin->core` and delegate to shared `_impl` functions taking `xce_sched_core_t* core` honestly.
**Dependencies**
- Inbound: Plugin Hook Dispatchers, via `job_election_hook`/`prepare_allocation_hook` (P0)
- Outbound: `instantiate_limit()`/`update_limit_res_value()` (`xcelimit.c`, unchanged) — registers the matched limit as a virtual resource into `core->sharedInst`/`w->resvals`, the same arrays `sched_account_jobs()` already resums every round in both drivers (P0)
**Contracts**: Service [x]
##### Service Interface
```c
/* plugins/limit.c */
static int limit_job_election_impl(xce_sched_core_t* core, xce_sched_plugin_t* plugin, xce_sched_job_t* job);
static int limit_prepare_allocation_impl(xce_sched_core_t* core, xce_sched_plugin_t* plugin, xce_sched_ctx_t* ctx);
/* existing hook-registered wrappers, now delegating to the _impl functions above
using the core-safe xce_sched_core_t* the (now core-safe) dispatcher supplies */
static int limit_job_election(void* arg, xce_sched_job_t* job);
static int limit_prepare_allocation(void* arg, xce_sched_ctx_t* ctx);
```
- Preconditions: `plugin->core` non-NULL (guaranteed by either registration path once this fix lands), `plugin->data` (the limit plugin's private `limits`/`instances` kvmgr handles) already initialized by `limit_init()` — unchanged.
- Postconditions: a global limit match registers a vres into `core->sharedInst`; a host-scoped match registers into `w->resvals` — identical outcome in both drivers.
- Invariants: `match_job_global_limit_()`'s existing `obj->configver >= core.configver` idempotency guard is preserved unchanged — safe to call from both the live path and, for global limits only, from `sim_reconcile_unaccounted_jobs()`.
**Implementation Notes**
- Integration: no change to `info limit`'s REPL display logic (`repl_info.c`) or its read-only accessors (`limit_get_configured_limits()`/`limit_get_usage_instances()`, from `scheduler-sim-diagnostics`) — this component only makes the data those accessors read become real.
- Validation: `test_repl_limit_table.c`'s existing fixtures, extended with a real sim-mode-driven (not test-only-registered) matching scenario for both global and host-scoped limits under contention.
- Risks: none beyond the standard signature-migration ripple to white-box stubs.
### Driver Threading
#### Unified Picker/Allocator Threads
| Field | Detail |
|-------|--------|
| Intent | `sched_job_picker()`/`sched_allocator()` become the sole picker/allocator implementation, spawned by both drivers |
| Requirements | 1.1, 1.2, 4.2, 4.3, 5.1 |
**Responsibilities & Constraints**
- `sim_picker_thread_fn()` is removed entirely, not deprecated alongside the new path.
- Thread-spawn call sites: `sched_init()` (normal mode, unchanged call site, unchanged behavior) and `sched_sim_driver_run()` (simulation mode, changed from one `sim_picker_thread_fn` spawn to two spawns of the shared functions).
- Preserves the capacity-1 `jbuf` rendezvous — this component reuses the existing cbuf, it does not reimplement it.
- No new teardown/lifecycle work for simulation mode (Non-Goals) — the pre-existing "never joined, lives for process lifetime" behavior is preserved for both threads in sim mode, matching today's `sim_picker_thread_fn()`'s own documented lifecycle.
**Dependencies**
- Inbound: `sched_init()` (normal mode), `sched_sim_driver_run()` (simulation mode) (P0)
- Outbound: Core-Safe Allocation Chain, Plugin Hook Dispatchers (P0)
**Contracts**: Service [x]
##### Service Interface
```c
/* sched_election.c (new file, extracted from scheduler.c -- see File Structure Plan) */
static void* sched_job_picker(void* arg); /* arg is xce_sched_core_t* */
static void* sched_allocator(void* arg); /* arg is xce_sched_core_t* */
```
- Preconditions: `sched_resolve_job_picker(core, ...)` must succeed before either thread is spawned (existing gate, unchanged) — for simulation mode, this means the picker plugin must already be bootstrapped (`sim_bootstrap_picker()`, bug-1825) before `sched_sim_driver_run()` spawns these threads.
- Postconditions: identical per-round behavior to today's normal mode — pull a job, elect it, attempt allocation, commit a decision, hand off via `jbuf`/`decisions`.
- Invariants: exactly one producer (`sched_job_picker`) and one consumer (`sched_allocator`) per session, per the jbuf rendezvous invariant.
**Implementation Notes**
- Integration: `sched_sim_driver_run()`'s existing `pthread_create(&picker_tid, NULL, sim_picker_thread_fn, &(sim->core))` becomes two `pthread_create()` calls (`sched_job_picker`, `sched_allocator`), both passed `&(sim->core)`.
- Validation: `test_repl_equivalence.c`, `test_sim_contention_parity.c`, `test_sim_equivalence.c` must pass unchanged or with investigated, understood deltas (see `research.md` Risks) — this is the highest-risk integration point in the whole design.
- Risks: (1) decision-timing/ordering shift from the fused-thread-to-two-thread change in simulation mode; mitigated by the validation above, not by avoiding the change. (2) **Recurring bug class, not a one-off**: `xce_sched_sim_driver_t sim` held stack-local while a never-joined thread (`sim_picker_thread_fn()` today) keeps a pointer into it past the owning function's return has caused ASAN stack-use-after-return at least three times in this codebase's history (bug-1598, and its own explicitly-documented recurrences bug-1632 and bug-1763, the latter still marked NOT FIXED, out of scope for the spec that found it). This design doubles the number of long-lived threads holding `&(sim->core)` (two instead of one), doubling the surface area for any remaining unaudited instance of this same pattern. Do not assume bug-1598's single fixture fix closed this class — any new or touched test fixture for this component must be positively confirmed to use a `static` or heap-allocated `xce_sched_sim_driver_t`, not merely assumed safe by analogy.
### Sim Driver
#### Live/Recorded Authority Rule
| Field | Detail |
|-------|--------|
| Intent | The live picker/allocator's decision is authoritative wherever it reaches a job; a disagreement with the recording is reported, not resolved silently |
| Requirements | 3.3, 3.5, 4.1 |
**Responsibilities & Constraints**
- This is a behavioral rule, not a new function — it falls out of (a) the live path now producing real decisions and (b) `sim_reconcile_unaccounted_jobs()`'s existing `job->accounts == NULL` guard naturally skipping any job the live path already decided.
- The existing `--sim-verify` positional comparison (`sim_log_mismatch()`) is the reporting mechanism — unchanged, now actually exercised meaningfully for the first time against live-computed decisions instead of only against reconstructed-from-recording ones.
**Dependencies**
- Inbound: none (a property of the interaction between the two components below) (P0)
- Outbound: Unified Picker/Allocator Threads, Sim-Mode Reconciliation Narrowing (P0)
**Contracts**: State [x]
##### State Management
- State model: a job's `accounts == NULL` marks it as not-yet-decided; either the live allocator (via `sched_commit_decision`) or `sim_reconcile_unaccounted_jobs()`'s FOUND branch sets it, whichever reaches the job first this round.
- Persistence & consistency: no new persisted state; this is in-memory per-session state, already covered by the field-completeness audit (dimension 4, PASS, unaffected).
- Concurrency strategy: the existing per-round ordering (`sim_reconcile_unaccounted_jobs()` runs before `xce_core_signal_go()` unblocks the picker thread each round) means reconciliation always gets first look at whether a job is already decided from an EARLIER round's live processing before deciding whether to reconstruct it from the recording this round — no race, since the live path's own decision-commit for a job from a prior round has already set `job->accounts` before this round's reconciliation check runs.
**Implementation Notes**
- Integration: no new code path — the reporting behavior already exists (`--sim-verify`); this component's job in the design is to make sure nothing is added that would suppress or override a genuine mismatch.
- Validation: Scope item 9 (`requirements.md`) — run `--sim-verify` against real recorded sessions and treat any reported mismatch as a finding to root-cause.
- Risks: a mismatch found this way could indicate either a defect in this feature's own migration, or a genuine pre-existing issue the old always-PEND behavior could never have exposed — both are valid, actionable outcomes, not a reason to suppress the report.
#### Sim-Mode Reconciliation Narrowing
| Field | Detail |
|-------|--------|
| Intent | `sim_reconcile_unaccounted_jobs()` keeps existing, narrowed to the async timing-gap safety net |
| Requirements | 3.4 |
**Responsibilities & Constraints**
- No structural change to this function's existing logic — the narrowing is a consequence of the live path now handling most jobs, not a new code path inside this function.
- One addition: call the core-safe global-limit-election entry point (`limit_job_election_impl`, via the now-core-safe `plugin_job_election`) for a job this function reconstructs, so a job whose outcome comes from the recording still gets its global-limit vres registered (idempotency guard already prevents double-registration if the live path also happened to touch it).
**Dependencies**
- Inbound: `sched_sim_driver_run()`'s per-round sequence, unchanged position (before `sched_account_jobs()`) (P0)
- Outbound: Plugin Hook Dispatchers (new, for the limit-election call) (P1)
**Contracts**: Batch [x]
##### Batch / Job Contract
- Trigger: once per round, unconditionally, for every job in `core->jobs` with `accounts == NULL` and not terminal.
- Input / validation: the round's decoded `ReplaySession.decisions` array (unchanged).
- Output / destination: `job->allocs`/`accounts`/`state` set directly from the recording; decision reported via the existing `sched_record_session_add_decision()`/`sim_print_decision_gate()`/`xce_repl_check_decision()` triple (unchanged).
- Idempotency & recovery: `job->accounts != NULL` permanently excludes the job from reconsideration (unchanged, existing guard).
**Implementation Notes**
- Integration: add one call (`plugin_job_election(&(sim->core), job)` — global-limit matching only, since host-scoped matching needs a resolved worker context this function doesn't independently construct) in the FOUND branch, before or alongside the existing account-reconstruction logic.
- Validation: `test_sim_reconcile_unaccounted.c`'s existing fixtures, extended to confirm a reconciled job's global limit is also registered.
- Risks: none beyond the standard ripple risk; this function's own scope and trigger condition are unchanged.
## Error Handling
### Error Strategy
No new error categories. Every migrated function preserves its existing error/return-code contract (`-EINVAL`, `-ENOENT`, etc.) — this is a signature migration, not a behavior change, so existing error-handling call sites (`ERROR(...)`, `WARN(...)`, fail-return-NULL patterns) are preserved verbatim.
### Error Categories and Responses
- **Picker resolution failure at thread start** (both drivers): existing fail-return-NULL-and-log behavior, unchanged (`sched_resolve_job_picker` failing in `sched_job_picker()`/`sched_allocator()`).
- **`unsatisfiable_lookup_cb`/`unsatisfiable_populate_cb` NULL**: not an error — the documented, correct fallback (always compute fresh).
- **Live/recorded decision divergence**: not an error in the traditional sense — reported via the existing `--sim-verify` mismatch mechanism (a diagnostic finding, not a crash or an aborted session).
### Monitoring
No new logging infrastructure. Existing `TRACE`/`WARN`/`ERROR` call sites in the migrated functions are preserved; `sim_log_mismatch()` (unchanged) remains the mechanism for surfacing a live/recorded divergence.
## Testing Strategy
- **Unit Tests**:
1. Each migrated function (`sched_meet_host_selection`, `sched_check_host_satisfy`, `sched_try_alloc_on_host_`, `sched_commit_decision`, `plugin_job_election`, `plugin_prepare_allocation`, `plugin_commit_decision`) called directly with an `xce_sched_core_t*` from EITHER an `xce_scheduler_t.core` or `xce_sched_sim_driver_t.core` fixture, confirming identical results.
2. `unsatisfiable_lookup_cb`/`unsatisfiable_populate_cb` NULL fallback: confirm `sched_make_primary_alloc_()` always computes fresh with no crash and no incorrect skip.
3. `limit_job_election_impl`/`limit_prepare_allocation_impl` called against a sim-registered (`plugin_register_core()`) plugin instance, confirming a match registers correctly even though `plugin->sched` is NULL.
- **Integration Tests**:
1. A real `xce_sched_sim_driver_t` driven through `sched_sim_driver_run()` against a replay whose original session had global AND host-scoped limits configured and actively enforced — `info limit` shows real usage for both (Requirement 2.1-2.4, the concrete bug-1826 acceptance test).
2. A resource-contention fixture (multiple jobs, insufficient capacity, including limit-constrained capacity) driven through both drivers, confirming identical admission/pend outcomes (Requirement 3.2, per `scheduler-driver-equivalence.md`'s testing discipline).
3. `sim_reconcile_unaccounted_jobs()` and the live picker/allocator both active in the same session, confirming no double-processing of a single job (Requirement 3.4).
- **E2E / Real-Recording Tests** (Scope item 9):
1. `--sim-verify` run against a range of real recorded sessions (not only crafted fixtures), treating any reported mismatch as a finding to root-cause.
2. `test_repl_equivalence.c`, `test_sim_contention_parity.c`, `test_sim_equivalence.c` re-run in full, with any assertion change investigated and explained, not silently updated to match new output.
- **Performance**: none targeted (Non-Goals) — no throughput/latency test required for this feature.

122
.kiro/specs/scheduler-sim-live-allocation/requirements.md Normal file
View file

@ -0,0 +1,122 @@
# Requirements Document
## Project Description (Input)
Give simulation/replay mode genuine live allocation -- today `sim_picker_thread_fn()` unconditionally PENDs every job it considers instead of running the real election/allocation path normal mode runs, because that path is signatured on `xce_scheduler_t*` (normal-mode-only) rather than the shared `xce_sched_core_t*`. Unify the picker/allocator thread bodies and the allocation-chain functions they call into core-safe, single implementations shared verbatim by both drivers, replacing `sim_picker_thread_fn()` entirely.
The concrete acceptance test for this capability is bug-1826: with real live allocation wired in, `info limit`'s live-usage display shows real data for a replay (both global and host-scoped limits), instead of "no active usage instance yet" -- because limit matching (`plugin_job_election()`/`match_job_global_limit_()`/`match_job_host_limit_()`) is one of the plugin hooks that only ever fires along the now-shared election/allocation path. Limit matching is the feature that surfaced this gap and the feature this spec proves the fix against -- it is not the whole of what this spec delivers.
Background: bug-1825 (already fixed) made the limit plugin's CONFIGURED SETTING visible in sim mode by bootstrapping registration (plugin_register_core + limit_seed_config). But limit MATCHING -- the mechanism that populates live usage instances (xce_vres_t) as jobs get evaluated -- is never invoked during replay, because plugin_job_election() (plugin.c) is normal-mode-only: it casts void* arg to xce_scheduler_t* and reads sched->core.plugins, and xce_sched_sim_driver_t deliberately has no xce_scheduler_t* field.
## GOVERNING PRINCIPLE (stated repeatedly by the user during discussion -- this is the design's first-class constraint, not one bullet among many)
**The scheduler must use the SAME core for both normal and simulator mode, with only the driver differing.** `src/scheduler/CLAUDE.md` already states this as "one core, two consumers": `driver.c` (normal mode) and `sim_loop.c` (simulation mode) are meant to drive the identical `xce_sched_core_t` state machine and decision-callback protocol, never separate reimplementations of the same logic. `.kiro/steering/scheduler-driver-equivalence.md` makes this an enforced DESIGN INVARIANT (state equivalence, not just decision equivalence -- identical resource-accounting state, not just a matching decision sequence, at every point in a replay).
This spec's scope is deliberately governed by that principle end to end: **both** global limit matching and host-scoped limit matching must be wired into sim mode by having sim mode call the literal SAME functions normal mode calls (after a mechanical core-safe migration), not by an approximation or a sim-only reimplementation. An earlier draft of this spec proposed accepting an "approximate" host-scoped match (against only the recorded winning host, since sim mode has no live per-candidate-host search) as a lesser but shippable compromise -- the user explicitly rejected that: prefer completing the core-safe migration so genuine parity is achievable for both, over shipping an approximation that violates the governing principle.
**Why this matters more than a style preference (user's own framing, verbatim intent):** the entire reason simulation/replay mode exists is to debug the REAL running behavior of the scheduler core without needing a live controller/worker connection. That premise only holds if sim mode reproduces the EXACT behavior of the core -- not an approximation, not a differently-shaped reimplementation that happens to usually agree. An operator debugging a production issue via `xce_scheduler -i -s <replay>` must be able to trust that what they see is what actually happened, including every plugin-driven side effect (limit matching among them) that live jobs went through. This is not a nice-to-have generalization of bug-1826's fix -- it is the founding premise sim mode was built to satisfy, and any place two separate implementations exist for what should be one core behavior is a latent trust gap in the debugging tool's core promise, not just a code-quality issue.
**Consequently, this spec's scope extends one level higher than the limit-matching hooks alone: unify the picker/allocator thread bodies themselves, not only the functions they call.** Today these are two separate hand-written implementations:
- Normal mode: `sched_job_picker()` (`scheduler.c:80`) + `sched_allocator()` (`scheduler.c:429`) -- two threads, handed off via `core->jbuf`.
- Sim mode: `sim_picker_thread_fn()` (`sim_loop.c:541`) -- one fused thread, `jbuf` unused entirely (confirmed by the audit's dimension-3 finding).
Once this spec's planned core-safe migration lands (the 7-function table below, `plugin_job_election()`, and the `unsatisfiable_cache` callback), every single thing `sched_allocator()` calls is core-safe. At that point there is no remaining technical reason `sched_job_picker()`/`sched_allocator()` themselves cannot be re-typed to `xce_sched_core_t*` and reused VERBATIM by sim mode -- one implementation, two call sites (normal mode's init spawns them against `&(sched->core)`, sim mode's startup spawns the identical two functions against `&(sim->core)`) -- entirely replacing `sim_picker_thread_fn()` rather than leaving it as a second, hand-maintained implementation of the same loop.
**Known behavioral risk to verify, not a reason to avoid this:** sim mode would start genuinely using `core->jbuf` (a real two-thread hand-off) instead of today's single-fused-thread sequencing. The capacity-1 rendezvous invariant still holds either way (dimension 3, PASS), but the *timing/interleaving* of when PEND/DISPATCH decisions land on `core->decisions` could shift with two concurrently-running threads instead of one sequential one. `test_repl_equivalence.c`, `test_sim_contention_parity.c`, and `test_sim_equivalence.c` must be checked against this change before it ships -- treat any assertion that breaks as a signal to look closely (it may be asserting on an artifact of the old fused-thread sequencing rather than a real invariant), not as a reason to revert to two implementations.
## Audit findings that make full parity achievable (from a completed `/xce-audit-invariants scheduler` run, 2026-09-18 -- do not re-derive, verify current status only)
**Dimension 1 (driver state-equivalence) -- the key finding: the remaining gap is smaller than it first appeared.** Tracing every function in the host-scoped allocation chain down to what it actually touches (not just its declared type) shows almost all of it is PURELY MECHANICALLY coupled to `xce_scheduler_t*` -- i.e. it only ever reads/writes `sched->core.*` or worker/ctx fields, with zero genuine dependency on anything `xce_scheduler_t`-specific:
| Function | Location | What it actually touches |
|---|---|---|
| `sched_meet_host_selection(xce_scheduler_t* sched, w, s)` | `resource.c:221` | Only `sched->core.sel_eval_ctx` |
| `sched_check_shared_satisfy(xce_scheduler_t* sched, ...)` | `resource.c:305` | Only `sched->core.sharedInst`/`sharedInUse` |
| `sched_check_host_satisfy(xce_scheduler_t* sched, w, ...)` | `resource.c:378` | Only `w->resvals`/`w->resInUse` + delegates to the above |
| `plugin_prepare_allocation(void* arg, ctx)` | `plugin.c:591` | Only `sched->core.plugins` (same void*-cast pattern as `plugin_job_election`) |
| `sched_try_alloc_on_host_(w, xce_scheduler_t* sched, ctx)` | `resource.c:497` | Only calls the three rows above |
| `sched_commit_decision(xce_scheduler_t* sched, decision)` | `scheduler.c:1607` | Only `sched->core.*`; already-core-safe `sched_commit_job_alloc(core,...)` underneath |
| `plugin_commit_decision(void* arg1, decision)` | `plugin.c:442` | Only `sched->core.plugins` (same void*-cast pattern) |
| `sched_ctx_get_uncomit_host(host, ctx, idx)` | `schedobj.c:522` | Already core-safe -- no `xce_scheduler_t*` at all |
This is the exact same shape `plugin_begin_session`, `plugin_end_session`, `sched_resolve_job_picker`, `sched_get_job`, `sched_account_jobs`, and `sched_commit_job_alloc` were in before their own (already-completed) migration to `xce_sched_core_t*` -- a pattern this codebase has proven safe six times over, not a new one to invent.
**The one genuine exception:** `xce_scheduler_t.unsatisfiable_cache`, consulted/populated only around `sched_make_primary_alloc_()`'s top-level dispatch chokepoint (`scheduler.c`, wired in by the completed `scheduler-allocation` spec's task 5.1). It is a pure performance optimization (skip recomputing an allocation for an already-known-unsatisfiable job fingerprint) with no correctness effect -- excise it behind a NULL-safe, driver-registered accessor mirroring the already-proven `cache_clear_cb` pattern (`xce_sched_core_t.cache_clear_cb`, set only in the driver-mode branch of `main.c`; that pattern exists specifically because a `__attribute__((weak))` cast-based shortcut around this same field once silently corrupted `xce_sched_sim_driver_t.alt_events_file` at the same byte offset, invisible to ASAN and the whole test suite -- do not repeat that mistake by casting instead of using the callback). Sim mode simply always takes the slow (always-compute-fresh) path; this is correct, not an approximation, since the cache never changes what a computation yields, only whether it's skipped.
**`sim_picker_thread_fn()`'s current behavior must also change**: its own header comment documents that sim mode makes NO real allocation attempt today -- it unconditionally PENDs every job the picker offers, specifically because `sched_make_allocation()` needed the live `xce_scheduler_t*` sim mode doesn't carry. Once the chain above is core-safe, this function needs to actually attempt allocation (mirroring `sched_allocator()`'s structure: resolve picker -> loop: `next_job()` -> job election -> allocation attempt -> decide) instead of always PENDing.
**Dimension 2 (core I/O isolation) -- unrelated, logged separately as bug-1827, does not block this spec.** `core_clock.c` is missing from two of `test_core_isolation.c`'s three enforcement checks (a test-coverage gap, not a confirmed live violation). Tracked independently; no action needed here.
**Dimension 3 (jbuf capacity-1 rendezvous) -- PASS as audited; this spec deliberately supersedes that finding, not by accident.** At audit time, sim mode bypassed `jbuf` entirely (fused picker+allocator into one thread). Once this spec's scope grew to unify `sched_job_picker()`/`sched_allocator()` themselves (Scope item 3 below; Requirement 3 AC3/AC4; Requirement 5), sim mode intentionally starts using `core->jbuf` for real, via the identical two-thread producer/consumer pattern normal mode already uses -- not a new, second mechanism. The invariant this dimension protects (capacity hardcoded to 1, exactly one producer and one consumer per session) still holds, because it's the same already-verified code, not a new implementation of the rendezvous. Do not read this paragraph's original "sim mode bypasses jbuf" framing as still current -- it described the state before that scope decision, not the target.
**Dimension 4 (snapshot field-completeness) -- PASS, not relevant to this spec.** No undocumented gaps found; no interaction with limit-matching wiring.
## Scope (revised: full parity, both limit hooks)
1. **Core-safe migration** of the ~7 functions in the table above (mechanical, `xce_scheduler_t*` -> `xce_sched_core_t*`, following the already-proven pattern) plus the `unsatisfiable_cache` callback-indirection design (a new `xce_sched_core_t` callback slot, registered only by `main.c`'s driver-mode branch, mirroring `cache_clear_cb`).
2. **`plugin_job_election()` (plugin.c:554)** made core-safe the same way, single implementation shared by both drivers (mirrors `plugin_begin_session`'s own already-completed migration). Its one normal-mode call site (`scheduler.c`'s `sched_allocator()`, called once per job right after `sched_get_job()` and before `sched_make_allocation()`) gets updated to pass `&(sched->core)`.
3. **`sched_job_picker()` (scheduler.c:80) and `sched_allocator()` (scheduler.c:429) made core-safe (`xce_sched_core_t*`) and reused VERBATIM by sim mode, replacing `sim_picker_thread_fn()` entirely** -- not just the functions they call. Sim mode's startup spawns these same two thread functions against `&(sim->core)` instead of maintaining a separate, hand-written `sim_picker_thread_fn()` implementation. This means sim mode genuinely uses `core->jbuf` for the first time (a real two-thread hand-off, not the current single-fused-thread shortcut). `sim_reconcile_unaccounted_jobs()` should also call the core-safe job-election function for a reconciled job (idempotency already guarded via `obj->configver >= core.configver`, so calling from both places is safe). Required verification: `test_repl_equivalence.c`, `test_sim_contention_parity.c`, and `test_sim_equivalence.c` must be re-run and any decision-timing/ordering change investigated (see governing-principle section above) before this is considered done.
4. **Mechanical gap in `xce_sched_plugin_t.sched`**: it is a bare `void*`, set ONLY by `plugin_register()` (normal mode). `plugin_register_core()` (sim-mode registration, bug-1825) never sets it, so `limit_job_election()`/`match_job_global_limit_()`/`match_job_host_limit_()`'s existing `xce_scheduler_t* sched = (xce_scheduler_t*)plugin->sched;` derivation reads NULL in sim mode today. Extend the same honest-`xce_sched_core_t*`-parameter treatment bug-1825 already established for `limit_seed_config()`/`limit_seed_config_impl()` (config-hook path) to the job-election and prepare-allocation call chains, rather than relying on `plugin->sched`.
5. **Both** `match_job_global_limit_()` (global/shared limits, via `job_election_hook`) and `match_job_host_limit_()` (host-scoped limits, via `prepare_allocation_hook`, reached through the now-core-safe `sched_try_alloc_on_host_()`/`plugin_prepare_allocation()` chain) end up genuinely invoked from sim mode by the SAME code path normal mode uses -- no approximation, no sim-only reimplementation.
6. **Ordering constraint (existing `sim_loop.c` comment, must be preserved):** any new per-round election/allocation call must complete strictly between `xce_core_begin_session()`'s resval reset and `xce_core_signal_go()` unblocking the picker -- the same window `sim_reconcile_unaccounted_jobs()`/`plugin_begin_session()`/`sched_account_jobs()` already occupy.
7. **Testing discipline (per `scheduler-driver-equivalence.md`):** validate with a resource-contention fixture (multiple jobs competing for capacity insufficient for all of them, including limit-constrained capacity), not only a no-contention fixture -- a no-contention fixture cannot distinguish "accounting/limit-matching is correct" from "it's wrong but nothing currently reads it."
8. **Precedent:** bug-1825 (commit 487e7b94) is the direct precedent for code style (factory extraction, honest-core-typed entry points, no first-member struct cross-casting -- `core.h`'s `xce_core_cache_clear_cb_t` doc comment documents the prior memory-corruption incident from that exact anti-pattern, structurally identical to the `unsatisfiable_cache` incident found during this investigation's audit).
9. **Validation discipline, distinct from item 7's contention-fixture discipline:** before this feature is considered done, run `--sim-verify` against a range of REAL recorded sessions (not only crafted unit fixtures) -- this is the most direct way to catch a subtle divergence the migration itself introduces, precisely because sim mode's live decision engine is new here and was never previously exercised against real production recordings. Per Requirement 3 AC5, a mismatch surfaced this way is a finding to root-cause (either a defect in this migration, or a genuine pre-existing issue the old fake-PEND behavior was hiding), never something to explain away or suppress to make the check pass.
## Introduction
Simulation/replay mode (`xce_scheduler -i` / `-s <replay>`) exists so an operator can debug the scheduler's real decision-making from a recorded live session, without a connected controller or worker. Today it falls short of that promise for one whole class of behavior: job election and allocation. Simulation mode currently pends every candidate job unconditionally rather than running the same election/allocation logic normal mode runs, so any plugin-driven behavior that depends on election or allocation never happens during replay — most visibly, resource-limit matching, so `info limit` cannot show a replay's real limit usage even when the original live session actively enforced it. This spec makes simulation mode reproduce normal mode's genuine job-election, allocation, and limit-matching behavior, so that replaying a recording reliably reproduces what actually happened.
## Boundary Context
- **In scope**: job election and allocation logic (including global and host-scoped resource-limit matching) running identically in simulation mode and normal mode, for both freshly-considered jobs and jobs reconstructed from a recording; simulation mode's resource-accounting state (worker and shared resource usage, per-job resource accounts, limit usage) matching normal mode's under the same input, including under resource contention; `info limit` showing real usage data for a replay whose original session had active limits.
- **Out of scope**: any change to normal mode's own observable scheduling decisions or behavior; the previously-identified, unrelated core-module-isolation test-coverage gap (tracked separately, not part of this feature); any new REPL command or `info` subcommand (the existing `info limit` command's display logic is unchanged — only the underlying data it reads becomes real); silently resolving a disagreement between the live-computed decision and the recording in either direction — such a disagreement must be surfaced, not hidden, by this feature; achieving simulation mode's allocation-decision throughput or latency parity with normal mode (correctness of decisions and accounting state is the requirement, not performance).
- **Adjacent expectations**: this feature depends on existing snapshot and recording formats, and existing REPL/CLI behaviors (`info limit`, `info worker`, `--sim-verify`, interactive vs. batch simulation), continuing to work unchanged; it does not require any change to how a live session records its decisions or how a replay file is structured. The existing recorded-decision reconciliation mechanism continues to exist as a narrower safety net for a job the live election/allocation path has not yet reached (the timing gap between an event's arrival and the live path's own asynchronous processing of it) — it is no longer expected to be the general-case source of truth for dispatch decisions once the live path can reach and decide most jobs directly. This feature does not require simulation mode to gain a clean shutdown/teardown path for its worker threads beyond what exists today.
## Requirements
### Requirement 1: Live Job Election and Allocation in Simulation Mode
**Objective:** As a scheduler operator debugging via replay, I want simulation mode to make real dispatch/pend decisions using the same job-election and allocation logic as normal mode, so that replay-based debugging reflects genuine scheduler behavior rather than a simplified stand-in.
#### Acceptance Criteria
1. When a job becomes a candidate for dispatch during a simulation-mode session, the Scheduler shall evaluate it using the same job-election and allocation logic normal mode uses for an equivalent job.
2. While operating in simulation mode, the Scheduler shall not unconditionally pend every candidate job regardless of actual resource availability.
3. When a job is dispatched during a simulation-mode session, whether newly decided or reconstructed from a recording, the Scheduler shall apply the identical resource-accounting effects (worker and shared resource usage) that normal mode would apply for the same dispatch.
### Requirement 2: Resource-Limit Matching Parity
**Objective:** As a scheduler operator inspecting resource limits during replay, I want simulation mode to evaluate configured limits the same way normal mode does, so that `info limit` shows a limit's real, currently-accounted usage rather than reporting no usage at all.
#### Acceptance Criteria
1. When a job in a simulation-mode session matches a configured global (non-host-scoped) limit, the Scheduler shall record that match and its usage the same way normal mode would.
2. When a job in a simulation-mode session matches a configured host-scoped limit, the Scheduler shall record that match and its usage the same way normal mode would.
3. When an operator runs `info limit <name>` or `info limit all` during a simulation-mode session against a replay whose original live session had active limits, the Scheduler shall display the limit's live usage reflecting jobs actually dispatched in that replay, instead of reporting no active usage instance.
4. The Scheduler shall continue to report no active usage instance, via `info limit`, for a limit that was never matched by any job in the replay.
### Requirement 3: State Equivalence Between Simulation and Normal Mode
**Objective:** As a scheduler operator relying on replay for debugging, I want simulation mode's resource-accounting state to match what the same events produced in the original live run, so that I can trust conclusions drawn from replay-based debugging.
#### Acceptance Criteria
1. When a simulation-mode session replays the exact recorded events of a normal-mode run, the Scheduler shall reproduce identical scheduling decisions and identical resource-accounting state (worker resource usage, shared resource usage, per-job resource accounts, and limit usage) at every point in the replay, not only a matching final decision sequence.
2. While multiple jobs in a replay compete for resource capacity insufficient to satisfy all of them concurrently, including capacity constrained by a matching limit, the Scheduler shall produce the same admission or pend outcome in simulation mode that normal mode produced for the same contention.
3. When the Scheduler's live job-election and allocation logic reaches and decides a job during a simulation-mode session, that live decision shall be authoritative for the job's outcome in the replay.
4. If a job's dispatch is found in the replay's recording before the live election/allocation logic has reached that job, the Scheduler shall reconstruct that job's outcome from the recording so the job is not lost or indefinitely delayed, and shall not subsequently re-decide or re-charge that same job once its outcome has been established this way.
5. If the Scheduler's live-computed decision for a job differs from that job's decision in the original recording, the Scheduler shall report this divergence to the operator rather than silently discarding either outcome or resolving the disagreement without visibility.
### Requirement 4: No Regression to Existing Simulation-Mode Guarantees
**Objective:** As a scheduler operator using existing simulation-mode tooling, I want the introduction of live allocation to preserve today's determinism and comparison guarantees, so that existing replay-based workflows continue to work correctly.
#### Acceptance Criteria
1. When a simulation-mode session is run with live-vs-recorded verification enabled, the Scheduler shall continue to compare each live decision against its corresponding recorded decision and report any mismatch, using its existing comparison mechanism.
2. The Scheduler's interactive and batch simulation modes shall remain decision-equivalent for the same input.
3. While a simulation-mode session is running, the Scheduler shall preserve its existing guarantee that only one job is being selected and allocated at a time, with no job double-processed or over-committed.
### Requirement 5: Single Shared Implementation, Not Independently-Verified Equivalence
**Objective:** As a maintainer of this codebase, I want simulation mode and normal mode to run the same job-election and allocation implementation rather than two separately-maintained implementations that are merely tested to currently agree, so that the two modes cannot silently drift apart the way this project's history shows they otherwise do.
#### Acceptance Criteria
1. The Scheduler shall implement job-election and allocation logic as a single implementation used by both simulation mode and normal mode, not as separate mode-specific implementations whose agreement is established only by testing.
2. Where a piece of state or behavior is genuinely mode-specific (present in normal mode but not simulation mode, or vice versa), the Scheduler shall isolate that difference behind an explicit, narrow seam rather than duplicating the surrounding shared logic to accommodate it.

77
.kiro/specs/scheduler-sim-live-allocation/research.md Normal file
View file

@ -0,0 +1,77 @@
# Research & Design Decisions
## Summary
- **Feature**: `scheduler-sim-live-allocation`
- **Discovery Scope**: Extension (existing system) — light discovery per `design-discovery-light.md`, since the integration points, existing patterns, and constraints were already exhaustively traced against live source during requirements development (see `requirements.md`'s Project Description for the full evidence trail; this document focuses on decisions, alternatives, and risk, not re-deriving what's already cited there).
- **Key Findings**:
- Six functions (`plugin_begin_session`, `plugin_end_session`, `sched_resolve_job_picker`, `sched_get_job`, `sched_account_jobs`, `sched_commit_job_alloc`) already completed the exact `xce_scheduler_t*` → `xce_sched_core_t*` migration this spec needs to repeat for the remaining allocation chain — a proven, six-times-repeated pattern, not a new one.
- Every function in the host-scoped allocation chain (`sched_meet_host_selection`, `sched_check_shared_satisfy`, `sched_check_host_satisfy`, `plugin_prepare_allocation`, `sched_try_alloc_on_host_`, `sched_commit_decision`, `plugin_commit_decision`) only ever touches `sched->core.*` or worker/ctx fields — zero genuine `xce_scheduler_t`-specific dependency, confirmed by reading every function body, not inferred from signatures.
- The one genuine exception, `xce_scheduler_t.unsatisfiable_cache`, is a pure performance optimization with a documented prior memory-corruption incident (scheduler-allocation task 5.2 round 1) from trying to shortcut it via a struct-prefix cast — the fix pattern (a driver-registered callback, `cache_clear_cb`) is proven and must be repeated, not re-invented.
- `sched_allocator()`'s full body (verified, not assumed) shows every single call it makes becomes core-safe once the above migration lands — the function itself needs only its top cast changed (`xce_scheduler_t* sched` → `xce_sched_core_t* core`).
- Sim mode's thread-spawn point is `sched_sim_driver_run()` (`sim_loop.c`), NOT `sched_sim_driver_start()` (`sim.c`) — the latter only does core init and snapshot/replay-file setup; the `pthread_create(&picker_tid, NULL, sim_picker_thread_fn, &(sim->core))` call lives inside the former.
## Research Log
### Is the allocation chain's `xce_scheduler_t*` coupling mechanical or genuine?
- **Context**: The spec's central bet is that sim mode can reuse normal mode's allocation code verbatim. This only holds if the coupling is superficial (never-migrated signature) rather than load-bearing (genuinely needs normal-mode-only state).
- **Sources Consulted**: Direct reads of `resource.c` (`sched_meet_host_selection:221`, `sched_check_shared_satisfy:305`/`sched_check_host_satisfy:378`/`sched_try_alloc_on_host_:497`), `plugin.c` (`plugin_prepare_allocation:591`, `plugin_commit_decision:442`, `plugin_job_election:554`), `scheduler.c` (`sched_commit_decision:1607`, `sched_allocator:429`), `schedobj.c` (`sched_ctx_get_uncomit_host:522`, already core-safe).
- **Findings**: Every one of the 8 functions in the allocation chain reads/writes only `sched->core.*` fields (or worker/ctx fields passed in separately) — no function reads `sched->apictx`, `sched->cluster_cfg`, `sched->unsatisfiable_cache`, or any other genuinely `xce_scheduler_t`-only field, with the single exception noted below.
- **Implications**: The migration is safe to do mechanically, function by function, following the exact precedent of the six already-migrated functions. No redesign of the allocation algorithm itself is needed.
### Is there a genuinely `xce_scheduler_t`-only dependency anywhere in the chain?
- **Context**: A prior investigation (scheduler-allocation spec, task 5.2) hit exactly this problem once already, for a different field.
- **Sources Consulted**: `scheduler-allocation`'s own Implementation Notes (task 5.1, 5.2), `core.h`'s `xce_core_cache_clear_cb_t` doc comment.
- **Findings**: `xce_scheduler_t.unsatisfiable_cache` (a fingerprint → pend-reason-category cache, populated when `sched_make_primary_alloc_()` exhausts all candidates for a job, consulted before attempting allocation again for an identically-shaped job) has no sim-mode equivalent field, by deliberate design (Req 5 AC 17 simulation-isolation boundary from `scheduler-core`). A first attempt to bridge this via `__attribute__((weak))` + a cast of the shared `xce_sched_core_t core` prefix back to `xce_scheduler_t*` silently corrupted `xce_sched_sim_driver_t.alt_events_file` (same byte offset as the cache field) — invisible to ASAN and the whole test suite until caught by review.
- **Implications**: This one field needs a callback seam, not a cast, mirroring `cache_clear_cb` exactly. Consult/populate calls must go through a NULL-checked function-pointer pair the normal-mode driver registers and sim mode leaves NULL. A NULL seam means "always take the slow path" (recompute fully), which is correct behavior, not degraded correctness — see Design Decision below.
### Where does sim mode actually spawn its picker/allocator thread?
- **Context**: Needed to know precisely which function's thread-spawn code changes.
- **Sources Consulted**: `sim.c`'s `sched_sim_driver_start()` (full body read) and `sim_loop.c`'s `sched_sim_driver_run()` (partial body read, confirmed `pthread_create` call site at line ~974).
- **Findings**: `sched_sim_driver_start()` only initializes `sim->core` and opens/validates the replay file and snapshot — it spawns no threads. `sched_sim_driver_run()` is the actual long-running entry point (design.md's own "literal SimDriver entry point" per its header comment) and is where the picker thread is created, once, for the process's sim-session lifetime.
- **Implications**: The unification touches `sched_sim_driver_run()`'s thread-spawn call, not `sched_sim_driver_start()`.
### Open question not fully resolved — flagged as a risk, not asserted either way
- **Context**: `sched_register_plugins()` (`scheduler.c:596`) also calls `register_seqjob_allocator(sched)` / `register_parjob_allocator(sched)`, and `xce_sched_plugin_t` has a `job_alloc_fn` field (`plugin_job_allocate_fn_t`) alongside `job_elect_fn`. This raises the question of whether primary-resreq allocation is ALSO dispatched through a generic plugin-hook indirection (a `plugin_job_allocate()` analogous to `plugin_job_election()`) that would need the same core-safety migration.
- **Findings so far**: The `scheduler-allocation` spec's own Implementation Notes describe `sched_make_primary_alloc_()` as directly branching between `seqjob_allocator.c`/`parjob_allocator.c`'s own `my_alloc()` functions by name (based on `primary->maxTasks`), which reads as a direct function-call dispatch, not a generic plugin-hook indirection — suggesting `job_alloc_fn`/`register_seqjob_allocator`/`register_parjob_allocator` may be a legacy/inactive registration path (parallel to `job_elect_fn`'s own documented legacy-compat status for pickers) rather than something on the live call path.
- **Not confirmed with certainty.** Flagged in Risks below as a design-time verification item for whoever implements task work in this area — if `sched_make_primary_alloc_()` turns out to dispatch through a generic hook after all, that hook needs the identical core-safe treatment as `plugin_job_election`/`plugin_prepare_allocation`/`plugin_commit_decision`.
## Architecture Pattern Evaluation
| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Unify picker/allocator (selected) | Re-type `sched_job_picker()`/`sched_allocator()` to `xce_sched_core_t*`, spawn the identical functions from both drivers | Single implementation, cannot drift (directly satisfies Requirement 5); reuses 6x-proven migration pattern | Sim mode gains a real two-thread `jbuf` hand-off for the first time — needs the 3 named regression tests re-verified | Selected: matches the project's own stated governing principle and its documented drift-prevention rationale |
| Behavioral-equivalence only (rejected) | Keep `sim_picker_thread_fn()` as a separate implementation, extend it to call the same core-safe sub-functions, verify equivalence via tests | Smaller diff, no `jbuf` behavior change in sim mode | Reintroduces exactly the two-implementations-must-independently-stay-in-sync failure mode this project has already hit three times (bug-1611, bug-1613, bug-1681) | Rejected per Requirement 5 and the user's explicit direction during spec discussion |
| Recording-authoritative (rejected) | Keep `sim_reconcile_unaccounted_jobs()` as the general-case decision source; new election/allocation logic only populates side effects (limit vres) for an already-known outcome | Smallest behavior change; no live/recorded divergence handling needed | Contradicts the project's own driver-equivalence invariant (decisions should be independently reproduced, not merely replayed) and doesn't scale to future plugin hooks with other side effects | Rejected during requirements review — see `requirements.md` Requirement 3 AC3-5 |
## Design Decisions
### Decision: Live election/allocation is authoritative; recorded-decision reconciliation narrows to a timing-gap safety net
- **Context**: Once sim mode can independently compute real decisions, two sources exist for a job's outcome: the live picker/allocator, and `sim_reconcile_unaccounted_jobs()`'s trust-the-recording reconstruction.
- **Alternatives Considered**:
1. Recording stays authoritative always; live path only supplies side effects — rejected (see table above).
2. Live path authoritative, recording is a fallback only for jobs it hasn't reached in time — selected.
- **Selected Approach**: The live picker/allocator decides for any job it reaches. `sim_reconcile_unaccounted_jobs()`'s existing `job->accounts == NULL` trigger guard already means it only acts on jobs the live path hasn't yet processed — no new guard needed, this narrowing falls out of the existing idempotency check once the live path starts actually producing outcomes.
- **Rationale**: Matches `scheduler-driver-equivalence.md`'s own state-equivalence invariant (decisions should be independently reproduced by the core state machine, not replayed from a trusted log) and the project's explicit governing principle for this spec.
- **Trade-offs**: A live-computed decision that disagrees with the recording is now possible and must be surfaced (Requirement 3 AC5) rather than silently reconciled either way — this is by design: a divergence is a genuine finding (a migration defect, or a previously-hidden issue the old fake-PEND behavior could never have exposed).
- **Follow-up**: Verify via `--sim-verify` against real recorded sessions (Scope item 9 in requirements.md) before shipping.
### Decision: `unsatisfiable_cache` becomes an optional, driver-registered callback pair, not a relocated field
- **Context**: Need the primary-allocation dispatch chokepoint to remain core-safe without giving sim mode a cache it structurally cannot support cleanly today.
- **Alternatives Considered**:
1. Move the cache itself onto `xce_sched_core_t` so both drivers have it — rejected: sim mode has no natural place to construct/own it (no `sched_fini()`-equivalent teardown exists in sim mode to destroy it), and giving sim mode a working cache is not something any requirement asks for (explicitly a non-goal — see Boundary Context, "achieving simulation mode's allocation-decision throughput or latency parity").
2. A NULL-safe callback pair (`lookup`/`populate`) on `xce_sched_core_t`, registered only by normal mode's `main.c`, mirroring `cache_clear_cb` — selected.
- **Selected Approach**: `xce_sched_core_t` gains two optional function-pointer fields (`unsatisfiable_lookup_cb`/`unsatisfiable_populate_cb`), each taking `xce_sched_core_t* core` directly — the exact same shape as the existing `cache_clear_cb` field, not a new wrapper struct with an opaque `impl`/user-data pointer (an earlier design draft added one; removed as unnecessary indirection with only one registrant and no foreseeable second). Normal mode's `main.c` registers both, pointing at two small adapter functions in `alloc_cache.c` (alongside the existing `xce_alloc_cache_clear_cb()` adapter) that safely recover `xce_scheduler_t*` via `(xce_scheduler_t*)core` — safe specifically because these adapters are normal-mode-exclusive and only ever invoked with a `core` genuinely embedded in an `xce_scheduler_t`, unlike the round-1 mistake which cast inside code shared by both drivers. Sim mode leaves both NULL. `sched_make_primary_alloc_()` treats NULL as "no cache available — always compute fresh," which is correct (never wrong), not a degraded-correctness mode.
- **Rationale**: Directly reuses the proven pattern and its documented rationale (`core.h`'s `xce_core_cache_clear_cb_t` doc comment); avoids repeating the exact class of memory-corruption bug the earlier cast-based attempt caused.
- **Trade-offs**: Sim mode never gets the cache's performance benefit — accepted, explicitly out of scope (see Boundary Context).
## Risks & Mitigations
- **Decision-timing/ordering shift in sim mode once `jbuf` is genuinely used** — Mitigation: re-run `test_repl_equivalence.c`, `test_sim_contention_parity.c`, `test_sim_equivalence.c` before considering this feature done; treat a broken assertion as a signal to investigate (it may be asserting on an artifact of the old single-fused-thread sequencing), not a reason to revert to two implementations.
- **Unconfirmed `job_alloc_fn`/`register_seqjob_allocator`/`register_parjob_allocator` dispatch path** — Mitigation: verify at implementation time whether `sched_make_primary_alloc_()` dispatches through a generic plugin hook or calls `seqjob`/`parjob` directly; if a hook exists, it needs the same core-safe migration as the other four plugin-hook dispatchers.
- **A live-computed decision may legitimately diverge from a real recording during validation** (Scope item 9) — Mitigation: this is treated as expected and valuable, not a blocker; root-cause every divergence found against real recordings before shipping, per Requirement 3 AC5.
- **`plugin->sched` (bare `void*`, set only by `plugin_register()`) staying NULL for sim-registered plugins** — Mitigation: extend the `limit_seed_config()`/`limit_seed_config_impl()` honest-core-parameter pattern (already proven by bug-1825) to the job-election and prepare-allocation call chains, rather than relying on `plugin->sched`.
## References
- `.kiro/steering/scheduler-driver-equivalence.md` — the governing state-equivalence invariant.
- `.wolf/buglog.json` bug-1611, bug-1613, bug-1681, bug-1825, bug-1826, bug-1827 — the evidence trail of this project's own recurring drift failure mode and its prior fixes.
- `.kiro/specs/scheduler-allocation/` — completed spec whose task 5.2 Implementation Notes document the `unsatisfiable_cache` memory-corruption incident and the `cache_clear_cb` fix pattern being reused here.
- `src/scheduler/CLAUDE.md` — "one core, two consumers" architecture statement.

22
.kiro/specs/scheduler-sim-live-allocation/spec.json Normal file
View file

@ -0,0 +1,22 @@
{
"feature_name": "scheduler-sim-live-allocation",
"created_at": "2026-09-18T06:48:27Z",
"updated_at": "2026-09-18T09:45:00Z",
"language": "en",
"phase": "tasks-generated",
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": true
}
},
"ready_for_implementation": false
}

112
.kiro/specs/scheduler-sim-live-allocation/tasks.md Normal file
View file

@ -0,0 +1,112 @@
# Implementation Plan: scheduler-sim-live-allocation
- [ ] 1. Foundation: independent core-safe seams and leaf migrations
- [x] 1.1 (P) Give the allocation dispatch chokepoint an optional, normal-mode-only performance cache seam
- Add two NULL-by-default callback fields to the shared core state for consulting and populating the unsatisfiable-resreq cache, mirroring the existing session-reset callback's proven shape (a bare function pointer taking the core state directly, no wrapper struct, no opaque handle)
- Implement the two adapter functions against the real cache, in the same file the existing session-reset adapter already lives in, registered only during normal-mode startup
- TDD: add a test asserting that with both callbacks left NULL, the allocation dispatch chokepoint always computes fresh with no crash and no incorrect skip; add a normal-mode test confirming the cache is still consulted/populated exactly as before this change
- Observable: a simulation-mode session (which never registers these callbacks) never touches the real cache implementation at all, while a normal-mode session's cache behavior is provably unchanged
- _Requirements: 5.2_
- _Boundary: Unsatisfiable-Cache Callback Seam_
- [ ] 1.2 (P) Make plugin hook dispatch driver-agnostic and give every plugin an honest reference to shared core state
- Re-point the three plugin-hook dispatch functions (job election, prepare-allocation, commit-decision) to iterate the shared plugin registry directly instead of deriving it from a normal-mode-only driver struct, mirroring the session-begin/session-end dispatchers' own already-completed migration
- Add a new, honestly-typed core-state reference field to the plugin instance struct, set by BOTH existing plugin-registration entry points (the normal-mode one and the simulation-mode one) — leave the plugin's existing, differently-typed driver-reference field completely untouched, since an unrelated legacy picker plugin still depends on its current meaning
- TDD: extend the existing plugin-hook dispatch tests to run unchanged against a core-state fixture built either way (embedded in a normal-mode-shaped or simulation-mode-shaped container), and add a test confirming a plugin registered via the simulation-mode entry point has a non-NULL core-state reference (closing today's gap where it silently stays unset)
- Observable: a plugin registered through either registration path can reach shared core state without ever reading the normal-mode-only field, and the existing dispatch behavior (each registered plugin's hook invoked exactly once, in registration order) is unchanged
- _Requirements: 1.1, 2.1, 2.2, 5.1_
- _Boundary: Plugin Hook Dispatchers_
- [ ] 1.3 (P) Make the per-host and shared capacity checks driver-agnostic
- Re-type the host-selection match, shared-capacity check, and per-host capacity check functions to operate on shared core state directly instead of a normal-mode-only driver struct
- TDD: re-run their existing unit tests unchanged (pure signature migration) plus a new test calling each function with core state drawn from a simulation-mode-shaped container, confirming identical results to the normal-mode-shaped case
- Observable: these three functions compile and pass identically regardless of which driver's core state they're given
- _Requirements: 1.3, 3.1, 3.2_
- _Boundary: Core-Safe Allocation Chain_
- [ ] 2. Core: driver-agnostic allocation search and dispatch
- [ ] 2.1 Thread shared core state through the per-job allocation context and the per-host allocation attempt
- Change the per-job allocation context's driver-reference field to carry shared core state instead of a normal-mode-only pointer, updating its handful of existing cast sites
- Re-type the per-host allocation-attempt function (which calls the three capacity checks from 1.3 and the prepare-allocation hook from 1.2) to take shared core state directly
- Thread the same change through the existing multi-host allocation search's entry points, preserving their host-iteration and bucket/plan-construction algorithm exactly as-is
- TDD: re-run the existing multi-host allocation and per-host-attempt test suites unchanged; add a test driving the per-host attempt function with simulation-mode-shaped core state end to end
- Observable: the per-host allocation attempt and the multi-host search both run correctly when given either driver's core state, with zero change to which hosts get selected or in what order
- _Depends: 1.2, 1.3_
- _Requirements: 1.3, 3.1, 3.2_
- _Boundary: Core-Safe Allocation Chain_
- [ ] 2.2 Make primary/compound/alter allocation dispatch and decision commit driver-agnostic
- Re-type the top-level allocation dispatch switch and its primary/compound/alter sub-functions, and the decision-commit function, to take shared core state directly
- Wire the one genuine normal-mode-only touch point (the unsatisfiable-resreq cache) through the seam built in 1.1 instead of reading it directly
- Confirm by reading each sub-function's body before migrating it that it touches nothing beyond core state, worker/context fields, and the one cache touch point. If a sub-function is found to touch additional normal-mode-only state, stop and split that state's seam into a new follow-up task (mirroring 1.1's pattern) rather than designing and building it within this task
- TDD: re-run the existing allocation-dispatch and decision-commit test suites unchanged; add a normal-mode test confirming the cache fast path still activates correctly through the new seam
- Observable: a job's dispatch decision and resulting resource-accounting state are identical before and after this migration, for every existing allocation-dispatch test
- _Depends: 2.1, 1.1_
- _Requirements: 1.1, 1.3, 3.1, 3.2, 5.2_
- _Boundary: Core-Safe Allocation Chain, Unsatisfiable-Cache Callback Seam_
- [ ] 2.3 (P) Make global and host-scoped resource-limit matching driver-agnostic
- Change the limit plugin's global-limit and host-scoped-limit matching functions to read the new honest core-state reference from 1.2 instead of deriving anything from the plugin's normal-mode-only field
- Keep the existing plugin-hook-registered entry points' external signature unchanged; have them resolve core state from the new field and delegate to the now-driver-agnostic matching logic
- TDD: extend the limit plugin's existing test suite with a case that registers the plugin via the simulation-mode entry point and confirms a match still succeeds and still registers the matched limit's usage, where today it would silently no-op
- Observable: a limit configured on a job registered via the simulation-mode plugin path is matched and its usage instance created, verifiable by reading the plugin's own configured-limits/usage-instances accessors
- _Depends: 1.2_
- _Requirements: 2.1, 2.2, 2.3, 2.4_
- _Boundary: Limit Plugin Core-Safe Matching_
- [ ] 3. Core: unify job election and allocation into one implementation
- [ ] 3.1 Extract the job-picking and allocation loop into a single implementation shared by both drivers
- Move the job-picking thread body and the allocation thread body out of the normal-mode-only file they live in today, into their own file, re-typed to operate on shared core state directly
- Confirm every call this pair makes is already driver-agnostic after tasks 1.2/2.1/2.2 — no remaining reference to anything normal-mode-only
- Update normal mode's own startup to spawn these (now-relocated) threads unchanged in behavior
- TDD: re-run the full existing normal-mode scheduling test suite unchanged (behavior-preserving relocation); add a test spawning both threads against a simulation-mode-shaped core state fixture and confirming a job flows all the way from being picked to a committed decision
- Observable: normal mode's own scheduling behavior is bit-for-bit unchanged, and the identical thread pair now also runs correctly when spawned against simulation-mode's own core state in a test fixture
- _Depends: 2.2, 2.3_
- _Requirements: 1.1, 1.2, 5.1_
- _Boundary: Unified Picker/Allocator Threads_
- [ ] 4. Integration: wire simulation mode onto the shared implementation
- [ ] 4.1 Replace simulation mode's placeholder picker with the shared picker/allocator pair
- Replace simulation mode's own always-pend placeholder thread with the shared job-picking and allocation thread pair from 3.1, spawned against simulation mode's own core state
- Confirm the existing one-job-at-a-time hand-off guarantee between picking and allocating still holds now that simulation mode genuinely uses it for the first time
- TDD: write a new fixture using a non-stack-local (static or heap-allocated) simulation driver instance — a prior, still-recurring bug class in this exact area came from a stack-local instance outliving a never-joined thread, and this change doubles the number of such threads, so this must be positively confirmed, not assumed safe by analogy
- Observable: a simulation-mode session, driven from a real replay with no changes to the replay format, produces a genuine dispatch or pend decision for a job based on its actual resource availability, instead of unconditionally pending it
- _Depends: 3.1_
- _Requirements: 1.1, 1.2, 3.3, 4.2, 4.3, 5.1_
- _Boundary: Unified Picker/Allocator Threads, Sim Driver_
- [ ] 4.2 Narrow recorded-decision reconciliation to the timing-gap safety net
- Add a call to the now-driver-agnostic job-election dispatch (from 1.2) for a job this mechanism reconstructs from the recording, so a job whose outcome comes from the recording still gets its global-limit usage registered
- Confirm the existing trigger guard (a job already decided is never reconsidered) needs no change, since it already excludes any job the live path from 4.1 has already handled
- TDD: extend the existing reconciliation test suite with a case confirming a reconciled job's global limit is registered, and a case confirming a job the live path already decided is never reprocessed by this mechanism
- Observable: replaying a recorded session in which a job's dispatch was found only via reconciliation still shows that job's limit usage through the existing limit-usage accessors
- _Depends: 4.1, 2.3_
- _Requirements: 3.4, 3.5_
- _Boundary: Sim-Mode Reconciliation Narrowing_
- [ ] 5. Validation: prove equivalence and the concrete acceptance test
- [ ] 5.1 Prove identical decisions and accounting state under resource contention across both drivers
- Build a resource-contention fixture (multiple jobs, capacity insufficient for all of them, including capacity constrained by a matching limit) and drive it through both drivers
- Observable: both drivers produce the same admission/pend outcome and the same final resource-accounting state for the identical input
- _Depends: 4.1_
- _Requirements: 3.1, 3.2_
- [ ] 5.2 Prove the concrete acceptance test: real global and host-scoped limit usage visible against a real replay
- Drive a real simulation session through a replay whose original live session had both a global and a host-scoped limit actively enforced
- Observable: the existing limit-inspection command displays real, non-zero usage for both limits against this replay, where today it reports no active usage instance for either
- _Depends: 4.1, 4.2_
- _Requirements: 2.1, 2.2, 2.3, 2.4_
- [ ] 5.3 Re-run existing simulation-mode equivalence suites and reconcile any behavior change
- Re-run the REPL/batch decision-equivalence suite and the existing normal-mode-recording-vs-simulation-replay decision-identity suites in full
- Investigate and explain, rather than silently update, any assertion whose expected output changes as a result of this feature
- Observable: all three suites pass, or every changed assertion has a written explanation of why the new behavior is correct
- _Depends: 4.1_
- _Requirements: 4.1, 4.2, 4.3_
- [ ] 5.4 Validate against real recorded sessions and root-cause any divergence found
- Run live-vs-recorded verification against a range of real recorded sessions, not only crafted fixtures
- Treat any reported mismatch between a live-computed decision and its recorded counterpart as a finding to root-cause (either a defect in this feature's own migration, or a genuine issue the old always-pend behavior could never have exposed), never as something to suppress
- Observable: a written disposition (fixed, or explained as a pre-existing issue) exists for every mismatch found this way
- _Depends: 5.1, 5.2, 5.3_
- _Requirements: 3.3, 3.5_

67
src/scheduler/include/core.h
View file

@ -109,6 +109,61 @@ typedef int (*xce_core_decision_cb_t)(void*, xce_decision_t*);
*/
typedef void (*xce_core_cache_clear_cb_t)(xce_sched_core_t*);
/*
* xce_unsatisfiable_lookup_cb_t / xce_unsatisfiable_populate_cb_t -
* driver-registered, OPTIONAL per-session unsatisfiable-resreq cache
* consult/populate seam (scheduler-sim-live-allocation spec, task 1.1,
* Req 5.2). Mirrors xce_core_cache_clear_cb_t's own shape and rationale
* immediately above exactly: a bare function pointer taking
* xce_sched_core_t* directly, no wrapper struct, no opaque "impl"/
* user-data pointer -- single registrant (main.c's normal-mode branch),
* no foreseeable second implementation.
*
* Parameter shape mirrors the REAL cache's own existing entry points
* (alloc_cache.c's cache_lookup()/cache_insert(), design.md's
* "Unsatisfiable-Cache Callback Seam" component) rather than inventing a
* new bare-integer "fingerprint" type: xce_sched_obj_t* is the exact
* object cache_lookup()/cache_insert() already key on (its embedded
* xce_resreq_fingerprint_t plus user/queue/project identity, sched/
* alloc_cache.h) -- this seam adds no new type, only a NULL-safe
* indirection over the existing one. xce_sched_obj_t is already visible
* here as a forward-declared pointer type (sched/types.h, included
* above); only a pointer, never the full struct, is needed in this
* header.
*
* WHY a hook instead of a direct call (mirrors xce_core_cache_clear_cb_t's
* own documented incident above): the allocation dispatch chokepoint this
* seam wraps (sched_make_primary_alloc_(), scheduler.c) is reached
* identically by both drivers once a later task's Core-Safe Allocation
* Chain migration lands -- but the real cache
* (xce_scheduler_t.unsatisfiable_cache) exists ONLY on a normal-mode
* xce_scheduler_t, never on a simulation-mode xce_sched_sim_driver_t. A
* NULL-checked, driver-registered callback keeps that difference behind
* an explicit seam instead of a struct-prefix cast into mode-specific
* state from code both drivers share -- the same anti-pattern
* xce_core_cache_clear_cb_t's own doc comment documents a real, prior
* memory-corruption incident from.
*
* NULL (the xce_sched_core_init() memset() default, unless a driver
* registers one) means "always compute fresh" for BOTH callbacks -- never
* an error, never a degraded-correctness path, only a degraded-
* performance one. Wiring the actual consult/populate call sites into
* sched_make_primary_alloc_() is a LATER task's job (tasks.md 2.2,
* boundary "Core-Safe Allocation Chain, Unsatisfiable-Cache Callback
* Seam") -- this task adds only the seam (these two fields, their
* adapters in alloc_cache.c, and main.c's normal-mode-only registration),
* not that call site. Every simulation-mode core leaves both fields NULL
* forever (sim_loop.c never registers them, Req 5 AC 17 "Simulation
* isolation"), so a simulation-mode session never touches the real cache
* implementation at all.
*/
typedef int (*xce_unsatisfiable_lookup_cb_t)(xce_sched_core_t* core,
xce_sched_obj_t* obj,
int* out_reason_category);
typedef void (*xce_unsatisfiable_populate_cb_t)(xce_sched_core_t* core,
xce_sched_obj_t* obj,
int reason_category);
struct xce_sched_core_s {
mylock_t lock;
int stop;
@ -159,6 +214,18 @@ struct xce_sched_core_s {
* call -- true for every simulation-mode xce_sched_core_t, which never
* registers this hook. */
xce_core_cache_clear_cb_t cache_clear_cb;
/* Req 5.2 (scheduler-sim-live-allocation spec, task 1.1): driver-
* registered, OPTIONAL unsatisfiable-resreq cache consult/populate
* seam -- see xce_unsatisfiable_lookup_cb_t/xce_unsatisfiable_
* populate_cb_t's own doc comment above for the full rationale. NULL
* (xce_sched_core_init()'s memset() default) until main.c's
* normal-mode branch registers both, alongside cache_clear_cb just
* above. A simulation-mode core never registers either, so
* consulting/populating this seam is always a true no-op for sim
* mode ("always compute fresh"), never a fallback onto real cache
* state. */
xce_unsatisfiable_lookup_cb_t unsatisfiable_lookup_cb;
xce_unsatisfiable_populate_cb_t unsatisfiable_populate_cb;
_Atomic int session_active; /* 1 during scheduling session, 0 between */
_Atomic int eval_in_progress; /* 1 while eval_select loop runs */
_Atomic int staging_merge_complete; /* 1 after staging merge completes */

18
src/scheduler/include/sched/alloc_cache.h
View file

@ -141,4 +141,22 @@ extern void cache_clear(xce_scheduler_t* sched);
*/
extern void xce_alloc_cache_clear_cb(xce_sched_core_t* core);
/*
* xce_alloc_cache_lookup_cb / xce_alloc_cache_populate_cb - the
* xce_unsatisfiable_lookup_cb_t / xce_unsatisfiable_populate_cb_t (core.h)
* adapters for cache_lookup()/cache_insert() above (scheduler-sim-live-
* allocation spec, task 1.1, Req 5.2). Registered on the real
* xce_scheduler_t's core via direct field assignment in main.c's
* normal-mode-only startup branch, alongside xce_alloc_cache_clear_cb()
* just above. See alloc_cache.c's own doc comment on these functions for
* the full rationale, including why they live here rather than in
* driver.c.
*/
extern int xce_alloc_cache_lookup_cb(xce_sched_core_t* core,
xce_sched_obj_t* obj,
int* out_reason_category);
extern void xce_alloc_cache_populate_cb(xce_sched_core_t* core,
xce_sched_obj_t* obj,
int reason_category);
#endif

48
src/scheduler/src/alloc_cache.c
View file

@ -266,3 +266,51 @@ void xce_alloc_cache_clear_cb(xce_sched_core_t* core)
{
cache_clear((xce_scheduler_t*)core);
}
/*
* xce_alloc_cache_lookup_cb / xce_alloc_cache_populate_cb -
* scheduler-sim-live-allocation spec, task 1.1 (Req 5.2): the
* xce_unsatisfiable_lookup_cb_t / xce_unsatisfiable_populate_cb_t (core.h)
* adapters for cache_lookup()/cache_insert() above, mirroring
* xce_alloc_cache_clear_cb()'s own shape and rationale exactly.
*
* Deliberately defined HERE (alloc_cache.c), not in driver.c or
* scheduler.c, for the IDENTICAL white-box-test-linkage reason
* xce_alloc_cache_clear_cb()'s own doc comment above documents: several
* white-box test TUs #include ../src/driver.c directly WITHOUT linking
* alloc_cache.c. Had either adapter lived anywhere those TUs don't
* already link, every one of them would fail to LINK the moment a future
* task wires a call site (main.c/a shared allocation-chain function)
* against it. Keeping both adapters here means those TUs stay entirely
* unaffected: none of them link alloc_cache.c, none of them reference
* either adapter, and none of them ever assign core->unsatisfiable_
* lookup_cb/unsatisfiable_populate_cb either -- both fields stay NULL,
* matching production simulation-mode behavior exactly (see
* xce_unsatisfiable_lookup_cb_t/xce_unsatisfiable_populate_cb_t's own doc
* comment, core.h, for the full "why NULL is correct for sim mode"
* rationale).
*
* The (xce_scheduler_t*)core cast below is safe for the identical reason
* xce_alloc_cache_clear_cb()'s own cast is safe just above: these two
* adapters are normal-mode-exclusive -- only ever assigned onto
* core->unsatisfiable_lookup_cb/unsatisfiable_populate_cb by main.c's
* normal-mode branch, only ever invoked with a `core` pointer genuinely
* embedded in an xce_scheduler_t (xce_sched_core_t core is
* xce_scheduler_t's first member, scheduler.h) -- a simulation-mode core
* never registers either callback, so this cast is never reached with a
* xce_sched_sim_driver_t*-embedded core. This is exactly the "small
* adapter function... itself normal-mode-exclusive" allowance design.md's
* "Unsatisfiable-Cache Callback Seam" component documents -- the cast
* must never appear in code shared by both drivers, but is safe here.
*/
int xce_alloc_cache_lookup_cb(xce_sched_core_t* core, xce_sched_obj_t* obj,
int* out_reason_category)
{
return cache_lookup((xce_scheduler_t*)core, obj, out_reason_category);
}
void xce_alloc_cache_populate_cb(xce_sched_core_t* core, xce_sched_obj_t* obj,
int reason_category)
{
cache_insert((xce_scheduler_t*)core, obj, reason_category);
}

22
src/scheduler/src/main.c
View file

@ -471,6 +471,28 @@ int main(int argc, char* argv[])
* "Simulation isolation"). */
xce_core_set_cache_clear_cb(&(sched.core), xce_alloc_cache_clear_cb);
/* Req 5.2 (scheduler-sim-live-allocation spec, task 1.1): register the
* optional unsatisfiable-resreq cache consult/populate seam -- see
* xce_unsatisfiable_lookup_cb_t/xce_unsatisfiable_populate_cb_t's own
* doc comment (core.h) for the full rationale. Direct field
* assignment, not a setter call (unlike cache_clear_cb just above):
* xce_sched_core_t is not opaque, and these two fields carry no
* locking/session-active discipline beyond the plain NULL-check the
* seam's own contract already gives them -- main.c already writes
* xce_sched_core_t fields directly elsewhere (e.g. sig_handler()'s
* `sched->core.stop = 1;`, above), so this is a precedented idiom, not
* a new one. Registered once, alongside every other driver-registered
* core hook, before the first scheduling round. The simulation driver
* (sim_loop.c) deliberately never sets either field -- a
* simulation-mode core has no cache to consult or populate (Req 5 AC
* 17 "Simulation isolation", same rationale as cache_clear_cb just
* above). Wiring the actual consult/populate call site into
* sched_make_primary_alloc_() (scheduler.c) is a LATER task's job
* (tasks.md 2.2) -- until then, both fields being non-NULL here has
* no observable effect, since nothing yet reads them. */
sched.core.unsatisfiable_lookup_cb = xce_alloc_cache_lookup_cb;
sched.core.unsatisfiable_populate_cb = xce_alloc_cache_populate_cb;
signal(SIGINT, sig_handler);
INFO("xce scheduler is running");

137
src/scheduler/testsuite/test_alloc_cache.c
View file

@ -420,6 +420,140 @@ static void test_argument_validation(void)
fini_sched(&sched);
}
/* ---------------------------------------------------------------------- */
/* Task 1.1 (scheduler-sim-live-allocation spec, Req 5.2): the */
/* unsatisfiable-cache callback seam. */
/* ---------------------------------------------------------------------- */
/*
* A freshly-zeroed xce_sched_core_t (the same memset() default
* xce_sched_core_init() applies -- core.c is not linked into this
* white-box TU, so this test zeroes a bare instance directly rather than
* calling that function) leaves BOTH new callback fields NULL. This is
* the exact state every simulation-mode xce_sched_core_t is in forever --
* sim_loop.c never registers either callback (Req 5 AC 17 "Simulation
* isolation").
*/
static void test_seam_fields_default_null_on_zeroed_core(void)
{
xce_sched_core_t core;
memset(&core, 0, sizeof(core));
TEST_ASSERT_NULL(core.unsatisfiable_lookup_cb);
TEST_ASSERT_NULL(core.unsatisfiable_populate_cb);
}
/*
* THE Observable this task's own TDD note names: "with both callbacks
* left NULL, the allocation dispatch chokepoint always computes fresh
* with no crash and no incorrect skip." The real chokepoint
* (sched_make_primary_alloc_(), scheduler.c) is migrated to actually
* consult this seam by a LATER task (tasks.md 2.2, boundary "Core-Safe
* Allocation Chain, Unsatisfiable-Cache Callback Seam") -- this task adds
* only the seam itself, not that call site. This test proves the seam's
* own NULL-safety contract in isolation, using the identical NULL-check
* shape design.md's own invariant specifies for that future call site
* ("the allocation chain ... only ever NULL-checks and calls these"):
* with both fields NULL, a NULL-checked consult never dereferences either
* function pointer (no crash), is never mistaken for a cache HIT (no
* incorrect skip -- `hit` stays 0, meaning "compute fresh"), and a
* NULL-checked populate attempt is silently skipped, not a fatal error.
*/
static void test_null_callbacks_mean_always_compute_fresh(void)
{
xce_sched_core_t core;
xce_sched_obj_t obj;
xce_primary_schedobj_t primary;
int category = -1;
int hit = -1; /* poison: neither 0 nor 1 yet */
memset(&core, 0, sizeof(core));
init_primary_obj(&obj, &primary, BASE_MIN_TASKS, BASE_MAX_TASKS,
BASE_SELECT_NODE_ID, BASE_RUSAGE_HASH, BASE_USER,
BASE_QUEUE, BASE_PROJECT);
/* Prove the guard's PRECONDITION explicitly, immediately before relying
* on it below -- not just that the end result happens to be "compute
* fresh", but that the NULL-ness this test's two guarded calls depend
* on genuinely holds at the exact point each guard is evaluated. A
* design that accidentally called through a non-NULL stale/garbage
* pointer here (e.g. an uninitialized field instead of a true NULL)
* could still coincidentally return 0/not crash on some platforms;
* asserting NULL directly, right where each guard fires, rules that
* out. */
TEST_ASSERT_NULL(core.unsatisfiable_lookup_cb);
if (core.unsatisfiable_lookup_cb != NULL) {
hit = core.unsatisfiable_lookup_cb(&core, &obj, &category);
} else {
hit = 0; /* no cache registered => always a miss => compute fresh */
}
TEST_ASSERT_EQUAL_INT(0, hit);
/* A NULL-checked populate attempt must be a true no-op, not a crash --
* same "assert the guard's own precondition, right at the guard"
* discipline as the lookup check just above. */
TEST_ASSERT_NULL(core.unsatisfiable_populate_cb);
if (core.unsatisfiable_populate_cb != NULL) {
core.unsatisfiable_populate_cb(&core, &obj, 7);
}
}
/*
* Normal-mode observable (this task's own note): "a normal-mode test
* confirming the cache is still consulted/populated exactly as before
* this change." xce_alloc_cache_lookup_cb()/xce_alloc_cache_populate_cb()
* (alloc_cache.c) are the two adapters main.c registers on the real
* xce_scheduler_t's core (xce_sched_core_t core is xce_scheduler_t's
* first member, scheduler.h -- the same pointer-to-first-member idiom
* xce_alloc_cache_clear_cb() already uses). This test calls them through
* core.unsatisfiable_lookup_cb/unsatisfiable_populate_cb exactly as a
* normal-mode driver would, and confirms the result is identical to
* calling cache_lookup()/cache_insert() directly -- proving the adapters
* are a transparent passthrough, not a behavior change.
*/
static void test_lookup_and_populate_cb_adapters_match_direct_cache_calls(void)
{
xce_scheduler_t sched;
xce_sched_obj_t obj;
xce_primary_schedobj_t primary;
int out_direct = -1;
int out_via_cb = -1;
int rc;
init_sched(&sched);
sched.core.unsatisfiable_lookup_cb = xce_alloc_cache_lookup_cb;
sched.core.unsatisfiable_populate_cb = xce_alloc_cache_populate_cb;
init_primary_obj(&obj, &primary, BASE_MIN_TASKS, BASE_MAX_TASKS,
BASE_SELECT_NODE_ID, BASE_RUSAGE_HASH, BASE_USER,
BASE_QUEUE, BASE_PROJECT);
/* Miss before any insert, via the callback -- same as a direct
* cache_lookup() call (test_lookup_before_any_insert_is_a_miss
* above). */
TEST_ASSERT_NOT_NULL(sched.core.unsatisfiable_lookup_cb);
rc = sched.core.unsatisfiable_lookup_cb(&(sched.core), &obj, &out_via_cb);
TEST_ASSERT_EQUAL_INT(0, rc);
/* Populate via the callback... */
TEST_ASSERT_NOT_NULL(sched.core.unsatisfiable_populate_cb);
sched.core.unsatisfiable_populate_cb(&(sched.core), &obj, 55);
/* ...then confirm a DIRECT cache_lookup() call sees the SAME entry --
* the adapter genuinely populated the real cache, not a shadow copy. */
rc = cache_lookup(&sched, &obj, &out_direct);
TEST_ASSERT_EQUAL_INT(1, rc);
TEST_ASSERT_EQUAL_INT(55, out_direct);
/* And the callback-based lookup now also reports the same hit. */
rc = sched.core.unsatisfiable_lookup_cb(&(sched.core), &obj, &out_via_cb);
TEST_ASSERT_EQUAL_INT(1, rc);
TEST_ASSERT_EQUAL_INT(55, out_via_cb);
fini_sched(&sched);
}
int main(void)
{
UNITY_BEGIN();
@ -436,5 +570,8 @@ int main(void)
RUN_TEST(test_clear_empties_all_entries);
RUN_TEST(test_insert_duplicate_key_returns_eexist);
RUN_TEST(test_argument_validation);
RUN_TEST(test_seam_fields_default_null_on_zeroed_core);
RUN_TEST(test_null_callbacks_mean_always_compute_fresh);
RUN_TEST(test_lookup_and_populate_cb_adapters_match_direct_cache_calls);
return UNITY_END();
}