Fully closes bug-1829's own tracked follow-up from task 4.1: retypes seqjob_allocator.c/parjob_allocator.c's init_fn AND runtime hooks (prepare_allocation_hook, job_alloc_fn) to read the plugin's own honest plugin->core field (task 1.2) instead of deriving an xce_scheduler_t* from plugin->sched -- mirroring task 2.3's limit.c migration exactly. The gap was broader than initially scoped: not just init_fn, but every runtime hook independently derived sched from plugin->sched on each allocation attempt. Adds seqjob_allocator_new_plugin()/parjob_allocator_new_plugin() factories (plugin.h, mirroring limit_new_plugin()), letting sim_alloc.c's bootstrap functions register both plugins DIRECTLY against &(sim->core) via the existing plugin_register_core() -- the same call sim_bootstrap_picker()/sim_bootstrap_limit_plugin() already use. Task 4.1's throwaway xce_scheduler_t wrapper is gone from the plugin-registry/fan-out path entirely; plugin_prepare_allocation()'s fan-out during a live seqjob/parjob dispatch now genuinely reaches sim->core.plugins, the same registry the limit plugin is registered in. One narrow, necessary exception remains, independently verified confined: my_alloc() still needs a small xce_scheduler_t wrapper (only .core.resmgr aliased to sim->core) to satisfy sched_make_allocs() (scheduler.c, out of this task's boundary, unmodified) -- set on plugin->sched only AFTER plugin_register_core() has run, never read by the plugin registry or any prepare_allocation_hook fan-out. TDD: new test_sim_alloc_limit_fanout.c drives a real live dispatch (sched_job_picker()/sched_allocator() thread pair) with a host-scoped limit registered directly against sim->core, proving a real usage instance is created -- isolated from both reconciliation and global-limit matching. RED: dispatch succeeded but the usage-instance iterator was empty. GREEN: 82/82, zero regressions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
513 lines
56 KiB
Markdown
513 lines
56 KiB
Markdown
# Technical Design: scheduler-sim-live-allocation
|
|
|
|
## Overview
|
|
|
|
**Purpose**: This feature gives the scheduler's simulation/replay driver (`xce_scheduler -i` / `-s <replay>`) a genuine, live job-election and allocation path, sharing a single implementation with the normal-mode driver, instead of unconditionally pending every job as it does today.
|
|
|
|
**Users**: Scheduler operators and developers debugging production scheduling behavior offline via replay, and this codebase's own maintainers, who need normal-mode and simulation-mode to stay behaviorally identical without relying on two independently-verified implementations.
|
|
|
|
**Impact**: Today, `sim_picker_thread_fn()` (`sim_loop.c`) unconditionally PENDs every job it considers; every real dispatch decision in simulation mode instead comes from a separate mechanism, `sim_reconcile_unaccounted_jobs()`, that trusts a recorded decision rather than computing one. This feature replaces `sim_picker_thread_fn()` with the literal `sched_job_picker()`/`sched_allocator()` functions normal mode already uses (after migrating their remaining `xce_scheduler_t*`-coupled dependencies to the shared `xce_sched_core_t*`), makes limit matching (the concrete acceptance test, bug-1826) work as a side effect of that shared path, and narrows `sim_reconcile_unaccounted_jobs()` to a safety net for the async timing gap between an event's arrival and the live path's processing of it.
|
|
|
|
### Goals
|
|
- Job election and allocation run as one implementation, used verbatim by both drivers.
|
|
- Global and host-scoped resource-limit matching work in simulation mode with the same fidelity as normal mode.
|
|
- A divergence between a live-computed decision and its recorded counterpart is surfaced, never silently resolved.
|
|
- Zero change to normal mode's own observable behavior.
|
|
|
|
### Non-Goals
|
|
- Simulation-mode throughput/latency parity with normal mode (the `unsatisfiable_cache` optimization stays normal-mode-only).
|
|
- A new clean shutdown/teardown path for simulation-mode worker threads (out of scope; pre-existing asymmetry, not introduced by this feature).
|
|
- Any new REPL command, `info` subcommand, or replay-file format change.
|
|
- Resolving the (currently unconfirmed) `job_alloc_fn`/`register_seqjob_allocator`/`register_parjob_allocator` plugin-hook question definitively before implementation — flagged as a verify-at-implementation-time risk in `research.md`, not blocking this design.
|
|
|
|
## Boundary Commitments
|
|
|
|
### This Spec Owns
|
|
- The core-safe (`xce_sched_core_t*`) signatures of the allocation-chain functions listed in the Architecture section's migration table.
|
|
- `plugin_job_election()`, `plugin_prepare_allocation()`, and `plugin_commit_decision()`'s dispatcher signatures and their single shared implementation across both drivers.
|
|
- The limit plugin's honest-core-parameter entry points for job election and host-scoped matching (extending the `limit_seed_config()` pattern bug-1825 established for the config-hook path).
|
|
- `sched_job_picker()`/`sched_allocator()`'s core-safe signatures and their use as the sole picker/allocator implementation for both drivers, replacing `sim_picker_thread_fn()`.
|
|
- The `unsatisfiable_cache` callback seam on `xce_sched_core_t` and its NULL-safe fallback behavior.
|
|
- The authority rule between live-computed decisions and `sim_reconcile_unaccounted_jobs()`'s recorded-decision reconstruction, and the divergence-reporting behavior when they disagree.
|
|
|
|
### Out of Boundary
|
|
- The `scheduler-allocation` spec's own algorithm internals (`host_iter.c`, `parjob_allocator.c`, the unsatisfiable-resreq cache's own lookup/populate logic) — this spec only wraps that cache behind a seam, it does not change what the cache does or how it decides.
|
|
- bug-1827 (the `core_clock.c` isolation-test coverage gap) — unrelated, tracked separately.
|
|
- Any change to `.replay` file format, recording/capture (`recording.c`/`recording_events.c`), or snapshot format (`snapshot_wire.c`/`snapshot_load.c`) — confirmed field-complete and unaffected by the audit (dimension 4).
|
|
- Giving simulation mode a working `unsatisfiable_cache` implementation of its own.
|
|
- Simulation-mode thread lifecycle/teardown (`sched_fini()`-equivalent for sim mode does not exist today and this spec does not add one).
|
|
|
|
### Allowed Dependencies
|
|
- `core.c`/`core_events.c`/`core_session.c`/`core_clock.c` (the mechanically-isolated core module) — read-only from this spec's perspective; no changes needed there beyond the new optional callback field on `xce_sched_core_t`.
|
|
- The completed `scheduler-allocation` spec's `host_iter.c`/`parjob_allocator.c`/`seqjob_allocator.c` — this design depends on their *algorithm* (host iteration order, bucket/plan construction) staying unmodified; it only touches the type threaded through their existing entry points (see File Structure Plan for the specific, minimal changes each file needs).
|
|
- bug-1825's plugin-bootstrap functions (`sim_bootstrap_picker()`, `sim_bootstrap_limit_plugin()`, `limit_seed_config()`) — this spec's honest-core-parameter treatment of job-election/prepare-allocation extends the same pattern those functions already established.
|
|
- `cbuf`/`mylock` (`src/lib`) — the existing rendezvous primitive; no changes.
|
|
|
|
### Revalidation Triggers
|
|
- A future change to `xce_sched_plugin_t`'s hook signatures (`plugin_hooks_t`) must re-verify every hook dispatcher this spec touches (`job_election_hook`, `prepare_allocation_hook`, `commit_hook`) stays core-safe.
|
|
- A future change to `xce_scheduler_t.unsatisfiable_cache`'s data shape must re-verify the callback seam's contract still matches.
|
|
- A future spec giving simulation mode a working live allocator with its own cache, or a clean teardown path, must re-check this spec's Non-Goals are still accurate.
|
|
- Any change to `sched_job_picker()`/`sched_allocator()`'s signature or behavior automatically changes simulation mode too (single implementation) — normal-mode-only test coverage is no longer sufficient; both drivers' test suites must be re-run.
|
|
|
|
## Architecture
|
|
|
|
### Existing Architecture Analysis
|
|
|
|
This is an extension of the already-established "one core, two consumers" split (`scheduler-core` spec): `xce_sched_core_t` (passive state + synchronization primitives, mechanically isolated from controller I/O) is driven by two interchangeable top-level structs, `xce_scheduler_t` (normal mode) and `xce_sched_sim_driver_t` (simulation mode), each embedding `xce_sched_core_t core` as a real field. Six functions in the picker/allocator path (`plugin_begin_session`, `plugin_end_session`, `sched_resolve_job_picker`, `sched_get_job`, `sched_account_jobs`, `sched_commit_job_alloc`) already completed the `xce_scheduler_t*` → `xce_sched_core_t*` migration and are called identically by both drivers today. This design completes that migration for the remaining allocation-chain functions and the picker/allocator thread bodies themselves.
|
|
|
|
**Project invariants checked** (`/xce-audit-invariants scheduler`, run during this spec's requirements phase — see `requirements.md`'s Project Description and `research.md` for full evidence):
|
|
- **Driver state-equivalence** (`scheduler-driver-equivalence.md`) — this spec's entire purpose is closing the remaining gap in this dimension. Checked against every function in the migration table below by reading its body, not inferring from its signature.
|
|
- **Core module I/O isolation** — not touched by this spec (only `xce_sched_core_t`'s field list grows by one optional callback pointer; no I/O, no new controller/netio dependency). The unrelated coverage gap found in this dimension (bug-1827) does not block this design.
|
|
- **jbuf capacity-1 rendezvous** — this spec deliberately changes simulation mode from bypassing `jbuf` entirely to using the same already-verified two-thread producer/consumer pattern normal mode uses. The invariant (capacity hardcoded to 1, one producer, one consumer) is unaffected because it's the same code, not a new implementation.
|
|
- **Snapshot field-completeness** — not touched; no new persisted field, no change to `snapshot_wire.c`/`snapshot_load.c`.
|
|
|
|
### Architecture Pattern & Boundary Map
|
|
|
|
```mermaid
|
|
graph TB
|
|
subgraph NormalMode
|
|
NormalDriver[NormalModeRound]
|
|
end
|
|
subgraph SimMode
|
|
SimDriver[SimDriverRun]
|
|
end
|
|
subgraph SharedCore
|
|
CoreState[xce_sched_core_t]
|
|
PickerThread[SchedJobPicker]
|
|
AllocatorThread[SchedAllocator]
|
|
ElectionDispatch[PluginJobElection]
|
|
AllocationChain[AllocationChain]
|
|
CommitDispatch[PluginCommitDecision]
|
|
end
|
|
subgraph SharedPlugins
|
|
LimitPlugin[LimitPlugin GlobalAndHostMatch]
|
|
end
|
|
subgraph NormalOnly
|
|
CacheSeam[UnsatisfiableCacheOps]
|
|
end
|
|
subgraph SimOnly
|
|
Reconcile[SimReconcileUnaccountedJobs]
|
|
end
|
|
|
|
NormalDriver --> CoreState
|
|
SimDriver --> CoreState
|
|
CoreState --> PickerThread
|
|
PickerThread --> AllocatorThread
|
|
AllocatorThread --> ElectionDispatch
|
|
ElectionDispatch --> LimitPlugin
|
|
AllocatorThread --> AllocationChain
|
|
AllocationChain --> LimitPlugin
|
|
AllocationChain --> CacheSeam
|
|
AllocatorThread --> CommitDispatch
|
|
SimDriver --> Reconcile
|
|
Reconcile -.-> CoreState
|
|
```
|
|
|
|
**Architecture Integration**:
|
|
- Selected pattern: single shared implementation for election/allocation (Requirement 5), with mode-specific differences isolated behind two narrow seams — the `UnsatisfiableCacheOps` callback (normal-mode-only, NULL in sim mode) and `SimReconcileUnaccountedJobs` (sim-mode-only, narrowed to the timing-gap safety net).
|
|
- Domain/feature boundaries: `SharedCore` is driver-agnostic and callable from both; `NormalOnly`/`SimOnly` subgraphs are the only places behavior legitimately differs.
|
|
- Existing patterns preserved: `xce_sched_core_t` embedding, the `cache_clear_cb`-style optional-callback seam, the plugin-hook-dispatcher pattern (`plugin_begin_session`/`plugin_end_session` already core-safe).
|
|
- New components rationale: `UnsatisfiableCacheOps` is new (a seam that didn't need to exist while the allocation chain was normal-mode-only); everything else is an existing component moved to a core-safe signature, not a new component.
|
|
- Steering compliance: directly satisfies `scheduler-driver-equivalence.md`'s state-equivalence invariant and `src/scheduler/CLAUDE.md`'s "one core, two consumers" statement.
|
|
|
|
### Technology Stack
|
|
|
|
| Layer | Choice / Version | Role in Feature | Notes |
|
|
|-------|------------------|-----------------|-------|
|
|
| Backend / Core | C (gnu17) | All changes are within the existing scheduler C codebase | No new language/runtime |
|
|
| Concurrency | POSIX pthreads, existing `cbuf`/`mylock` (`src/lib`) | Simulation mode gains a genuine two-thread producer/consumer pair, reusing the exact primitives normal mode already uses | No new synchronization primitive introduced |
|
|
| Build | Meson | New/changed files wired into `sched_sim_isolation_lib` and normal build targets | Follows existing `meson.build` conventions per `src/scheduler/CLAUDE.md` |
|
|
|
|
## File Structure Plan
|
|
|
|
### Modified Files
|
|
- `src/scheduler/include/core.h` — add `unsatisfiable_lookup_cb`/`unsatisfiable_populate_cb` (two optional function-pointer fields, NULL unless registered) to `xce_sched_core_t`, alongside the existing `cache_clear_cb` field and documented with the same `xce_core_cache_clear_cb_t`-style rationale comment.
|
|
- `src/scheduler/src/alloc_cache.c` — add the two small adapter functions implementing `unsatisfiable_lookup_cb`/`unsatisfiable_populate_cb`, alongside the existing `xce_alloc_cache_clear_cb()` adapter (same file, same reason: normal-mode-exclusive code that safely recovers `xce_scheduler_t*` from `core`, kept out of `driver.c` so the ~11 white-box test files that `#include driver.c` without linking `alloc_cache.c` are unaffected).
|
|
- `src/scheduler/src/schedobj.c` — `sched_ctx_init()` populates `xce_sched_ctx_t.sched` with `core` instead of `sched`; `sched_ctx_set_primary()`'s existing `(xce_scheduler_t*)ctx->sched` cast becomes `(xce_sched_core_t*)ctx->sched`; `sched_ctx_get_uncomit_host()` is already core-safe, unchanged.
|
|
- `src/scheduler/src/host_iter.c`, `src/scheduler/src/parjob_allocator.c`, `src/scheduler/src/seqjob_allocator.c` (`scheduler-allocation` spec's own files) — no algorithm change; their entry points (`my_alloc()`, `host_iter_init()`) accept whatever type their caller (`sched_make_primary_allocation()`, now core-safe) and `ctx->sched` (now `xce_sched_core_t*`) hand them. Confirm the exact signature ripple at implementation time (see `research.md` Risks) — this design commits to the seam (`ctx->sched`'s type), not to enumerating every internal call site in these files.
|
|
- `src/scheduler/include/plugin.h` — no signature change to `plugin_hooks_t` itself; `plugin_job_election()`, `plugin_prepare_allocation()`, `plugin_commit_decision()` declarations change from `void*` to `xce_sched_core_t*`; `xce_sched_plugin_t` gains a new `xce_sched_core_t* core` field (alongside the existing, untouched `sched` field — see Limit Plugin Core-Safe Matching for why they stay separate).
|
|
- `src/scheduler/src/plugin.c` — re-type `plugin_job_election()`, `plugin_prepare_allocation()`, `plugin_commit_decision()` to take `xce_sched_core_t* core` and iterate `core->plugins` directly (mirrors `plugin_begin_session()`'s existing body); `plugin_register()` additionally sets `plugin->core = &(sched->core);`; `plugin_register_core()` additionally sets `plugin->core = core;`.
|
|
- `src/scheduler/src/resource.c` — re-type `sched_meet_host_selection()`, `sched_check_shared_satisfy()`/`sched_check_shared_satisfy_()`, `sched_check_host_satisfy()`/`sched_check_host_satisfy_()`, `sched_try_alloc_on_host_()` to take `xce_sched_core_t* core`.
|
|
- `src/scheduler/src/scheduler.c` — re-type `sched_commit_decision()` and the full allocation-dispatch switch (`sched_make_allocation()`, `sched_make_primary_allocation()`, `sched_make_compound_allocation()`, `sched_make_alter_allocation()`, and whichever internal function turns out to hold the direct `sched->unsatisfiable_cache` touch — believed to be `sched_make_primary_alloc_()`, confirm at implementation time) to take `xce_sched_core_t* core`; update their bodies' internal `sched->core.X` references to `core->X`; replace the one direct `sched->unsatisfiable_cache` touch with NULL-checked calls to `core->unsatisfiable_lookup_cb`/`core->unsatisfiable_populate_cb`; **remove** `sched_job_picker()`/`sched_allocator()` (moved out, see new file below); update `sched_init()` to spawn the moved functions and pass `&(sched->core)`.
|
|
- `src/scheduler/src/sched_election.c` (**new file**) — houses the core-safe `sched_job_picker()`/`sched_allocator()`, extracted verbatim (signature migrated) from `scheduler.c`. Two reasons, not one: (1) it is the single implementation both drivers spawn, so it should not live inside a file (`scheduler.c`) that is otherwise entirely normal-mode-only and controller-coupled; (2) `scheduler.c` is already 1382 lines, over this project's own 800-line cap, and `src/scheduler/CLAUDE.md` explicitly asks for opportunistic splitting rather than further growth — this migration is exactly such an opportunity, not scope creep invented for its own sake. Declared in a small new header or an existing shared one (`scheduler.h`), whichever this codebase's own convention for a two-function extraction favors — confirm at implementation time.
|
|
- `src/scheduler/src/plugins/limit.c` — add `limit_job_election_core()`/`limit_prepare_allocation_core()` (or equivalently-named honest-core entry points) alongside the existing hook-registered `limit_job_election()`/`limit_prepare_allocation()`, following the `limit_seed_config()`/`limit_seed_config_impl()` split bug-1825 established; `match_job_global_limit_()`/`match_job_host_limit_()` take `xce_sched_core_t* core` directly instead of deriving `sched` from `plugin->sched`.
|
|
- `src/scheduler/src/sim_loop.c` — replace the `pthread_create(&picker_tid, NULL, sim_picker_thread_fn, &(sim->core))` call in `sched_sim_driver_run()` with two `pthread_create()` calls against `sched_job_picker`/`sched_allocator`, passing `&(sim->core)`; remove `sim_picker_thread_fn()` entirely; `sim_reconcile_unaccounted_jobs()` gains a call to the core-safe job-election entry point before its FOUND-branch decision reconstruction (idempotency already guarded by `obj->configver >= core.configver`).
|
|
- `src/scheduler/src/main.c` — register `unsatisfiable_lookup_cb`/`unsatisfiable_populate_cb` only in the normal-mode (controller-connected) startup branch, mirroring the existing `cache_clear_cb` registration's placement.
|
|
- `src/scheduler/meson.build` — add `sched_election.c` to the normal build's `scheduler_srcs` (both drivers link it, since it lives in the single `xce_scheduler` binary alongside `scheduler.c`/`sim_loop.c`); confirm whether it also needs adding to `sched_sim_isolation_lib`'s source set (Req 5 AC 17 coverage), mirroring how `sim_limit.c` was added there for the analogous reason during `scheduler-sim-diagnostics` task 6.
|
|
- White-box test files (`test_sim_*.c`, `test_repl_*.c`, `test_scheduler_*.c`, per this codebase's established "duplicate, don't share" stub convention) — any file that `#include`s `plugin.c`/`resource.c`/`scheduler.c` without linking its full dependency set will need its link-satisfying stubs updated for the new `xce_sched_core_t*` signatures, following the exact ripple-fix pattern already used for `limit_get_configured_limits()`/`limit_get_usage_instances()` (scheduler-sim-diagnostics task 2.1's Implementation Note) and `limit_seed_config()` (bug-1825's own remediation). Extracting `sched_job_picker()`/`sched_allocator()` into `sched_election.c` specifically limits this ripple for `sim_loop.c`'s own white-box tests: they now only need to link the small new file to exercise real election/allocation, not the whole of `scheduler.c`.
|
|
|
|
- `src/scheduler/src/sim_alloc.c` (**new file**, added during task 4.1's implementation) — `sim_bootstrap_seqjob_allocator()`/`sim_bootstrap_parjob_allocator()`, bootstrapping `core->seqjob_allocator`/`core->parjob_allocator` for simulation mode, mirroring the established `sim_picker.c`/`sim_limit.c` bootstrap precedent exactly. Fixes bug-1829: these two fields (read directly by `sched_make_seqjob_allocation()`/`sched_make_parjob_allocation()`, not looked up via `core->plugins`) were never populated for the simulation driver at all, which was undiscovered until task 4.1 made `sched_allocator()` genuinely reach `sched_make_allocation()` in sim mode for the first time — before that, this code path was structurally unreachable in simulation mode, so the gap had zero prior symptoms. Without this fix, task 4.1's own acceptance criteria (a live dispatch decision, not an unconditional pend) could not be met at all. See "Known Seam Gap: Host-Scoped Limit Fan-Out During Live Seqjob/Parjob Dispatch" below for a limitation this fix's own mechanism introduces.
|
|
|
|
> Two new files are introduced: `sched_election.c` (task 3.1) and `sim_alloc.c` (task 4.1, fixing bug-1829). Both extract or bootstrap existing behavior rather than adding new behavior — justified by the isolation-boundary/file-size reasons above (`sched_election.c`) and by a genuinely blocking pre-existing gap discovered only once live sim-mode allocation became reachable (`sim_alloc.c`), not by any new component or abstraction layer. Every other change is a signature/call-site migration of an existing function, or a small addition inside an existing file.
|
|
|
|
### Known Seam Gap: Host-Scoped Limit Fan-Out During Live Seqjob/Parjob Dispatch
|
|
|
|
**Confirmed during task 4.1's review, mechanism-traced end to end — not resolved by that task, closed by task 4.3 (see task 4.3's own commit/buglog-1829 entry for the actual fix: `plugin->core`-based retyping of `seqjob_allocator.c`/`parjob_allocator.c`'s `init_fn` and runtime hooks, plus direct `plugin_register_core()` registration against `sim->core`, eliminating the wrapper-registry entirely from the fan-out path).** `register_seqjob_allocator()`/`register_parjob_allocator()`'s own `init_fn` (a pre-existing, real, out-of-boundary dependency this feature's tasks 1.2/2.3 never touched, since those covered `job_election_hook`/`prepare_allocation_hook`/`commit_hook`, not `init_fn`) still requires a real `xce_scheduler_t*`, which simulation mode does not have. `sim_bootstrap_seqjob_allocator()`/`sim_bootstrap_parjob_allocator()` satisfy this with a throwaway `xce_scheduler_t` wrapper, calling `plugin_register()` (not `plugin_register_core()`) against it — which means the seqjob/parjob plugin instances' own `xce_sched_plugin_t.core` field (task 1.2) ends up honestly pointing at the **wrapper's** core, not `sim->core`. `seqjob_allocator.c`/`parjob_allocator.c`'s `my_alloc()` then derives its own `sched`/`core` reference from `plugin->sched`/`plugin->core` (its OWN plugin instance, not the `core` parameter threaded down from `sched_make_allocation()`), so `sched_try_alloc_on_host_()` → `plugin_prepare_allocation()`'s fan-out during a REAL sim-mode seqjob/parjob dispatch attempt iterates the **wrapper's** plugin registry, not `sim->core.plugins` — meaning any plugin registered only against `sim->core` directly (the limit plugin, via `sim_bootstrap_limit_plugin()`) is never reached for host-scoped matching during a live sim-mode dispatch.
|
|
|
|
This is a genuine functional gap (not a harmless unused-registry artifact — the wrapper's registry is on the real, executed code path), confined specifically to host-scoped limit matching during a live seqjob/parjob allocation attempt in simulation mode. It does not affect: global-limit matching (reached via `plugin_job_election()`, called directly against `sim->core` before allocation, per the "Plugin Hook Dispatchers"/"Limit Plugin Core-Safe Matching" components above); normal mode (unaffected, its own `xce_scheduler_t` is the real one throughout); or the jbuf rendezvous, decision-commit, or accounting-state correctness of the dispatch itself (the allocation DECISION and its resource charge are correct; only a specific plugin hook's visibility during that decision's host search is narrowed). **Closed by task 4.3** (see above) — task 5.2 verifies the resulting end-to-end behavior as part of its own acceptance test, rather than closing the gap itself.
|
|
|
|
## System Flows
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant Driver as DriverPerMode
|
|
participant Core as SchedCore
|
|
participant Picker as SchedJobPicker
|
|
participant Allocator as SchedAllocator
|
|
participant Election as PluginJobElection
|
|
participant Limit as LimitPlugin
|
|
participant Alloc as AllocationChain
|
|
participant Commit as PluginCommitDecision
|
|
|
|
Driver->>Core: PushEvents
|
|
Driver->>Core: BeginSession
|
|
Driver->>Core: SignalGo
|
|
Core->>Picker: wbuf go signal
|
|
Picker->>Core: next_job via jbuf
|
|
Core->>Allocator: jbuf handoff
|
|
Allocator->>Election: job election
|
|
Election->>Limit: match global limit
|
|
Allocator->>Alloc: attempt allocation
|
|
Alloc->>Limit: match host scoped limit
|
|
Alloc-->>Allocator: dispatch or pend
|
|
Allocator->>Commit: commit decision
|
|
Commit-->>Core: decisions cbuf
|
|
Core-->>Driver: drain decisions
|
|
```
|
|
|
|
This flow is identical for both drivers — the only per-mode difference is what `Driver` does before pushing events (capnp translation for normal mode, replay decode for simulation mode) and after draining decisions (publish/commit for normal mode, print/record/`--sim-verify`-compare for simulation mode).
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
Start[Job accounts is null this round] --> LiveReached{Live election allocation already reached this job}
|
|
LiveReached -->|Yes| LiveWins[Live decision is authoritative]
|
|
LiveReached -->|No| RecordingFound{Recorded dispatch found for this job}
|
|
RecordingFound -->|Yes| Reconstruct[Reconstruct outcome from recording]
|
|
RecordingFound -->|No| Wait[Leave pending re-check next round]
|
|
LiveWins --> Compare{Verify mode enabled}
|
|
Reconstruct --> Compare
|
|
Compare -->|Yes and outcomes differ| Report[Report divergence to operator]
|
|
Compare -->|Yes and outcomes match| Continue[Continue]
|
|
Compare -->|No| Continue
|
|
```
|
|
|
|
Key decision: `Reconstruct` never re-fires for a job the `LiveWins` path already handled — both branches set `job->accounts`, and the existing `job->accounts == NULL` trigger guard on `sim_reconcile_unaccounted_jobs()` naturally excludes an already-decided job from being reconsidered (Requirement 3 AC4).
|
|
|
|
## Requirements Traceability
|
|
|
|
| Requirement | Summary | Components | Interfaces | Flows |
|
|
|-------------|---------|------------|------------|-------|
|
|
| 1.1 | Job election/allocation uses same logic in both modes | Core-Safe Allocation Chain, Unified Picker/Allocator Threads | `sched_allocator(core)`, `plugin_job_election(core, job)` | Round flow (sequence) |
|
|
| 1.2 | Sim mode no longer always pends | Unified Picker/Allocator Threads | `sched_allocator(core)` | Round flow |
|
|
| 1.3 | Dispatch applies identical accounting effects | Core-Safe Allocation Chain | `sched_commit_decision(core, decision)`, `sched_commit_job_alloc(core, job)` | Round flow |
|
|
| 2.1 | Global limit match recorded same way | Limit Plugin Core-Safe Matching | `limit_job_election_core(core, job)` | Round flow (Election → Limit) |
|
|
| 2.2 | Host-scoped limit match recorded same way | Limit Plugin Core-Safe Matching, Core-Safe Allocation Chain | `limit_prepare_allocation_core(core, ctx)` | Round flow (Alloc → Limit) |
|
|
| 2.3 | `info limit` shows real usage | Limit Plugin Core-Safe Matching (no REPL change; reads existing accessors) | `limit_get_usage_instances()` (unchanged, from scheduler-sim-diagnostics) | — |
|
|
| 2.4 | Unmatched limit still reports no instance | Limit Plugin Core-Safe Matching | (unchanged accessor behavior) | — |
|
|
| 3.1 | Identical decisions/state at every replay point | Core-Safe Allocation Chain, Unified Picker/Allocator Threads | full chain | Round flow |
|
|
| 3.2 | Same contention outcome | Core-Safe Allocation Chain | `sched_check_host_satisfy`, `sched_check_shared_satisfy` | Round flow |
|
|
| 3.3 | Live decision authoritative when reached | Live/Recorded Authority Rule | — | Decision-authority flowchart |
|
|
| 3.4 | Recording safety net for timing gap | Sim-Mode Reconciliation Narrowing | `sim_reconcile_unaccounted_jobs` (unchanged guard) | Decision-authority flowchart |
|
|
| 3.5 | Divergence reported, not resolved silently | Live/Recorded Authority Rule | existing `--sim-verify` comparison/`sim_log_mismatch` | Decision-authority flowchart |
|
|
| 4.1 | `--sim-verify` continues comparing/reporting | Live/Recorded Authority Rule | unchanged comparison mechanism | Decision-authority flowchart |
|
|
| 4.2 | Interactive/batch remain decision-equivalent | Unified Picker/Allocator Threads | unchanged (`sim->interactive` gate) | — |
|
|
| 4.3 | One-job-at-a-time preserved | Unified Picker/Allocator Threads | `jbuf` (capacity 1, unchanged) | Round flow |
|
|
| 5.1 | Single shared implementation | Unified Picker/Allocator Threads, Core-Safe Allocation Chain | all core-safe signatures | — |
|
|
| 5.2 | Mode-specific differences isolated behind a seam | Unsatisfiable-Cache Callback Seam, Sim-Mode Reconciliation Narrowing | `unsatisfiable_lookup_cb`, `unsatisfiable_populate_cb` | — |
|
|
|
|
## Components and Interfaces
|
|
|
|
| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|
|
|-----------|--------------|--------|--------------|--------------------------|-----------|
|
|
| Core-Safe Allocation Chain | Scheduler Core | Migrate the 7 audit-identified functions plus the 4-function `sched_make_allocation()` dispatch switch from `xce_scheduler_t*` to `xce_sched_core_t*` | 1.1, 1.3, 3.1, 3.2 | Existing `resource.c`/`scheduler.c` internals (P0) | Service |
|
|
| Unsatisfiable-Cache Callback Seam | Scheduler Core | Isolate the one genuinely mode-specific piece of state behind a NULL-safe seam | 5.2 | `main.c` registration (P0) | Service |
|
|
| Plugin Hook Dispatchers | Plugin Framework | Core-safe `plugin_job_election`/`plugin_prepare_allocation`/`plugin_commit_decision` | 1.1, 2.1, 2.2 | `core->plugins` registry (P0) | Service |
|
|
| Limit Plugin Core-Safe Matching | Limit Plugin | Honest-core-parameter entry points for global/host-scoped matching | 2.1, 2.2, 2.3, 2.4 | Plugin Hook Dispatchers (P0), bug-1825's `limit_seed_config` pattern (P1) | Service |
|
|
| Unified Picker/Allocator Threads | Driver Threading | Single `sched_job_picker`/`sched_allocator` implementation, spawned by both drivers | 1.1, 1.2, 4.2, 4.3, 5.1 | Core-Safe Allocation Chain (P0), `jbuf`/`decisions` cbufs (P0) | Service |
|
|
| Live/Recorded Authority Rule | Sim Driver | Live decision wins when reached; divergence from recording is reported | 3.3, 3.5, 4.1 | Unified Picker/Allocator Threads (P0), existing `--sim-verify` comparison (P0) | Service |
|
|
| Sim-Mode Reconciliation Narrowing | Sim Driver | `sim_reconcile_unaccounted_jobs()` continues, scoped to the async timing gap | 3.4 | Unified Picker/Allocator Threads (P1) | Service |
|
|
|
|
### Scheduler Core
|
|
|
|
#### Core-Safe Allocation Chain
|
|
|
|
| Field | Detail |
|
|
|-------|--------|
|
|
| Intent | Every function between "a job is picked" and "a decision is committed" operates on `xce_sched_core_t*`, callable from either driver |
|
|
| Requirements | 1.1, 1.3, 3.1, 3.2 |
|
|
|
|
**Responsibilities & Constraints**
|
|
- Owns the per-host/shared capacity check and commit-decision state transition (`job->allocs`/`accounts`/`state`).
|
|
- Must not read or write anything outside `core->*` and the `xce_sched_worker_t*`/`xce_sched_ctx_t*` passed to it — this is the boundary that makes verbatim reuse safe.
|
|
- Must not introduce a first-member struct cast to reach normal-mode-only state (the documented anti-pattern, `core.h`'s `xce_core_cache_clear_cb_t` comment) — the one case that needs mode-specific state (the cache) goes through the seam below instead.
|
|
|
|
**Dependencies**
|
|
- Inbound: Unified Picker/Allocator Threads — the sole caller (P0)
|
|
- Outbound: Unsatisfiable-Cache Callback Seam — consulted at the primary-dispatch chokepoint (P0)
|
|
- Outbound: Plugin Hook Dispatchers — `plugin_prepare_allocation`/`plugin_commit_decision` (P0)
|
|
|
|
**Contracts**: Service [x]
|
|
|
|
##### Service Interface
|
|
```c
|
|
/* resource.c */
|
|
int sched_meet_host_selection(xce_sched_core_t* core, xce_sched_worker_t* w, xce_select_t* s);
|
|
int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, xvec_idval_t* wanted, xvec_idval_t* unmeet);
|
|
int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, xvec_t* uncomitted, xvec_idval_t* wanted, xvec_idval_t* unmeet);
|
|
int sched_try_alloc_on_host_(xce_sched_worker_t* w, xce_sched_core_t* core, xce_sched_ctx_t* ctx);
|
|
|
|
/* scheduler.c -- the full dispatch switch, not just the leaf commit, must migrate
|
|
together: sched_allocator() (core-safe, see Unified Picker/Allocator Threads)
|
|
calls sched_make_allocation() directly, so every function between that call
|
|
and the resource-check leaves above must share the same xce_sched_core_t*
|
|
the caller now has -- there is no xce_scheduler_t* left in that call chain
|
|
to pass down otherwise. */
|
|
int sched_make_allocation(xce_sched_core_t* core, xce_sched_obj_t* obj);
|
|
int sched_make_primary_allocation(xce_sched_core_t* core, xce_sched_obj_t* obj);
|
|
int sched_make_compound_allocation(xce_sched_core_t* core, xce_sched_obj_t* obj);
|
|
int sched_make_alter_allocation(xce_sched_core_t* core, xce_sched_obj_t* obj);
|
|
int sched_commit_decision(xce_sched_core_t* core, xce_decision_t* decision);
|
|
```
|
|
- Preconditions: `core` non-NULL; `w`/`ctx`/`decision`/`obj` non-NULL where used (matches existing NULL-guard behavior of the pre-migration functions).
|
|
- Postconditions: identical to today's `xce_scheduler_t*`-typed behavior — this is a pure signature migration, not a behavior change.
|
|
- Invariants: no function in this chain reads any field outside `core->*`/`w->*`/`ctx->*`, **except** the one primary-resreq dispatch chokepoint inside `sched_make_primary_allocation()`'s own call chain (believed to be `sched_make_primary_alloc_()`, per `scheduler-allocation`'s task 5.1 notes — not yet directly read in full; verify at implementation time) that currently reads `sched->unsatisfiable_cache` directly. That one call site routes through the Unsatisfiable-Cache Callback Seam instead of a direct field read once migrated — it is the seam's entire reason to exist.
|
|
|
|
**Implementation Notes**
|
|
- Integration: this migration is wider than just the 7 functions originally identified during the audit (the table in `requirements.md`) — it necessarily also includes the whole `sched_make_allocation()` dispatch switch and its three primary/compound/alter sub-functions, since `sched_allocator()` calls into this switch directly and there is no `xce_scheduler_t*` available to hand it after `sched_allocator()` itself becomes core-safe. Confirm during implementation that `sched_make_primary_allocation()`/`sched_make_compound_allocation()`/`sched_make_alter_allocation()` (not yet read in full at design time) touch nothing beyond `core->*`, worker/ctx fields, and the one cache touch point — if any of them reads other `xce_scheduler_t`-only state, that state needs its own seam, following the same NULL-safe-callback pattern as the cache.
|
|
- Validation: contention-fixture tests (per `scheduler-driver-equivalence.md`'s testing discipline) exercising this chain from both drivers.
|
|
- Risks: (1) the open question about `job_alloc_fn`/`register_seqjob_allocator`/`register_parjob_allocator` (see `research.md`) — verify before assuming this chain is fully closed; (2) `sched_make_primary_allocation`/`sched_make_compound_allocation`/`sched_make_alter_allocation`'s full bodies were not read in full during design — treat their migration as believed-mechanical based on the pattern established by every sibling function in this chain, but confirm by reading each body before migrating it, not by assumption.
|
|
|
|
#### Unsatisfiable-Cache Callback Seam
|
|
|
|
| Field | Detail |
|
|
|-------|--------|
|
|
| Intent | Isolate the one genuinely `xce_scheduler_t`-only piece of state behind a NULL-safe seam, so the surrounding allocation code never needs a mode-specific branch |
|
|
| Requirements | 5.2 |
|
|
|
|
**Responsibilities & Constraints**
|
|
- Owns nothing itself — a thin, optional indirection over `xce_scheduler_t.unsatisfiable_cache`'s existing lookup/populate behavior.
|
|
- A NULL callback must mean "always compute fresh" — never an error, never a degraded-correctness path, only a degraded-performance one (Non-Goals).
|
|
- Must never be bridged via a struct-prefix cast **inside code shared by both drivers** — but the small adapter function that implements each callback is itself normal-mode-exclusive (only ever registered by `main.c`'s normal-mode branch, only ever invoked with a `core` pointer that is genuinely embedded in an `xce_scheduler_t`, since sim mode never registers it), so it may safely recover `xce_scheduler_t*` via `(xce_scheduler_t*)core` — the identical, already-reviewed-safe pattern `alloc_cache.c`'s existing `xce_alloc_cache_clear_cb()` adapter uses for `cache_clear_cb`. No separate opaque `impl`/user-data pointer is needed; simplification over an earlier draft of this design that added one unnecessarily (single registrant, no foreseeable second implementation).
|
|
|
|
**Dependencies**
|
|
- Inbound: Core-Safe Allocation Chain, at `sched_make_primary_alloc_()`'s existing chokepoint (P0)
|
|
- Outbound: `xce_scheduler_t.unsatisfiable_cache`'s existing lookup/populate implementation, reached via a small adapter in `alloc_cache.c` (P0), the same file the existing `xce_alloc_cache_clear_cb()` adapter already lives in and for the same reason (white-box tests that `#include driver.c` without linking `alloc_cache.c`)
|
|
|
|
**Contracts**: Service [x]
|
|
|
|
##### Service Interface
|
|
```c
|
|
/* core.h -- mirrors cache_clear_cb's existing shape exactly: a bare function
|
|
pointer taking xce_sched_core_t*, NULL unless registered, no wrapper struct */
|
|
typedef int (*xce_unsatisfiable_lookup_cb_t)(xce_sched_core_t* core, xce_sched_obj_t* obj, int* out_reason_category);
|
|
typedef void (*xce_unsatisfiable_populate_cb_t)(xce_sched_core_t* core, xce_sched_obj_t* obj, int reason_category);
|
|
|
|
/* two new fields on xce_sched_core_t, alongside the existing cache_clear_cb field */
|
|
xce_unsatisfiable_lookup_cb_t unsatisfiable_lookup_cb; /* NULL unless registered */
|
|
xce_unsatisfiable_populate_cb_t unsatisfiable_populate_cb; /* NULL unless registered */
|
|
```
|
|
Corrected during task 1.1's implementation: the parameter list is `xce_sched_obj_t* obj` + `int*`/`int` reason-category, matching the REAL `cache_lookup()`/`cache_insert()` API (`alloc_cache.c`) exactly, not the `uint64_t fingerprint`/`int category` shape drafted above at design time — the real cache already keys on the full `xce_sched_obj_t*` (its embedded `xce_resreq_fingerprint_t` plus user/queue/project identity), so this seam needed no invented fingerprint abstraction on top of it.
|
|
- Preconditions: registered (non-NULL) only by normal mode's `main.c`, exactly once, before `sched_init()` spawns any thread — identical registration discipline to `cache_clear_cb`.
|
|
- Postconditions: NULL callbacks mean every call to `sched_make_primary_alloc_()` computes fully, bypassing the cache fast path — behaviorally correct, matching what happens today whenever the cache has no entry for a fingerprint.
|
|
- Invariants: the allocation chain (shared code) only ever NULL-checks and calls these — it never interprets `core`'s contents itself; only the mode-exclusive adapter functions in `alloc_cache.c` do, and only when they are guaranteed (by registration discipline, not by a runtime check) to have a normal-mode `core`.
|
|
|
|
**Implementation Notes**
|
|
- Integration: registered in `main.c`'s normal-mode branch only, alongside the existing `cache_clear_cb` registration — same placement pattern, same rationale, same file for the adapter implementation (`alloc_cache.c`).
|
|
- Validation: a unit test confirming NULL callbacks never crash and always take the compute-fresh path; a normal-mode test confirming the cache is still consulted/populated exactly as before this change.
|
|
- Risks: none identified beyond the general struct-cast anti-pattern this design explicitly avoids in shared code.
|
|
|
|
### Plugin Framework
|
|
|
|
#### Plugin Hook Dispatchers
|
|
|
|
| Field | Detail |
|
|
|-------|--------|
|
|
| Intent | `plugin_job_election`, `plugin_prepare_allocation`, `plugin_commit_decision` iterate `core->plugins` directly, callable from either driver |
|
|
| Requirements | 1.1, 2.1, 2.2 |
|
|
|
|
**Responsibilities & Constraints**
|
|
- Mirrors `plugin_begin_session()`/`plugin_end_session()`'s already-completed migration exactly — same iteration pattern, same NULL-guard shape.
|
|
- Does not change `plugin_hooks_t`'s function-pointer signatures (`job_election_hook`, `prepare_allocation_hook`, `commit_hook` keep their existing `(void* arg, ...)` shape, since `arg` there is the plugin instance, not the driver struct — only the dispatcher functions that iterate plugins change).
|
|
|
|
**Dependencies**
|
|
- Inbound: Unified Picker/Allocator Threads, Core-Safe Allocation Chain (P0)
|
|
- Outbound: Limit Plugin Core-Safe Matching, via the hook function pointers (P0)
|
|
|
|
**Contracts**: Service [x]
|
|
|
|
##### Service Interface
|
|
```c
|
|
int plugin_job_election(xce_sched_core_t* core, xce_sched_job_t* job);
|
|
int plugin_prepare_allocation(xce_sched_core_t* core, xce_sched_ctx_t* ctx);
|
|
int plugin_commit_decision(xce_sched_core_t* core, xce_decision_t* decision);
|
|
```
|
|
- Preconditions: `core` non-NULL.
|
|
- Postconditions: every registered plugin's corresponding hook is invoked exactly once per call, in registration order — unchanged from today's behavior.
|
|
- Invariants: iterates `core->plugins` (a field both `xce_scheduler_t.core` and `xce_sched_sim_driver_t.core` genuinely have), never a driver-specific field.
|
|
|
|
**Implementation Notes**
|
|
- Integration: single normal-mode call site each (`sched_allocator()`, `sched_try_alloc_on_host_()`, `sched_commit_decision()`) gets updated to pass `&(sched->core)` / `core` (already core-typed after that function's own migration); simulation mode's `sched_allocator()` (now the same function) passes `&(sim->core)`.
|
|
- Validation: existing plugin-hook unit tests re-run unchanged (behavior-preserving refactor); white-box test files that `#include plugin.c` need their link-satisfying stubs' signatures updated.
|
|
- Risks: none beyond the standard ripple-to-white-box-stubs risk already seen in bug-1825/scheduler-sim-diagnostics. Separately, **bug-1771** (an unfixed ASAN SEGV in `fcfs_del_job()`, reached via `plugin_commit_decision()`'s `commit_hook` dispatch to `fcfs_commit`) is a pre-existing, unrelated defect in exactly this dispatch path, discovered during `scheduler-allocation`'s own end-to-end testing. This migration is a pure signature change with no behavior change, so it neither fixes nor worsens bug-1771 — noted here only so an implementer who hits this crash during testing attributes it correctly instead of assuming their own migration caused it.
|
|
|
|
### Limit Plugin
|
|
|
|
#### Limit Plugin Core-Safe Matching
|
|
|
|
| Field | Detail |
|
|
|-------|--------|
|
|
| Intent | `match_job_global_limit_()`/`match_job_host_limit_()` derive `xce_sched_core_t*` honestly instead of via `plugin->sched` (a bare `void*`, NULL for sim-registered plugins) |
|
|
| Requirements | 2.1, 2.2, 2.3, 2.4 |
|
|
|
|
**Responsibilities & Constraints**
|
|
- Extends the exact split bug-1825 established for the config-hook path (`limit_update_config()` / `sim_bootstrap_limit_plugin()` both delegating to a shared `limit_seed_config_impl()`) to the job-election and prepare-allocation call chains.
|
|
- **Concrete mechanism (corrected during task planning — the hook signature itself cannot carry `core`):** `job_election_hook`/`prepare_allocation_hook` keep their existing `(void* arg, ...)` shape where `arg` is the *plugin instance*, not `core` — so `plugin_job_election(core, job)`'s own `core` parameter does not automatically reach the leaf hook function it calls. `xce_sched_plugin_t` (`plugin.h`) gains a **new, honestly-typed field**, `xce_sched_core_t* core`, set by BOTH `plugin_register()` (`plugin->core = &(sched->core);`) and `plugin_register_core()` (`plugin->core = core;` — closing the gap bug-1825 left, since that function never sets anything today). `limit_job_election()`/`limit_prepare_allocation()`/`match_job_global_limit_()`/`match_job_host_limit_()` read `plugin->core` (never NULL after either registration path), not `plugin->sched`.
|
|
- **The existing `plugin->sched` field is deliberately left untouched** — still `xce_scheduler_t*`-flavored, still set only by `plugin_register()`, still read by `plugins/fcfs.c`'s legacy `job_elect_fn` path exactly as today. Repurposing `plugin->sched` itself (an earlier draft of this design considered this) was rejected: it is a *shared* field across every plugin category, and `fcfs.c`'s legacy picker also reads it expecting `xce_scheduler_t*` — changing its meaning would silently break that unrelated, normal-mode-only consumer. Adding a new field costs one pointer per plugin and avoids that regression entirely, consistent with this codebase's own preference (the `limit_seed_config()` precedent) for an honest new entry point over reinterpreting an existing ambiguous one.
|
|
- The plugin-hook-registered entry points (`limit_job_election()`, `limit_prepare_allocation()`, unchanged `(void* arg, ...)` signature) become thin wrappers that read `plugin->core` and delegate to shared `_impl` functions taking `xce_sched_core_t* core` honestly.
|
|
|
|
**Dependencies**
|
|
- Inbound: Plugin Hook Dispatchers, via `job_election_hook`/`prepare_allocation_hook` (P0)
|
|
- Outbound: `instantiate_limit()`/`update_limit_res_value()` (`xcelimit.c`, unchanged) — registers the matched limit as a virtual resource into `core->sharedInst`/`w->resvals`, the same arrays `sched_account_jobs()` already resums every round in both drivers (P0)
|
|
|
|
**Contracts**: Service [x]
|
|
|
|
##### Service Interface
|
|
```c
|
|
/* plugins/limit.c */
|
|
static int limit_job_election_impl(xce_sched_core_t* core, xce_sched_plugin_t* plugin, xce_sched_job_t* job);
|
|
static int limit_prepare_allocation_impl(xce_sched_core_t* core, xce_sched_plugin_t* plugin, xce_sched_ctx_t* ctx);
|
|
|
|
/* existing hook-registered wrappers, now delegating to the _impl functions above
|
|
using the core-safe xce_sched_core_t* the (now core-safe) dispatcher supplies */
|
|
static int limit_job_election(void* arg, xce_sched_job_t* job);
|
|
static int limit_prepare_allocation(void* arg, xce_sched_ctx_t* ctx);
|
|
```
|
|
- Preconditions: `plugin->core` non-NULL (guaranteed by either registration path once this fix lands), `plugin->data` (the limit plugin's private `limits`/`instances` kvmgr handles) already initialized by `limit_init()` — unchanged.
|
|
- Postconditions: a global limit match registers a vres into `core->sharedInst`; a host-scoped match registers into `w->resvals` — identical outcome in both drivers.
|
|
- Invariants: `match_job_global_limit_()`'s existing `obj->configver >= core.configver` idempotency guard is preserved unchanged — safe to call from both the live path and, for global limits only, from `sim_reconcile_unaccounted_jobs()`.
|
|
|
|
**Implementation Notes**
|
|
- Integration: no change to `info limit`'s REPL display logic (`repl_info.c`) or its read-only accessors (`limit_get_configured_limits()`/`limit_get_usage_instances()`, from `scheduler-sim-diagnostics`) — this component only makes the data those accessors read become real.
|
|
- Validation: `test_repl_limit_table.c`'s existing fixtures, extended with a real sim-mode-driven (not test-only-registered) matching scenario for both global and host-scoped limits under contention.
|
|
- Risks: none beyond the standard signature-migration ripple to white-box stubs.
|
|
|
|
### Driver Threading
|
|
|
|
#### Unified Picker/Allocator Threads
|
|
|
|
| Field | Detail |
|
|
|-------|--------|
|
|
| Intent | `sched_job_picker()`/`sched_allocator()` become the sole picker/allocator implementation, spawned by both drivers |
|
|
| Requirements | 1.1, 1.2, 4.2, 4.3, 5.1 |
|
|
|
|
**Responsibilities & Constraints**
|
|
- `sim_picker_thread_fn()` is removed entirely, not deprecated alongside the new path.
|
|
- Thread-spawn call sites: `sched_init()` (normal mode, unchanged call site, unchanged behavior) and `sched_sim_driver_run()` (simulation mode, changed from one `sim_picker_thread_fn` spawn to two spawns of the shared functions).
|
|
- Preserves the capacity-1 `jbuf` rendezvous — this component reuses the existing cbuf, it does not reimplement it.
|
|
- No new teardown/lifecycle work for simulation mode (Non-Goals) — the pre-existing "never joined, lives for process lifetime" behavior is preserved for both threads in sim mode, matching today's `sim_picker_thread_fn()`'s own documented lifecycle.
|
|
|
|
**Dependencies**
|
|
- Inbound: `sched_init()` (normal mode), `sched_sim_driver_run()` (simulation mode) (P0)
|
|
- Outbound: Core-Safe Allocation Chain, Plugin Hook Dispatchers (P0)
|
|
|
|
**Contracts**: Service [x]
|
|
|
|
##### Service Interface
|
|
```c
|
|
/* sched_election.c (new file, extracted from scheduler.c -- see File Structure Plan) */
|
|
static void* sched_job_picker(void* arg); /* arg is xce_sched_core_t* */
|
|
static void* sched_allocator(void* arg); /* arg is xce_sched_core_t* */
|
|
```
|
|
- Preconditions: `sched_resolve_job_picker(core, ...)` must succeed before either thread is spawned (existing gate, unchanged) — for simulation mode, this means the picker plugin must already be bootstrapped (`sim_bootstrap_picker()`, bug-1825) before `sched_sim_driver_run()` spawns these threads.
|
|
- Postconditions: identical per-round behavior to today's normal mode — pull a job, elect it, attempt allocation, commit a decision, hand off via `jbuf`/`decisions`.
|
|
- Invariants: exactly one producer (`sched_job_picker`) and one consumer (`sched_allocator`) per session, per the jbuf rendezvous invariant.
|
|
|
|
**Implementation Notes**
|
|
- Integration: `sched_sim_driver_run()`'s existing `pthread_create(&picker_tid, NULL, sim_picker_thread_fn, &(sim->core))` becomes two `pthread_create()` calls (`sched_job_picker`, `sched_allocator`), both passed `&(sim->core)`.
|
|
- Validation: `test_repl_equivalence.c`, `test_sim_contention_parity.c`, `test_sim_equivalence.c` must pass unchanged or with investigated, understood deltas (see `research.md` Risks) — this is the highest-risk integration point in the whole design.
|
|
- Risks: (1) decision-timing/ordering shift from the fused-thread-to-two-thread change in simulation mode; mitigated by the validation above, not by avoiding the change. (2) **Recurring bug class, not a one-off**: `xce_sched_sim_driver_t sim` held stack-local while a never-joined thread (`sim_picker_thread_fn()` today) keeps a pointer into it past the owning function's return has caused ASAN stack-use-after-return at least three times in this codebase's history (bug-1598, and its own explicitly-documented recurrences bug-1632 and bug-1763, the latter still marked NOT FIXED, out of scope for the spec that found it). This design doubles the number of long-lived threads holding `&(sim->core)` (two instead of one), doubling the surface area for any remaining unaudited instance of this same pattern. Do not assume bug-1598's single fixture fix closed this class — any new or touched test fixture for this component must be positively confirmed to use a `static` or heap-allocated `xce_sched_sim_driver_t`, not merely assumed safe by analogy.
|
|
|
|
### Sim Driver
|
|
|
|
#### Live/Recorded Authority Rule
|
|
|
|
| Field | Detail |
|
|
|-------|--------|
|
|
| Intent | The live picker/allocator's decision is authoritative wherever it reaches a job; a disagreement with the recording is reported, not resolved silently |
|
|
| Requirements | 3.3, 3.5, 4.1 |
|
|
|
|
**Responsibilities & Constraints**
|
|
- This is a behavioral rule, not a new function — it falls out of (a) the live path now producing real decisions and (b) `sim_reconcile_unaccounted_jobs()`'s existing `job->accounts == NULL` guard naturally skipping any job the live path already decided.
|
|
- The existing `--sim-verify` positional comparison (`sim_log_mismatch()`) is the reporting mechanism — unchanged, now actually exercised meaningfully for the first time against live-computed decisions instead of only against reconstructed-from-recording ones.
|
|
|
|
**Dependencies**
|
|
- Inbound: none (a property of the interaction between the two components below) (P0)
|
|
- Outbound: Unified Picker/Allocator Threads, Sim-Mode Reconciliation Narrowing (P0)
|
|
|
|
**Contracts**: State [x]
|
|
|
|
##### State Management
|
|
- State model: a job's `accounts == NULL` marks it as not-yet-decided; either the live allocator (via `sched_commit_decision`) or `sim_reconcile_unaccounted_jobs()`'s FOUND branch sets it, whichever reaches the job first this round.
|
|
- Persistence & consistency: no new persisted state; this is in-memory per-session state, already covered by the field-completeness audit (dimension 4, PASS, unaffected).
|
|
- Concurrency strategy: the existing per-round ordering (`sim_reconcile_unaccounted_jobs()` runs before `xce_core_signal_go()` unblocks the picker thread each round) means reconciliation always gets first look at whether a job is already decided from an EARLIER round's live processing before deciding whether to reconstruct it from the recording this round — no race, since the live path's own decision-commit for a job from a prior round has already set `job->accounts` before this round's reconciliation check runs.
|
|
|
|
**Implementation Notes**
|
|
- Integration: no new code path — the reporting behavior already exists (`--sim-verify`); this component's job in the design is to make sure nothing is added that would suppress or override a genuine mismatch.
|
|
- Validation: Scope item 9 (`requirements.md`) — run `--sim-verify` against real recorded sessions and treat any reported mismatch as a finding to root-cause.
|
|
- Risks: a mismatch found this way could indicate either a defect in this feature's own migration, or a genuine pre-existing issue the old always-PEND behavior could never have exposed — both are valid, actionable outcomes, not a reason to suppress the report.
|
|
|
|
#### Sim-Mode Reconciliation Narrowing
|
|
|
|
| Field | Detail |
|
|
|-------|--------|
|
|
| Intent | `sim_reconcile_unaccounted_jobs()` keeps existing, narrowed to the async timing-gap safety net |
|
|
| Requirements | 3.4 |
|
|
|
|
**Responsibilities & Constraints**
|
|
- No structural change to this function's existing logic — the narrowing is a consequence of the live path now handling most jobs, not a new code path inside this function.
|
|
- One addition: call the core-safe global-limit-election entry point (`limit_job_election_impl`, via the now-core-safe `plugin_job_election`) for a job this function reconstructs, so a job whose outcome comes from the recording still gets its global-limit vres registered (idempotency guard already prevents double-registration if the live path also happened to touch it).
|
|
|
|
**Dependencies**
|
|
- Inbound: `sched_sim_driver_run()`'s per-round sequence, unchanged position (before `sched_account_jobs()`) (P0)
|
|
- Outbound: Plugin Hook Dispatchers (new, for the limit-election call) (P1)
|
|
|
|
**Contracts**: Batch [x]
|
|
|
|
##### Batch / Job Contract
|
|
- Trigger: once per round, unconditionally, for every job in `core->jobs` with `accounts == NULL` and not terminal.
|
|
- Input / validation: the round's decoded `ReplaySession.decisions` array (unchanged).
|
|
- Output / destination: `job->allocs`/`accounts`/`state` set directly from the recording; decision reported via the existing `sched_record_session_add_decision()`/`sim_print_decision_gate()`/`xce_repl_check_decision()` triple (unchanged).
|
|
- Idempotency & recovery: `job->accounts != NULL` permanently excludes the job from reconsideration (unchanged, existing guard).
|
|
|
|
**Implementation Notes**
|
|
- Integration: add one call (`plugin_job_election(&(sim->core), job)` — global-limit matching only, since host-scoped matching needs a resolved worker context this function doesn't independently construct) in the FOUND branch, before or alongside the existing account-reconstruction logic.
|
|
- Validation: `test_sim_reconcile_unaccounted.c`'s existing fixtures, extended to confirm a reconciled job's global limit is also registered.
|
|
- Risks: none beyond the standard ripple risk; this function's own scope and trigger condition are unchanged.
|
|
|
|
## Error Handling
|
|
|
|
### Error Strategy
|
|
No new error categories. Every migrated function preserves its existing error/return-code contract (`-EINVAL`, `-ENOENT`, etc.) — this is a signature migration, not a behavior change, so existing error-handling call sites (`ERROR(...)`, `WARN(...)`, fail-return-NULL patterns) are preserved verbatim.
|
|
|
|
### Error Categories and Responses
|
|
- **Picker resolution failure at thread start** (both drivers): existing fail-return-NULL-and-log behavior, unchanged (`sched_resolve_job_picker` failing in `sched_job_picker()`/`sched_allocator()`).
|
|
- **`unsatisfiable_lookup_cb`/`unsatisfiable_populate_cb` NULL**: not an error — the documented, correct fallback (always compute fresh).
|
|
- **Live/recorded decision divergence**: not an error in the traditional sense — reported via the existing `--sim-verify` mismatch mechanism (a diagnostic finding, not a crash or an aborted session).
|
|
|
|
### Monitoring
|
|
No new logging infrastructure. Existing `TRACE`/`WARN`/`ERROR` call sites in the migrated functions are preserved; `sim_log_mismatch()` (unchanged) remains the mechanism for surfacing a live/recorded divergence.
|
|
|
|
## Testing Strategy
|
|
|
|
- **Unit Tests**:
|
|
1. Each migrated function (`sched_meet_host_selection`, `sched_check_host_satisfy`, `sched_try_alloc_on_host_`, `sched_commit_decision`, `plugin_job_election`, `plugin_prepare_allocation`, `plugin_commit_decision`) called directly with an `xce_sched_core_t*` from EITHER an `xce_scheduler_t.core` or `xce_sched_sim_driver_t.core` fixture, confirming identical results.
|
|
2. `unsatisfiable_lookup_cb`/`unsatisfiable_populate_cb` NULL fallback: confirm `sched_make_primary_alloc_()` always computes fresh with no crash and no incorrect skip.
|
|
3. `limit_job_election_impl`/`limit_prepare_allocation_impl` called against a sim-registered (`plugin_register_core()`) plugin instance, confirming a match registers correctly even though `plugin->sched` is NULL.
|
|
- **Integration Tests**:
|
|
1. A real `xce_sched_sim_driver_t` driven through `sched_sim_driver_run()` against a replay whose original session had global AND host-scoped limits configured and actively enforced — `info limit` shows real usage for both (Requirement 2.1-2.4, the concrete bug-1826 acceptance test).
|
|
2. A resource-contention fixture (multiple jobs, insufficient capacity, including limit-constrained capacity) driven through both drivers, confirming identical admission/pend outcomes (Requirement 3.2, per `scheduler-driver-equivalence.md`'s testing discipline).
|
|
3. `sim_reconcile_unaccounted_jobs()` and the live picker/allocator both active in the same session, confirming no double-processing of a single job (Requirement 3.4).
|
|
- **E2E / Real-Recording Tests** (Scope item 9):
|
|
1. `--sim-verify` run against a range of real recorded sessions (not only crafted fixtures), treating any reported mismatch as a finding to root-cause.
|
|
2. `test_repl_equivalence.c`, `test_sim_contention_parity.c`, `test_sim_equivalence.c` re-run in full, with any assertion change investigated and explained, not silently updated to match new output.
|
|
- **Performance**: none targeted (Non-Goals) — no throughput/latency test required for this feature.
|