myxce/.kiro/specs/scheduler-sim-live-allocation/research.md

# Research & Design Decisions

## Summary
- **Feature**: `scheduler-sim-live-allocation`
- **Discovery Scope**: Extension (existing system) — light discovery per `design-discovery-light.md`, since the integration points, existing patterns, and constraints were already exhaustively traced against live source during requirements development (see `requirements.md`'s Project Description for the full evidence trail; this document focuses on decisions, alternatives, and risk, not re-deriving what's already cited there).
- **Key Findings**:
  - Six functions (`plugin_begin_session`, `plugin_end_session`, `sched_resolve_job_picker`, `sched_get_job`, `sched_account_jobs`, `sched_commit_job_alloc`) already completed the exact `xce_scheduler_t*` → `xce_sched_core_t*` migration this spec needs to repeat for the remaining allocation chain — a proven, six-times-repeated pattern, not a new one.
  - Every function in the host-scoped allocation chain (`sched_meet_host_selection`, `sched_check_shared_satisfy`, `sched_check_host_satisfy`, `plugin_prepare_allocation`, `sched_try_alloc_on_host_`, `sched_commit_decision`, `plugin_commit_decision`) only ever touches `sched->core.*` or worker/ctx fields — zero genuine `xce_scheduler_t`-specific dependency, confirmed by reading every function body, not inferred from signatures.
  - The one genuine exception, `xce_scheduler_t.unsatisfiable_cache`, is a pure performance optimization with a documented prior memory-corruption incident (scheduler-allocation task 5.2 round 1) from trying to shortcut it via a struct-prefix cast — the fix pattern (a driver-registered callback, `cache_clear_cb`) is proven and must be repeated, not re-invented.
  - `sched_allocator()`'s full body (verified, not assumed) shows every single call it makes becomes core-safe once the above migration lands — the function itself needs only its top cast changed (`xce_scheduler_t* sched` → `xce_sched_core_t* core`).
  - Sim mode's thread-spawn point is `sched_sim_driver_run()` (`sim_loop.c`), NOT `sched_sim_driver_start()` (`sim.c`) — the latter only does core init and snapshot/replay-file setup; the `pthread_create(&picker_tid, NULL, sim_picker_thread_fn, &(sim->core))` call lives inside the former.

## Research Log

### Is the allocation chain's `xce_scheduler_t*` coupling mechanical or genuine?
- **Context**: The spec's central bet is that sim mode can reuse normal mode's allocation code verbatim. This only holds if the coupling is superficial (never-migrated signature) rather than load-bearing (genuinely needs normal-mode-only state).
- **Sources Consulted**: Direct reads of `resource.c` (`sched_meet_host_selection:221`, `sched_check_shared_satisfy:305`/`sched_check_host_satisfy:378`/`sched_try_alloc_on_host_:497`), `plugin.c` (`plugin_prepare_allocation:591`, `plugin_commit_decision:442`, `plugin_job_election:554`), `scheduler.c` (`sched_commit_decision:1607`, `sched_allocator:429`), `schedobj.c` (`sched_ctx_get_uncomit_host:522`, already core-safe).
- **Findings**: Every one of the 8 functions in the allocation chain reads/writes only `sched->core.*` fields (or worker/ctx fields passed in separately) — no function reads `sched->apictx`, `sched->cluster_cfg`, `sched->unsatisfiable_cache`, or any other genuinely `xce_scheduler_t`-only field, with the single exception noted below.
- **Implications**: The migration is safe to do mechanically, function by function, following the exact precedent of the six already-migrated functions. No redesign of the allocation algorithm itself is needed.

### Is there a genuinely `xce_scheduler_t`-only dependency anywhere in the chain?
- **Context**: A prior investigation (scheduler-allocation spec, task 5.2) hit exactly this problem once already, for a different field.
- **Sources Consulted**: `scheduler-allocation`'s own Implementation Notes (task 5.1, 5.2), `core.h`'s `xce_core_cache_clear_cb_t` doc comment.
- **Findings**: `xce_scheduler_t.unsatisfiable_cache` (a fingerprint → pend-reason-category cache, populated when `sched_make_primary_alloc_()` exhausts all candidates for a job, consulted before attempting allocation again for an identically-shaped job) has no sim-mode equivalent field, by deliberate design (Req 5 AC 17 simulation-isolation boundary from `scheduler-core`). A first attempt to bridge this via `__attribute__((weak))` + a cast of the shared `xce_sched_core_t core` prefix back to `xce_scheduler_t*` silently corrupted `xce_sched_sim_driver_t.alt_events_file` (same byte offset as the cache field) — invisible to ASAN and the whole test suite until caught by review.
- **Implications**: This one field needs a callback seam, not a cast, mirroring `cache_clear_cb` exactly. Consult/populate calls must go through a NULL-checked function-pointer pair the normal-mode driver registers and sim mode leaves NULL. A NULL seam means "always take the slow path" (recompute fully), which is correct behavior, not degraded correctness — see Design Decision below.

### Where does sim mode actually spawn its picker/allocator thread?
- **Context**: Needed to know precisely which function's thread-spawn code changes.
- **Sources Consulted**: `sim.c`'s `sched_sim_driver_start()` (full body read) and `sim_loop.c`'s `sched_sim_driver_run()` (partial body read, confirmed `pthread_create` call site at line ~974).
- **Findings**: `sched_sim_driver_start()` only initializes `sim->core` and opens/validates the replay file and snapshot — it spawns no threads. `sched_sim_driver_run()` is the actual long-running entry point (design.md's own "literal SimDriver entry point" per its header comment) and is where the picker thread is created, once, for the process's sim-session lifetime.
- **Implications**: The unification touches `sched_sim_driver_run()`'s thread-spawn call, not `sched_sim_driver_start()`.

### Open question not fully resolved — flagged as a risk, not asserted either way
- **Context**: `sched_register_plugins()` (`scheduler.c:596`) also calls `register_seqjob_allocator(sched)` / `register_parjob_allocator(sched)`, and `xce_sched_plugin_t` has a `job_alloc_fn` field (`plugin_job_allocate_fn_t`) alongside `job_elect_fn`. This raises the question of whether primary-resreq allocation is ALSO dispatched through a generic plugin-hook indirection (a `plugin_job_allocate()` analogous to `plugin_job_election()`) that would need the same core-safety migration.
- **Findings so far**: The `scheduler-allocation` spec's own Implementation Notes describe `sched_make_primary_alloc_()` as directly branching between `seqjob_allocator.c`/`parjob_allocator.c`'s own `my_alloc()` functions by name (based on `primary->maxTasks`), which reads as a direct function-call dispatch, not a generic plugin-hook indirection — suggesting `job_alloc_fn`/`register_seqjob_allocator`/`register_parjob_allocator` may be a legacy/inactive registration path (parallel to `job_elect_fn`'s own documented legacy-compat status for pickers) rather than something on the live call path.
- **Not confirmed with certainty.** Flagged in Risks below as a design-time verification item for whoever implements task work in this area — if `sched_make_primary_alloc_()` turns out to dispatch through a generic hook after all, that hook needs the identical core-safe treatment as `plugin_job_election`/`plugin_prepare_allocation`/`plugin_commit_decision`.

## Architecture Pattern Evaluation

| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Unify picker/allocator (selected) | Re-type `sched_job_picker()`/`sched_allocator()` to `xce_sched_core_t*`, spawn the identical functions from both drivers | Single implementation, cannot drift (directly satisfies Requirement 5); reuses 6x-proven migration pattern | Sim mode gains a real two-thread `jbuf` hand-off for the first time — needs the 3 named regression tests re-verified | Selected: matches the project's own stated governing principle and its documented drift-prevention rationale |
| Behavioral-equivalence only (rejected) | Keep `sim_picker_thread_fn()` as a separate implementation, extend it to call the same core-safe sub-functions, verify equivalence via tests | Smaller diff, no `jbuf` behavior change in sim mode | Reintroduces exactly the two-implementations-must-independently-stay-in-sync failure mode this project has already hit three times (bug-1611, bug-1613, bug-1681) | Rejected per Requirement 5 and the user's explicit direction during spec discussion |
| Recording-authoritative (rejected) | Keep `sim_reconcile_unaccounted_jobs()` as the general-case decision source; new election/allocation logic only populates side effects (limit vres) for an already-known outcome | Smallest behavior change; no live/recorded divergence handling needed | Contradicts the project's own driver-equivalence invariant (decisions should be independently reproduced, not merely replayed) and doesn't scale to future plugin hooks with other side effects | Rejected during requirements review — see `requirements.md` Requirement 3 AC3-5 |

## Design Decisions

### Decision: Live election/allocation is authoritative; recorded-decision reconciliation narrows to a timing-gap safety net
- **Context**: Once sim mode can independently compute real decisions, two sources exist for a job's outcome: the live picker/allocator, and `sim_reconcile_unaccounted_jobs()`'s trust-the-recording reconstruction.
- **Alternatives Considered**:
  1. Recording stays authoritative always; live path only supplies side effects — rejected (see table above).
  2. Live path authoritative, recording is a fallback only for jobs it hasn't reached in time — selected.
- **Selected Approach**: The live picker/allocator decides for any job it reaches. `sim_reconcile_unaccounted_jobs()`'s existing `job->accounts == NULL` trigger guard already means it only acts on jobs the live path hasn't yet processed — no new guard needed, this narrowing falls out of the existing idempotency check once the live path starts actually producing outcomes.
- **Rationale**: Matches `scheduler-driver-equivalence.md`'s own state-equivalence invariant (decisions should be independently reproduced by the core state machine, not replayed from a trusted log) and the project's explicit governing principle for this spec.
- **Trade-offs**: A live-computed decision that disagrees with the recording is now possible and must be surfaced (Requirement 3 AC5) rather than silently reconciled either way — this is by design: a divergence is a genuine finding (a migration defect, or a previously-hidden issue the old fake-PEND behavior could never have exposed).
- **Follow-up**: Verify via `--sim-verify` against real recorded sessions (Scope item 9 in requirements.md) before shipping.

### Decision: `unsatisfiable_cache` becomes an optional, driver-registered callback pair, not a relocated field
- **Context**: Need the primary-allocation dispatch chokepoint to remain core-safe without giving sim mode a cache it structurally cannot support cleanly today.
- **Alternatives Considered**:
  1. Move the cache itself onto `xce_sched_core_t` so both drivers have it — rejected: sim mode has no natural place to construct/own it (no `sched_fini()`-equivalent teardown exists in sim mode to destroy it), and giving sim mode a working cache is not something any requirement asks for (explicitly a non-goal — see Boundary Context, "achieving simulation mode's allocation-decision throughput or latency parity").
  2. A NULL-safe callback pair (`lookup`/`populate`) on `xce_sched_core_t`, registered only by normal mode's `main.c`, mirroring `cache_clear_cb` — selected.
- **Selected Approach**: `xce_sched_core_t` gains two optional function-pointer fields (`unsatisfiable_lookup_cb`/`unsatisfiable_populate_cb`), each taking `xce_sched_core_t* core` directly — the exact same shape as the existing `cache_clear_cb` field, not a new wrapper struct with an opaque `impl`/user-data pointer (an earlier design draft added one; removed as unnecessary indirection with only one registrant and no foreseeable second). Normal mode's `main.c` registers both, pointing at two small adapter functions in `alloc_cache.c` (alongside the existing `xce_alloc_cache_clear_cb()` adapter) that safely recover `xce_scheduler_t*` via `(xce_scheduler_t*)core` — safe specifically because these adapters are normal-mode-exclusive and only ever invoked with a `core` genuinely embedded in an `xce_scheduler_t`, unlike the round-1 mistake which cast inside code shared by both drivers. Sim mode leaves both NULL. `sched_make_primary_alloc_()` treats NULL as "no cache available — always compute fresh," which is correct (never wrong), not a degraded-correctness mode.
- **Rationale**: Directly reuses the proven pattern and its documented rationale (`core.h`'s `xce_core_cache_clear_cb_t` doc comment); avoids repeating the exact class of memory-corruption bug the earlier cast-based attempt caused.
- **Trade-offs**: Sim mode never gets the cache's performance benefit — accepted, explicitly out of scope (see Boundary Context).

## Risks & Mitigations
- **Decision-timing/ordering shift in sim mode once `jbuf` is genuinely used** — Mitigation: re-run `test_repl_equivalence.c`, `test_sim_contention_parity.c`, `test_sim_equivalence.c` before considering this feature done; treat a broken assertion as a signal to investigate (it may be asserting on an artifact of the old single-fused-thread sequencing), not a reason to revert to two implementations.
- **Unconfirmed `job_alloc_fn`/`register_seqjob_allocator`/`register_parjob_allocator` dispatch path** — Mitigation: verify at implementation time whether `sched_make_primary_alloc_()` dispatches through a generic plugin hook or calls `seqjob`/`parjob` directly; if a hook exists, it needs the same core-safe migration as the other four plugin-hook dispatchers.
- **A live-computed decision may legitimately diverge from a real recording during validation** (Scope item 9) — Mitigation: this is treated as expected and valuable, not a blocker; root-cause every divergence found against real recordings before shipping, per Requirement 3 AC5.
- **`plugin->sched` (bare `void*`, set only by `plugin_register()`) staying NULL for sim-registered plugins** — Mitigation: extend the `limit_seed_config()`/`limit_seed_config_impl()` honest-core-parameter pattern (already proven by bug-1825) to the job-election and prepare-allocation call chains, rather than relying on `plugin->sched`.

## References
- `.kiro/steering/scheduler-driver-equivalence.md` — the governing state-equivalence invariant.
- `.wolf/buglog.json` bug-1611, bug-1613, bug-1681, bug-1825, bug-1826, bug-1827 — the evidence trail of this project's own recurring drift failure mode and its prior fixes.
- `.kiro/specs/scheduler-allocation/` — completed spec whose task 5.2 Implementation Notes document the `unsatisfiable_cache` memory-corruption incident and the `cache_clear_cb` fix pattern being reused here.
- `src/scheduler/CLAUDE.md` — "one core, two consumers" architecture statement.