myxce/.kiro/specs/scheduler-sim-decision-parity/research.md

# Research & Design Decisions

## Summary
- **Feature**: `scheduler-sim-decision-parity`
- **Discovery Scope**: Extension (existing system) — light discovery
- **Key Findings**:
  - `sched_check_host_satisfy_()`/`sched_check_shared_satisfy_()` (resource.c) read `w->resInUse`/`core->sharedInUse` directly as inputs to the live allocator's own capacity check — bug-1613's accounting gap is **confirmed** to affect simulation-mode scheduling decisions under contention, not just displayed data. This resolves the open question from the brief and raises this spec's priority.
  - Every decision the normal-mode scheduler makes (including `DISPATCH` with its full `xce_job_alloc_t` payload) is written into the `.replay` stream via `sched_record_session_add_decision()` (`xce_sched_driver_run_normal_mode_round()`, driver.c). `sim_loop.c` already decodes this per round (`decoded_session->decisions`) but only reads it under `sim->verify`, purely for comparison.
  - `job->accounts == NULL` is already the exact signal `sched_account_jobs()` uses to skip a job it can't re-charge — the same signal cleanly identifies "this RUN-state job has never been reconstructed" and, once set, naturally prevents double-application from a later pass (idempotent by construction, no new bookkeeping field needed).
  - `sim_verify_job_alloc_eq()`/`sim_verify_host_alloc_eq()`/`sim_verify_alloc_eq()` (sim_verify.c) already implement field-by-field equality for exactly the `xce_job_alloc_t`/`xce_host_alloc_t`/`xce_alloc_t` payload shapes this feature needs to reconstruct `job->allocs` from — reusable, not something to reinvent.
  - The actual round-level comparison driver in sim_verify.c (as opposed to the eq-helpers read this session) was not read — how `--sim-verify` currently handles a recorded decision with no live-drained counterpart (which is exactly the case this feature changes the meaning of) needs to be confirmed at implementation time, flagged as a risk below.

## Research Log

### Does resInUse feed actual scheduling decisions, or only display data?
- **Context**: The brief flagged this as unconfirmed and consequential — determines whether bug-1613 is a display bug or a decision-correctness bug.
- **Sources Consulted**: `resource.c` (`sched_check_host_satisfy_`, `sched_check_shared_satisfy_`, `sched_try_alloc_on_host_`).
- **Findings**: `sched_check_host_satisfy_()` reads `w->resvals` (total) and `w->resInUse` (in-use) and passes both to `unsatisfy_()`, whose result determines whether `sched_try_alloc_on_host_()` (the live allocator's per-host evaluation, called from the picker/allocator's job-election path) treats a host as viable. `sched_check_shared_satisfy_()` does the identical thing against `core->sharedInUse` for shared resources.
- **Implications**: Confirmed, not hypothetical. Requirement 2 exists because this mechanism is real: with `resInUse` permanently zero, simulation-mode capacity checks can systematically pass where normal mode would fail, meaning simulated scheduling under contention can diverge from what actually happened.

### Where does the recorded decision stream carry allocation data, and what does the normal-mode driver already do with decisions?
- **Context**: Confirming decoded_session->decisions is a complete, reusable data source rather than a partial one.
- **Sources Consulted**: `driver.c` (`xce_sched_driver_run_normal_mode_round`'s decision-drain loop), `sim_verify.c` (equality helpers).
- **Findings**: Every decision drained from `sched->core.decisions` in a normal-mode round — before being handed to `sched->core.on_decision` — is recorded via `sched_record_session_add_decision()` if recording is active, with its full payload (`xce_job_alloc_t` for DISPATCH, including per-host and shared-resource allocation lists). `sim_verify.c` already has mature, tested field-by-field equality functions for this exact payload shape.
- **Implications**: `decoded_session->decisions` is a complete historical record of every decision (not just PEND, which is the only op the live sim-mode picker currently re-derives for already-resolved jobs). Reconstructing `job->allocs` from a recorded `DispatchDecision` is a straightforward field copy, not a reverse-engineering exercise — and the equality-checking precedent in `sim_verify.c` shows the exact shape to build against.

## Architecture Pattern Evaluation

| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Reconstruct from recorded decisions, keyed on `job->accounts == NULL` (chosen) | At the round-boundary accounting point, for every RUN-state job with no accounts, look up and apply its recorded decision from this round's `decoded_session->decisions` | Reuses data already decoded every round; reuses `sim_verify.c`'s existing payload-equality precedent for the reconstruction shape; the NULL-accounts check is naturally idempotent (no new dedup bookkeeping) | Requires care around `--sim-verify`'s existing comparison semantics (see Risks) | Directly extends the point Requirement 1's accounting fix already establishes — one round-boundary pass, not a second parallel mechanism |
| Reconstruct from `JOB_UPDATE`'s wire `allocs` field | Extend `core_events.c`'s `XCE_CORE_EVENT_JOB_UPDATE` handler to check `updated->allocs != NULL`, mirroring `core_build_job()`'s existing JOBS_PUSH pattern | Simpler, localized to one file | **Rejected as primary mechanism** — confirmed this session that allocation data does not travel via `JOB_UPDATE` events at all; it travels via the recorded decision stream exclusively (see Research Log). This approach has no data to act on for the dominant case. | Not viable as the primary fix; not pursued further |
| Re-run the live picker/allocator against every job every round, discard non-matching results | Force full re-derivation regardless of recording | Would not depend on recorded-decision structure at all | Breaks the entire premise of replay (a deterministic capture-and-replay, not a live re-simulation) and risks non-determinism/performance cost re-running allocation logic that already ran once | Rejected — contradicts `scheduler-core`'s own design intent for what "replay" means |

## Design Decisions

### Decision: Key reconstruction and dedup on `job->accounts == NULL`, not a separate tracking set
- **Context**: Need to (a) know which RUN-state jobs need reconstruction from a recorded decision, and (b) avoid double-applying when a job also gets a fresh live decision in the same round.
- **Alternatives Considered**: (1) a separate per-job "already reconciled" flag/set; (2) reuse the existing `job->accounts == NULL` check `sched_account_jobs()` already performs.
- **Selected Approach**: (2).
- **Rationale**: `job->accounts` is set exactly once, at the moment a job's allocation becomes known — either via a fresh `sched_commit_decision()` (live path, unchanged by this feature) or via this feature's reconstruction pass. Once set, both `sched_account_jobs()`'s re-charge and this feature's own reconstruction pass naturally skip the job on every subsequent round — no new state, no drift risk between two independently-maintained bookkeeping mechanisms.
- **Trade-offs**: None significant — this is a strict simplification over introducing new state.
- **Follow-up**: none.

### Decision: Feed a reconstructed decision through the same reporting path as a fresh one
- **Context**: Requirement 3 needs decision *visibility*, not just accounting correctness — an operator must see the decision, not only have `resInUse` quietly become correct.
- **Alternatives Considered**: (1) only fix accounting, leave decision visibility as a separate/future concern; (2) synthesize a decision object from the recorded data and route it through the exact same callback (`on_decision`/`xce_repl_check_decision`) a freshly-drained decision uses.
- **Selected Approach**: (2).
- **Rationale**: Requirement 3.1 explicitly requires "the same decision-reporting channel used for a freshly-computed decision" — a parallel/different reporting path would be a second thing to keep in sync, exactly the class of bug this whole spec exists to close.
- **Trade-offs**: The reconstructed decision must be a genuine `xce_decision_t` (or converted to one), not a display-only string — slightly more construction work than a printf, but necessary for it to be indistinguishable from a live decision to any consumer (REPL, `scheduler-sim-repl-verbosity`'s round rollup, etc.).
- **Follow-up**: none.

## Risks & Mitigations
- **`--sim-verify` interaction — RESOLVED during design validation** (see `design.md`'s Recorded-Decision Reconciliation Implementation Notes for the full mechanism). The round-level comparison driver (`sim_loop.c`'s `while (cbuf_get(&(sim->core.decisions), &d) ...)` loop) walks `decoded_session->decisions` positionally via an index `di`. Confirmed: a job this feature reconciles never produces a live-drained counterpart, so the reconciliation step must remove (properly freed, not just index-dropped — see design.md's memory-ownership note) each consumed recorded decision from `decoded_session->decisions` before the verify loop runs, keeping the remaining entries positionally aligned by shifting down (not swap-with-last, which would reorder them). `sim_verify.c`'s comparison predicates need no change.
- **Snapshot-restored jobs already running at replay start — RESOLVED**: confirmed by reading `snapshot_load.c` that `xce_sched_core_load_snapshot()` restores `job->state` but never `job->accounts`/`job->allocs`. Such a job is therefore always `job->accounts == NULL` with no in-window recorded decision — unambiguously Requirement 4's graceful-degradation case, no separate handling needed.
- **Relocating `sched_account_jobs()`'s signature touches a function with production callers** — mitigated by the fact the relocation is signature-only (body unchanged, per the `sched_resolve_job_picker()` precedent already proven safe for the identical pattern) and both call sites (`driver.c`, `sim_loop.c`) are updated in the same change. Independently re-confirmed during design validation: `grep "sched->" resource.c | grep -v "sched->core"` returns zero matches across the entire file — every one of the six relocated functions' bodies (not just `sched_account_jobs()` itself) genuinely touches only `sched->core`.
- **New risk surfaced while resolving the `--sim-verify` risk above**: naively freeing a consumed DISPATCH entry to avoid the leak would double-free, because the live path (`scheduler.c:1440`, `job->allocs = decision->u.alloc;`) takes ownership of `alloc` by pointer move, not copy — reconciliation reconstructing `job->allocs` the same way would alias memory about to be freed. Mitigation: null out the source entry's `alloc` field immediately after the takeover assignment, before freeing the rest of the entry (every `_ptr`-style free helper in this codebase already no-ops on `NULL`). See design.md for the full writeup.

## References
- `.wolf/buglog.json` bug-1611, bug-1613 — the originating findings.
- `.kiro/steering/scheduler-driver-equivalence.md` — the invariant this spec restores, including its testing discipline (resource-contention fixtures required).
- `.kiro/specs/scheduler-sim-repl-verbosity/` — sibling spec; its round-rollup dispatch counts benefit from this fix once implemented but do not depend on it.