myxce/.kiro/specs/scheduler-sim-decision-parity/research.md

Research & Design Decisions

Summary

• Feature: scheduler-sim-decision-parity
• Discovery Scope: Extension (existing system) — light discovery
• Key Findings:
  • sched_check_host_satisfy_()/sched_check_shared_satisfy_() (resource.c) read w->resInUse/core->sharedInUse directly as inputs to the live allocator's own capacity check — bug-1613's accounting gap is confirmed to affect simulation-mode scheduling decisions under contention, not just displayed data. This resolves the open question from the brief and raises this spec's priority.
  • Every decision the normal-mode scheduler makes (including DISPATCH with its full xce_job_alloc_t payload) is written into the .replay stream via sched_record_session_add_decision() (xce_sched_driver_run_normal_mode_round(), driver.c). sim_loop.c already decodes this per round (decoded_session->decisions) but only reads it under sim->verify, purely for comparison.
  • job->accounts == NULL is already the exact signal sched_account_jobs() uses to skip a job it can't re-charge — the same signal cleanly identifies "this RUN-state job has never been reconstructed" and, once set, naturally prevents double-application from a later pass (idempotent by construction, no new bookkeeping field needed).
  • sim_verify_job_alloc_eq()/sim_verify_host_alloc_eq()/sim_verify_alloc_eq() (sim_verify.c) already implement field-by-field equality for exactly the xce_job_alloc_t/xce_host_alloc_t/xce_alloc_t payload shapes this feature needs to reconstruct job->allocs from — reusable, not something to reinvent.
  • The actual round-level comparison driver in sim_verify.c (as opposed to the eq-helpers read this session) was not read — how --sim-verify currently handles a recorded decision with no live-drained counterpart (which is exactly the case this feature changes the meaning of) needs to be confirmed at implementation time, flagged as a risk below.

Research Log

Does resInUse feed actual scheduling decisions, or only display data?

• Context: The brief flagged this as unconfirmed and consequential — determines whether bug-1613 is a display bug or a decision-correctness bug.
• Sources Consulted: resource.c (sched_check_host_satisfy_, sched_check_shared_satisfy_, sched_try_alloc_on_host_).
• Findings: sched_check_host_satisfy_() reads w->resvals (total) and w->resInUse (in-use) and passes both to unsatisfy_(), whose result determines whether sched_try_alloc_on_host_() (the live allocator's per-host evaluation, called from the picker/allocator's job-election path) treats a host as viable. sched_check_shared_satisfy_() does the identical thing against core->sharedInUse for shared resources.
• Implications: Confirmed, not hypothetical. Requirement 2 exists because this mechanism is real: with resInUse permanently zero, simulation-mode capacity checks can systematically pass where normal mode would fail, meaning simulated scheduling under contention can diverge from what actually happened.

Where does the recorded decision stream carry allocation data, and what does the normal-mode driver already do with decisions?

• Context: Confirming decoded_session->decisions is a complete, reusable data source rather than a partial one.
• Sources Consulted: driver.c (xce_sched_driver_run_normal_mode_round's decision-drain loop), sim_verify.c (equality helpers).
• Findings: Every decision drained from sched->core.decisions in a normal-mode round — before being handed to sched->core.on_decision — is recorded via sched_record_session_add_decision() if recording is active, with its full payload (xce_job_alloc_t for DISPATCH, including per-host and shared-resource allocation lists). sim_verify.c already has mature, tested field-by-field equality functions for this exact payload shape.
• Implications: decoded_session->decisions is a complete historical record of every decision (not just PEND, which is the only op the live sim-mode picker currently re-derives for already-resolved jobs). Reconstructing job->allocs from a recorded DispatchDecision is a straightforward field copy, not a reverse-engineering exercise — and the equality-checking precedent in sim_verify.c shows the exact shape to build against.

Architecture Pattern Evaluation

Option	Description	Strengths	Risks / Limitations	Notes
Reconstruct from recorded decisions, keyed on job->accounts == NULL (chosen)	At the round-boundary accounting point, for every RUN-state job with no accounts, look up and apply its recorded decision from this round's decoded_session->decisions	Reuses data already decoded every round; reuses sim_verify.c's existing payload-equality precedent for the reconstruction shape; the NULL-accounts check is naturally idempotent (no new dedup bookkeeping)	Requires care around --sim-verify's existing comparison semantics (see Risks)	Directly extends the point Requirement 1's accounting fix already establishes — one round-boundary pass, not a second parallel mechanism
Reconstruct from JOB_UPDATE's wire allocs field	Extend core_events.c's XCE_CORE_EVENT_JOB_UPDATE handler to check updated->allocs != NULL, mirroring core_build_job()'s existing JOBS_PUSH pattern	Simpler, localized to one file	Rejected as primary mechanism — confirmed this session that allocation data does not travel via JOB_UPDATE events at all; it travels via the recorded decision stream exclusively (see Research Log). This approach has no data to act on for the dominant case.	Not viable as the primary fix; not pursued further
Re-run the live picker/allocator against every job every round, discard non-matching results	Force full re-derivation regardless of recording	Would not depend on recorded-decision structure at all	Breaks the entire premise of replay (a deterministic capture-and-replay, not a live re-simulation) and risks non-determinism/performance cost re-running allocation logic that already ran once	Rejected — contradicts scheduler-core's own design intent for what "replay" means

Design Decisions

Decision: Key reconstruction and dedup on job->accounts == NULL, not a separate tracking set

• Context: Need to (a) know which RUN-state jobs need reconstruction from a recorded decision, and (b) avoid double-applying when a job also gets a fresh live decision in the same round.
• Alternatives Considered: (1) a separate per-job "already reconciled" flag/set; (2) reuse the existing job->accounts == NULL check sched_account_jobs() already performs.
• Selected Approach: (2).
• Rationale: job->accounts is set exactly once, at the moment a job's allocation becomes known — either via a fresh sched_commit_decision() (live path, unchanged by this feature) or via this feature's reconstruction pass. Once set, both sched_account_jobs()'s re-charge and this feature's own reconstruction pass naturally skip the job on every subsequent round — no new state, no drift risk between two independently-maintained bookkeeping mechanisms.
• Trade-offs: None significant — this is a strict simplification over introducing new state.
• Follow-up: none.

Decision: Feed a reconstructed decision through the same reporting path as a fresh one

• Context: Requirement 3 needs decision visibility, not just accounting correctness — an operator must see the decision, not only have resInUse quietly become correct.
• Alternatives Considered: (1) only fix accounting, leave decision visibility as a separate/future concern; (2) synthesize a decision object from the recorded data and route it through the exact same callback (on_decision/xce_repl_check_decision) a freshly-drained decision uses.
• Selected Approach: (2).
• Rationale: Requirement 3.1 explicitly requires "the same decision-reporting channel used for a freshly-computed decision" — a parallel/different reporting path would be a second thing to keep in sync, exactly the class of bug this whole spec exists to close.
• Trade-offs: The reconstructed decision must be a genuine xce_decision_t (or converted to one), not a display-only string — slightly more construction work than a printf, but necessary for it to be indistinguishable from a live decision to any consumer (REPL, scheduler-sim-repl-verbosity's round rollup, etc.).
• Follow-up: none.

Risks & Mitigations

• --sim-verify interaction — RESOLVED during design validation (see design.md's Recorded-Decision Reconciliation Implementation Notes for the full mechanism). The round-level comparison driver (sim_loop.c's while (cbuf_get(&(sim->core.decisions), &d) ...) loop) walks decoded_session->decisions positionally via an index di. Confirmed: a job this feature reconciles never produces a live-drained counterpart, so the reconciliation step must remove (properly freed, not just index-dropped — see design.md's memory-ownership note) each consumed recorded decision from decoded_session->decisions before the verify loop runs, keeping the remaining entries positionally aligned by shifting down (not swap-with-last, which would reorder them). sim_verify.c's comparison predicates need no change.
• Snapshot-restored jobs already running at replay start — RESOLVED: confirmed by reading snapshot_load.c that xce_sched_core_load_snapshot() restores job->state but never job->accounts/job->allocs. Such a job is therefore always job->accounts == NULL with no in-window recorded decision — unambiguously Requirement 4's graceful-degradation case, no separate handling needed.
• Relocating sched_account_jobs()'s signature touches a function with production callers — mitigated by the fact the relocation is signature-only (body unchanged, per the sched_resolve_job_picker() precedent already proven safe for the identical pattern) and both call sites (driver.c, sim_loop.c) are updated in the same change. Independently re-confirmed during design validation: grep "sched->" resource.c | grep -v "sched->core" returns zero matches across the entire file — every one of the six relocated functions' bodies (not just sched_account_jobs() itself) genuinely touches only sched->core.
• New risk surfaced while resolving the --sim-verify risk above: naively freeing a consumed DISPATCH entry to avoid the leak would double-free, because the live path (scheduler.c:1440, job->allocs = decision->u.alloc;) takes ownership of alloc by pointer move, not copy — reconciliation reconstructing job->allocs the same way would alias memory about to be freed. Mitigation: null out the source entry's alloc field immediately after the takeover assignment, before freeing the rest of the entry (every _ptr-style free helper in this codebase already no-ops on NULL). See design.md for the full writeup.

References

• .wolf/buglog.json bug-1611, bug-1613 — the originating findings.
• .kiro/steering/scheduler-driver-equivalence.md — the invariant this spec restores, including its testing discipline (resource-contention fixtures required).
• .kiro/specs/scheduler-sim-repl-verbosity/ — sibling spec; its round-rollup dispatch counts benefit from this fix once implemented but do not depend on it.