myxce/.kiro/specs/scheduler-sim-eager-reconciliation/research.md

# Research & Design Decisions

## Summary
- **Feature**: `scheduler-sim-eager-reconciliation`
- **Discovery Scope**: Extension (widens an existing, already-shipped mechanism's trigger condition; no new subsystem, no external dependency)
- **Key Findings**:
  - The fix is a single-condition removal in `sim_reconcile_unaccounted_jobs()`'s trigger gate (`sim_loop.c:592`), not a new cross-round data structure. The per-round `decoded_session` decode/reconcile/free cycle already runs every round regardless of any job's state; the only reason cross-round matching fails today is that the gate additionally requires `JOB_IS_RUN(job->state)` before a job is even considered for a search, so a job still PEND at the round its own DISPATCH decision appears is skipped outright and never re-checked once it later becomes RUN (by then that round's `decoded_session` is long freed).
  - `sched_account_jobs()` (the function that actually turns `job->accounts` into `w->resInUse`/`core->sharedInUse`) already independently gates on `JOB_IS_RUN(job->state)` before charging. This means capturing `job->accounts` early (while a job is still PEND) is inert until the job's real state later transitions to RUN — the existing charging gate is exactly the state-appropriate deferral this feature needs, and it requires no change.
  - One existing test, `test_reconcile_skips_non_run_state_job` (`test_sim_reconcile_unaccounted.c:507-531`), asserts the CURRENT (buggy) behavior by name and message ("a non-RUN-state job must never be reconciled here") — this test encodes the bug and must be replaced (inverted), not left alongside new coverage.

## Research Log

### Where does resInUse actually go to zero, and what re-populates it?
- **Context**: bug-1613's buglog root-cause trail (confirmed) plus `.kiro/steering/scheduler-driver-equivalence.md`.
- **Sources Consulted**: `src/scheduler/src/scheduler.c` (`sched_account_jobs`, `sched_commit_decision`), `src/scheduler/src/resource.c` (`sched_update_res_accounts`, `sched_charge_account`), `.wolf/buglog.json` bug-1611/bug-1613 full history.
- **Findings**: `xce_core_begin_session()` zeroes `resInUse`/`sharedInUse` every round; `sched_account_jobs()` is the only re-summing mechanism, iterating `core->jobs` and charging every job that is both `JOB_IS_RUN` and has `job->accounts != NULL`. Normal mode's `sched_commit_decision()` sets `job->accounts` synchronously at DISPATCH-decision time, independent of `job->state` — so by the time a job's state reaches RUN, its accounts are already long-set. Simulation mode has no equivalent synchronous path (its own live picker never emits a DISPATCH — confirmed in the bug-1613 trail — it is PEND-only), so `job->accounts` for a replayed job can only come from `sim_reconcile_unaccounted_jobs()` reconstructing it from the recorded decision stream.
- **Implications**: the fix belongs entirely in how/when `sim_reconcile_unaccounted_jobs()` captures `job->accounts` — `sched_account_jobs()` itself is correct today and must not change.

### Why does today's reconciliation only work same-round?
- **Context**: bug-1611's confirmed root-cause note and the `scheduler-sim-decision-parity` spec's task 4.3 implementation notes (both quoted in `.wolf/buglog.json`).
- **Sources Consulted**: `src/scheduler/src/sim_loop.c:566-709` (`sim_reconcile_unaccounted_jobs`, read in full), `src/scheduler/src/sim_loop.c:840-960` (round-boundary call sequence, read in full).
- **Findings**: `decoded_session` is decoded fresh from the replay stream once per round (`decode_ReplaySession_ptr`, `sim_loop.c:871`) and freed at the end of the same round (`free_ReplaySession_ptr`, `sim_loop.c:1006`) — it never survives past its own round. `sim_reconcile_unaccounted_jobs()` is called unconditionally every round (`sim_loop.c:948`, between `xce_core_begin_session()` and `sched_account_jobs()`), but its trigger gate (`sim_loop.c:592`) is:
  ```c
  if ((job == NULL) || !JOB_IS_RUN(job->state) || (job->accounts != NULL)) {
      continue;
  }
  ```
  A job whose DISPATCH decision is recorded at round N but whose RUN-state confirmation is only replayed at round N+k (realistic dispatch-to-confirmation latency) is skipped at round N (state is still PEND) and is never re-considered once it reaches RUN at round N+k, because round N's `decoded_session` — the only place that DISPATCH decision ever existed — is already gone.
- **Implications**: no new persistent, cross-round data structure is needed. The function already re-runs every round with a live view of the round's own recorded decisions; it only needs to stop requiring `JOB_IS_RUN` before searching. Removing that one condition means every not-yet-captured job is (re-)checked against every round's own decoded stream until either found (captured, permanently, via the existing `job->accounts != NULL` guard) or the replay ends.

### Does removing the RUN-state gate risk fabricating or double-capturing?
- **Context**: Requirement 3's explicit no-fabrication / no-double-capture acceptance criteria.
- **Sources Consulted**: `sim_loop.c:592-693` (full match/capture logic, read in full).
- **Findings**: The match loop already restricts a capture to the job's own jobid (`entry->body.dispatch->alloc->jobid != job->jobid`, `sim_loop.c:605`) — this is state-independent and unaffected by the gate change. The `job->accounts != NULL` continue-guard (unchanged) already prevents a second capture on any later round, for any job, regardless of state. Widening the state condition therefore only widens *which jobs are eligible to be searched for*, never *what counts as a match* or *whether a second capture can happen*.
- **Implications**: Requirement 3 (no fabrication, no double-capture) is satisfied by logic that already exists and needs no modification — only new test coverage proving it still holds once the gate widens.

### Interaction with `--sim-verify`
- **Context**: Requirement 4.2 (`--sim-verify` must not report a new mismatch caused by this feature).
- **Sources Consulted**: `sim_loop.c:925-1004` (reconciliation call site through the verify positional-walk loop, read in full).
- **Findings**: Reconciliation runs and compacts `decoded_session->decisions` *before* the `--sim-verify` positional walk (`sim_loop.c:970-1004`) reads that same array. Today, a job whose confirmation lands in a later round leaves its DISPATCH entry sitting **unconsumed** in the round-N `decoded_session` (reconciliation skipped it), which `--sim-verify`'s walk would then see as a recorded entry with no matching live decision — a latent false-mismatch risk in `--sim-verify` mode that already exists independent of this fix. Widening the gate causes reconciliation to consume that entry at the correct round instead, which removes this latent risk rather than introducing a new one.
- **Implications**: this fix is verify-neutral-to-positive, not verify-risky. Still needs an explicit test per Requirement 4.2's own wording (a `--sim-verify` run across a multi-round-separated dispatch must not report a new mismatch).

### Does early capture risk an incorrect uncharge when a captured-but-not-yet-RUN job later leaves the job set entirely?
- **Context**: capturing `job->accounts` while a job is still PEND (this feature's whole point) means, for the first time, a job can carry non-NULL `accounts` without ever having been charged. If any cleanup path unconditionally treats "accounts non-NULL" as "this job was charged, so subtract it," early capture would cause `resInUse` to under-count (a phantom decrement for a charge that was never added) for any captured-but-never-actually-run job (e.g. cancelled while still PEND).
- **Sources Consulted**: every production (non-test) reference to `job->accounts`/`sched_uncharge_account` across `src/scheduler/src/*.c`, via grep: `resource.c:591-602` (`sched_uncharge_account`/`sched_commit_job_alloc` definitions), `scheduler.c:964-980` (`sched_job_state_change`), `core_events.c:898-921` (`XCE_CORE_EVENT_JOB_STATE_CHANGE` handler) — the two places a job's `job->accounts` gets freed outside of normal completion.
- **Findings**: `sched_uncharge_account()` has **zero production callers** anywhere in the scheduler (confirmed by grep across every `.c` file in `src/scheduler/src/` — its only callers are test files exercising it in isolation). Both places `job->accounts` gets freed on a non-RUN state transition (`scheduler.c:974-977`, `core_events.c:915-918`) do exactly one thing: `free_job_account(job->accounts); job->accounts = NULL;` — memory release only, no `resInUse` mutation, no call to `sched_uncharge_account`. Production accounting is entirely a **reset-and-resum** model, not incremental charge/uncharge: `xce_core_begin_session()` zeroes `resInUse`/`sharedInUse` every round, and `sched_account_jobs()` re-sums it from scratch by summing every *currently* `JOB_IS_RUN` job with non-NULL `accounts`. A job never counts toward `resInUse` until a round where it is both RUN and has accounts — regardless of how early those accounts were captured.
- **Implications**: a job captured while still PEND that later transitions directly to a terminal state (cancelled, failed) **without ever passing through RUN** is simply never summed by `sched_account_jobs()` at any round (it never satisfies `JOB_IS_RUN`), and its `job->accounts` is freed harmlessly with zero `resInUse` impact when its terminal state is applied. Early capture cannot cause an under-count. This is not a risk this design needs to mitigate — it is a consequence of an accounting model (reset+resum, no incremental uncharge) that was already correct before this feature and remains correct with a wider capture-eligibility gate.

### Does bug-1624's normal-mode fix (job->state = RUN set synchronously at dispatch-commit) already close this gap for current recordings?
- **Context**: `bug-1624` (fixed 2026-08-23, `scheduler-dispatch-commit` tasks 2.1/3.1) made `sched_commit_decision()` set `job->state = JOB_STAT_RUN` synchronously in the same operation as `job->accounts`/`job->allocs`, specifically to close a *live* normal-mode over-commit window. Since this predates the current investigation, it's worth confirming it doesn't also incidentally collapse the round-separation this spec fixes for any recording made after 2026-08-23.
- **Sources Consulted**: `grep -n "job->state\|JOB_STAT_RUN" src/scheduler/src/sim_loop.c` (only hit: the trigger gate itself, `sim_loop.c:592`); `src/scheduler/src/scheduler.c:1441-1447` (`sched_commit_decision`, read in full earlier); `.wolf/buglog.json` bug-1624's full fix note.
- **Findings**: `sim_reconcile_unaccounted_jobs()` never sets `job->state` anywhere — replay-mode state transitions come exclusively from the separately-recorded `XCE_CORE_EVENT_JOB_STATE_CHANGE` event stream, applied via `core_events.c`, independent of whatever the decision stream contains. `sched_commit_decision()`'s `job->state = JOB_STAT_RUN` (bug-1624's fix) is a direct in-memory field write in the *live* scheduler process — not a call into the core's event-push pipeline (`xce_core_push_events()`) — so it is not, by itself, a recordable core event. The recorded `JOB_STATE_CHANGE` event a replay later applies still originates from whatever asynchronous signal produces it for recording purposes (the controller's own worker-confirmation path), independent of bug-1624's live in-memory synchronization.
- **Implications**: bug-1624 fixed a live-mode admission-safety window; it does not change what ends up in a recording's event stream or its round timing. This spec's premise — a recorded DISPATCH decision and its recorded RUN-state-change event can land in different replay rounds — holds for recordings made before *and* after bug-1624, and is not something a future reader should mistake as already resolved by that unrelated fix.

### Existing test locking in the bug
- **Context**: identifying what existing coverage this feature must invalidate, not just extend.
- **Sources Consulted**: `src/scheduler/testsuite/test_sim_reconcile_unaccounted.c` (grepped test names, read `test_reconcile_skips_non_run_state_job` in full at lines 507-531).
- **Findings**: `test_reconcile_skips_non_run_state_job` adds a `JOB_STAT_PEND` job with a matching recorded DISPATCH entry and asserts `job->accounts` stays `NULL`, with the message "a non-RUN-state job must never be reconciled here, regardless of what recorded data exists for it this round." This is the literal bug this spec fixes, encoded as a passing test.
- **Implications**: this test must be replaced (inverted expectation, renamed), not left in place alongside new coverage — the design's File Structure Plan and the eventual task list must call this out explicitly, or the fix will be blocked by its own existing test suite.

## Architecture Pattern Evaluation

| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Widen existing gate (chosen) | Remove `!JOB_IS_RUN(job->state)` from `sim_reconcile_unaccounted_jobs()`'s trigger condition; rely on the function's existing every-round re-invocation and the existing `job->accounts != NULL` guard for persistence/idempotency | Minimal surface area (one condition), reuses every existing invariant (dedup, ownership transfer, decision reporting) untouched, symmetric with normal mode's own state-independent capture | Every not-yet-captured job is now re-searched against every round's decoded stream until found — O(jobs × rounds) worst case for a job with no recorded origin at all (Req 4.4 case) | Chosen: smallest design satisfying all requirements (design-synthesis Simplification lens) |
| New persistent cross-round pending-capture table | Maintain a separate `jobid -> pending decision` map populated across rounds, independent of `sim->core.jobs` iteration | Could in principle capture before a job even exists in `core->jobs` | Solves a problem that does not exist here — `decoded_session` already only ever contains decisions for jobs `sim->core.jobs` already knows about at that round; adds a new owned lifetime/lookup structure and a new invalidation surface for no behavioral gain | Rejected — violates the Simplification lens; not pursued further |

## Design Decisions

### Decision: Remove the `JOB_IS_RUN` pre-filter instead of adding a new capture-tracking structure
- **Context**: bug-1611/1613's residual gap is that reconciliation only matches within the same round a job happens to already be RUN.
- **Alternatives Considered**:
  1. New persistent `jobid -> captured allocation` structure spanning rounds, decoupled entirely from `sim->core.jobs` iteration.
  2. Widen the existing trigger gate on the existing per-round iteration (chosen).
- **Selected Approach**: Delete `!JOB_IS_RUN(job->state)` from the trigger condition at `sim_loop.c:592`; leave every other line of the function (match search, ownership transfer, decision reporting, compaction) unchanged.
- **Rationale**: The function already iterates `sim->core.jobs` every round and already searches that round's own decoded stream; the only defect is an overly-narrow precondition on which jobs are eligible. `sched_account_jobs()`'s own unchanged RUN-state gate already provides the correct "don't charge until actually running" deferral this feature's Requirement 1.2 needs — building a second deferral mechanism would duplicate an already-correct one.
- **Trade-offs**: A job that never has a recorded origin anywhere in the replay (Requirement 4.4) is now re-searched, and its "not found" TRACE-level log line re-emitted, every round for the rest of the replay instead of never being searched again after one failed same-round attempt. Accepted: TRACE-level (filtered by default log level in normal operation) and structurally required by Requirement 4.4's "shall continue processing... without terminating" wording, which implies retrying, not giving up.
- **Follow-up**: Verify during implementation that no other caller or test assumes today's "same-round-only" contract as load-bearing behavior beyond the one test identified above.

## Risks & Mitigations
- **Existing test asserts the bug as correct behavior** (`test_reconcile_skips_non_run_state_job`) — Mitigation: replace it with an inverted-expectation test in the same task that changes the gate, not as a follow-up; do not let old and new expectations coexist even briefly.
- **Resource-contention blind spot** (`scheduler-driver-equivalence.md`'s own documented testing discipline: a no-contention fixture cannot distinguish "accounting correct" from "accounting wrong but unread") — Mitigation: Testing Strategy must include at least one fixture where a worker's capacity is insufficient for all candidate jobs, so a wrongly-zero `resInUse` would visibly change a scheduling decision, not just a REPL display value.
- **`--sim-verify` false-mismatch risk already latent today** — Mitigation: add an explicit `--sim-verify` regression test spanning a multi-round dispatch-to-confirmation gap; confirm it reports zero mismatches after the fix (and, ideally, confirm it currently free of new mismatches — the fix is expected to strictly improve this, not just avoid regressing it).
- **Log noise from repeated "not found" TRACE lines** across many rounds for a job with no recorded origin — Mitigation: accepted as documented trade-off (see Design Decision above); TRACE level already filtered in default operation, not escalated further by this spec.
- **Wider per-round search cost**: every not-yet-captured job (previously only RUN-state ones) is now linearly searched against that round's decoded decision list every round until captured or the replay ends — a round with a large PEND backlog does more search work than before. Mitigation: the per-job, per-round search cost itself is unchanged (same linear scan the existing mechanism already performs); only the set of jobs subjected to it widens. Replay is a debugging/tuning tool, not a real-time or production hot path, and `--sim-speed` already exists to control replay pacing — accepted without further mitigation; revisit only if a real replay is observed to become impractically slow.

## References
- `.kiro/steering/scheduler-driver-equivalence.md` — the enforced state-equivalence invariant and its testing discipline this feature closes the remaining gap in.
- `.kiro/specs/scheduler-sim-decision-parity/` — the parent spec whose `sim_reconcile_unaccounted_jobs()` mechanism this feature extends (implementation-complete, not reopened).
- `.wolf/buglog.json` bug-1611, bug-1613 — full confirmed root-cause and partial-fix history.