myxce/.kiro/specs/scheduler-sim-eager-reconciliation/design.md

# Design Document

## Overview

**Purpose**: This feature closes the residual gap in `scheduler-sim-decision-parity`'s Recorded-Decision Reconciliation mechanism (`sim_reconcile_unaccounted_jobs()`, `sim_loop.c`) so a simulation-mode replay correctly captures a job's resource allocation regardless of how many rounds separate its recorded DISPATCH decision from its RUN-state confirmation — the realistic case in any real deployment, and the dominant real-world manifestation of bug-1611/bug-1613.

**Users**: Scheduler operators debugging or tuning via `xce_scheduler -i -s <replay>` (interactive) or batch `--sim` replay.

**Impact**: Changes when `sim_reconcile_unaccounted_jobs()` is willing to search a round's recorded decision stream for a job — from "only if the job is already RUN this round" to "whenever the job has no captured allocation yet, regardless of its current state." No other function changes.

### Goals
- Capture a job's recorded DISPATCH allocation at the round it actually occurred in the original recording, independent of the job's state at that moment.
- Preserve correct, state-gated resource-in-use accounting: a captured-but-not-yet-running job still contributes zero to `resInUse` until it genuinely transitions to RUN.
- Make the dispatch decision observable (printed, recorded, REPL-checked) at the historically accurate round.

### Non-Goals
- Any change to `job->state` transition timing or triggers.
- Any change to simulation mode's live picker (still PEND-only for freshly-arriving jobs, per `scheduler-allocation`).
- Any change to `--sim-verify`'s comparison semantics beyond not introducing a new false mismatch.
- Reopening or modifying `scheduler-sim-decision-parity`'s own requirements, design, or tasks, or any normal-mode (`driver.c`) behavior.

## Boundary Commitments

### This Spec Owns
- The trigger condition of `sim_reconcile_unaccounted_jobs()` (`sim_loop.c`) — which jobs are eligible to be searched for a matching recorded DISPATCH decision each round.
- The test coverage proving capture now works across round separation, still respects the existing no-fabrication/no-double-capture guarantees, and does not regress `--sim-verify` or interactive/batch parity.

### Out of Boundary
- `sched_account_jobs()`'s own `JOB_IS_RUN` charging gate (`scheduler.c:1258`) — unchanged; it is exactly the correct deferral mechanism this feature relies on, not something it modifies.
- The match/capture mechanics once a job is eligible (jobid matching, ownership transfer of `xce_job_alloc_t`, `make_job_account()`, shift-down compaction, decision reporting via `make_dispatch_decision()`/`sim_print_decision()`/`xce_repl_check_decision()`) — all unchanged, all already correct per `scheduler-sim-decision-parity`.
- Normal-mode (`driver.c`) behavior, simulation mode's live picker, `--sim-verify`'s comparison algorithm itself, and any `job->state` transition logic.

### Allowed Dependencies
- `sim->core.jobs` iteration and `decoded_session->decisions` (both already owned/populated by `sim_reconcile_unaccounted_jobs()`'s existing implementation) — P0.
- `sched_account_jobs()`'s existing `JOB_IS_RUN` gate as the sole mechanism deferring actual resource charging — P0, must remain unmodified for this design to be correct.
- `make_job_account()`/`make_dispatch_decision()` (existing, unmodified) — P0.

### Revalidation Triggers
- Any future change to `sched_account_jobs()`'s charging gate (e.g. adding a new precondition beyond `JOB_IS_RUN`) must re-verify this feature's Requirement 1.2/1.3 still hold, since this design relies on that gate — not a new one of its own — for correct deferral.
- Any future change that makes simulation mode's live picker emit a DISPATCH decision (currently PEND-only, `scheduler-allocation`'s stated scope) must re-verify Requirement 3.3's dedup assumption, which currently holds structurally (no live DISPATCH ever competes with a reconciled capture) rather than via an explicit check in this feature's own code.
- Any change to `decoded_session`'s per-round decode/free lifecycle (`sim_loop.c:871`/`:1006`) must re-verify this feature's core premise (every round's own decisions are only ever visible during that round).

## Architecture

### Existing Architecture Analysis
`sim_reconcile_unaccounted_jobs()` runs unconditionally once per simulation round, strictly between `xce_core_begin_session()` (which zeroes `resInUse`/`sharedInUse`) and `sched_account_jobs()` (which re-sums them from every `JOB_IS_RUN` job with `job->accounts != NULL`) — see `sim_loop.c:912-957`. Its current trigger gate additionally requires `JOB_IS_RUN(job->state)` before considering a job at all:

```c
if ((job == NULL) || !JOB_IS_RUN(job->state) || (job->accounts != NULL)) {
    continue;
}
```

`decoded_session` (this round's decoded recorded-decision stream) is freed at the end of the same round (`sim_loop.c:1006`) and never persisted — so a job skipped at round N because it was still PEND has no later opportunity to be matched against round N's data, even after it reaches RUN at round N+k.

### Architecture Pattern & Boundary Map

```mermaid
flowchart TD
    RoundStart[Round begins] --> Decode[decode_ReplaySession this round]
    Decode --> Begin[xce_core_begin_session resets resInUse]
    Begin --> Reconcile[sim_reconcile_unaccounted_jobs]
    Reconcile --> Gate{job accounts is NULL}
    Gate -- no --> NextJob[skip already-captured job]
    Gate -- yes --> Search[search this rounds decoded decisions for jobid]
    Search --> Found{matching DISPATCH found}
    Found -- yes --> Capture[capture allocs and accounts, report decision]
    Found -- no --> Retry[leave accounts NULL, retry next round]
    Capture --> Charge[sched_account_jobs]
    Retry --> Charge
    NextJob --> Charge
    Charge --> ChargeGate{job state is RUN and accounts set}
    ChargeGate -- yes --> ResInUse[include in resInUse]
    ChargeGate -- no --> Defer[exclude from resInUse]
```

**Architecture Integration**:
- Selected pattern: single-condition removal from an existing, already-correct-in-every-other-respect mechanism — not a new component.
- Domain/feature boundaries: capture (this feature, `Gate`/`Search`/`Capture`/`Retry` above) stays fully decoupled from charging (`sched_account_jobs`, `ChargeGate` above, untouched) — the same separation normal mode already has between `sched_commit_decision()` (capture) and `sched_account_jobs()` (charge).
- Existing patterns preserved: per-round re-invocation, jobid-scoped matching, ownership-transfer capture, shift-down compaction, three-call decision reporting — all untouched.
- New components rationale: none — this is a boundary-widening change to one existing condition.
- Steering compliance: directly closes the gap `.kiro/steering/scheduler-driver-equivalence.md` documents as bug-1611's residual; testing discipline (contention fixture) from that same steering doc drives the Testing Strategy below.

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Scheduler core (C, gnu17) | Existing `sim_loop.c`/`scheduler.c` | Trigger-condition change only | No new dependency; no schema/wire-format change |

## File Structure Plan

### Modified Files
- `src/scheduler/src/sim_loop.c` — remove `!JOB_IS_RUN(job->state)` from `sim_reconcile_unaccounted_jobs()`'s trigger condition (`sim_loop.c:592`); update the function's header doc comment (`sim_loop.c:~505-565`) to describe the new state-independent trigger and reframe the "double-application guard" description to also cover "this is now the sole capture-eligibility gate."
- `src/scheduler/testsuite/test_sim_reconcile_unaccounted.c` — replace `test_reconcile_skips_non_run_state_job` (asserts the bug: a PEND job with a matching recorded entry must stay uncaptured) with an inverted-expectation test proving a PEND job **is** now captured; add new tests for cross-round separation, `job->state` transitioning after capture, the no-recorded-origin-ever case, and the "captured-but-not-RUN contributes zero to resInUse" case (Requirement 1.2) driven through `sched_account_jobs()`.
- `src/scheduler/testsuite/test_sim_contention_parity.c` and/or a new contention-focused test file — add a resource-contention fixture (capacity insufficient for all candidate jobs) exercising cross-round dispatch-to-confirmation separation, per `scheduler-driver-equivalence.md`'s explicit testing discipline that a no-contention fixture cannot prove accounting correctness.
- A `--sim-verify` regression test (existing `test_sim_*` verify-mode file, or a new one if none fits) spanning a multi-round-separated dispatch — proves Requirement 4.2.

### No New Files
This feature introduces no new production source file — see Simplification decision in `research.md`.

## System Flows

```mermaid
sequenceDiagram
    participant Replay as Replay stream
    participant Recon as sim_reconcile_unaccounted_jobs
    participant Job as job in core jobs
    participant Acct as sched_account_jobs

    Replay->>Recon: round N decoded decisions include DISPATCH for job X
    Note over Job: job X state is PEND at round N
    Recon->>Job: capture allocs and accounts from DISPATCH
    Recon->>Recon: report decision at round N
    Acct->>Job: check state is RUN
    Note over Acct: state is PEND, skip charging
    Replay->>Job: round N+k replays job X state change to RUN
    Recon->>Job: accounts already set, skip re-search
    Acct->>Job: check state is RUN and accounts set
    Acct->>Acct: charge resInUse from round N+k onward
```

**Key decisions not obvious from the diagram**: the decision is reported once, at round N (the historically accurate round, Requirement 2.1) — not at round N+k when charging actually begins. Charging timing (round N+k) and reporting timing (round N) are intentionally decoupled, mirroring how normal mode's own `job->state` display and `resInUse` accounting are already decoupled from `sched_commit_decision()`'s synchronous capture.

## Requirements Traceability

| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|------------|------------|-------|
| 1.1 | Capture at originating round regardless of state | `sim_reconcile_unaccounted_jobs` | trigger gate | Search/Capture |
| 1.2 | Captured-but-not-RUN excludes from resInUse | `sched_account_jobs` (unchanged, relied upon) | `JOB_IS_RUN` charge gate | ChargeGate/Defer |
| 1.3 | RUN transition includes in resInUse from that point | `sched_account_jobs` (unchanged, relied upon) | `JOB_IS_RUN` charge gate | ChargeGate/ResInUse |
| 1.4 | Matches normal-mode's equivalent-point reporting | `sched_account_jobs` (shared with normal mode) | n/a | n/a |
| 2.1 | Decision visible at historically accurate round | `sim_reconcile_unaccounted_jobs` (decision reporting, unchanged) | `make_dispatch_decision`/`sim_print_decision`/`xce_repl_check_decision` | Capture step |
| 2.2 | Original decision type/detail preserved | `make_dispatch_decision` (unchanged) | n/a | Capture step |
| 2.3 | No duplicate report when live decision also exists | live picker (unchanged, PEND-only — structural invariant) | n/a | n/a |
| 3.1 | Never capture from a non-originating decision | jobid match (unchanged) | `entry->body.dispatch->alloc->jobid` check | Search |
| 3.2 | Never re-capture once captured | `job->accounts != NULL` guard (unchanged, now the sole gate) | trigger gate | Gate |
| 3.3 | Already-captured job not overwritten by a later live decision | structural (live picker never emits DISPATCH) | n/a | n/a |
| 4.1 | Same-round case still works | trigger gate (RUN is a subset of "any state") | trigger gate | Gate/Search |
| 4.2 | No new `--sim-verify` mismatch | compaction ordering (unchanged), verified by new test | decoded_session compaction | Capture step |
| 4.3 | Interactive/batch parity preserved | shared `sched_sim_driver_run()` call site (unchanged) | n/a | n/a |
| 4.4 | No recorded origin anywhere: report unavailable, keep going | "not found" branch (unchanged) | TRACE log, `job->accounts` stays NULL | Retry |
| 4.5 | Normal mode unaffected | `driver.c` untouched | n/a | n/a |

## Components and Interfaces

| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|--------------|--------|---------------|---------------------------|-----------|
| `sim_reconcile_unaccounted_jobs` | Scheduler sim driver | Widen capture eligibility to any not-yet-captured job, regardless of state | 1.1, 2.1, 3.1, 3.2, 4.1, 4.4 | `sched_account_jobs` (P0, relied on unmodified) | Service |

### Scheduler sim driver

#### `sim_reconcile_unaccounted_jobs`

| Field | Detail |
|-------|--------|
| Intent | Capture a job's recorded DISPATCH allocation as soon as the replay reaches the round it occurred in, independent of the job's current state |
| Requirements | 1.1, 2.1, 3.1, 3.2, 4.1, 4.4 |

**Responsibilities & Constraints**
- Owns only the capture-eligibility decision (which jobs are searched each round) and the search/capture/report mechanics already delivered by `scheduler-sim-decision-parity` (unchanged).
- Has no knowledge of resource charging — that remains `sched_account_jobs()`'s exclusive responsibility.
- Never fabricates data: a job with no matching recorded decision in a given round is left uncaptured and re-checked the following round (Requirement 4.4).

**Dependencies**
- Inbound: `sched_sim_driver_run()`'s round loop — calls this once per round, unchanged call site and unchanged position relative to `xce_core_begin_session()`/`sched_account_jobs()`/`xce_core_signal_go()` (P0).
- Outbound: `sched_account_jobs()` — this feature supplies `job->accounts` earlier than before; `sched_account_jobs()`'s own unmodified `JOB_IS_RUN` gate determines when that data starts being charged (P0).

**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

##### Service Interface
```c
/* Unchanged signature. Behavior change: a job is now eligible for capture
 * whenever job->accounts == NULL, regardless of job->state (previously
 * additionally required JOB_IS_RUN(job->state)). Capturing a job's
 * allocation while it is not yet RUN is inert until sched_account_jobs()'s
 * own JOB_IS_RUN gate later includes it -- this function performs no
 * charging itself. */
static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim,
                                            replay_session_t* decoded_session,
                                            sched_record_session_ctx_t* record_ctx);
```
- Preconditions: unchanged — `sim`/`decoded_session` NULL is a safe no-op; `record_ctx` may legitimately be NULL.
- Postconditions: unchanged — a matched job has `job->allocs`/`job->accounts` set and its entry removed from `decoded_session->decisions`; an unmatched job is left exactly as it was.
- Invariants (new): a job is captured **at most once**, at the first round (in replay order) whose decoded stream contains its originating DISPATCH decision — independent of that round's job state.

**Implementation Notes**
- Integration: delete `!JOB_IS_RUN(job->state)` from the trigger condition at `sim_loop.c:592`; no other line in the match/capture/report path changes.
- Validation: see Testing Strategy — contention fixture, cross-round separation, `--sim-verify`, interactive/batch parity, and the inverted `test_reconcile_skips_non_run_state_job` replacement are all required, not optional.
- Risks: repeated per-round "not found" TRACE logging for a job with no recorded origin anywhere in the replay (Requirement 4.4 case) — accepted, TRACE-level, see `research.md`'s Risks section.
- **Verified safe (early capture vs. cleanup)**: capturing `job->accounts` while a job is still PEND is safe against every existing job-cleanup path. `sched_uncharge_account()` has zero production callers anywhere in the scheduler — production resource accounting is a reset-and-resum model (`xce_core_begin_session()` zeroes `resInUse`/`sharedInUse` every round; `sched_account_jobs()` re-sums it from scratch from currently-`JOB_IS_RUN` jobs only). Both places `job->accounts` is freed on a non-RUN transition (`scheduler.c:974-977`, `core_events.c:915-918`) release memory only, with no `resInUse` mutation. A job captured while PEND that later transitions directly to a terminal state without ever passing through RUN is therefore never summed by `sched_account_jobs()` at any round and its accounts are freed with zero accounting impact — early capture cannot under-count `resInUse`. See `research.md`'s corresponding Research Log entry for the full trace.

## Error Handling

### Error Strategy
- **No matching recorded decision this round**: not an error — `job->accounts`/`job->allocs` stay `NULL`, a TRACE line is logged, the job is re-checked next round (unchanged behavior, now reached by more jobs per round since the gate widened).
- **No recorded origin anywhere in the replay** (Requirement 4.4): the same "not found" path applies every round for the rest of the replay; the scheduler continues processing without terminating, and that job's resource usage is reported as unavailable (never fabricated) wherever it is inspected (e.g. `info worker`).
- **`decoded_session` decode failure**: unchanged — already handled upstream of this function (`sim_loop.c:874-879`) by skipping verification for that round; this feature does not alter that path.

## Testing Strategy

### Unit Tests (`test_sim_reconcile_unaccounted.c`)
- Replace `test_reconcile_skips_non_run_state_job` with `test_reconcile_captures_non_run_state_job_when_dispatch_recorded` — a `JOB_STAT_PEND` job with a matching recorded DISPATCH entry now gets `job->allocs`/`job->accounts` captured and the entry consumed (Requirement 1.1, 3.1).
- New: `test_reconcile_still_skips_already_captured_job_regardless_of_state` — a job with `job->accounts` already set (any state) is never re-searched, proving the state-widened gate did not weaken the existing no-double-capture guard (Requirement 3.2).
- New: `test_reconcile_no_match_leaves_pend_job_uncaptured_and_retries` — a PEND job with no matching entry this round stays uncaptured and is eligible again next round (Requirement 4.4, state-independent retry).
- New: `test_reconcile_multiple_jobs_captured_independently_across_different_rounds` — several PEND jobs simultaneously outstanding, each with its own recorded DISPATCH decision spread across different rounds (some matching this round, some not yet recorded, some already captured from an earlier round); assert each job's `job->accounts` reflects only its own jobid's allocation data, captured at its own correct round, with zero cross-job interference from the wider per-round scan (Requirement 1.1, 3.1 under concurrent load — the single-job tests above cannot distinguish "correct" from "correct by coincidence, only one candidate available").
- Existing `test_reconcile_reconstructs_allocs_and_accounts_from_matching_dispatch`, `test_reconcile_ignores_non_dispatch_entry_with_matching_jobid`, `test_reconcile_compacts_via_shift_down_preserving_order`, `test_reconcile_is_safe_to_recheck_on_a_later_round`, `test_reconcile_then_release_via_job_state_change_is_asan_clean`, `test_reconcile_null_args_are_a_safe_noop` — re-run unmodified as regression coverage (Requirement 4.1); no expected behavior change for an already-RUN job.

### Integration Tests (round-boundary + charging interaction)
- New: `test_captured_pend_job_excluded_from_resinuse_until_run` — drive `sim_reconcile_unaccounted_jobs()` then `sched_account_jobs()` back to back for a PEND job with a freshly-captured account; assert `w->resInUse`/`core->sharedInUse` stay zero (Requirement 1.2), then transition the job to RUN and re-run `sched_account_jobs()`; assert `resInUse` now reflects it (Requirement 1.3).
- New (resource-contention fixture, per `scheduler-driver-equivalence.md`'s explicit discipline): a worker with capacity for N jobs, more than N candidate jobs recorded, dispatch decisions and RUN confirmations separated across multiple rounds — assert the live allocator's own capacity check (`sched_check_host_satisfy_`) sees correct `resInUse` at each round, not permanently-zero capacity. A no-contention fixture is explicitly insufficient here (`scheduler-driver-equivalence.md`).
- New: `test_sim_verify_no_false_mismatch_across_round_separated_dispatch` — a `--sim-verify` run over a replay where dispatch and RUN confirmation are several rounds apart reports zero mismatches (Requirement 4.2).
- Existing `test_repl_equivalence.c`-style interactive/batch parity coverage re-run unmodified — this feature's call site is shared between both modes, so no new test is needed to prove Requirement 4.3, only confirmation the existing suite still passes.