myxce/.kiro/specs/scheduler-sim-eager-reconciliation/design.md

Design Document

Overview

Purpose: This feature closes the residual gap in scheduler-sim-decision-parity's Recorded-Decision Reconciliation mechanism (sim_reconcile_unaccounted_jobs(), sim_loop.c) so a simulation-mode replay correctly captures a job's resource allocation regardless of how many rounds separate its recorded DISPATCH decision from its RUN-state confirmation — the realistic case in any real deployment, and the dominant real-world manifestation of bug-1611/bug-1613.

Users: Scheduler operators debugging or tuning via xce_scheduler -i -s <replay> (interactive) or batch --sim replay.

Impact: Changes when sim_reconcile_unaccounted_jobs() is willing to search a round's recorded decision stream for a job — from "only if the job is already RUN this round" to "whenever the job has no captured allocation yet, regardless of its current state." No other function changes.

Goals

• Capture a job's recorded DISPATCH allocation at the round it actually occurred in the original recording, independent of the job's state at that moment.
• Preserve correct, state-gated resource-in-use accounting: a captured-but-not-yet-running job still contributes zero to resInUse until it genuinely transitions to RUN.
• Make the dispatch decision observable (printed, recorded, REPL-checked) at the historically accurate round.

Non-Goals

• Any change to job->state transition timing or triggers.
• Any change to simulation mode's live picker (still PEND-only for freshly-arriving jobs, per scheduler-allocation).
• Any change to --sim-verify's comparison semantics beyond not introducing a new false mismatch.
• Reopening or modifying scheduler-sim-decision-parity's own requirements, design, or tasks, or any normal-mode (driver.c) behavior.

Boundary Commitments

This Spec Owns

• The trigger condition of sim_reconcile_unaccounted_jobs() (sim_loop.c) — which jobs are eligible to be searched for a matching recorded DISPATCH decision each round.
• The test coverage proving capture now works across round separation, still respects the existing no-fabrication/no-double-capture guarantees, and does not regress --sim-verify or interactive/batch parity.

Out of Boundary

• sched_account_jobs()'s own JOB_IS_RUN charging gate (scheduler.c:1258) — unchanged; it is exactly the correct deferral mechanism this feature relies on, not something it modifies.
• The match/capture mechanics once a job is eligible (jobid matching, ownership transfer of xce_job_alloc_t, make_job_account(), shift-down compaction, decision reporting via make_dispatch_decision()/sim_print_decision()/xce_repl_check_decision()) — all unchanged, all already correct per scheduler-sim-decision-parity.
• Normal-mode (driver.c) behavior, simulation mode's live picker, --sim-verify's comparison algorithm itself, and any job->state transition logic.

Allowed Dependencies

• sim->core.jobs iteration and decoded_session->decisions (both already owned/populated by sim_reconcile_unaccounted_jobs()'s existing implementation) — P0.
• sched_account_jobs()'s existing JOB_IS_RUN gate as the sole mechanism deferring actual resource charging — P0, must remain unmodified for this design to be correct.
• make_job_account()/make_dispatch_decision() (existing, unmodified) — P0.

Revalidation Triggers

• Any future change to sched_account_jobs()'s charging gate (e.g. adding a new precondition beyond JOB_IS_RUN) must re-verify this feature's Requirement 1.2/1.3 still hold, since this design relies on that gate — not a new one of its own — for correct deferral.
• Any future change that makes simulation mode's live picker emit a DISPATCH decision (currently PEND-only, scheduler-allocation's stated scope) must re-verify Requirement 3.3's dedup assumption, which currently holds structurally (no live DISPATCH ever competes with a reconciled capture) rather than via an explicit check in this feature's own code.
• Any change to decoded_session's per-round decode/free lifecycle (sim_loop.c:871/:1006) must re-verify this feature's core premise (every round's own decisions are only ever visible during that round).

Architecture

Existing Architecture Analysis

sim_reconcile_unaccounted_jobs() runs unconditionally once per simulation round, strictly between xce_core_begin_session() (which zeroes resInUse/sharedInUse) and sched_account_jobs() (which re-sums them from every JOB_IS_RUN job with job->accounts != NULL) — see sim_loop.c:912-957. Its current trigger gate additionally requires JOB_IS_RUN(job->state) before considering a job at all:

if ((job == NULL) || !JOB_IS_RUN(job->state) || (job->accounts != NULL)) {
    continue;
}

decoded_session (this round's decoded recorded-decision stream) is freed at the end of the same round (sim_loop.c:1006) and never persisted — so a job skipped at round N because it was still PEND has no later opportunity to be matched against round N's data, even after it reaches RUN at round N+k.

Architecture Pattern & Boundary Map

flowchart TD
    RoundStart[Round begins] --> Decode[decode_ReplaySession this round]
    Decode --> Begin[xce_core_begin_session resets resInUse]
    Begin --> Reconcile[sim_reconcile_unaccounted_jobs]
    Reconcile --> Gate{job accounts is NULL}
    Gate -- no --> NextJob[skip already-captured job]
    Gate -- yes --> Search[search this rounds decoded decisions for jobid]
    Search --> Found{matching DISPATCH found}
    Found -- yes --> Capture[capture allocs and accounts, report decision]
    Found -- no --> Retry[leave accounts NULL, retry next round]
    Capture --> Charge[sched_account_jobs]
    Retry --> Charge
    NextJob --> Charge
    Charge --> ChargeGate{job state is RUN and accounts set}
    ChargeGate -- yes --> ResInUse[include in resInUse]
    ChargeGate -- no --> Defer[exclude from resInUse]

Architecture Integration:

• Selected pattern: single-condition removal from an existing, already-correct-in-every-other-respect mechanism — not a new component.
• Domain/feature boundaries: capture (this feature, Gate/Search/Capture/Retry above) stays fully decoupled from charging (sched_account_jobs, ChargeGate above, untouched) — the same separation normal mode already has between sched_commit_decision() (capture) and sched_account_jobs() (charge).
• Existing patterns preserved: per-round re-invocation, jobid-scoped matching, ownership-transfer capture, shift-down compaction, three-call decision reporting — all untouched.
• New components rationale: none — this is a boundary-widening change to one existing condition.
• Steering compliance: directly closes the gap .kiro/steering/scheduler-driver-equivalence.md documents as bug-1611's residual; testing discipline (contention fixture) from that same steering doc drives the Testing Strategy below.

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Scheduler core (C, gnu17)	Existing sim_loop.c/scheduler.c	Trigger-condition change only	No new dependency; no schema/wire-format change

File Structure Plan

Modified Files

• src/scheduler/src/sim_loop.c — remove !JOB_IS_RUN(job->state) from sim_reconcile_unaccounted_jobs()'s trigger condition (sim_loop.c:592); update the function's header doc comment (sim_loop.c:~505-565) to describe the new state-independent trigger and reframe the "double-application guard" description to also cover "this is now the sole capture-eligibility gate."
• src/scheduler/testsuite/test_sim_reconcile_unaccounted.c — replace test_reconcile_skips_non_run_state_job (asserts the bug: a PEND job with a matching recorded entry must stay uncaptured) with an inverted-expectation test proving a PEND job is now captured; add new tests for cross-round separation, job->state transitioning after capture, the no-recorded-origin-ever case, and the "captured-but-not-RUN contributes zero to resInUse" case (Requirement 1.2) driven through sched_account_jobs().
• src/scheduler/testsuite/test_sim_contention_parity.c and/or a new contention-focused test file — add a resource-contention fixture (capacity insufficient for all candidate jobs) exercising cross-round dispatch-to-confirmation separation, per scheduler-driver-equivalence.md's explicit testing discipline that a no-contention fixture cannot prove accounting correctness.
• A --sim-verify regression test (existing test_sim_* verify-mode file, or a new one if none fits) spanning a multi-round-separated dispatch — proves Requirement 4.2.

No New Files

This feature introduces no new production source file — see Simplification decision in research.md.

System Flows

sequenceDiagram
    participant Replay as Replay stream
    participant Recon as sim_reconcile_unaccounted_jobs
    participant Job as job in core jobs
    participant Acct as sched_account_jobs

    Replay->>Recon: round N decoded decisions include DISPATCH for job X
    Note over Job: job X state is PEND at round N
    Recon->>Job: capture allocs and accounts from DISPATCH
    Recon->>Recon: report decision at round N
    Acct->>Job: check state is RUN
    Note over Acct: state is PEND, skip charging
    Replay->>Job: round N+k replays job X state change to RUN
    Recon->>Job: accounts already set, skip re-search
    Acct->>Job: check state is RUN and accounts set
    Acct->>Acct: charge resInUse from round N+k onward

Key decisions not obvious from the diagram: the decision is reported once, at round N (the historically accurate round, Requirement 2.1) — not at round N+k when charging actually begins. Charging timing (round N+k) and reporting timing (round N) are intentionally decoupled, mirroring how normal mode's own job->state display and resInUse accounting are already decoupled from sched_commit_decision()'s synchronous capture.

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1	Capture at originating round regardless of state	sim_reconcile_unaccounted_jobs	trigger gate	Search/Capture
1.2	Captured-but-not-RUN excludes from resInUse	sched_account_jobs (unchanged, relied upon)	JOB_IS_RUN charge gate	ChargeGate/Defer
1.3	RUN transition includes in resInUse from that point	sched_account_jobs (unchanged, relied upon)	JOB_IS_RUN charge gate	ChargeGate/ResInUse
1.4	Matches normal-mode's equivalent-point reporting	sched_account_jobs (shared with normal mode)	n/a	n/a
2.1	Decision visible at historically accurate round	sim_reconcile_unaccounted_jobs (decision reporting, unchanged)	make_dispatch_decision/sim_print_decision/xce_repl_check_decision	Capture step
2.2	Original decision type/detail preserved	make_dispatch_decision (unchanged)	n/a	Capture step
2.3	No duplicate report when live decision also exists	live picker (unchanged, PEND-only — structural invariant)	n/a	n/a
3.1	Never capture from a non-originating decision	jobid match (unchanged)	entry->body.dispatch->alloc->jobid check	Search
3.2	Never re-capture once captured	job->accounts != NULL guard (unchanged, now the sole gate)	trigger gate	Gate
3.3	Already-captured job not overwritten by a later live decision	structural (live picker never emits DISPATCH)	n/a	n/a
4.1	Same-round case still works	trigger gate (RUN is a subset of "any state")	trigger gate	Gate/Search
4.2	No new --sim-verify mismatch	compaction ordering (unchanged), verified by new test	decoded_session compaction	Capture step
4.3	Interactive/batch parity preserved	shared sched_sim_driver_run() call site (unchanged)	n/a	n/a
4.4	No recorded origin anywhere: report unavailable, keep going	"not found" branch (unchanged)	TRACE log, job->accounts stays NULL	Retry
4.5	Normal mode unaffected	driver.c untouched	n/a	n/a

Components and Interfaces

Component	Domain/Layer	Intent	Req Coverage	Key Dependencies (P0/P1)	Contracts
sim_reconcile_unaccounted_jobs	Scheduler sim driver	Widen capture eligibility to any not-yet-captured job, regardless of state	1.1, 2.1, 3.1, 3.2, 4.1, 4.4	sched_account_jobs (P0, relied on unmodified)	Service

Scheduler sim driver

sim_reconcile_unaccounted_jobs

Field	Detail
Intent	Capture a job's recorded DISPATCH allocation as soon as the replay reaches the round it occurred in, independent of the job's current state
Requirements	1.1, 2.1, 3.1, 3.2, 4.1, 4.4

Responsibilities & Constraints

• Owns only the capture-eligibility decision (which jobs are searched each round) and the search/capture/report mechanics already delivered by scheduler-sim-decision-parity (unchanged).
• Has no knowledge of resource charging — that remains sched_account_jobs()'s exclusive responsibility.
• Never fabricates data: a job with no matching recorded decision in a given round is left uncaptured and re-checked the following round (Requirement 4.4).

Dependencies

• Inbound: sched_sim_driver_run()'s round loop — calls this once per round, unchanged call site and unchanged position relative to xce_core_begin_session()/sched_account_jobs()/xce_core_signal_go() (P0).
• Outbound: sched_account_jobs() — this feature supplies job->accounts earlier than before; sched_account_jobs()'s own unmodified JOB_IS_RUN gate determines when that data starts being charged (P0).

Contracts: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Service Interface

/* Unchanged signature. Behavior change: a job is now eligible for capture
 * whenever job->accounts == NULL, regardless of job->state (previously
 * additionally required JOB_IS_RUN(job->state)). Capturing a job's
 * allocation while it is not yet RUN is inert until sched_account_jobs()'s
 * own JOB_IS_RUN gate later includes it -- this function performs no
 * charging itself. */
static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim,
                                            replay_session_t* decoded_session,
                                            sched_record_session_ctx_t* record_ctx);

• Preconditions: unchanged — sim/decoded_session NULL is a safe no-op; record_ctx may legitimately be NULL.
• Postconditions: unchanged — a matched job has job->allocs/job->accounts set and its entry removed from decoded_session->decisions; an unmatched job is left exactly as it was.
• Invariants (new): a job is captured at most once, at the first round (in replay order) whose decoded stream contains its originating DISPATCH decision — independent of that round's job state.

Implementation Notes

• Integration: delete !JOB_IS_RUN(job->state) from the trigger condition at sim_loop.c:592; no other line in the match/capture/report path changes.
• Validation: see Testing Strategy — contention fixture, cross-round separation, --sim-verify, interactive/batch parity, and the inverted test_reconcile_skips_non_run_state_job replacement are all required, not optional.
• Risks: repeated per-round "not found" TRACE logging for a job with no recorded origin anywhere in the replay (Requirement 4.4 case) — accepted, TRACE-level, see research.md's Risks section.
• Verified safe (early capture vs. cleanup): capturing job->accounts while a job is still PEND is safe against every existing job-cleanup path. sched_uncharge_account() has zero production callers anywhere in the scheduler — production resource accounting is a reset-and-resum model (xce_core_begin_session() zeroes resInUse/sharedInUse every round; sched_account_jobs() re-sums it from scratch from currently-JOB_IS_RUN jobs only). Both places job->accounts is freed on a non-RUN transition (scheduler.c:974-977, core_events.c:915-918) release memory only, with no resInUse mutation. A job captured while PEND that later transitions directly to a terminal state without ever passing through RUN is therefore never summed by sched_account_jobs() at any round and its accounts are freed with zero accounting impact — early capture cannot under-count resInUse. See research.md's corresponding Research Log entry for the full trace.

Error Handling

Error Strategy

• No matching recorded decision this round: not an error — job->accounts/job->allocs stay NULL, a TRACE line is logged, the job is re-checked next round (unchanged behavior, now reached by more jobs per round since the gate widened).
• No recorded origin anywhere in the replay (Requirement 4.4): the same "not found" path applies every round for the rest of the replay; the scheduler continues processing without terminating, and that job's resource usage is reported as unavailable (never fabricated) wherever it is inspected (e.g. info worker).
• decoded_session decode failure: unchanged — already handled upstream of this function (sim_loop.c:874-879) by skipping verification for that round; this feature does not alter that path.

Testing Strategy

Unit Tests (test_sim_reconcile_unaccounted.c)

• Replace test_reconcile_skips_non_run_state_job with test_reconcile_captures_non_run_state_job_when_dispatch_recorded — a JOB_STAT_PEND job with a matching recorded DISPATCH entry now gets job->allocs/job->accounts captured and the entry consumed (Requirement 1.1, 3.1).
• New: test_reconcile_still_skips_already_captured_job_regardless_of_state — a job with job->accounts already set (any state) is never re-searched, proving the state-widened gate did not weaken the existing no-double-capture guard (Requirement 3.2).
• New: test_reconcile_no_match_leaves_pend_job_uncaptured_and_retries — a PEND job with no matching entry this round stays uncaptured and is eligible again next round (Requirement 4.4, state-independent retry).
• New: test_reconcile_multiple_jobs_captured_independently_across_different_rounds — several PEND jobs simultaneously outstanding, each with its own recorded DISPATCH decision spread across different rounds (some matching this round, some not yet recorded, some already captured from an earlier round); assert each job's job->accounts reflects only its own jobid's allocation data, captured at its own correct round, with zero cross-job interference from the wider per-round scan (Requirement 1.1, 3.1 under concurrent load — the single-job tests above cannot distinguish "correct" from "correct by coincidence, only one candidate available").
• Existing test_reconcile_reconstructs_allocs_and_accounts_from_matching_dispatch, test_reconcile_ignores_non_dispatch_entry_with_matching_jobid, test_reconcile_compacts_via_shift_down_preserving_order, test_reconcile_is_safe_to_recheck_on_a_later_round, test_reconcile_then_release_via_job_state_change_is_asan_clean, test_reconcile_null_args_are_a_safe_noop — re-run unmodified as regression coverage (Requirement 4.1); no expected behavior change for an already-RUN job.

Integration Tests (round-boundary + charging interaction)

• New: test_captured_pend_job_excluded_from_resinuse_until_run — drive sim_reconcile_unaccounted_jobs() then sched_account_jobs() back to back for a PEND job with a freshly-captured account; assert w->resInUse/core->sharedInUse stay zero (Requirement 1.2), then transition the job to RUN and re-run sched_account_jobs(); assert resInUse now reflects it (Requirement 1.3).
• New (resource-contention fixture, per scheduler-driver-equivalence.md's explicit discipline): a worker with capacity for N jobs, more than N candidate jobs recorded, dispatch decisions and RUN confirmations separated across multiple rounds — assert the live allocator's own capacity check (sched_check_host_satisfy_) sees correct resInUse at each round, not permanently-zero capacity. A no-contention fixture is explicitly insufficient here (scheduler-driver-equivalence.md).
• New: test_sim_verify_no_false_mismatch_across_round_separated_dispatch — a --sim-verify run over a replay where dispatch and RUN confirmation are several rounds apart reports zero mismatches (Requirement 4.2).
• Existing test_repl_equivalence.c-style interactive/batch parity coverage re-run unmodified — this feature's call site is shared between both modes, so no new test is needed to prove Requirement 4.3, only confirmation the existing suite still passes.