myxce/.kiro/specs/scheduler-sim-eager-reconciliation/design.md

Design Document

Overview

Purpose: This feature closes the residual gap in scheduler-sim-decision-parity's Recorded-Decision Reconciliation mechanism (sim_reconcile_unaccounted_jobs(), sim_loop.c) so a simulation-mode replay correctly captures a job's resource allocation regardless of how many rounds separate its recorded DISPATCH decision from its RUN-state confirmation — the realistic case in any real deployment, and the dominant real-world manifestation of bug-1611/bug-1613.

Users: Scheduler operators debugging or tuning via xce_scheduler -i -s <replay> (interactive) or batch --sim replay.

Impact: Changes when sim_reconcile_unaccounted_jobs() is willing to search a round's recorded decision stream for a job — from "only if the job is already RUN this round" to "whenever the job has no captured allocation yet, regardless of its current state." No other function changes.

Goals

• Capture a job's recorded DISPATCH allocation at the round it actually occurred in the original recording, independent of the job's state at that moment.
• Transition a captured job to a running state synchronously, at the SAME round its dispatch decision is captured -- mirroring normal-mode's own sched_commit_decision() semantics (bug-1624) -- so resource-in-use accounting reflects it starting that same round via sched_account_jobs()'s existing unchanged JOB_IS_RUN gate. (Revised 2026-09-11 -- see below.)
• Make the dispatch decision observable (printed, recorded, REPL-checked) at the historically accurate round.

Non-Goals

• Any change to how/when a job transitions to a NON-running state (DONE/EXIT/preemption). (The RUN transition itself is IN scope as of the 2026-09-11 revision below -- see Correction.)
• Any change to simulation mode's live picker (still PEND-only for freshly-arriving jobs, per scheduler-allocation).
• Any change to --sim-verify's comparison semantics beyond not introducing a new false mismatch.
• Reopening or modifying scheduler-sim-decision-parity's own requirements, design, or tasks, or any normal-mode (driver.c) behavior.

Correction (2026-09-11, user-directed, live replay session): the original version of this Overview described capture and RUN-state transition as fully decoupled, with sched_account_jobs()'s existing JOB_IS_RUN gate providing "correct, state-gated" deferral until a job's SEPARATELY-recorded JOB_STATE_CHANGE confirmation event replayed -- potentially many rounds after capture. Live use surfaced the resulting operator confusion directly (info jobs showing PEND for a job [sim] DISPATCH had already printed), and investigation confirmed normal-mode's own sched_commit_decision() (scheduler.c, bug-1624) does NOT defer this way -- it sets job->state = JOB_STAT_RUN synchronously in the same operation that captures the allocation. This design is revised so simulation-mode reconciliation now does the same: sim_reconcile_unaccounted_jobs()'s capture step ALSO sets job->state = JOB_STAT_RUN, so sched_account_jobs()'s unchanged JOB_IS_RUN gate now includes the job starting the SAME round as capture, not a later one. The later, separately-recorded JOB_STATE_CHANGE/JOB_UPDATE-to-RUN event, when it replays, becomes a safe no-op re-affirmation (RUN -> RUN) rather than the sole state-transition trigger. See Requirement 1's own 2026-09-11 revision (requirements.md) and research.md's corresponding Research Log entry for the full rationale.

Correction (2026-09-07, bug-1759/bug-1779): the "live picker" Non-Goal above was never meant to license sim_reconcile_unaccounted_jobs() leaving a reconciled job as a live picker candidate — it means "do not give the live picker real resource-matching allocation logic" (still out of scope, still scheduler-allocation's territory). Whether a reconciled job stays in the picker's own pendJobs bookkeeping is a scheduler-sim-decision-parity concern (see that spec's design.md, "Integration with --sim-verify", which already documented "the picker never re-decides an already-resolved job" as the correct assumption). The implementation had drifted from that documented assumption: sim_reconcile_unaccounted_jobs() never called picker_iface->on_dispatch(), so a reconciled job stayed a live candidate and was live-observed being re-offered and re-PENDed every subsequent round (bug-1759: 22/50 already-dispatched jobs in one batch; bug-1779: 5/10 array elements). Fixed by adding that missing on_dispatch() call, mirroring sched_allocator()'s own live-dispatch ordering (scheduler.c) exactly — a correction to already-agreed-upon behavior, not a new capability, so it did not need to reopen this spec's own requirements/tasks. test_sim_verify_no_false_mismatch_across_round_separated_dispatch (test_sim_verify.c) was updated to match: rounds 2-3 of its fixture are now empty ReplaySessions (no more picker-echoed PEND for an already-reconciled job) rather than baking in the pre-fix stale-PEND behavior as "expected."

Boundary Commitments

This Spec Owns

• The trigger condition of sim_reconcile_unaccounted_jobs() (sim_loop.c) — which jobs are eligible to be searched for a matching recorded DISPATCH decision each round.
• The test coverage proving capture now works across round separation, still respects the existing no-fabrication/no-double-capture guarantees, and does not regress --sim-verify or interactive/batch parity.

Out of Boundary

• sched_account_jobs()'s own JOB_IS_RUN charging gate (scheduler.c:1258) — unchanged; it is exactly the mechanism this feature relies on to turn a captured job's RUN state (now set at capture time, per the 2026-09-11 revision above) into an actual resource charge.
• The match/capture mechanics once a job is eligible (jobid matching, ownership transfer of xce_job_alloc_t, make_job_account(), shift-down compaction, decision reporting via make_dispatch_decision()/sim_print_decision()/xce_repl_check_decision()) — all unchanged, all already correct per scheduler-sim-decision-parity.
• Normal-mode (driver.c) behavior, simulation mode's live picker, --sim-verify's comparison algorithm itself, and any NON-running-state job transition logic (DONE/EXIT/preemption) — the RUN transition triggered by capture is now IN boundary as of the 2026-09-11 revision (see Overview Correction).

Allowed Dependencies

• sim->core.jobs iteration and decoded_session->decisions (both already owned/populated by sim_reconcile_unaccounted_jobs()'s existing implementation) — P0.
• sched_account_jobs()'s existing JOB_IS_RUN gate as the sole mechanism deferring actual resource charging — P0, must remain unmodified for this design to be correct.
• make_job_account()/make_dispatch_decision() (existing, unmodified) — P0.

Revalidation Triggers

• Any future change to sched_account_jobs()'s charging gate (e.g. adding a new precondition beyond JOB_IS_RUN) must re-verify this feature's Requirement 1.2/1.3 still hold, since this design relies on that gate — not a new one of its own — for correct deferral.
• Any future change that makes simulation mode's live picker emit a DISPATCH decision (currently PEND-only, scheduler-allocation's stated scope) must re-verify Requirement 3.3's dedup assumption, which currently holds structurally (no live DISPATCH ever competes with a reconciled capture) rather than via an explicit check in this feature's own code.
• Any change to decoded_session's per-round decode/free lifecycle (sim_loop.c:871/:1006) must re-verify this feature's core premise (every round's own decisions are only ever visible during that round).

Architecture

Existing Architecture Analysis

sim_reconcile_unaccounted_jobs() runs unconditionally once per simulation round, strictly between xce_core_begin_session() (which zeroes resInUse/sharedInUse) and sched_account_jobs() (which re-sums them from every JOB_IS_RUN job with job->accounts != NULL) — see sim_loop.c:912-957. Its current trigger gate additionally requires JOB_IS_RUN(job->state) before considering a job at all:

if ((job == NULL) || !JOB_IS_RUN(job->state) || (job->accounts != NULL)) {
    continue;
}

decoded_session (this round's decoded recorded-decision stream) is freed at the end of the same round (sim_loop.c:1006) and never persisted — so a job skipped at round N because it was still PEND has no later opportunity to be matched against round N's data, even after it reaches RUN at round N+k.

Architecture Pattern & Boundary Map

flowchart TD
    RoundStart[Round begins] --> Decode[decode_ReplaySession this round]
    Decode --> Begin[xce_core_begin_session resets resInUse]
    Begin --> Reconcile[sim_reconcile_unaccounted_jobs]
    Reconcile --> Gate{job accounts is NULL}
    Gate -- no --> NextJob[skip already-captured job]
    Gate -- yes --> Search[search this rounds decoded decisions for jobid]
    Search --> Found{matching DISPATCH found}
    Found -- yes --> Capture["capture allocs and accounts, set state=RUN synchronously (2026-09-11), report decision"]
    Found -- no --> Retry[leave accounts NULL and state unchanged, retry next round]
    Capture --> Charge[sched_account_jobs]
    Retry --> Charge
    NextJob --> Charge
    Charge --> ChargeGate{job state is RUN and accounts set}
    ChargeGate -- yes --> ResInUse["include in resInUse -- now true the SAME round as capture, not a later one"]
    ChargeGate -- no --> Defer[exclude from resInUse]

Architecture Integration:

• Selected pattern: single-condition removal from an existing, already-correct-in-every-other-respect mechanism — not a new component.
• Domain/feature boundaries: capture (this feature, Gate/Search/Capture/Retry above) stays fully decoupled from charging (sched_account_jobs, ChargeGate above, untouched) — the same separation normal mode already has between sched_commit_decision() (capture) and sched_account_jobs() (charge).
• Existing patterns preserved: per-round re-invocation, jobid-scoped matching, ownership-transfer capture, shift-down compaction, three-call decision reporting — all untouched.
• New components rationale: none — this is a boundary-widening change to one existing condition.
• Steering compliance: directly closes the gap .kiro/steering/scheduler-driver-equivalence.md documents as bug-1611's residual; testing discipline (contention fixture) from that same steering doc drives the Testing Strategy below.

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Scheduler core (C, gnu17)	Existing sim_loop.c/scheduler.c	Trigger-condition change only	No new dependency; no schema/wire-format change

File Structure Plan

Modified Files

• src/scheduler/src/sim_loop.c — remove !JOB_IS_RUN(job->state) from sim_reconcile_unaccounted_jobs()'s trigger condition (sim_loop.c:592); update the function's header doc comment (sim_loop.c:~505-565) to describe the new state-independent trigger and reframe the "double-application guard" description to also cover "this is now the sole capture-eligibility gate."
• src/scheduler/testsuite/test_sim_reconcile_unaccounted.c — replace test_reconcile_skips_non_run_state_job (asserts the bug: a PEND job with a matching recorded entry must stay uncaptured) with an inverted-expectation test proving a PEND job is now captured; add new tests for cross-round separation, job->state transitioning after capture, the no-recorded-origin-ever case, and the "captured-but-not-RUN contributes zero to resInUse" case (Requirement 1.2) driven through sched_account_jobs().
• src/scheduler/testsuite/test_sim_contention_parity.c and/or a new contention-focused test file — add a resource-contention fixture (capacity insufficient for all candidate jobs) exercising cross-round dispatch-to-confirmation separation, per scheduler-driver-equivalence.md's explicit testing discipline that a no-contention fixture cannot prove accounting correctness.
• A --sim-verify regression test (existing test_sim_* verify-mode file, or a new one if none fits) spanning a multi-round-separated dispatch — proves Requirement 4.2.

No New Files

This feature introduces no new production source file — see Simplification decision in research.md.

System Flows

sequenceDiagram
    participant Replay as Replay stream
    participant Recon as sim_reconcile_unaccounted_jobs
    participant Job as job in core jobs
    participant Acct as sched_account_jobs

    Replay->>Recon: round N decoded decisions include DISPATCH for job X
    Note over Job: job X state is PEND at round N
    Recon->>Job: capture allocs and accounts from DISPATCH; set state=RUN synchronously (2026-09-11)
    Recon->>Recon: report decision at round N
    Acct->>Job: check state is RUN and accounts set
    Acct->>Acct: charge resInUse starting round N (same round as capture)
    Replay->>Job: round N+k replays job X's separately-recorded state-change-to-RUN event
    Note over Job: state already RUN -- idempotent re-affirmation, not a new transition
    Recon->>Job: accounts already set, skip re-search

Key decisions not obvious from the diagram (revised 2026-09-11): the decision is reported at round N (the historically accurate round, Requirement 2.1). As of the 2026-09-11 revision, charging ALSO begins at round N, not round N+k — job->state is set to RUN synchronously with capture, mirroring normal-mode's own sched_commit_decision() exactly (bug-1624), so reporting and charging are no longer decoupled by round. The separately-recorded JOB_STATE_CHANGE/JOB_UPDATE-to-RUN event at round N+k, when it later replays, is a safe no-op re-affirmation of an already-RUN job (core_events.c's handler simply reassigns job->state = new_state, RUN → RUN, with no guard triggered) — it is preserved in the replay stream and still processed, but no longer the operative trigger for the job's own running-state transition or its resource charge.

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1	Capture at originating round regardless of state	sim_reconcile_unaccounted_jobs	trigger gate	Search/Capture
1.2 (revised 2026-09-11)	Capture ALSO transitions job to RUN synchronously, same round	sim_reconcile_unaccounted_jobs	Capture step (new job->state = JOB_STAT_RUN write)	Capture
1.3 (revised 2026-09-11)	resInUse includes the job starting the SAME round as capture	sched_account_jobs (unchanged, relied upon)	JOB_IS_RUN charge gate	ChargeGate/ResInUse
1.4 (new 2026-09-11)	Later separately-recorded confirmation event is an idempotent re-affirmation	core_events.c's XCE_CORE_EVENT_JOB_STATE_CHANGE handler (unchanged, verified safe)	n/a	n/a
1.5	Matches normal-mode's equivalent-point reporting	sched_account_jobs (shared with normal mode)	n/a	n/a
2.1	Decision visible at historically accurate round	sim_reconcile_unaccounted_jobs (decision reporting, unchanged)	make_dispatch_decision/sim_print_decision/xce_repl_check_decision	Capture step
2.2	Original decision type/detail preserved	make_dispatch_decision (unchanged)	n/a	Capture step
2.3	No duplicate report when live decision also exists	live picker (unchanged, PEND-only — structural invariant)	n/a	n/a
3.1	Never capture from a non-originating decision	jobid match (unchanged)	entry->body.dispatch->alloc->jobid check	Search
3.2	Never re-capture once captured	job->accounts != NULL guard (unchanged, now the sole gate)	trigger gate	Gate
3.3	Already-captured job not overwritten by a later live decision	structural (live picker never emits DISPATCH)	n/a	n/a
4.1	Same-round case still works	trigger gate (RUN is a subset of "any state")	trigger gate	Gate/Search
4.2	No new --sim-verify mismatch	compaction ordering (unchanged), verified by new test	decoded_session compaction	Capture step
4.3	Interactive/batch parity preserved	shared sched_sim_driver_run() call site (unchanged)	n/a	n/a
4.4	No recorded origin anywhere: report unavailable, keep going	"not found" branch (unchanged)	TRACE log, job->accounts stays NULL	Retry
4.5	Normal mode unaffected	driver.c untouched	n/a	n/a

Components and Interfaces

Component	Domain/Layer	Intent	Req Coverage	Key Dependencies (P0/P1)	Contracts
sim_reconcile_unaccounted_jobs	Scheduler sim driver	Widen capture eligibility to any not-yet-captured job, regardless of state	1.1, 2.1, 3.1, 3.2, 4.1, 4.4	sched_account_jobs (P0, relied on unmodified)	Service

Scheduler sim driver

sim_reconcile_unaccounted_jobs

Field	Detail
Intent	Capture a job's recorded DISPATCH allocation as soon as the replay reaches the round it occurred in, independent of the job's current state
Requirements	1.1, 2.1, 3.1, 3.2, 4.1, 4.4

Responsibilities & Constraints

• Owns only the capture-eligibility decision (which jobs are searched each round) and the search/capture/report mechanics already delivered by scheduler-sim-decision-parity (unchanged).
• Has no knowledge of resource charging — that remains sched_account_jobs()'s exclusive responsibility.
• Never fabricates data: a job with no matching recorded decision in a given round is left uncaptured and re-checked the following round (Requirement 4.4).

Dependencies

• Inbound: sched_sim_driver_run()'s round loop — calls this once per round, unchanged call site and unchanged position relative to xce_core_begin_session()/sched_account_jobs()/xce_core_signal_go() (P0).
• Outbound: sched_account_jobs() — this feature supplies job->accounts earlier than before; sched_account_jobs()'s own unmodified JOB_IS_RUN gate determines when that data starts being charged (P0).

Contracts: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Service Interface

/* Unchanged signature. Behavior change #1 (original): a job is now
 * eligible for capture whenever job->accounts == NULL, regardless of
 * job->state (previously additionally required JOB_IS_RUN(job->state)).
 * Behavior change #2 (2026-09-11 revision): a successful capture ALSO
 * sets job->state = JOB_STAT_RUN synchronously, in the same operation --
 * mirroring scheduler.c's sched_commit_decision() (bug-1624) exactly --
 * so sched_account_jobs()'s own unchanged JOB_IS_RUN gate includes the
 * job starting THIS round, not a later one. This function still performs
 * no charging itself; it now also performs no NON-running state
 * transition (DONE/EXIT/preemption remain exclusively triggered by their
 * own separately-recorded events, unchanged). */
static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim,
                                            replay_session_t* decoded_session,
                                            sched_record_session_ctx_t* record_ctx);

• Preconditions: unchanged — sim/decoded_session NULL is a safe no-op; record_ctx may legitimately be NULL.
• Postconditions: unchanged — a matched job has job->allocs/job->accounts set and its entry removed from decoded_session->decisions; an unmatched job is left exactly as it was.
• Invariants (new): a job is captured at most once, at the first round (in replay order) whose decoded stream contains its originating DISPATCH decision — independent of that round's job state.

Implementation Notes

• Integration: delete !JOB_IS_RUN(job->state) from the trigger condition at sim_loop.c:592; no other line in the match/capture/report path changes.
• Validation: see Testing Strategy — contention fixture, cross-round separation, --sim-verify, interactive/batch parity, and the inverted test_reconcile_skips_non_run_state_job replacement are all required, not optional.
• Risks: repeated per-round "not found" TRACE logging for a job with no recorded origin anywhere in the replay (Requirement 4.4 case) — accepted, TRACE-level, see research.md's Risks section.
• Verified safe (early capture vs. cleanup): capturing job->accounts while a job is still PEND is safe against every existing job-cleanup path. sched_uncharge_account() has zero production callers anywhere in the scheduler — production resource accounting is a reset-and-resum model (xce_core_begin_session() zeroes resInUse/sharedInUse every round; sched_account_jobs() re-sums it from scratch from currently-JOB_IS_RUN jobs only). Both places job->accounts is freed on a non-RUN transition (scheduler.c:974-977, core_events.c:915-918) release memory only, with no resInUse mutation. A job captured while PEND that later transitions directly to a terminal state without ever passing through RUN is therefore never summed by sched_account_jobs() at any round and its accounts are freed with zero accounting impact — early capture cannot under-count resInUse. See research.md's corresponding Research Log entry for the full trace.

Error Handling

Error Strategy

• No matching recorded decision this round: not an error — job->accounts/job->allocs stay NULL, a TRACE line is logged, the job is re-checked next round (unchanged behavior, now reached by more jobs per round since the gate widened).
• No recorded origin anywhere in the replay (Requirement 4.4): the same "not found" path applies every round for the rest of the replay; the scheduler continues processing without terminating, and that job's resource usage is reported as unavailable (never fabricated) wherever it is inspected (e.g. info worker).
• decoded_session decode failure: unchanged — already handled upstream of this function (sim_loop.c:874-879) by skipping verification for that round; this feature does not alter that path.

Testing Strategy

Unit Tests (test_sim_reconcile_unaccounted.c)

• Replace test_reconcile_skips_non_run_state_job with test_reconcile_captures_non_run_state_job_when_dispatch_recorded — a JOB_STAT_PEND job with a matching recorded DISPATCH entry now gets job->allocs/job->accounts captured and the entry consumed (Requirement 1.1, 3.1).
• New: test_reconcile_still_skips_already_captured_job_regardless_of_state — a job with job->accounts already set (any state) is never re-searched, proving the state-widened gate did not weaken the existing no-double-capture guard (Requirement 3.2).
• New: test_reconcile_no_match_leaves_pend_job_uncaptured_and_retries — a PEND job with no matching entry this round stays uncaptured and is eligible again next round (Requirement 4.4, state-independent retry).
• New: test_reconcile_multiple_jobs_captured_independently_across_different_rounds — several PEND jobs simultaneously outstanding, each with its own recorded DISPATCH decision spread across different rounds (some matching this round, some not yet recorded, some already captured from an earlier round); assert each job's job->accounts reflects only its own jobid's allocation data, captured at its own correct round, with zero cross-job interference from the wider per-round scan (Requirement 1.1, 3.1 under concurrent load — the single-job tests above cannot distinguish "correct" from "correct by coincidence, only one candidate available").
• Existing test_reconcile_reconstructs_allocs_and_accounts_from_matching_dispatch, test_reconcile_ignores_non_dispatch_entry_with_matching_jobid, test_reconcile_compacts_via_shift_down_preserving_order, test_reconcile_is_safe_to_recheck_on_a_later_round, test_reconcile_then_release_via_job_state_change_is_asan_clean, test_reconcile_null_args_are_a_safe_noop — re-run unmodified as regression coverage (Requirement 4.1); no expected behavior change for an already-RUN job.

Integration Tests (round-boundary + charging interaction)

• Revised 2026-09-11: test_captured_pend_job_excluded_from_resinuse_until_run — RENAMED/INVERTED to test_captured_job_included_in_resinuse_same_round_as_capture. Drive sim_reconcile_unaccounted_jobs() for a PEND job with a matching recorded DISPATCH entry, then assert job->state == JOB_STAT_RUN immediately (Requirement 1.2) and, after running sched_account_jobs() in the SAME round, that w->resInUse/core->sharedInUse already reflect it (Requirement 1.3) — NOT deferred to a later round. Add a companion assertion that a later, separately-recorded JOB_STATE_CHANGE-to-RUN event for the same job (applied via the normal XCE_CORE_EVENT_JOB_STATE_CHANGE path) is a safe no-op that does not double-charge or corrupt resInUse (Requirement 1.4).
• New (resource-contention fixture, per scheduler-driver-equivalence.md's explicit discipline): a worker with capacity for N jobs, more than N candidate jobs recorded, dispatch decisions and RUN confirmations separated across multiple rounds — assert the live allocator's own capacity check (sched_check_host_satisfy_) sees correct resInUse starting at the CAPTURE round (revised from "the confirmation round"), not permanently-zero capacity. A no-contention fixture is explicitly insufficient here (scheduler-driver-equivalence.md).
• New: test_sim_verify_no_false_mismatch_across_round_separated_dispatch — a --sim-verify run over a replay where dispatch and RUN confirmation are several rounds apart reports zero mismatches (Requirement 4.2).
• Existing test_repl_equivalence.c-style interactive/batch parity coverage re-run unmodified — this feature's call site is shared between both modes, so no new test is needed to prove Requirement 4.3, only confirmation the existing suite still passes.