myxce/.kiro/specs/scheduler-sim-eager-reconciliation/research.md

Research & Design Decisions

Summary

• Feature: scheduler-sim-eager-reconciliation
• Discovery Scope: Extension (widens an existing, already-shipped mechanism's trigger condition; no new subsystem, no external dependency)
• Key Findings:
  • The fix is a single-condition removal in sim_reconcile_unaccounted_jobs()'s trigger gate (sim_loop.c:592), not a new cross-round data structure. The per-round decoded_session decode/reconcile/free cycle already runs every round regardless of any job's state; the only reason cross-round matching fails today is that the gate additionally requires JOB_IS_RUN(job->state) before a job is even considered for a search, so a job still PEND at the round its own DISPATCH decision appears is skipped outright and never re-checked once it later becomes RUN (by then that round's decoded_session is long freed).
  • sched_account_jobs() (the function that actually turns job->accounts into w->resInUse/core->sharedInUse) already independently gates on JOB_IS_RUN(job->state) before charging. This means capturing job->accounts early (while a job is still PEND) is inert until the job's real state later transitions to RUN — the existing charging gate is exactly the state-appropriate deferral this feature needs, and it requires no change.
  • One existing test, test_reconcile_skips_non_run_state_job (test_sim_reconcile_unaccounted.c:507-531), asserts the CURRENT (buggy) behavior by name and message ("a non-RUN-state job must never be reconciled here") — this test encodes the bug and must be replaced (inverted), not left alongside new coverage.

Research Log

Where does resInUse actually go to zero, and what re-populates it?

• Context: bug-1613's buglog root-cause trail (confirmed) plus .kiro/steering/scheduler-driver-equivalence.md.
• Sources Consulted: src/scheduler/src/scheduler.c (sched_account_jobs, sched_commit_decision), src/scheduler/src/resource.c (sched_update_res_accounts, sched_charge_account), .wolf/buglog.json bug-1611/bug-1613 full history.
• Findings: xce_core_begin_session() zeroes resInUse/sharedInUse every round; sched_account_jobs() is the only re-summing mechanism, iterating core->jobs and charging every job that is both JOB_IS_RUN and has job->accounts != NULL. Normal mode's sched_commit_decision() sets job->accounts synchronously at DISPATCH-decision time, independent of job->state — so by the time a job's state reaches RUN, its accounts are already long-set. Simulation mode has no equivalent synchronous path (its own live picker never emits a DISPATCH — confirmed in the bug-1613 trail — it is PEND-only), so job->accounts for a replayed job can only come from sim_reconcile_unaccounted_jobs() reconstructing it from the recorded decision stream.
• Implications: the fix belongs entirely in how/when sim_reconcile_unaccounted_jobs() captures job->accounts — sched_account_jobs() itself is correct today and must not change.

Why does today's reconciliation only work same-round?

• Context: bug-1611's confirmed root-cause note and the scheduler-sim-decision-parity spec's task 4.3 implementation notes (both quoted in .wolf/buglog.json).
• Sources Consulted: src/scheduler/src/sim_loop.c:566-709 (sim_reconcile_unaccounted_jobs, read in full), src/scheduler/src/sim_loop.c:840-960 (round-boundary call sequence, read in full).
• Findings: decoded_session is decoded fresh from the replay stream once per round (decode_ReplaySession_ptr, sim_loop.c:871) and freed at the end of the same round (free_ReplaySession_ptr, sim_loop.c:1006) — it never survives past its own round. sim_reconcile_unaccounted_jobs() is called unconditionally every round (sim_loop.c:948, between xce_core_begin_session() and sched_account_jobs()), but its trigger gate (sim_loop.c:592) is:

  if ((job == NULL) || !JOB_IS_RUN(job->state) || (job->accounts != NULL)) {
      continue;
  }
  

  A job whose DISPATCH decision is recorded at round N but whose RUN-state confirmation is only replayed at round N+k (realistic dispatch-to-confirmation latency) is skipped at round N (state is still PEND) and is never re-considered once it reaches RUN at round N+k, because round N's decoded_session — the only place that DISPATCH decision ever existed — is already gone.
• Implications: no new persistent, cross-round data structure is needed. The function already re-runs every round with a live view of the round's own recorded decisions; it only needs to stop requiring JOB_IS_RUN before searching. Removing that one condition means every not-yet-captured job is (re-)checked against every round's own decoded stream until either found (captured, permanently, via the existing job->accounts != NULL guard) or the replay ends.

Does removing the RUN-state gate risk fabricating or double-capturing?

• Context: Requirement 3's explicit no-fabrication / no-double-capture acceptance criteria.
• Sources Consulted: sim_loop.c:592-693 (full match/capture logic, read in full).
• Findings: The match loop already restricts a capture to the job's own jobid (entry->body.dispatch->alloc->jobid != job->jobid, sim_loop.c:605) — this is state-independent and unaffected by the gate change. The job->accounts != NULL continue-guard (unchanged) already prevents a second capture on any later round, for any job, regardless of state. Widening the state condition therefore only widens which jobs are eligible to be searched for, never what counts as a match or whether a second capture can happen.
• Implications: Requirement 3 (no fabrication, no double-capture) is satisfied by logic that already exists and needs no modification — only new test coverage proving it still holds once the gate widens.

Interaction with --sim-verify

• Context: Requirement 4.2 (--sim-verify must not report a new mismatch caused by this feature).
• Sources Consulted: sim_loop.c:925-1004 (reconciliation call site through the verify positional-walk loop, read in full).
• Findings: Reconciliation runs and compacts decoded_session->decisions before the --sim-verify positional walk (sim_loop.c:970-1004) reads that same array. Today, a job whose confirmation lands in a later round leaves its DISPATCH entry sitting unconsumed in the round-N decoded_session (reconciliation skipped it), which --sim-verify's walk would then see as a recorded entry with no matching live decision — a latent false-mismatch risk in --sim-verify mode that already exists independent of this fix. Widening the gate causes reconciliation to consume that entry at the correct round instead, which removes this latent risk rather than introducing a new one.
• Implications: this fix is verify-neutral-to-positive, not verify-risky. Still needs an explicit test per Requirement 4.2's own wording (a --sim-verify run across a multi-round-separated dispatch must not report a new mismatch).

Does early capture risk an incorrect uncharge when a captured-but-not-yet-RUN job later leaves the job set entirely?

• Context: capturing job->accounts while a job is still PEND (this feature's whole point) means, for the first time, a job can carry non-NULL accounts without ever having been charged. If any cleanup path unconditionally treats "accounts non-NULL" as "this job was charged, so subtract it," early capture would cause resInUse to under-count (a phantom decrement for a charge that was never added) for any captured-but-never-actually-run job (e.g. cancelled while still PEND).
• Sources Consulted: every production (non-test) reference to job->accounts/sched_uncharge_account across src/scheduler/src/*.c, via grep: resource.c:591-602 (sched_uncharge_account/sched_commit_job_alloc definitions), scheduler.c:964-980 (sched_job_state_change), core_events.c:898-921 (XCE_CORE_EVENT_JOB_STATE_CHANGE handler) — the two places a job's job->accounts gets freed outside of normal completion.
• Findings: sched_uncharge_account() has zero production callers anywhere in the scheduler (confirmed by grep across every .c file in src/scheduler/src/ — its only callers are test files exercising it in isolation). Both places job->accounts gets freed on a non-RUN state transition (scheduler.c:974-977, core_events.c:915-918) do exactly one thing: free_job_account(job->accounts); job->accounts = NULL; — memory release only, no resInUse mutation, no call to sched_uncharge_account. Production accounting is entirely a reset-and-resum model, not incremental charge/uncharge: xce_core_begin_session() zeroes resInUse/sharedInUse every round, and sched_account_jobs() re-sums it from scratch by summing every currently JOB_IS_RUN job with non-NULL accounts. A job never counts toward resInUse until a round where it is both RUN and has accounts — regardless of how early those accounts were captured.
• Implications: a job captured while still PEND that later transitions directly to a terminal state (cancelled, failed) without ever passing through RUN is simply never summed by sched_account_jobs() at any round (it never satisfies JOB_IS_RUN), and its job->accounts is freed harmlessly with zero resInUse impact when its terminal state is applied. Early capture cannot cause an under-count. This is not a risk this design needs to mitigate — it is a consequence of an accounting model (reset+resum, no incremental uncharge) that was already correct before this feature and remains correct with a wider capture-eligibility gate.

Does bug-1624's normal-mode fix (job->state = RUN set synchronously at dispatch-commit) already close this gap for current recordings?

• Context: bug-1624 (fixed 2026-08-23, scheduler-dispatch-commit tasks 2.1/3.1) made sched_commit_decision() set job->state = JOB_STAT_RUN synchronously in the same operation as job->accounts/job->allocs, specifically to close a live normal-mode over-commit window. Since this predates the current investigation, it's worth confirming it doesn't also incidentally collapse the round-separation this spec fixes for any recording made after 2026-08-23.
• Sources Consulted: grep -n "job->state\|JOB_STAT_RUN" src/scheduler/src/sim_loop.c (only hit: the trigger gate itself, sim_loop.c:592); src/scheduler/src/scheduler.c:1441-1447 (sched_commit_decision, read in full earlier); .wolf/buglog.json bug-1624's full fix note.
• Findings: sim_reconcile_unaccounted_jobs() never sets job->state anywhere — replay-mode state transitions come exclusively from the separately-recorded XCE_CORE_EVENT_JOB_STATE_CHANGE event stream, applied via core_events.c, independent of whatever the decision stream contains. sched_commit_decision()'s job->state = JOB_STAT_RUN (bug-1624's fix) is a direct in-memory field write in the live scheduler process — not a call into the core's event-push pipeline (xce_core_push_events()) — so it is not, by itself, a recordable core event. The recorded JOB_STATE_CHANGE event a replay later applies still originates from whatever asynchronous signal produces it for recording purposes (the controller's own worker-confirmation path), independent of bug-1624's live in-memory synchronization.
• Implications: bug-1624 fixed a live-mode admission-safety window; it does not change what ends up in a recording's event stream or its round timing. This spec's premise — a recorded DISPATCH decision and its recorded RUN-state-change event can land in different replay rounds — holds for recordings made before and after bug-1624, and is not something a future reader should mistake as already resolved by that unrelated fix.

Existing test locking in the bug

• Context: identifying what existing coverage this feature must invalidate, not just extend.
• Sources Consulted: src/scheduler/testsuite/test_sim_reconcile_unaccounted.c (grepped test names, read test_reconcile_skips_non_run_state_job in full at lines 507-531).
• Findings: test_reconcile_skips_non_run_state_job adds a JOB_STAT_PEND job with a matching recorded DISPATCH entry and asserts job->accounts stays NULL, with the message "a non-RUN-state job must never be reconciled here, regardless of what recorded data exists for it this round." This is the literal bug this spec fixes, encoded as a passing test.
• Implications: this test must be replaced (inverted expectation, renamed), not left in place alongside new coverage — the design's File Structure Plan and the eventual task list must call this out explicitly, or the fix will be blocked by its own existing test suite.

Architecture Pattern Evaluation

Option	Description	Strengths	Risks / Limitations	Notes
Widen existing gate (chosen)	Remove !JOB_IS_RUN(job->state) from sim_reconcile_unaccounted_jobs()'s trigger condition; rely on the function's existing every-round re-invocation and the existing job->accounts != NULL guard for persistence/idempotency	Minimal surface area (one condition), reuses every existing invariant (dedup, ownership transfer, decision reporting) untouched, symmetric with normal mode's own state-independent capture	Every not-yet-captured job is now re-searched against every round's decoded stream until found — O(jobs × rounds) worst case for a job with no recorded origin at all (Req 4.4 case)	Chosen: smallest design satisfying all requirements (design-synthesis Simplification lens)
New persistent cross-round pending-capture table	Maintain a separate jobid -> pending decision map populated across rounds, independent of sim->core.jobs iteration	Could in principle capture before a job even exists in core->jobs	Solves a problem that does not exist here — decoded_session already only ever contains decisions for jobs sim->core.jobs already knows about at that round; adds a new owned lifetime/lookup structure and a new invalidation surface for no behavioral gain	Rejected — violates the Simplification lens; not pursued further

Design Decisions

Decision: Remove the JOB_IS_RUN pre-filter instead of adding a new capture-tracking structure

• Context: bug-1611/1613's residual gap is that reconciliation only matches within the same round a job happens to already be RUN.
• Alternatives Considered:
  1. New persistent jobid -> captured allocation structure spanning rounds, decoupled entirely from sim->core.jobs iteration.
  2. Widen the existing trigger gate on the existing per-round iteration (chosen).
• Selected Approach: Delete !JOB_IS_RUN(job->state) from the trigger condition at sim_loop.c:592; leave every other line of the function (match search, ownership transfer, decision reporting, compaction) unchanged.
• Rationale: The function already iterates sim->core.jobs every round and already searches that round's own decoded stream; the only defect is an overly-narrow precondition on which jobs are eligible. sched_account_jobs()'s own unchanged RUN-state gate already provides the correct "don't charge until actually running" deferral this feature's Requirement 1.2 needs — building a second deferral mechanism would duplicate an already-correct one.
• Trade-offs: A job that never has a recorded origin anywhere in the replay (Requirement 4.4) is now re-searched, and its "not found" TRACE-level log line re-emitted, every round for the rest of the replay instead of never being searched again after one failed same-round attempt. Accepted: TRACE-level (filtered by default log level in normal operation) and structurally required by Requirement 4.4's "shall continue processing... without terminating" wording, which implies retrying, not giving up.
• Follow-up: Verify during implementation that no other caller or test assumes today's "same-round-only" contract as load-bearing behavior beyond the one test identified above.

Risks & Mitigations

• Existing test asserts the bug as correct behavior (test_reconcile_skips_non_run_state_job) — Mitigation: replace it with an inverted-expectation test in the same task that changes the gate, not as a follow-up; do not let old and new expectations coexist even briefly.
• Resource-contention blind spot (scheduler-driver-equivalence.md's own documented testing discipline: a no-contention fixture cannot distinguish "accounting correct" from "accounting wrong but unread") — Mitigation: Testing Strategy must include at least one fixture where a worker's capacity is insufficient for all candidate jobs, so a wrongly-zero resInUse would visibly change a scheduling decision, not just a REPL display value.
• --sim-verify false-mismatch risk already latent today — Mitigation: add an explicit --sim-verify regression test spanning a multi-round dispatch-to-confirmation gap; confirm it reports zero mismatches after the fix (and, ideally, confirm it currently free of new mismatches — the fix is expected to strictly improve this, not just avoid regressing it).
• Log noise from repeated "not found" TRACE lines across many rounds for a job with no recorded origin — Mitigation: accepted as documented trade-off (see Design Decision above); TRACE level already filtered in default operation, not escalated further by this spec.
• Wider per-round search cost: every not-yet-captured job (previously only RUN-state ones) is now linearly searched against that round's decoded decision list every round until captured or the replay ends — a round with a large PEND backlog does more search work than before. Mitigation: the per-job, per-round search cost itself is unchanged (same linear scan the existing mechanism already performs); only the set of jobs subjected to it widens. Replay is a debugging/tuning tool, not a real-time or production hot path, and --sim-speed already exists to control replay pacing — accepted without further mitigation; revisit only if a real replay is observed to become impractically slow.

References

• .kiro/steering/scheduler-driver-equivalence.md — the enforced state-equivalence invariant and its testing discipline this feature closes the remaining gap in.
• .kiro/specs/scheduler-sim-decision-parity/ — the parent spec whose sim_reconcile_unaccounted_jobs() mechanism this feature extends (implementation-complete, not reopened).
• .wolf/buglog.json bug-1611, bug-1613 — full confirmed root-cause and partial-fix history.