myxce/.kiro/specs/scheduler-sim-eager-reconciliation/tasks.md

Implementation Plan

• 1. Core: widen reconciliation's capture eligibility to any not-yet-captured job

• 1.1 Prove and implement state-independent capture of a job's recorded dispatch allocation

  • Replace the existing test that locks in the bug (asserting a not-yet-running job must never be reconciled) with one asserting the opposite: a job whose recorded dispatch decision is found this round gets its allocation captured regardless of whether it has reached a running state yet
  • Run the new/updated test against the current code first to confirm it fails for the expected reason (the old state-gated behavior), then remove the state precondition from the capture-eligibility check so only "not yet captured" remains
  • Update the function's own explanatory comments so they describe the new state-independent trigger instead of the old state-gated one
  • Observable: the updated unit test suite shows the not-yet-running job's allocation is captured and its recorded entry consumed, where it previously stayed uncaptured
  • Requirements: 1.1, 3.1
  • Boundary: sim_reconcile_unaccounted_jobs

• 1.2 Extend unit coverage for capture-safety guarantees under the widened eligibility

  • Add a test proving a job whose allocation was already captured is never re-searched or re-captured on a later round, regardless of its state
  • Add a test proving a job with no matching recorded decision this round stays uncaptured and remains eligible to be found on a later round
  • Add a test with several simultaneously outstanding jobs whose recorded dispatch decisions land in different rounds, proving each job's captured allocation reflects only its own recorded decision with no cross-job interference
  • Observable: the unit test suite demonstrates no-double-capture, retry-on-no-match, and per-job isolation all hold under the widened eligibility check
  • Requirements: 3.1, 3.2, 4.1, 4.4
  • Boundary: sim_reconcile_unaccounted_jobs

• 2. Integration: prove captured-but-not-running allocations stay excluded from resource-in-use accounting until the job actually runs

• 2.1 Verify resource-in-use accounting defers to the job's real running state, not its capture time

  • Drive a captured, not-yet-running job's allocation through the resource-in-use recomputation and confirm it contributes nothing to worker or shared resource usage
  • Transition that same job to a running state and drive resource-in-use recomputation again, confirming its allocation now contributes from that point forward
  • Observable: a test demonstrates the same captured job's resource usage is zero before its running-state transition and correctly non-zero after, using the exact allocation captured earlier
  • Requirements: 1.2, 1.3, 1.4
  • Boundary: sim_reconcile_unaccounted_jobs, sched_account_jobs
  • Depends: 1.1

• 3. Validation: end-to-end proof under contention, decision-verification mode, and full regression

• 3.1 (P) Verify correct capacity accounting under resource contention with realistic dispatch-to-confirmation separation

  • Build a replay fixture where a worker's capacity is insufficient for every candidate job, with dispatch decisions and their running-state confirmations recorded several rounds apart
  • Confirm the live capacity check sees the correct, non-zero resource-in-use total at each round rather than treating capacity as permanently free
  • Observable: a resource-contention test demonstrates the capacity check's view of resource usage matches what actually happened in the original recording at each round, not a permanently-zero reading
  • Requirements: 1.1, 1.2, 1.3, 1.4
  • Boundary: sim_reconcile_unaccounted_jobs, sched_account_jobs
  • Depends: 1.1

• 3.2 (P) Verify decision-verification mode reports no new mismatch across a round-separated dispatch

  • Run decision-verification mode over a replay where a job's dispatch decision and its running-state confirmation are recorded several rounds apart
  • Confirm the verification pass reports zero mismatches for that job's decision
  • Observable: a decision-verification test run over a multi-round-separated dispatch scenario reports zero mismatches
  • Requirements: 2.1, 2.2, 2.3, 4.2
  • Boundary: sim_reconcile_unaccounted_jobs
  • Depends: 1.1

• 3.3 Confirm full regression: existing reconciliation, interactive/batch parity, and the full scheduler suite are unaffected

  • Re-run the existing reconciliation unit tests that were not modified by this feature and confirm their already-running-job scenarios still pass unchanged
  • Re-run the existing interactive-versus-batch simulation parity coverage and confirm it still passes unchanged
  • Run the full scheduler test suite and confirm no new failures beyond any already-tracked, unrelated pre-existing ones
  • Observable: a full scheduler test suite run shows the same pass/fail baseline as before this feature, with only the new tests added by this feature as net-new coverage
  • Requirements: 3.3, 4.1, 4.3, 4.5
  • Depends: 3.1, 3.2

• 4. Revision (2026-09-11, user-directed): align captured job's job->state with normal-mode's synchronous dispatch-commit semantics

• 4.1 Set job->state = JOB_STAT_RUN synchronously at capture, mirroring sched_commit_decision() (bug-1624)

  • Add job->state = JOB_STAT_RUN; to sim_reconcile_unaccounted_jobs()'s FOUND branch (sim_loop.c), in the same place job->allocs/job->accounts are set, so a captured job is RUN from the scheduler's own perspective starting the SAME round as capture -- not deferred to whenever its separately-recorded JOB_STATE_CHANGE confirmation event later replays
  • Update the function's own header/doc comments to describe the new synchronous state transition alongside the existing state-independent capture-eligibility description
  • Observable: info jobs/info worker show a job as RUN in the same session its [sim] DISPATCH line prints, matching what a live scheduler process's own job->state field would have shown at that point
  • Requirements: 1.2 (revised), 1.3 (revised)
  • Boundary: sim_reconcile_unaccounted_jobs

• 4.2 Update/replace the test(s) that pinned the deferred-until-confirmation behavior as correct

  • Rename/invert test_captured_pend_job_excluded_from_resinuse_until_run to test_captured_job_included_in_resinuse_same_round_as_capture: assert job->state == JOB_STAT_RUN immediately after capture, and resInUse reflects the charge after the SAME round's sched_account_jobs() call, not a later one
  • Add a companion assertion that a later, separately-recorded JOB_STATE_CHANGE-to-RUN event replayed for the same (already-RUN) job is a safe idempotent no-op -- no double-charge, no crash, no regressed terminal-state guard
  • Confirm RED first (temporarily revert the job->state = JOB_STAT_RUN line and rebuild) before confirming GREEN with it restored
  • Observable: the updated/new tests fail without the job->state = JOB_STAT_RUN line and pass with it; full scheduler suite shows no new regressions
  • Requirements: 1.2 (revised), 1.3 (revised), 1.4 (new)
  • Boundary: sim_reconcile_unaccounted_jobs, sched_account_jobs, core_events.c (verification only, no code change expected there)
  • Depends: 4.1

Implementation Notes

• Task 1.2: the pre-existing test_recordstart_symlink_escape_replies_error failure (bug-1560, test_admin_request.c) is unrelated to this spec and appears in every task's full-suite run — do not treat it as a regression.
• Task 3.1: any NEW xce_sched_sim_driver_t fixture passed to sched_sim_driver_run() in a scheduler test file MUST be declared static, not stack-local — sched_sim_driver_run() spawns a picker thread that is never joined and keeps reading the struct for the life of the process, so a stack-local instance is a latent stack-use-after-return once the test function returns (already-documented class, bug-1598/bug-1632; this spec's own task 3.1 round-1 review caught a live recurrence). sched_sim_driver_start() unconditionally memset()s the struct first, so static storage is safe as long as each fixture is written by exactly one call site. Caveat (task 3.2): this static fix is NOT safe for a helper called multiple times per process (e.g. run_sim_and_capture_stdout()) — an earlier call's never-joined zombie picker thread could still be alive when a later call's memset() reuses the same static storage. Leave multiply-invoked fixtures stack-local; only single-invocation fixtures should convert.
• Task 3.2: sim mode's live picker (sim_picker_thread_fn + builtin_fcfs) never calls on_dispatch() — it unconditionally PENDs every candidate, including a job whose allocation sim_reconcile_unaccounted_jobs() already captured. Such a job stays in the picker's pendJobs and gets re-PENDed every round until its recorded JOB_STATE_CHANGE event actually removes it. Any --sim-verify/decision-stream fixture spanning a captured-but-not-yet-RUN job's intervening rounds must include a matching PEND(jobid) entry at each of those rounds, or the fixture will produce a false mismatch (or mask a real one) unrelated to the behavior under test.