myxce/.kiro/specs/scheduler-snapshot-job-allocation/tasks.md

Implementation Plan

• 1. Core: independent snapshot-reconstruction primitives

• 1.1 (P) Build a deep-copy primitive for a job's resource allocation

  • Write failing unit tests first (TDD red): single-host/single-resource copy, multi-host copy, multi-resource-per-host copy, shared-resource-pool copy, an independence proof (mutate or free the source after copying and confirm the copy is unaffected), and a fidelity check that the copy's allocation start-time/duration match the source exactly rather than being reset to copy-time.
  • Implement the primitive by composing the existing allocation-construction helpers, explicitly carrying over the source's original start-time/duration rather than accepting a fresh default.
  • On an allocation failure partway through, return no result rather than a partial/inconsistent one, matching this area's existing convention (no forced full-rollback machinery).
  • Observable completion: the new unit test suite goes from failing to passing, and a copy produced from a multi-host, multi-resource, shared-pool source is byte-for-byte equivalent in content to the source while sharing no pointers with it.
  • Requirements: 1.1, 1.4
  • Boundary: xcealloc (job allocation deep-copy)

• 1.2 (P) Construct a valid scheduling representation for every job restored from a snapshot

  • Write a failing test first (TDD red) asserting that a job reconstructed from a snapshot has a non-NULL, correctly-shaped internal scheduling representation for a primary, a compound, and an alternative resource request.
  • Implement construction using only information already available by the time a restored job's resource request has been parsed (its queue, user, project, and resource request) — no new snapshot data is captured or required.
  • Skip construction (leaving today's behavior unchanged) when a restored job's resource request itself failed to parse — do not attempt to fix that separate, pre-existing failure path as part of this task.
  • Observable completion: the new test goes from failing to passing, and a restored job's scheduling representation is structurally identical to what the same job would get if it had just been freshly submitted with the same queue/user/project/resource request.
  • Requirements: 2.1, 2.2, 2.3, 2.4
  • Boundary: snapshot load (scheduling-representation construction)

• 2. Core: wire the allocation restoration into the save and load paths

• 2.1 Populate a job's resource allocation on the save side

  • Write a failing test first (TDD red) asserting that saving a running job with a resource allocation produces a saved representation carrying that allocation.
  • Populate the already-existing allocation slot on the saved job representation using the deep-copy primitive from 1.1 whenever the live job has an allocation; leave it absent otherwise, matching how every other optional value here is already handled.
  • Observable completion: the new test goes from failing to passing, and saving a job with a multi-host allocation produces a saved job whose allocation content matches the live job's.
  • Requirements: 1.1
  • Depends: 1.1
  • Boundary: snapshot save (allocation population)
  • Note: not marked parallel with 2.2 — both tasks' tests land in the same test file (test_snapshot_save_load.c), a shared-file conflict even though the production code they touch is disjoint.

• 2.2 Restore a job's resource allocation and accounting on the load side

  • Write a failing test first (TDD red) asserting that loading a saved job that carries a resource allocation reconstructs both the job's allocation and its per-host/per-resource accounting.
  • Restore the allocation via the deep-copy primitive from 1.1 (never sharing the saved representation's own copy), then rebuild the job's accounting from the restored allocation using the existing accounting constructor.
  • Position this restoration immediately after the scheduling-representation construction from 1.2 within the same job-reconstruction step, before the job becomes visible to the rest of the scheduler.
  • Treat a job with no saved allocation data as the normal case (no error, no warning) — leave its allocation and accounting unset, identical to today's behavior.
  • Observable completion: the new test goes from failing to passing, and loading a saved job with a multi-host, multi-resource allocation produces a restored job whose worker and shared resource usage accounting immediately reflects that allocation, with no further activity required.
  • Requirements: 1.1, 1.2, 1.3, 1.4, 3.5
  • Depends: 1.1, 1.2
  • Boundary: snapshot load (allocation and accounting restoration)

• 3. Integration: prove the save/load round trip end to end

• 3.1 Verify a full cross-instance save/load round trip preserves allocation, accounting, and scheduling representation together

  • Save a scheduler instance containing a mix of running jobs (spanning multiple hosts, multiple resources, and a shared-resource-pool allocation) and pending jobs, then load that saved state into a second, independent scheduler instance.
  • Assert every running job's allocation and accounting match the source instance, every job (running or pending) has a valid scheduling representation, and that ending either instance's job list afterward does not corrupt or crash the other.
  • Observable completion: the round-trip test passes, demonstrating the restored instance's resource-in-use totals exactly match what the source instance reported before saving.
  • Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 2.3, 2.4
  • Depends: 1.2, 2.1, 2.2

• 3.2 Verify a snapshot with no allocation data loads cleanly

  • Extend the round-trip coverage with a saved state containing no allocation data for its jobs (the natural case for a job that was never dispatched, or a save taken before this feature existed).
  • Assert the load succeeds with no error, and every such job's resource contribution reads as zero/unavailable, identical to today's behavior.
  • Observable completion: the backward-compatibility test passes without any special-casing visible to the caller.
  • Requirements: 3.5
  • Depends: 2.2

• 4. Validation: close the reported symptom, prove no regressions, and keep documentation current

• 4.1 (P) Prove a restored pending job with a valid resource request can be evaluated for allocation without crashing

  • Load a saved state containing a pending job whose resource request parses successfully, then run that job through the scheduler's normal allocation-evaluation path.
  • Assert no crash and no undefined behavior — this is the direct regression test for the scheduling-representation gap this feature closes. This does not cover a pending job whose resource request itself fails to parse on load — that pre-existing, separate failure path is explicitly out of this feature's boundary (see design.md's Known Limitation) and is not exercised by this test.
  • Observable completion: the test passes cleanly under the project's memory-safety build configuration.
  • Requirements: 2.2, 2.4
  • Depends: 1.2
  • Boundary: snapshot load (scheduling-representation construction), test_snapshot_save_load.c

• 4.2 (P) Reproduce and close the originally-reported resource-accounting symptom end to end

  • Extend the existing simulation-mode scenario that already reproduces the reported symptom (a saved state with many running jobs, replayed forward) to assert that every running job's resource contribution is correctly reflected immediately after loading, not only the jobs whose dispatch also happens to appear later in the replay.
  • Observable completion: the extended end-to-end scenario passes, showing 100% of the saved running jobs' resource contribution accounted for immediately on load.
  • Requirements: 1.1, 1.2, 1.3
  • Depends: 3.1
  • Boundary: test_sim_e2e_original_motivation.c

• 4.3 (P) Confirm no regression to existing reconciliation and the rest of the snapshot format

  • Run the existing replay-based reconciliation test coverage and the full existing test suite unchanged; confirm they all still pass exactly as before this feature.
  • Confirm no other previously-supported snapshot field's save/load behavior changed.
  • Observable completion: a full test run shows zero new failures and zero behavior change outside the allocation/scheduling-representation fields this feature touches.
  • Requirements: 3.1, 3.4
  • Depends: 2.1, 2.2
  • Boundary: existing test suite (read-only run, no source changes)

• 4.4 (P) Keep the snapshot format's own scope-decision documentation current

  • Correct the existing source-level note that a job's resource allocation "is not captured" to reflect that it now is, and add a note documenting the job's latest resource-usage report as a known, still-uncaptured gap requiring a future format change.
  • Observable completion: the source-level documentation and the feature's actual behavior no longer contradict each other.
  • Requirements: 3.2, 3.3
  • Depends: 2.2
  • Boundary: snapshot.c (scope-decision header comment only)

• 4.5 Run the project's scheduler design-invariant audit as final validation

  • Run the scheduler component's field-completeness invariant audit dimension (or manually re-check it) against the finished implementation.
  • Confirm the audit finds no undocumented gap introduced or left behind by this feature.
  • Observable completion: the audit dimension reports clean, with this feature's fields correctly classified as fixed or documented.
  • Requirements: 3.2, 3.3, 3.4
  • Depends: 4.1, 4.2, 4.3, 4.4

Implementation Notes

• Task 4.4 follow-up (non-blocking, flag for 4.5): snapshot.c's "internal_state contents" comment section (~lines 217-220, distinct from the "Scope decisions" section this task corrected) still cites the now-stale premise "this task also does not restore job->allocs/accounts (see 'Scope decisions' below)" as part of the rationale for why sharedInUse/resInUse aren't serialized. The exclusion conclusion itself still holds (session-transient state, re-summed every session by sched_account_jobs()), but the cited premise contradicts the corrected "Scope decisions" bullet below it. Out of task 4.4's boundary (scoped strictly to the "Scope decisions" header comment); confirmed by two independent readings (implementer + reviewer). Should be corrected — drop/rephrase the stale clause, keep the still-valid rationale — before task 4.5's field-completeness audit signs off clean.
• Task 4.2 side-finding (not fixed here, out of boundary): core_upsert_queue()'s existing-queue branch (core_events.c:335-336) calls FREEUP(q->resreq) (plain free()) on a resreq that was set via the rtti-backed makeString() (mismatched allocator/free, same bug class as already-fixed bug-1503/bug-1600). Reproducible via any second QUEUE_UPDATE for a queue whose resreq was ever populated via makeString() — not specific to snapshot restoration; the near-identical pattern already exists in the legacy sched_update_existing_queue() (schedobj.c:573-575). Confirmed real by two independent readings (implementer + reviewer). Needs its own buglog.json entry and likely a separate follow-up task — tracked here so it isn't lost.