myxce/.kiro/specs/scheduler-allocator-runner/tasks.md

Implementation Plan

• 1. Foundation: plan-iterator contract and allocator contract

• 1.1 Retype the plan generator's initialisation to take the scheduling core

  • Change the plan-iterator initialisation parameter from the scheduler object to the scheduling core in the shared contract, in both built-in plan generators, in both allocators' current calls that initialise their plan generator (they pass the scheduling core instead of the scheduler object), and in the stub generator used by the plan-iterator contract tests
  • Update the parjob combinatorial tests' direct initialisation calls; turn the poisoned-scheduler case into a poisoned-core case that shows the plan sequence is identical with no core and with a poisoned core
  • Done when the full scheduler suite builds and passes under ASAN with no plan generator taking a scheduler object, and the poisoned-core test shows identical plan sequences for parjob
  • Requirements: 6.1, 6.2

• 1.2 Declare the allocator plugin contract and the plugin's reference to it

  • Add the allocator contract (optional acceptance guard plus plan initialise / next / finish) next to the plan-iterator contract, and declare the framework allocation entry point there
  • Add the plugin's reference to its allocator contract, documented with the same ownership rule as the job-picker contract: static storage in the plugin file, never freed by the registry, NULL for non-allocator plugins
  • Done when the scheduler and its suite build and pass unchanged with the new declarations present and no plugin using them yet
  • Requirements: 1.2, 1.6

• 2. Characterization tests against the current allocators

• 2.1 Characterize the sequential-job allocator through its registered entry point

  • New test target that drives the real seqjob plugin through its job-allocation entry point, using the existing prepare-allocation stub convention and an allocation-record stub with out-of-memory injection
  • Set up the out-of-memory injection that 2.2 reuses, using only injection points that stay the same across the switch: the existing allocation-record stub failing on its N-th call (per-host records in claim order, then shared records) and link-time wrapping (the testsuite's existing wrap precedent) of the libxce host-allocation and job-allocation constructors with fail-on-call-N counters; the host-array allocation is not injected here because its origin changes at the switch (seqjob's libxce helper and parjob's in-file allocation both become a runner allocation) and is covered by 3.3
  • Fixtures: success; pending on a shared resource where the job requests exactly the shared resources that are short (so bug-1893's extra reasons cannot appear); pending on hosts with the pre-filter state arranged so no host is skipped on stale state (assert host reasons only where 3.7 does not change them, and the main reason); out of memory at the host allocation, the job allocation and the shared records (today's result code and allocation state; for the host-allocation case the per-host debit is asserted separately so 4.2 can change it); task counts 0 and 2 reaching the plugin directly; the second primary of a COMPOUND job with non-overlapping resources (bug-1892 not exercised)
  • Each fixture asserts result code, host count and names, per-host resource entries and values, shared records, per-host and shared uncommitted amounts, and pend reasons
  • Done when the new target passes against the unchanged seqjob allocator
  • Requirements: 4.5, 5.4, 5.8, 7.2

• 2.2 Characterize the parallel-job allocator through its registered entry point

  • New test target for the real parjob plugin with the same stubs, injection mechanism and assertions as 2.1
  • Fixtures: multi-bucket success; pending on a shared resource requesting exactly the short resources; pending on hosts; mid-plan rollback where a later bucket fails and a later plan succeeds; task counts 0 and 1 (pending, no pend-reason record); missing core, scheduler back-reference or primary request (invalid-argument, no pend-reason record); out of memory at the allocation records of host 0 and of a later host, at host allocation 0 and at a later host allocation, at the job allocation and at the shared records; the second primary of a COMPOUND job with non-overlapping resources
  • Done when the new target passes against the unchanged parjob allocator
  • Requirements: 4.5, 5.1, 5.2, 5.4, 5.8, 7.2

• 3. Framework allocation routine

• 3.1 Build the routine's guards and shared-resource precheck, test first

  • Add the new runner source to the scheduler build and to the pend-reason factory audit's scanned files, and add a runner test target that uses a scripted fake allocator contract
  • Missing plugin, context, core, scheduler back-reference, primary request, contract or plan functions returns invalid-argument without creating a pend-reason record
  • A rejecting acceptance guard returns pending with no pend-reason record, no resource check and no plan initialisation; an absent guard accepts
  • The pend-reason record is created only after the guards; the shared check adds exactly one host-less "no resource" reason per unmet entry (bug-1893 fixture: two shared resources, one short); shared amounts are counted as uncommitted when met and are not released when the attempt then pends; each failure returns its own error
  • Done when the runner tests for guards and shared precheck pass under ASAN and the factory audit passes with the runner scanned
  • Depends: 1.1, 1.2
  • Requirements: 1.3, 1.4, 2.1, 2.2, 2.3, 2.4, 5.1, 7.2, 7.3

• 3.2 Build the plan and claim search with rollback, test first

  • Plans are tried in generator order and the first fully satisfied plan wins; each bucket claims the first unclaimed satisfying host in host-iteration order
  • Shared amounts are counted as uncommitted exactly once across an attempt that tries several plans
  • The per-host request is empty before every candidate host is pre-filtered (cleared at the start of each bucket and after every rejected candidate), so every selection-matching unclaimed short host gets its host/resource reasons; before each candidate the slot count is the bucket's task count and the host count is 1
  • A failing bucket releases every claim of the plan; a claim-record (snapshot) failure releases the candidate and earlier claims and returns the error; a plan with no buckets or a zero-slot bucket is skipped
  • Claims-list growth failure is injected with a call-counting replacement of the heap allocator applied only to the white-box-included runner source (reused by 3.3); snapshot failure through a new link-time wrap of the libxce vector-copy function (same precedent) that is armed to fail only its next call, because other libxce code also calls it
  • The claims list grows to the largest plan with no fixed limit; plan-generator initialisation failure returns pending; every return path after initialisation finishes the generator and frees the claims list
  • Done when runner tests covering order, single shared count, distinct hosts, rollback debits, empty-request-before-every-candidate with complete reasons, slot/host counts, growth, malformed plans and snapshot failure pass under ASAN
  • Requirements: 2.3, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 5.5, 5.7, 7.2

• 3.3 Build result assembly and its out-of-memory paths, test first

  • One host allocation per claim in claim order from the claimed host's name and its request snapshot; one shared record per shared-request entry, none when empty; job identifier 0, empty reservation name, no duration
  • The result is built privately and published to the attempt only on success, which returns a positive result
  • Out of memory at the host array, any host allocation, the job allocation or the shared records returns out-of-memory, leaves no allocation on the attempt, frees every partly built piece and releases the debits of claims not yet turned into a host allocation; claim snapshots are freed exactly once on every path
  • Out-of-memory steps are injected with the constructor wrapping and record stub from 2.1, plus a call-counting replacement of the heap allocator applied only to the white-box-included runner source (host array and claims list), so the host-array step Requirement 7 AC 2 needs is covered here
  • Done when runner tests for success and each out-of-memory step pass under ASAN with no leak or double free
  • Requirements: 4.1, 4.2, 4.3, 4.4, 5.4, 5.5, 5.6, 5.7, 7.2

• 4. Move both allocators onto the framework routine

• 4.1 Switch the parallel-job allocator

  • First rewrite the parjob characterization tests' _today host-reason assertions to the complete per-host reasons of 3.7 (every selection-matching unclaimed short host of each bucket search) and see them fail against the old code
  • Also add the parjob bug-1893 fixture (two shared resources, one short: exactly one host-less reason, naming the short one; today none is recorded) to the parjob characterization tests and see it fail against the old code
  • Give parjob its allocator contract with a task-count acceptance guard (more than one task), register the framework routine as its allocation entry point, and remove its private allocation routine and claim helpers
  • Build the runner into the parjob characterization target (2.2) so it keeps reaching the plugin through its entry point
  • Update the parjob combinatorial tests, the only permitted kind of change: build the runner into the white-box test, give their hand-built plugin structs the parjob allocator contract, and call the framework routine and the framework's claim search and rollback instead of the removed helpers
  • Done when every parjob characterization fixture (including the new one), the parjob combinatorial and end-to-end multi-slot tests, and the full scheduler suite pass under ASAN
  • Depends: 2.2, 3.3
  • Requirements: 1.1, 1.2, 1.5, 1.6, 2.2, 3.6, 3.7, 5.2, 7.1

• 4.2 Switch the sequential-job allocator

  • First add the seqjob intentional-change fixtures to the seqjob characterization tests and see them fail against the old code: a candidate host that the stale per-host request left by the job-level pass used to skip is now checked; complete per-host "no resource" reasons for every selection-matching short host with the main reason unchanged; bug-1893 (the 2.1 fixture that records no shared reason today now expects one host-less reason naming the short resource); release of the claimed host's debit when result building runs out of memory (this rewrites the 2.1 host-allocation out-of-memory fixture's per-host debit assertion to the new expected value)
  • Give seqjob its allocator contract with no acceptance guard, register the framework routine as its allocation entry point, and remove its private allocation routine
  • After the switch, add seqjob missing-input fixtures (invalid-argument, no pend-reason record)
  • Build the runner into the seqjob characterization target (2.1)
  • Update the seqjob allocator and seqjob rewrap-equivalence tests, the only permitted kind of change: build the runner into the white-box tests, give their hand-built plugin structs the seqjob allocator contract, and call the framework routine
  • Done when every seqjob characterization fixture, the seqjob allocator and rewrap-equivalence tests, and the full scheduler suite pass under ASAN
  • Depends: 2.1, 4.1
  • Requirements: 1.1, 1.2, 1.5, 1.6, 3.6, 3.7, 5.1, 5.3, 5.5, 7.1

• 5. Validation and closure

• 5.1 Remove stale references to the removed routines

  • Rewrite comments in both allocator plugins and the simulation allocator bootstrap that describe the removed private routines, so they describe the framework routine
  • Rewrite testsuite comments that describe current behaviour in terms of the removed routines (host-iterator, simulation limit fan-out, election simulation fixture and parjob end-to-end multi-slot tests); comments that record history may keep the old names
  • Done when no scheduler source or test comment presents the removed routines as current code, and the suite still passes
  • Requirements: 1.5

• 5.2 Prove equivalence across modes and close the tracked item

  • Run the full scheduler ASAN suite and confirm the existing allocation, task-count wiring, unsatisfiable-cache dispatch, simulation parity, simulation live-allocation, limit fan-out, election fixture and snapshot-load allocator tests pass with unchanged expectations
  • Run the scheduler invariant audit and confirm no new violations in the allocation path (driver state-equivalence between normal and simulation mode included)
  • Mark issues/new_requirements.md item 4 resolved with a reference to this spec
  • Done when the suite and the audit are clean and item 4 is marked resolved
  • Depends: 4.2, 5.1
  • Requirements: 6.3, 7.1, 7.4, 7.5

Implementation Notes

• Build/test only in the dedicated dir src/build.alloc (bash src/tools/xce-build.sh scheduler asan build.alloc --jobs N --incremental; baseline 90/90). Another session uses src/xce.build.
• Several white-box scheduler test targets run with detect_leaks=0 because their fixtures are not torn down (testsuite/meson.build:1756); a leak summary from running such a binary directly is pre-existing, not a regression. New tests that assert "no leak" (3.3) need leak detection on for their own target.
• bug-1893 (found in 2.1): vget() returns 0 on success, so today's shared-reason loop (rc <= 0 skip) records NO shared reason at all. The runner must skip only on rc < 0. Characterization fixtures record "no shared reason today" in a separate _today test that 4.1/4.2 rewrite.
• xce_sched_core_fini() does not free the string table xce_sched_core_init() creates; leak-checked test targets call xstring_table_fini() in teardown (see test_alloc_equivalence_seqjob.c).
• Per-host reasons (decided 2026-09-30 after 2.2): the runner clears alloc_req.host before every candidate (start of bucket + after each rejected candidate), both allocators, so every selection-matching unclaimed short host gets reasons. Today parjob gives reasons only for a bucket search's first rejected candidate; the 2.2 file keeps those counts in _today tests that 4.1 rewrites.
• test_alloc_equivalence_seqjob.c is at ~794/800 lines: 4.2's new fixtures go in a second file or after trimming.
• 4.1 expectations for the parjob _today host-reason test under the per-candidate clear: two-bucket success 2 reasons (A slots, B slots); pending on hosts 6 (A mem, A slots, B mem, B slots, A mem, B mem); mid-plan rollback 5 slots reasons (A, B, C from plan [4]; B, C from plan [2,2] bucket 2). eq_assert_reason_prefix checks >=, so 4.1 must pin exact counts. The parjob file is at 799/800 lines.
• Req 2.4 error-path test scope (3.1 review): make_pendreason -ENOMEM is tested (--wrap=make_pendreason). The uncommitted-read, shared-check and uncommitted-add error paths are returned correctly but not tested: the read cannot fail past the input guard (sched_ctx_get_uncomit_shared only rejects NULL args), and the other two fail only on a libxce-internal allocation failure (would need malloc-level injection).
• The runner ignores add_job_preason's return value, as the old allocators did (a failed add drops that reason).
• Incremental rebuilds in src/build.alloc can miss a file restored after a temporary RED edit if its mtime ends up older than the object; touch restored files before the GREEN run (seen in 3.1).
• Stale references for 5.1 (from 4.1 review): test_parjob_combinatorial.c lines ~6, 56, 106, 668, 806 and helper drive_my_alloc; scheduler.h:415; sim_alloc.c:48; test_host_iter.c:665; test_parjob_e2e_multislot.c:138 (plus seqjob ones after 4.2).
• Stale references for 5.1 (from 4.2 review): sim_alloc.c:36-175, scheduler.c:166, test names test_my_alloc_* and helper drive_my_alloc in test_seqjob_allocator.c / test_seqjob_rewrap_equivalence.c, and their my_alloc() comments.
• Follow-up (pre-existing gap, 5.2 review): no simulation-mode parity test drives parjob through allocation (sim parity fixtures pin maxTasks=1). Equivalence holds by construction (same factory + runner in both modes); a sim-driver parjob multi-host parity fixture would close the gap.