myxce/.kiro/specs/scheduler-allocator-runner/research.md

Research & Design Decisions

Summary

• Feature: scheduler-allocator-runner
• Discovery Scope: Extension (light discovery). Refactor of an existing plugin seam inside src/scheduler; no new external dependency.
• Key Findings:
  • The two my_alloc()s differ only in: the plan iterator they wire, parjob's maxTasks <= 1 guard and NULL-input guard, parjob's per-bucket alloc_req.host reset, per-candidate nslots/nhosts assignment, and claim bookkeeping (parjob snapshots each claim's host_req; seqjob keeps a single w and reads the live alloc_req.host at commit). Everything else is duplicated text.
  • xce_alloc_plan_t is an anonymous typedef declared in alloc_iter.h, which itself includes scheduler.h → plugin.h, so the full allocator iface cannot live in plugin.h without either a header cycle or a tag change. The picker iface already solves the same problem with a forward typedef in plugin.h.
  • Three white-box tests call the static my_alloc() / parjob_try_plan_() / parjob_rollback_claims_() directly and stub sched_make_allocs() (bug-1882 OOM injection); none asserts pend-reason content. test_parjob_combinatorial.c passes NULL and a poisoned xce_scheduler_t* as the plan iterator's third argument.

Research Log

Current allocator code paths

• Context: Establish exactly what the runner must reproduce (Req 3, 4, 5).
• Sources Consulted: src/scheduler/src/plugins/seqjob_allocator.c:213-420, parjob_allocator.c:300-760, src/scheduler/src/resource.c:493-620, src/scheduler/src/plugin.c:612-705, src/scheduler/src/host_iter.c:100-170, src/scheduler/src/scheduler.c:235-435.
• Findings:
  • Shared prep is byte-identical in both: lazy make_pendreason(0) → sched_ctx_get_uncomit_shared → sched_check_shared_satisfy → NO_RESOURCE fill → xvec_idval(uncomit, &shared, XVEC_ADD_FLOAT). The fill loop is bounded by alloc_req.shared.count over ctx->unmeet (bug-1893).
  • Plan-init failure returns 0 in both (no fini).
  • parjob: claimed[primary->maxTasks] VLA; per bucket: reset alloc_req.host, host iter, skip claimed, set nslots = bucket_slots[b], nhosts = 1, sched_try_alloc_on_host_() > 0 → snapshot alloc_req.host (OOM → uncommit candidate with live request + rollback earlier claims → return rc).
  • Commit tail: parjob calloc of hallocs, per claim sched_make_allocs(sched, "", nres, idxs, vals, NULL) + make_host_alloc; failure rolls back claims [i, n) and frees hallocs; then make_job_alloc(0, n, hallocs); then the shared-record block. seqjob uses make_host_allocs(1, halloc), which produces the same one-element array.
  • Both set ctx->alloc before building the shared records and free_JobAlloc_ptr(&ctx->alloc) (which NULLs it) on failure.
• Implications: The runner is parjob's algorithm, generalised, plus the shared-prep block; seqjob becomes a degenerate case (1 plan, 1 bucket of 1 slot).

Where nslots/nhosts are consumed

• Context: brief Constraints item 3.
• Findings: Only sched_fill_limit_requests() reads them (plugin.c:682 job-level pass, :699 per-host pass). seqjob sets 1/1 only in its job-level pass (seqjob_allocator.c:92).
• Implications: The runner sets both per candidate from the plan (Req 3.5); seqjob sees 1/1 everywhere.

The per-host request and the host iterator pre-filter

• Context: brief Constraints item 1, Req 3.6/3.7.
• Findings: plugin_prepare_allocation() with ctx->worker set clears and refills alloc_req.host per candidate (plugin.c:639), so the authoritative check always uses a fresh request. sched_host_iter_next() pre-filters against whatever alloc_req.host holds before that refill and skips failures silently. add_job_preason(..., host, res) writes res_based_reasons, never main_reason (lib/preason.c:147); the unsatisfiable cache reads main_reason only (scheduler.c:271); sim parity compares main_reason only (sim_verify.c:126).
• Implications: The unconditional per-bucket reset changes only seqjob's host/resource reasons.

Plugin ABI and header layout

• Context: Req 1.1–1.6, 6.1.
• Sources Consulted: include/plugin.h, include/sched/alloc_iter.h, include/scheduler.h, include/core.h.
• Findings: plugin.h includes only stdint.h, stdlib.h, decision.h, resreq.h and sched/types.h (corrected 2026-09-30 during task 1.2: discovery misread a truncated listing as "no includes"); core.h and scheduler.h include it. alloc_iter.h includes scheduler.h. picker_iface is a forward-declared xce_job_picker_iface_t* pointing at static storage that the registry never frees (plugin.c:111,149). Plugin templates use designated initialisers and are memcpy'd at registration.
• Implications: Forward typedef in plugin.h; full struct and runner prototype in alloc_iter.h; iface storage static const in each plugin file.

Simulation mode

• Context: Req 5.1, 6.3.
• Sources Consulted: src/scheduler/src/sim_alloc.c.
• Findings: sim registers the same plugins through seqjob_allocator_new_plugin() / parjob_allocator_new_plugin() against sim->core and sets plugin->sched to a stand-in wrapper used only by sched_make_allocs(). So plugin->sched is non-NULL in both modes.
• Implications: No sim code change beyond comments; the runner reaches sim through the plugin templates.

Build and test wiring

• Sources Consulted: src/scheduler/meson.build:25-45, testsuite/meson.build:1700-1760, testsuite/test_picker_pendreason_factory_audit.c:60-80.
• Findings: scheduler_srcs lists every source; many test targets link it. test_seqjob_allocator.c, test_seqjob_rewrap_equivalence.c, test_parjob_combinatorial.c white-box #include the allocator .c plus core.c, resource.c, schedobj.c, host_iter.c. The factory audit scans a hard-coded file list.
• Implications: New source goes into scheduler_srcs, into the white-box tests' #include lists, and into the audit's file list.

File size cap

• Findings: src/scheduler/CLAUDE.md: 800-line cap. parjob_allocator.c is 814 lines, scheduler.c 2002 (pre-existing exceptions, not precedent).
• Implications: The runner goes into a new file, not scheduler.c; parjob drops below the cap.

Architecture Pattern Evaluation

Option	Description	Strengths	Risks / Limitations	Notes
Template method via iface (chosen)	Framework runner owns the algorithm; plugin supplies plan iterator + optional accepts	One copy of the algorithm; matches picker-iface precedent; new allocators need only a plan iterator	Plugin ABI change	Brief Approach, "Framework runner, lean"
Shared helpers only	Extract helpers, keep each my_alloc()	Smallest diff	Plugins still own the loop; fails item 4	Rejected in brief
Runner + commit hook	Also let plugins override result assembly	Future flexibility	Unused extension point needing its own tests	Rejected in brief

Design Decisions

Decision: Iface lives in alloc_iter.h, forward-declared in plugin.h

• Context: Req 1.2, 1.6; plugin.h cannot see plan-iterator types.
• Alternatives Considered:
  1. Full struct in plugin.h (needs a named tag on xce_alloc_plan_t and forward declarations of resreq types).
  2. Forward typedef in plugin.h, full struct in alloc_iter.h (picker pattern).
• Selected Approach: 2.
• Rationale: No header cycle, no change to xce_alloc_plan_t; alloc_iter.h is already the allocation-contract header both plugins include.
• Trade-offs: The iface is not visible from plugin.h alone; plugin authors include sched/alloc_iter.h, which they already do.

Decision: Claims buffer grows with the largest plan, not bound by maxTasks

• Context: Req 3.8, 5.3; brief item 5 (no VLA). seqjob accepts any maxTasks (0 reaches it from direct callers), so maxTasks is not a safe bound.
• Selected Approach: A heap buffer owned by the attempt, grown with realloc to plan->nbuckets before trying a plan that needs more entries; freed on every exit.
• Trade-offs: One extra error path (-ENOMEM on growth, taken between plans with no claims outstanding); no stack-size risk for large task counts.

Decision: Result built into a local, published to ctx->alloc last

• Context: Req 4.4, 5.4, 5.6, bug-1882.
• Selected Approach: sched_finalize_job_alloc_() builds the xce_job_alloc_t including shared records in a local and assigns ctx->alloc only on success.
• Rationale: Today's code sets ctx->alloc early and relies on free_JobAlloc_ptr NULLing it; building locally makes Req 5.6 structural. Observable result is identical.

Decision: Staged migration

• Selected Approach: (1) retype plan-iterator init; (2) characterization tests that drive each plugin through plugin->job_alloc_fn against the current code; (3) runner + iface, parjob switched; (4) seqjob switched; (5) closure (comments, audit list, item 4). Each stage leaves the suite green.
• Rationale: Characterization tests written through job_alloc_fn pass before and after the switch, giving Req 7.2's equivalence evidence without keeping the old code.

Synthesis outcomes

• Generalization: seqjob is parjob's algorithm with one plan of one one-slot bucket; the runner is written once for the general case (N plans × M buckets), and seqjob needs no special path.
• Build vs adopt: internal refactor; nothing external applies. The claim/rollback/snapshot helpers are adopted from parjob unchanged in behaviour, only renamed and moved.
• Simplification: no commit hook; accepts is optional (NULL for seqjob); helper functions stay static in the runner file (tests reach them by white-box #include, the project's existing convention) instead of widening the scheduler header.

Decisions added in design review (2026-09-29)

• Snapshot ownership handoff: sched_try_plan_() owns claim snapshots until it returns 1, then sched_finalize_job_alloc_() owns and frees all of them; the runner never frees one. Prevents a double free between the finalize failure path and the runner's cleanup.
• Thread cancellation: plan_next and sched_host_iter_next() are cancellation points; no scheduler code uses pthread_cleanup_push(). Cancel happens only at shutdown, so a mid-attempt cancel keeps abandoning attempt-scoped heap as today (the claims buffer joins that set). No cleanup handlers added.
• Per-bucket reset side effects: seqjob now runs the prepare fan-out on hosts the stale pre-filter used to skip, so the limit plugin's lazy host-limit instantiation (limit.c match_job_host_limit_: obj->hlimits, instantiate_limit, worker->resvals) can happen earlier. Decision-neutral; extra cost per short host per pending seqjob attempt, bounded by the unsatisfiable cache. Recorded in brief item 1.
• Characterization tests reuse the existing plugin_prepare_allocation() stub convention (test_seqjob_allocator.c:84); limit fill stays covered by test_sim_alloc_limit_fanout through the real plugin.c. seqjob -EINVAL fixtures come after the switch (today's code dereferences NULL).
• Retype scope includes test_alloc_plan_iter.c's stub iterator.

Risks & Mitigations

• Unintended behaviour change in seqjob beyond the listed ones — characterization tests written before the switch, run unchanged after; sim parity tests.
• White-box tests fail to link after the move — the runner file is added to each white-box test's #include list in the same task that deletes the plugin code.
• bug-1892 (positional uncommitted read in unsatisfy_) makes some COMPOUND outcomes wrong today — out of scope; COMPOUND characterization tests avoid asserting outcomes that depend on it.
• parjob_allocator.c and seqjob_allocator.c comments reference my_alloc() extensively; stale comments mislead — comment cleanup is its own closure task.

References

• .kiro/specs/scheduler-allocator-runner/brief.md — decisions 1–6.
• .kiro/specs/scheduler-allocation/ — plan/host iterator contracts (upstream).
• .wolf/bugs/bug-1892.md, .wolf/bugs/bug-1893.md.