Rongsong Shen
31ad79d325
xce_sched_allocator_iface_t (optional accepts guard + plan_init/next/fini) and the sched_allocator_run() entry point in sched/alloc_iter.h; the plugin's const allocator_iface reference, forward-declared in plugin.h with the picker_iface ownership rule. No plugin uses them yet. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
109 lines
12 KiB
Markdown
109 lines
12 KiB
Markdown
# Research & Design Decisions
|
||
|
||
## Summary
|
||
- **Feature**: `scheduler-allocator-runner`
|
||
- **Discovery Scope**: Extension (light discovery). Refactor of an existing plugin seam inside `src/scheduler`; no new external dependency.
|
||
- **Key Findings**:
|
||
- The two `my_alloc()`s differ only in: the plan iterator they wire, parjob's `maxTasks <= 1` guard and NULL-input guard, parjob's per-bucket `alloc_req.host` reset, per-candidate `nslots/nhosts` assignment, and claim bookkeeping (parjob snapshots each claim's `host_req`; seqjob keeps a single `w` and reads the live `alloc_req.host` at commit). Everything else is duplicated text.
|
||
- `xce_alloc_plan_t` is an anonymous typedef declared in `alloc_iter.h`, which itself includes `scheduler.h` → `plugin.h`, so the full allocator iface cannot live in `plugin.h` without either a header cycle or a tag change. The picker iface already solves the same problem with a forward typedef in `plugin.h`.
|
||
- Three white-box tests call the static `my_alloc()` / `parjob_try_plan_()` / `parjob_rollback_claims_()` directly and stub `sched_make_allocs()` (bug-1882 OOM injection); none asserts pend-reason content. `test_parjob_combinatorial.c` passes `NULL` and a poisoned `xce_scheduler_t*` as the plan iterator's third argument.
|
||
|
||
## Research Log
|
||
|
||
### Current allocator code paths
|
||
- **Context**: Establish exactly what the runner must reproduce (Req 3, 4, 5).
|
||
- **Sources Consulted**: `src/scheduler/src/plugins/seqjob_allocator.c:213-420`, `parjob_allocator.c:300-760`, `src/scheduler/src/resource.c:493-620`, `src/scheduler/src/plugin.c:612-705`, `src/scheduler/src/host_iter.c:100-170`, `src/scheduler/src/scheduler.c:235-435`.
|
||
- **Findings**:
|
||
- Shared prep is byte-identical in both: lazy `make_pendreason(0)` → `sched_ctx_get_uncomit_shared` → `sched_check_shared_satisfy` → `NO_RESOURCE` fill → `xvec_idval(uncomit, &shared, XVEC_ADD_FLOAT)`. The fill loop is bounded by `alloc_req.shared.count` over `ctx->unmeet` (bug-1893).
|
||
- Plan-init failure returns 0 in both (no `fini`).
|
||
- parjob: `claimed[primary->maxTasks]` VLA; per bucket: reset `alloc_req.host`, host iter, skip claimed, set `nslots = bucket_slots[b]`, `nhosts = 1`, `sched_try_alloc_on_host_() > 0` → snapshot `alloc_req.host` (OOM → uncommit candidate with live request + rollback earlier claims → return rc).
|
||
- Commit tail: parjob `calloc` of `hallocs`, per claim `sched_make_allocs(sched, "", nres, idxs, vals, NULL)` + `make_host_alloc`; failure rolls back claims `[i, n)` and frees `hallocs`; then `make_job_alloc(0, n, hallocs)`; then the shared-record block. seqjob uses `make_host_allocs(1, halloc)`, which produces the same one-element array.
|
||
- Both set `ctx->alloc` before building the shared records and `free_JobAlloc_ptr(&ctx->alloc)` (which NULLs it) on failure.
|
||
- **Implications**: The runner is parjob's algorithm, generalised, plus the shared-prep block; seqjob becomes a degenerate case (1 plan, 1 bucket of 1 slot).
|
||
|
||
### Where `nslots/nhosts` are consumed
|
||
- **Context**: brief Constraints item 3.
|
||
- **Findings**: Only `sched_fill_limit_requests()` reads them (`plugin.c:682` job-level pass, `:699` per-host pass). seqjob sets 1/1 only in its job-level pass (`seqjob_allocator.c:92`).
|
||
- **Implications**: The runner sets both per candidate from the plan (Req 3.5); seqjob sees 1/1 everywhere.
|
||
|
||
### The per-host request and the host iterator pre-filter
|
||
- **Context**: brief Constraints item 1, Req 3.6/3.7.
|
||
- **Findings**: `plugin_prepare_allocation()` with `ctx->worker` set clears and refills `alloc_req.host` per candidate (`plugin.c:639`), so the authoritative check always uses a fresh request. `sched_host_iter_next()` pre-filters against whatever `alloc_req.host` holds before that refill and skips failures silently. `add_job_preason(..., host, res)` writes `res_based_reasons`, never `main_reason` (`lib/preason.c:147`); the unsatisfiable cache reads `main_reason` only (`scheduler.c:271`); sim parity compares `main_reason` only (`sim_verify.c:126`).
|
||
- **Implications**: The unconditional per-bucket reset changes only seqjob's host/resource reasons.
|
||
|
||
### Plugin ABI and header layout
|
||
- **Context**: Req 1.1–1.6, 6.1.
|
||
- **Sources Consulted**: `include/plugin.h`, `include/sched/alloc_iter.h`, `include/scheduler.h`, `include/core.h`.
|
||
- **Findings**: `plugin.h` includes only `stdint.h`, `stdlib.h`, `decision.h`, `resreq.h` and `sched/types.h` (corrected 2026-09-30 during task 1.2: discovery misread a truncated listing as "no includes"); `core.h` and `scheduler.h` include it. `alloc_iter.h` includes `scheduler.h`. `picker_iface` is a forward-declared `xce_job_picker_iface_t*` pointing at static storage that the registry never frees (`plugin.c:111,149`). Plugin templates use designated initialisers and are `memcpy`'d at registration.
|
||
- **Implications**: Forward typedef in `plugin.h`; full struct and runner prototype in `alloc_iter.h`; iface storage `static const` in each plugin file.
|
||
|
||
### Simulation mode
|
||
- **Context**: Req 5.1, 6.3.
|
||
- **Sources Consulted**: `src/scheduler/src/sim_alloc.c`.
|
||
- **Findings**: sim registers the same plugins through `seqjob_allocator_new_plugin()` / `parjob_allocator_new_plugin()` against `sim->core` and sets `plugin->sched` to a stand-in wrapper used only by `sched_make_allocs()`. So `plugin->sched` is non-NULL in both modes.
|
||
- **Implications**: No sim code change beyond comments; the runner reaches sim through the plugin templates.
|
||
|
||
### Build and test wiring
|
||
- **Sources Consulted**: `src/scheduler/meson.build:25-45`, `testsuite/meson.build:1700-1760`, `testsuite/test_picker_pendreason_factory_audit.c:60-80`.
|
||
- **Findings**: `scheduler_srcs` lists every source; many test targets link it. `test_seqjob_allocator.c`, `test_seqjob_rewrap_equivalence.c`, `test_parjob_combinatorial.c` white-box `#include` the allocator `.c` plus `core.c`, `resource.c`, `schedobj.c`, `host_iter.c`. The factory audit scans a hard-coded file list.
|
||
- **Implications**: New source goes into `scheduler_srcs`, into the white-box tests' `#include` lists, and into the audit's file list.
|
||
|
||
### File size cap
|
||
- **Findings**: `src/scheduler/CLAUDE.md`: 800-line cap. `parjob_allocator.c` is 814 lines, `scheduler.c` 2002 (pre-existing exceptions, not precedent).
|
||
- **Implications**: The runner goes into a new file, not `scheduler.c`; parjob drops below the cap.
|
||
|
||
## Architecture Pattern Evaluation
|
||
|
||
| Option | Description | Strengths | Risks / Limitations | Notes |
|
||
|--------|-------------|-----------|---------------------|-------|
|
||
| Template method via iface (chosen) | Framework runner owns the algorithm; plugin supplies plan iterator + optional `accepts` | One copy of the algorithm; matches picker-iface precedent; new allocators need only a plan iterator | Plugin ABI change | Brief Approach, "Framework runner, lean" |
|
||
| Shared helpers only | Extract helpers, keep each `my_alloc()` | Smallest diff | Plugins still own the loop; fails item 4 | Rejected in brief |
|
||
| Runner + `commit` hook | Also let plugins override result assembly | Future flexibility | Unused extension point needing its own tests | Rejected in brief |
|
||
|
||
## Design Decisions
|
||
|
||
### Decision: Iface lives in `alloc_iter.h`, forward-declared in `plugin.h`
|
||
- **Context**: Req 1.2, 1.6; `plugin.h` cannot see plan-iterator types.
|
||
- **Alternatives Considered**:
|
||
1. Full struct in `plugin.h` (needs a named tag on `xce_alloc_plan_t` and forward declarations of resreq types).
|
||
2. Forward typedef in `plugin.h`, full struct in `alloc_iter.h` (picker pattern).
|
||
- **Selected Approach**: 2.
|
||
- **Rationale**: No header cycle, no change to `xce_alloc_plan_t`; `alloc_iter.h` is already the allocation-contract header both plugins include.
|
||
- **Trade-offs**: The iface is not visible from `plugin.h` alone; plugin authors include `sched/alloc_iter.h`, which they already do.
|
||
|
||
### Decision: Claims buffer grows with the largest plan, not bound by `maxTasks`
|
||
- **Context**: Req 3.8, 5.3; brief item 5 (no VLA). seqjob accepts any `maxTasks` (0 reaches it from direct callers), so `maxTasks` is not a safe bound.
|
||
- **Selected Approach**: A heap buffer owned by the attempt, grown with `realloc` to `plan->nbuckets` before trying a plan that needs more entries; freed on every exit.
|
||
- **Trade-offs**: One extra error path (`-ENOMEM` on growth, taken between plans with no claims outstanding); no stack-size risk for large task counts.
|
||
|
||
### Decision: Result built into a local, published to `ctx->alloc` last
|
||
- **Context**: Req 4.4, 5.4, 5.6, bug-1882.
|
||
- **Selected Approach**: `sched_finalize_job_alloc_()` builds the `xce_job_alloc_t` including shared records in a local and assigns `ctx->alloc` only on success.
|
||
- **Rationale**: Today's code sets `ctx->alloc` early and relies on `free_JobAlloc_ptr` NULLing it; building locally makes Req 5.6 structural. Observable result is identical.
|
||
|
||
### Decision: Staged migration
|
||
- **Selected Approach**: (1) retype plan-iterator `init`; (2) characterization tests that drive each plugin through `plugin->job_alloc_fn` against the current code; (3) runner + iface, parjob switched; (4) seqjob switched; (5) closure (comments, audit list, item 4). Each stage leaves the suite green.
|
||
- **Rationale**: Characterization tests written through `job_alloc_fn` pass before and after the switch, giving Req 7.2's equivalence evidence without keeping the old code.
|
||
|
||
### Synthesis outcomes
|
||
- **Generalization**: seqjob is parjob's algorithm with one plan of one one-slot bucket; the runner is written once for the general case (N plans × M buckets), and seqjob needs no special path.
|
||
- **Build vs adopt**: internal refactor; nothing external applies. The claim/rollback/snapshot helpers are adopted from parjob unchanged in behaviour, only renamed and moved.
|
||
- **Simplification**: no `commit` hook; `accepts` is optional (NULL for seqjob); helper functions stay `static` in the runner file (tests reach them by white-box `#include`, the project's existing convention) instead of widening the scheduler header.
|
||
|
||
### Decisions added in design review (2026-09-29)
|
||
- **Snapshot ownership handoff**: `sched_try_plan_()` owns claim snapshots until it returns 1, then `sched_finalize_job_alloc_()` owns and frees all of them; the runner never frees one. Prevents a double free between the finalize failure path and the runner's cleanup.
|
||
- **Thread cancellation**: `plan_next` and `sched_host_iter_next()` are cancellation points; no scheduler code uses `pthread_cleanup_push()`. Cancel happens only at shutdown, so a mid-attempt cancel keeps abandoning attempt-scoped heap as today (the claims buffer joins that set). No cleanup handlers added.
|
||
- **Per-bucket reset side effects**: seqjob now runs the prepare fan-out on hosts the stale pre-filter used to skip, so the limit plugin's lazy host-limit instantiation (`limit.c` `match_job_host_limit_`: `obj->hlimits`, `instantiate_limit`, `worker->resvals`) can happen earlier. Decision-neutral; extra cost per short host per pending seqjob attempt, bounded by the unsatisfiable cache. Recorded in brief item 1.
|
||
- **Characterization tests** reuse the existing `plugin_prepare_allocation()` stub convention (`test_seqjob_allocator.c:84`); limit fill stays covered by `test_sim_alloc_limit_fanout` through the real `plugin.c`. seqjob `-EINVAL` fixtures come after the switch (today's code dereferences NULL).
|
||
- **Retype scope** includes `test_alloc_plan_iter.c`'s stub iterator.
|
||
|
||
## Risks & Mitigations
|
||
- Unintended behaviour change in seqjob beyond the listed ones — characterization tests written before the switch, run unchanged after; sim parity tests.
|
||
- White-box tests fail to link after the move — the runner file is added to each white-box test's `#include` list in the same task that deletes the plugin code.
|
||
- bug-1892 (positional uncommitted read in `unsatisfy_`) makes some COMPOUND outcomes wrong today — out of scope; COMPOUND characterization tests avoid asserting outcomes that depend on it.
|
||
- `parjob_allocator.c` and `seqjob_allocator.c` comments reference `my_alloc()` extensively; stale comments mislead — comment cleanup is its own closure task.
|
||
|
||
## References
|
||
- `.kiro/specs/scheduler-allocator-runner/brief.md` — decisions 1–6.
|
||
- `.kiro/specs/scheduler-allocation/` — plan/host iterator contracts (upstream).
|
||
- `.wolf/bugs/bug-1892.md`, `.wolf/bugs/bug-1893.md`.
|