Rongsong Shen
d1a3e158ae
Found during task 1.1 implementation: the Data Structure Changes Summary declared next() returning xce_worker_t* (the wire/config type), but the Composition Algorithm pseudocode in the same document already used xce_sched_worker_t* (the scheduler-internal type with resvals/resInUse/host_eval_ctx) for the identical call -- these are genuinely different types in this codebase, confirmed via make_sched_worker()'s xce_worker_t* -> xce_sched_worker_t* conversion. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
854 lines
69 KiB
Markdown
854 lines
69 KiB
Markdown
# Requirements Document
|
||
|
||
## Introduction
|
||
|
||
The XCE scheduler has a plugin architecture (`plugin.h`/`plugin.c`, `xce_sched_plugin_t`) but three gaps keep it from being a real framework. The first gap -- job selection policy is hardcoded (`sched_job_picker()` sets `job_pick_plugin = "xce_sched_fcfs"` as a literal) -- is addressed in the sibling **scheduler-core** spec.
|
||
|
||
This spec closes the remaining two allocation gaps:
|
||
|
||
1. **Parallel job allocation is a stub**: `parjob_allocator.c`'s `my_alloc()` takes `(plugin, ctx)` and immediately `return 0` -- any job with `maxTasks > 1` (routed there by `sched_make_primary_alloc_()` at `scheduler.c:93-103`) produces no allocation at all.
|
||
|
||
2. **Allocation is eager and one-shot**: the sibling `seqjob_allocator.c`'s `my_alloc()` (used when `maxTasks == 1`) calls `sched_collect_workers()` then `xlist_find()` for the first host that satisfies the request and stops -- there is no mechanism to generate alternative host/slot combinations and retry when the first plan fails.
|
||
|
||
This spec introduces an **iterator-driven allocation architecture** with these components:
|
||
|
||
- A lazy `xce_alloc_plan_iter_t` -- candidate plans (each a partition of the job's task count into per-host slot buckets, e.g. `[2,1]` for 3 tasks across 2 hosts), most-preferred first
|
||
- A lazy `xce_host_iter_t` -- candidate hosts for a given plan, filtered by the existing select-expression evaluator
|
||
- A `parjob_allocator` that walks the plan iterator to do real combinatorial multi-slot allocation
|
||
- A session-scoped unsatisfiable-resreq fingerprint cache so a job whose resreq no plan can satisfy doesn't re-pay full allocation cost for every later job with the same fingerprint in the same scheduling session
|
||
- A seqjob_allocator rewrap onto the iterator interfaces as a trivial 1-plan/1-slot case, validating the abstraction before applying it to the complex parjob case
|
||
|
||
> Full discovery context (problem, current state, alternatives considered, approach, boundary, upstream/downstream) is in `.kiro/specs/scheduler-allocation/brief.md`.
|
||
|
||
## Boundary Context
|
||
|
||
- **In scope**: `xce_alloc_plan_iter_t` and `xce_host_iter_t` interfaces (lazy, cancel-safe, composable); `seqjob_allocator` rewrapped onto the iterator interface as a trivial 1-plan/1-host-loop case (behavior unchanged, `maxTasks == 1` only); `parjob_allocator`'s `my_alloc()` implementing real combinatorial plan generation (1host x N, 2hosts x ceil(N/2), ..., Nhosts x 1) for `maxTasks > 1`; `xce_alloc_mode_t` on `xce_sched_ctx_t` for allocation mode (NORMAL/PREEMPT); session-level unsatisfiable-resreq fingerprint cache keyed by `xce_cache_key_t` (O(1) content hash from parsed resreq + identity); unit tests for plan generation, host iteration, cache hit/miss, seqjob rewrap equivalence; integration test for end-to-end multi-slot parallel job allocation.
|
||
|
||
- **Out of scope**: Pluggable job picker interface (`xce_job_picker_iface_t`) -- in **scheduler-core**; `builtin_fcfs` picker -- in **scheduler-core**; configurable picker plugin with per-queue override -- in **scheduler-core**; preemption (Req 6 of scheduler-framework) -- in **scheduler-preemption**; core/driver separation (Req 10 of scheduler-framework) -- in **scheduler-core**; Queue vs Project roles -- in **scheduler-core**; new job selection policies beyond the builtin FCFS picker; a resreq template registry (named/shared resreq objects); changes to the select-expression language or evaluator; controller-side scheduling-state replication / commit-barrier wiring; GPU/topology/affinity-aware allocation (left pluggable via future allocator plugins).
|
||
|
||
- **Adjacent expectations**: depends on the existing plugin framework (`plugin.h`/`plugin.c`), the existing select-expression evaluator (`resreq.h`, memoized via `select_eval_ctx_t`), and existing shared-resource tracking (`xce_scheduler_t.sharedInst`/`sharedInUse`) -- none of these change. Depends on scheduler-core for the core event interface that pushes worker and resource state; the iterators are independently unit-testable with constructed state. `ha-consistency-invariants`' commit barrier is a future consumer of scheduling decisions; this spec does not wire it.
|
||
|
||
## Allocation Architecture (Reqs 2-4)
|
||
|
||
Requirements 2, 3, and 4 form a three-layer allocation architecture. These layers compose according to the job's resource request type (PRIMARY, COMPOUND, or ALTER):
|
||
|
||
### Resreq Type Hierarchy
|
||
|
||
```
|
||
ALTER -> OR-of-compounds: first compound that fully allocates wins
|
||
└─ COMPOUND -> AND-of-primaries: ALL primaries must allocate for success
|
||
└─ PRIMARY -> plan_iter -> host_iter -> try_alloc (leaf level)
|
||
```
|
||
|
||
The iterator composition (`xce_alloc_plan_iter_t` + `xce_host_iter_t`) operates at the **PRIMARY leaf**. COMPOUND and ALTER are composition rules above the iterators -- they call into the same primary allocator, unchanged from today's `sched_make_compound_allocation()` and `sched_make_alter_allocation()` dispatch.
|
||
|
||
### Iterator Composition (PRIMARY leaf)
|
||
|
||
```
|
||
┌──────────────────────────┐
|
||
│ xce_alloc_plan_iter_t │ Req 2
|
||
│ │
|
||
│ init(primary_resreq, sched) │
|
||
│ next() -> plan │ generates per-host bucket
|
||
│ fini() │ partitions, most-preferred first
|
||
└────────────┬───────────────┘
|
||
│ plan
|
||
▼
|
||
┌──────────────────────────┐
|
||
│ xce_host_iter_t │ Req 3
|
||
│ │
|
||
│ init(plan, workers) │ for a given plan, iterate
|
||
│ next() -> worker │ candidate hosts matching
|
||
│ fini() │ select expression + availability
|
||
└────────────┬───────────────┘
|
||
│ worker
|
||
▼
|
||
┌──────────────────────────┐
|
||
│ try_alloc(worker, plan) │ Req 4 (parjob) or
|
||
│ │ existing seqjob logic
|
||
│ check slots, shared res, │
|
||
│ commit or advance │
|
||
└────────────────────────────┘
|
||
```
|
||
|
||
### Composition Algorithm (All Three Types)
|
||
|
||
A plan is a **partition** of the primary's task count into per-host buckets (e.g. for `maxTasks=3`: `[3]`, `[2,1]`, `[1,1,1]`) -- not a single `(nhosts, slots_per_host)` pair, which cannot express uneven splits. `allocate_primary()` must therefore satisfy every bucket in a plan against a **distinct** host before the plan counts as satisfied; satisfying only the first bucket and stopping (as a single-pass `try_alloc_on_host` loop would) is not a valid implementation of Requirement 4 -- it degenerates into the single-host plan and never actually exercises the multi-host case.
|
||
|
||
**Host exclusivity within a plan** is enforced by the caller, not `xce_host_iter_t` itself: `allocate_primary()` skips any host already present in `claimed` before calling `try_alloc_on_host` -- Requirement 3's `init(plan, workers)` signature is unchanged, host_iter stays a pure select+availability filter. This distinction matters: a host *offered and rejected* for one bucket (too small for that bucket's slot count) must remain eligible for a later, smaller bucket in the same plan -- only a host *actually claimed* by a bucket is permanently excluded for the rest of that plan attempt.
|
||
|
||
**Correction (this review): a fresh `host_iter` per bucket, not one shared across the whole plan.** An earlier version of this pseudocode created one `host_iter` before the bucket loop and let every bucket's `while` pull from the same cursor. Since `next()` is a forward-only cursor with no rewind, that silently breaks the "offered-and-rejected stays eligible for a later bucket" guarantee just stated: a host bucket 0 tries and rejects (too small for it) is gone from the cursor forever, even if it exactly satisfies bucket 1. Concretely, for plan `[2,1]` with host A (1 free slot) and host B (2 free slots): bucket 0 ("need 2") pulls A first, rejects it, pulls B, succeeds -- cursor now exhausted; bucket 1 ("need 1") gets nothing back even though A was sitting there the whole time. The fix: `host_iter_init(iter, plan, sched->workers)` is called fresh at the top of **each** bucket iteration (same plan, same host_filter every time -- only `is_claimed()`'s growing exclusion set differs call to call), and `fini()`'d at the end of that same bucket's search. This is a three-level nested loop (plan → bucket/partition → worker, the third level always starting fresh), not a two-level one with an inner index.
|
||
|
||
**Bucket order is not merely a fail-fast heuristic -- for v1's single-resource-dimension model, descending (largest bucket first) can succeed where ascending would fail, with the identical set of available hosts.** Same example: with buckets tried ascending `[1,2]` instead, a fresh scan for "need 1" could greedily claim host B (2 free slots -- trivially satisfies 1, but wastefully overqualified), leaving only A (1 free slot) for "need 2", which fails. Descending order avoids this by construction: the largest ask is checked against the full pool before a smaller ask has a chance to consume a host only the larger one could have used. This is why the v1 default (descending, "Data Structure Changes Summary" below) is a real correctness property here, not just an optimization -- and why generating the reverse order as a *second* candidate plan for the same partition isn't needed on top of it. Which specific host gets picked among *multiple* hosts that satisfy the same bucket (host_iter's own internal candidate order, `HOST_ORDER_ANY`/`HOST_ORDER_MOST_FREE` etc.) is a separate, softer question already deferred as future optimization -- it affects efficiency under adversarial cases, not whether a solvable case gets missed outright the way the shared-cursor bug did.
|
||
|
||
**Rollback scope.** `ctx->uncomitted` (`sched_ctx_init`/`sched_ctx_fini`, `schedobj.c`) is per-job-allocation-attempt scratch state, torn down at the end of one job's `sched_make_primary_allocation()`/`sched_make_compound_allocation()` call -- it never touches the real, session-wide `resInUse` books, which are only charged later, separately, once a job actually dispatches (`job->accounts`/`sched_account_jobs()`). So `rollback_plan_allocation()` can never corrupt cluster-wide resource state. It is still required, though: without it, a bucket that succeeded before a later bucket exhausted the host iterator leaves its provisional debit in `ctx->uncomitted.hosts[idx]` for the rest of that job's allocation attempt -- a *later* plan considering the same host would see it as falsely less available, purely from a phantom debit an earlier, abandoned plan left behind, which could wrongly pend a job a later plan could actually have satisfied. The reversal itself has no existing precedent to reuse (`sched_make_compound_allocation()`'s own on-failure path, `scheduler.c:206-238`, does not actually clean up partial state on a mid-loop failure -- a separate, pre-existing gap outside this spec's scope, tracked separately, not modeled here). The underlying subtract primitive already exists (`xvec_op(v1, v2, XVEC_SUB_FLOAT)`, `xvec.c`); it just needs a small new "uncommit" helper wired to it, mirroring the existing `XVEC_ADD_FLOAT` commit call in `sched_try_alloc_on_host_()`. For this to work, each `claimed[]` entry must retain the exact `alloc_req.host` snapshot captured when that bucket succeeded (needed for the merge step too -- see "Final allocation result" below), since `ctx->alloc_req` is reset and repopulated for every subsequent host tried.
|
||
|
||
**Allocation convention (this review): iterators are stack-allocated by the caller, then initialized in place** -- matching `sched_ctx_init(&ctx, ...)`'s existing convention in this codebase, NOT a heap-allocating factory. `init()`'s own declared signature (`int (*init)(xce_alloc_plan_iter_t* iter, ...)`, "Data Structure Changes Summary" below) takes the iterator's own address as its first argument for exactly this reason. An earlier version of this pseudocode wrote `plan_iter = plan_iter_init(primary_resreq, sched)` as if it were a 2-arg factory function returning a new iterator -- inconsistent with the 3-arg in-place-init method the struct actually declares. Corrected below.
|
||
|
||
```c
|
||
// === PRIMARY leaf: iterator composition ===
|
||
// Returns >0 on success, 0 on failure
|
||
int allocate_primary(sched, primary_resreq, ctx) {
|
||
xce_alloc_plan_iter_t plan_iter_storage;
|
||
xce_alloc_plan_iter_t* plan_iter = &plan_iter_storage;
|
||
if (plan_iter->init(plan_iter, primary_resreq, sched) != 0) {
|
||
return ALLOC_FAIL; // exact error-propagation contract: design.md
|
||
}
|
||
|
||
while ((plan = plan_iter->next(plan_iter)) != NULL) {
|
||
// Correction (kiro-validate-design pass): sized by maxTasks, not an
|
||
// undefined fixed constant -- nbuckets can never exceed maxTasks
|
||
// (each bucket needs >=1 slot), so this is the true, tight upper
|
||
// bound. A fixed compile-time guess risks either overflow for a
|
||
// large parallel job or wasted stack for a small one.
|
||
xce_sched_worker_t* claimed[primary_resreq->maxTasks];
|
||
int nclaimed = 0;
|
||
int plan_ok = 1;
|
||
|
||
// Three levels of iteration: plan (outer while, above) -> bucket/
|
||
// partition (this for) -> worker (host_iter, fresh per bucket, below).
|
||
for (int b = 0; b < plan->nbuckets; b++) {
|
||
// Correction (this review): a FRESH host_iter per bucket, not
|
||
// one shared across the whole plan -- next() is a forward-only
|
||
// cursor, so sharing it would drop any host an earlier, larger
|
||
// bucket tried and rejected, even if it exactly satisfies THIS,
|
||
// smaller bucket. Same plan (same select-expression host_filter)
|
||
// every call; only is_claimed()'s growing exclusion set differs.
|
||
xce_host_iter_t host_iter_storage;
|
||
xce_host_iter_t* host_iter = &host_iter_storage;
|
||
xce_sched_worker_t* w;
|
||
int bucket_ok = 0;
|
||
|
||
if (host_iter->init(host_iter, plan, sched->workers) == 0) {
|
||
while ((w = host_iter->next(host_iter)) != NULL) {
|
||
if (is_claimed(claimed, nclaimed, w)) {
|
||
continue;
|
||
}
|
||
if (try_alloc_on_host(w, plan->bucket_slots[b], ctx) > 0) {
|
||
// snapshot ctx->alloc_req.host here (it is reset on
|
||
// the next plugin_prepare_allocation() cycle) --
|
||
// needed for both rollback below and the merge step
|
||
// in commit_plan_allocation().
|
||
claimed[nclaimed++] = capture_claim(w, ctx);
|
||
bucket_ok = 1;
|
||
break;
|
||
}
|
||
}
|
||
host_iter->fini(host_iter);
|
||
}
|
||
// host_iter->init() failing is treated the same as "no
|
||
// candidates" for this bucket -- an infrastructure error, not a
|
||
// resource-availability determination; see design.md for the
|
||
// exact error-propagation contract.
|
||
|
||
if (!bucket_ok) {
|
||
plan_ok = 0;
|
||
break; // this bucket has no satisfying host -- abandon the plan
|
||
}
|
||
}
|
||
|
||
if (plan_ok) {
|
||
commit_plan_allocation(claimed, nclaimed, ctx);
|
||
plan_iter->fini(plan_iter);
|
||
return ALLOC_SUCCESS;
|
||
}
|
||
|
||
// Partial success within this plan attempt must not leak into the
|
||
// next plan attempt for the SAME job (ctx->uncomitted persists across
|
||
// plan attempts until sched_ctx_fini() -- see "Rollback scope" above):
|
||
// for each claimed host, subtract back exactly the debit
|
||
// sched_try_alloc_on_host_() added via XVEC_ADD_FLOAT, using the
|
||
// snapshot captured above (new xvec_op(..., XVEC_SUB_FLOAT)-based
|
||
// helper -- no existing "uncommit" primitive to call into).
|
||
rollback_plan_allocation(claimed, nclaimed, ctx);
|
||
}
|
||
plan_iter->fini(plan_iter);
|
||
return ALLOC_FAIL;
|
||
}
|
||
|
||
// === COMPOUND: AND of primaries ===
|
||
// ALL primaries must allocate. On any failure, roll back prior allocations
|
||
// and return failure. NOTE: today's sched_make_compound_allocation() does
|
||
// NOT actually do this rollback (see bug-1766, found during this spec's
|
||
// review) -- rollback_compound_allocations() below is new/corrective
|
||
// behavior this spec should introduce, not a port of existing logic. Out of
|
||
// this spec's own boundary per Req 9 AC2 (COMPOUND/ALTER dispatch
|
||
// unchanged) unless the user elects to fold bug-1766's fix in here.
|
||
int allocate_compound(sched, compound_resreq, ctx) {
|
||
for (int i = 0; i < compound_resreq->primaries.count; i++) {
|
||
primary = compound_resreq->primaries[i];
|
||
if (allocate_primary(sched, primary, ctx) <= 0) {
|
||
// Roll back all primaries allocated so far in this compound
|
||
rollback_compound_allocations(ctx, i);
|
||
return ALLOC_FAIL;
|
||
}
|
||
}
|
||
return ALLOC_SUCCESS;
|
||
}
|
||
|
||
// === ALTER: OR of compounds ===
|
||
// First compound that fully allocates wins. Same logic as today's
|
||
// sched_make_alter_allocation(). On failure, no rollback needed
|
||
// (prior compounds were never committed -- they failed their own checks).
|
||
int allocate_alter(sched, alter_resreq, ctx) {
|
||
for (int i = 0; i < alter_resreq->compounds.count; i++) {
|
||
compound = alter_resreq->compounds[i];
|
||
if (allocate_compound(sched, compound, ctx) > 0) {
|
||
return ALLOC_SUCCESS;
|
||
}
|
||
// This compound failed -- advance to next alternative
|
||
}
|
||
return ALLOC_FAIL;
|
||
}
|
||
|
||
// === Top-level dispatch ===
|
||
// Unchanged from today's sched_make_allocation() dispatch on obj->tag.
|
||
int allocate_job(sched, job, ctx) {
|
||
switch (job->resreq_type) {
|
||
case PRIMARY:
|
||
return allocate_primary(sched, job->primary_resreq, ctx);
|
||
case COMPOUND:
|
||
return allocate_compound(sched, job->compound_resreq, ctx);
|
||
case ALTER:
|
||
return allocate_alter(sched, job->alter_resreq, ctx);
|
||
}
|
||
}
|
||
```
|
||
|
||
**COMPOUND rollback note:** If primaries 1 and 2 of a compound allocate but primary 3 fails, primaries 1 and 2 *should* be rolled back -- the compound is meant to be atomic. **Correction (this review): today's `sched_make_compound_allocation()` does not actually do this** -- see bug-1766. It shares one `ctx` across the whole primaries loop, `break`s on the first failing primary without undoing earlier successes, and its unconditional post-loop `if (ctx.alloc != NULL) MOVE_PTR(obj->alloc, ctx.alloc)` can attach a partial, orphaned allocation to a compound job reported as failed. This spec's iterator rework does not touch that path (Req 9 AC2 keeps COMPOUND/ALTER dispatch unchanged) -- the pre-existing gap is tracked separately in bug-1766, not fixed here, unless scope is deliberately expanded to include it.
|
||
|
||
### Who Uses This Composition
|
||
|
||
| Allocator | `maxTasks` | Plan iterator | Host iterator | Behavior |
|
||
|-----------|------------|---------------|---------------|----------|
|
||
| `seqjob_allocator` | `== 1` | Trivial: single plan (1 host x 1 slot) | Full: filters hosts by select expression | Unchanged from today -- first matching host wins |
|
||
| `parjob_allocator` | `> 1` | Full combinatorial: 1host x N, 2hosts x ceil(N/2), ..., Nhosts x 1 | Full: filters hosts with enough available slots | New -- walks plans until one succeeds |
|
||
| Future allocators | any | Custom strategy (e.g., topology-aware, bin-packing) | Reuses same `xce_host_iter_t` | Pluggable via `job_alloc_fn` |
|
||
|
||
### Key Design Properties
|
||
|
||
- **Lazy**: `plan_iter->next()` generates one plan at a time. `host_iter->next()` returns one host at a time. No full materialization.
|
||
- **Cancel-safe**: both `next()` calls have `pthread_cancel` cancellation points (Req 2 AC 6 / Req 3 AC 6 -- a testable requirement, not just a design intent, since neither loop has a natural cancellation point of its own).
|
||
- **Independent lifecycles**: a `host_iter` is created from a plan, used, and destroyed -- and, for a multi-bucket plan, this happens **once per bucket**, not once per plan (a fresh instance each time, per Req 3 AC 3/4). The `plan_iter` is advanced to the next plan only after every bucket's `host_iter` for the current plan is done. `fini()` on one does not affect the other.
|
||
- **COMPOUND/ALTER unchanged**: the composition rules above the PRIMARY leaf (AND for compound, OR for alter) are unchanged from today. The iterator interfaces only change what happens inside `allocate_primary()`.
|
||
- **Seqjob rewrap**: `seqjob_allocator` is rewrapped onto the iterator interfaces as a trivial case -- its plan iterator always returns the single plan (1 host x 1 slot), and the inner loop is identical to today's first-match logic. This validates the abstraction before applying it to the complex parjob case.
|
||
- **Pluggable plan generation, shared host iteration**: plan generation is allocator-specific — each `job_alloc_fn` brings its own strategy (trivial, combinatorial, topology-aware, bin-packing) and creates its own `xce_alloc_plan_iter_t` internally. Host iteration is shared infrastructure — `xce_host_iter_t` has one implementation reused by all allocators, parameterized by sort order (`HOST_ORDER_ANY`, `HOST_ORDER_MOST_FREE`, etc.). Swapping allocation strategy = swapping the plugin; no core loop change.
|
||
|
||
## Requirements
|
||
|
||
### Requirement 2: Lazy allocation plan iterator
|
||
|
||
**Objective:** As a maintainer of the scheduler's allocator plugins, I want a reusable lazy plan-generation abstraction, so that every allocator (current and future) can propose alternative host/slot combinations without materializing the full combinatorial space up front.
|
||
|
||
#### Acceptance Criteria
|
||
|
||
1. The Scheduler shall define an `xce_alloc_plan_iter_t` interface with `init(resreq, sched)`, `next() -> plan`, and `fini()` operations. The `init()` signature passes the full `xce_primary_resreq_t` (including `span`, `same`, `affinity`) and scheduler state — sufficient for any allocator to derive its plan space without future interface changes.
|
||
|
||
**Plan-shaping fields** (on `xce_primary_resreq_t`, consumed by plan iterator internally):
|
||
- `span`: bounds host count (`nhosts_min`/`nhosts_max`) and per-host task cap (`max_per_host`, analogous to LSF `ptile`). v1 combinatorial generation ignores these (empty structs); fill when ptile/nhosts support is needed.
|
||
- `same`: co-location constraints ("all tasks on same rack"). v1 ignores; future.
|
||
- `affinity`: restrict to named host group. v1 ignores; future.
|
||
- `minTasks`/`maxTasks`: task count bounds — the primary driver of plan space. Used in v1.
|
||
- `select`: host filter expression — consumed by host iterator (Req 3), not plan iterator.
|
||
- `rusages`: resource requests — consumed by `try_alloc_on_host`, not plan iterator.
|
||
|
||
v1 implements simple combinatorial generation: `1host×N, 2hosts×ceil(N/2), ..., Nhosts×1`. Future allocators read `span`/`same`/`affinity` to constrain the plan space without touching the `init()` signature.
|
||
|
||
2. Each call to `next()` shall generate exactly one candidate plan -- a partition of `maxTasks` into per-host buckets (`xce_alloc_plan_t.bucket_slots[]`, see "Composition Algorithm" above; NOT a uniform `(nhosts, slots_per_host)` pair, which cannot express uneven splits) -- without generating or materializing any other plan in the same call.
|
||
|
||
3. The Scheduler shall order plans returned by successive `next()` calls from most-preferred to least-preferred, per the owning allocator's preference policy.
|
||
|
||
4. `next()` shall return an end-of-iteration signal (not a plan) once every candidate plan has been produced, and this signal shall be stable across repeated calls after exhaustion.
|
||
|
||
5. Calling `fini()` shall release all resources held by the iterator, and the interface shall define no other required cleanup step.
|
||
|
||
6. `next()` shall call `pthread_testcancel()` (or otherwise provide an equivalent, real cancellation point) at least once per invocation, matching the existing convention `sched_job_picker()`/`sched_allocator()` already establish at the top of their own loops (`scheduler.c`). This is required because, unlike those threads' blocking `cbuf_get()` calls (a POSIX cancellation point via `pthread_cond_wait`), combinatorial plan generation is fast, non-blocking pure computation with no cancellation point of its own -- an implementation satisfying AC 1-5 without this would silently violate the "Cancel-safe" design property stated for this iterator without failing any other criterion.
|
||
|
||
### Requirement 3: Lazy host candidate iterator
|
||
|
||
**Objective:** As a maintainer of the scheduler's allocator plugins, I want a reusable lazy host-candidate abstraction for finding a host satisfying one plan bucket at a time, so that host filtering reuses the existing select-expression evaluator instead of each allocator reimplementing worker collection and filtering. (Correction, this review: NOT "one `xce_host_iter_t` per plan" -- a plan with multiple buckets requires a fresh instance per bucket, per Req 4 AC 2.)
|
||
|
||
#### Acceptance Criteria
|
||
|
||
1. The Scheduler shall define an `xce_host_iter_t` interface with `init(plan, workers)`, `next() -> worker`, and `fini()` operations.
|
||
|
||
2. Each call to `next()` shall return the next candidate host satisfying the given plan's host filter (select expression plus resource availability), evaluated against the existing select-expression evaluator without changes to that evaluator.
|
||
|
||
3. `next()` shall return NULL once no further candidate host exists for the current bucket's search, and the caller shall not need to call `next()` again on that same `host_iter` instance after receiving NULL. This does NOT mean the caller is done with the plan: a plan with more buckets remaining requires a fresh `host_iter` instance (a new `init()` call) for the next bucket, per Req 4 AC 2 -- exhaustion is scoped to one `host_iter` instance, not to the plan as a whole.
|
||
|
||
4. The host iterator shall compose with the plan iterator: a new `xce_host_iter_t` shall be constructible from any plan produced by Requirement 2's `xce_alloc_plan_iter_t.next()` -- and, for a multi-bucket plan, constructible **repeatedly** from the same plan, once per bucket (Req 4 AC 2), each a fully independent instance.
|
||
|
||
5. Calling `fini()` shall release all resources held by the host iterator, independent of the plan iterator's lifecycle.
|
||
|
||
6. `next()` shall call `pthread_testcancel()` (or otherwise provide an equivalent, real cancellation point) at least once per invocation, for the same reason as Requirement 2 AC 6: scanning/filtering candidate hosts is non-blocking pure computation with no cancellation point of its own, so this must be explicit rather than assumed.
|
||
|
||
### Requirement 4: Parallel job combinatorial allocation
|
||
|
||
**Objective:** As a cluster operator submitting a job that needs multiple task slots, I want `parjob_allocator` to actually allocate across hosts, so that jobs with `maxTasks > 1` are no longer silently unallocatable.
|
||
|
||
#### Acceptance Criteria
|
||
|
||
1. When a job's `xce_primary_resreq_t.maxTasks > 1` reaches `parjob_allocator`'s `job_alloc_fn` (`my_alloc()` in `parjob_allocator.c`), the Scheduler shall generate candidate plans covering at minimum the host-count sequence `1..N` hosts, where each plan's bucket sizes are a partition of `maxTasks` into that many buckets (e.g. for `N=3`: `[3]`, `[2,1]`, `[1,1,1]` -- balanced ceil/floor split, not a uniform `slots_per_host` that cannot express uneven splits), using Requirement 2's plan iterator.
|
||
|
||
2. For each candidate plan, the Scheduler shall use Requirement 3's host iterator to find, for every bucket in the plan, a distinct host with enough available slots to satisfy that bucket -- accumulating claimed hosts one bucket at a time -- and shall attempt allocation before advancing to the next plan. A plan is satisfied only when every one of its buckets has a claimed host; if any bucket exhausts the host iterator without a match, the plan fails as a whole and any buckets already matched within that same plan attempt are released before advancing to the next plan (see Composition Algorithm's `rollback_plan_allocation`).
|
||
|
||
**A new `xce_host_iter_t` instance shall be constructed for each bucket** (three levels of iteration: plan -> bucket/partition -> worker, the third always starting fresh) -- never one shared, continuously-advancing instance reused across every bucket of a plan. Reusing one instance is an incorrect implementation: `next()`'s forward-only cursor would permanently drop any host an earlier, larger bucket tried and rejected as too small, even when that same host exactly satisfies a later, smaller bucket -- silently failing plans a correct implementation would have satisfied. Only `is_claimed()` (Criterion 2's own "distinct host" exclusion, applied by the caller across each bucket's fresh scan) may exclude a host across buckets; rejection by an earlier bucket's size check must not.
|
||
|
||
3. When every bucket of a plan is satisfied, the Scheduler shall commit that allocation as a single `xce_job_alloc_t` merging one `xce_host_alloc_t` per claimed bucket/host (`nhosts` == the plan's bucket count; see "Final allocation result" under parjob_allocator changes, below) and shall not evaluate any remaining plan for that job.
|
||
|
||
4. When no plan among those generated in Criterion 1 can be satisfied, `my_alloc()` shall return the same not-satisfied signal (`0`) that the current stub returns today, so pend-reason handling in `sched_make_primary_alloc_()` is unaffected.
|
||
|
||
5. `parjob_allocator`'s `job_alloc_fn` shall no longer unconditionally `return 0` regardless of `ctx` contents -- the current stub behavior -- for any job with `maxTasks > 1`.
|
||
|
||
6. Before each candidate host is checked against a plan's bucket, the Scheduler shall set `ctx->nslots` (and `ctx->nhosts`, for the shared/global-resource pass) to that bucket's slot count so that `parjob_prepare_allocation()` computes the per-host resource want via the existing `calc_wanted_host_resvals(..., ctx->nhosts, ctx->nslots)` call -- the same mechanism `seqjob_prepare_allocation()` already uses, currently hardcoded to `1, 1`. Concurrently, `seqjob_prepare_allocation()` shall gain a `primary->maxTasks == 1` guard (it currently has none, unlike `parjob_prepare_allocation()`, which already gates on `maxTasks`). This guard is required, not optional cleanup: `plugin_prepare_allocation()` (`plugin.c:591`) unconditionally calls every registered plugin's `prepare_allocation_hook`, and `xvec_idval_store()` (`xvec.c:247`) appends `(idx, val)` pairs with no dedup by resource idx -- so once `parjob_prepare_allocation()` stores real values, an unguarded `seqjob_prepare_allocation()` would *also* fire for the same `maxTasks > 1` job and append a second, wrong (`nslots=1`) entry for the same resource idx into `ctx->alloc_req`, corrupting the availability check that Criterion 2 depends on. This is a latent defect in the existing fan-out, invisible today only because `parjob_prepare_allocation()` is currently a no-op.
|
||
|
||
7. Shared-resource satisfaction (`sched_check_shared_satisfy()`, `ctx->alloc_req.shared`) shall be computed and checked once per primary resreq -- independent of which plan or bucket-shape is currently being tried -- exactly as it already is today for `seqjob_allocator`. Plan/bucket iteration (Criteria 1-3) governs only per-host slot allocation; it does not change when or how often the job-level shared-resource check runs.
|
||
|
||
### Requirement 7: Session-level unsatisfiable resreq cache
|
||
|
||
**Objective:** As a maintainer of scheduling-loop performance, I want jobs with a resreq the current session already proved unsatisfiable to skip allocation immediately, so that a batch of similarly-shaped unsatisfiable jobs doesn't each re-pay full plan/host iteration cost.
|
||
|
||
#### Acceptance Criteria
|
||
|
||
1. The Scheduler shall maintain a per-session unsatisfiable-resreq fingerprint cache on `xce_scheduler_t`, keyed by `xce_cache_key_t` — a value type composing `xce_resreq_fingerprint_t` (pre-computed `{minTasks, maxTasks, select_node_id, rusage_hash}`) and identity (`{user, queue, project}` as xstring pointers). The resreq fingerprint is stored on the scheduling object (`xce_primary_schedobj_t`) and computed once on job ingest. The full cache key is constructed on the stack at lookup time and looked up via a single `key_find()` call against a `kvtab_t` bound to a new `cache_key_ops` (`kv_key_ops_t`) — this codebase's real hash-table API binds hash/cmp/assign/destroy to the table at creation, not per call (see "New types" below; no free-function `kvtab_lookup()` taking hash/eq params exists). Lookup itself is O(1): no list walks, no string traversals; `key_find()` does not allocate (only `key_add()`, on insert, allocates the heap-owned key copy). This captures everything the allocator and limit plugin consume.
|
||
|
||
2. The Scheduler shall clear the unsatisfiable-resreq cache at the start of every scheduling session (`begin_session`), by placing the clear inside `core_begin_session_internal()` (`core_session.c`) alongside the existing, unlocked `core_reset_shared_resval()`/`core_reset_workers()` calls -- a core-level function both `driver.c` (normal mode) and `sim_loop.c`/`sim.c` (simulation mode) reach identically through `xce_core_begin_session()`, so no driver-specific clearing code is needed for either mode. No explicit lock is required: this matches the existing, already-safe pattern those two sibling resets use, which relies on `xce_core_begin_session()`'s own documented ordering guarantee -- the driver completes all begin-session reset work strictly before `xce_core_signal_go()` unblocks the picker thread (the one `cbuf_put()` that lets the picker/allocator touch anything for the new session), so the mutex/condvar inside that `cbuf_put()`/`cbuf_get()` pair provides the memory-barrier the reset needs, the same way it already does for worker `resvals`/`sharedInUse`. Neither driver touches the cache object anywhere else -- lookup (AC 4) and insert (AC 3) happen exclusively on the allocator thread, inside `allocate_primary()`.
|
||
|
||
**Not part of the snapshot/replay wire format.** The cache is pure in-memory, session-scoped state -- `sched_snapshot_data_t` (`snapshot.c`/`typemap.h`) does not capture it, and none should be added. Since AC 2 unconditionally clears it at the next `begin_session` regardless of what a loaded snapshot contained, any captured cache state would be discarded before a single lookup against it could occur -- persisting it would be dead weight with no observable effect.
|
||
|
||
3. When a job's primary resreq's plan iterator (Requirement 2) is exhausted without a successful allocation, the Scheduler shall populate the cache with that resreq's fingerprint before pending the job. In v1, cache population is preemption-agnostic: it fires on allocation failure alone. When the scheduler-preemption spec ships, preemption fires between allocation failure and cache population — if preemption succeeds, no cache entry is created; if preemption fails or is inapplicable, the cache entry is populated as normal. The allocation layer does not call into preemption; the caller (scheduling loop) orchestrates the sequence.
|
||
|
||
4. Before starting full allocation for a job, the Scheduler shall look up the job's resreq fingerprint in the cache; on a hit, it shall pend the job immediately without invoking the plan or host iterators or preemption.
|
||
|
||
5. A cache hit shall attribute the same pend reason category as the original unsatisfiable determination, so operators are not misled about why a later job was skipped.
|
||
|
||
**Note on `resreq_fp` and snapshot restore (informational, not an AC):** `resreq_fp` is computed at job-ingest time (`core_build_schedobj()`/`core_events.c`) but not by `snapshot_load.c`'s independent local `schedobj` reconstruction (scheduler-snapshot-job-allocation spec) -- so a session restored via `xce_scheduler -i -s <replay>` would run with a permanently-cold cache for this Requirement. Confirmed (`xce_sched_core_load_snapshot()`'s 12 call sites are all in `sim.c`; none in `driver.c`) that this can only ever affect offline simulation/investigation sessions, never live production scheduling throughput -- so this is *not* elevated to a formal Acceptance Criterion. Tracked in bug-1767 alongside the broader recurring-duplication pattern in that reconstruction path, should someone want to close it later.
|
||
|
||
### Requirement 9: No regression to existing allocators or HA soak
|
||
|
||
**Objective:** As a maintainer, I want the iterator-driven rework scoped so that existing single-slot allocation and COMPOUND/ALTER dispatch are unaffected, so that this framework change is safe to ship independently of any new allocator.
|
||
|
||
#### Acceptance Criteria
|
||
|
||
1. **Seqjob regression**: `seqjob_allocator`'s externally observable allocation behavior (which host a `maxTasks == 1` job lands on, given the same worker/resource state) shall be unchanged after it is rewrapped onto the `xce_host_iter_t` interface.
|
||
|
||
2. **COMPOUND/ALTER unchanged**: The `COMPOUND` and `ALTER` resreq allocation paths (`sched_make_compound_allocation`, `sched_make_alter_allocation` in `scheduler.c`) shall remain unchanged, continuing to call into the per-primary allocation path (now iterator-driven) exactly as they do today.
|
||
|
||
3. **Unit tests**: Existing scheduler unit tests (`scheduler` meson test suite) shall continue to pass unmodified in assertions, adjusted only where they directly exercised the eager-one-shot behavior or stub return value this feature replaces.
|
||
|
||
4. **HA soak**: A full HA soak (sequential and chaotic `/xce-test-ha` regression) shall show no new job loss, divergence, or liveness stall attributable to this feature, confirming the scheduler-side change does not perturb controller-side commit/replication behavior it reads state from.
|
||
|
||
5. **Allocation-mode extension point**: The Scheduler shall add an `xce_alloc_mode_t` enum (`XCE_ALLOC_NORMAL = 0`, `XCE_ALLOC_PREEMPT`) and an `alloc_mode` field on `xce_sched_ctx_t`, defaulting to `XCE_ALLOC_NORMAL`. This spec's own plan/host iterators and `try_alloc_on_host()` shall not branch on any value other than `XCE_ALLOC_NORMAL` -- implementing what `XCE_ALLOC_PREEMPT` (or any future mode) actually does to resource-availability checks is `scheduler-preemption`'s job, not this spec's. The field exists here, in this spec's own scope, purely so `scheduler-preemption` and later resource-pool modes (e.g. reservation-aware backfill) can extend `xce_sched_ctx_t` and the availability-check call sites without a further interface change to types this spec already owns -- avoiding a sibling spec's first task being "go modify scheduler-allocation's struct." Adding further enum values beyond the two declared here is explicitly out of this spec's scope.
|
||
|
||
**Note (informational, not an AC):** Requirement 7's `resreq_fp` field is only ever populated on the live job-ingest path in this spec's v1 -- `snapshot_load.c`'s independent restore path does not compute it (see Requirement 7's note above and bug-1767). Since that path is exclusively simulation/investigation tooling (`sim.c`, never `driver.c`), this does not warrant a dedicated `/xce-audit-invariants scheduler` pass as a formal AC here; ordinary unit-test coverage of Requirement 7's cache hit/miss behavior is sufficient for this spec's scope.
|
||
|
||
## Data Structure Changes Summary
|
||
|
||
### `xce_alloc_plan_iter_t` (new, Req 2)
|
||
|
||
```c
|
||
typedef struct xce_alloc_plan_iter_s xce_alloc_plan_iter_t;
|
||
|
||
struct xce_alloc_plan_iter_s {
|
||
// Initialize the iterator for a given primary resreq and scheduler state.
|
||
// Returns 0 on success, nonzero on error.
|
||
int (*init)(xce_alloc_plan_iter_t* iter, xce_primary_resreq_t* resreq,
|
||
xce_scheduler_t* sched);
|
||
|
||
// Return the next candidate plan, or NULL when exhausted.
|
||
// The returned plan pointer is valid until the next next() or fini() call.
|
||
xce_alloc_plan_t* (*next)(xce_alloc_plan_iter_t* iter);
|
||
|
||
// Release all resources held by the iterator.
|
||
void (*fini)(xce_alloc_plan_iter_t* iter);
|
||
|
||
// Private state: current position, strategy-specific data
|
||
void* private;
|
||
};
|
||
|
||
// A plan is a partition of maxTasks into per-host buckets, e.g. for
|
||
// maxTasks=3: [3], [2,1], [1,1,1] -- NOT a uniform (nhosts, slots_per_host)
|
||
// pair, which cannot express uneven splits.
|
||
typedef struct {
|
||
int nbuckets; // number of hosts this plan needs (== len(bucket_slots))
|
||
int* bucket_slots; // partition of maxTasks; bucket_slots[i] = slots
|
||
// needed on the i-th distinct host claimed for this plan
|
||
// host_filter: derived from primary_resreq->select, plus slot availability
|
||
} xce_alloc_plan_t;
|
||
|
||
// Generating which host-counts to try, in what order (plan-shape policy) and
|
||
// how to partition maxTasks for a given host-count (partition policy) are
|
||
// deliberately left as internal, swappable policy inside the plan iterator's
|
||
// reference implementation -- not fixed by this interface. v1 ships ONE
|
||
// concrete, fully-specified, deterministic default of each -- "swappable
|
||
// later" does not mean "unspecified now":
|
||
//
|
||
// Plan-shape order (v1 default): ascending host-count, k = 1, 2, ..., N
|
||
// (N = maxTasks). Most-consolidated (fewest hosts) tried first, most-spread
|
||
// (N hosts x 1 slot) tried last -- the brief's original "1hostxN, 2hostsx
|
||
// ceil(N/2), ..., Nhostsx1" sequence, generalized to real partitions below.
|
||
//
|
||
// Partition order (v1 default), for a given host-count k: the canonical
|
||
// BALANCED split -- (N mod k) buckets get ceil(N/k) slots, the remaining
|
||
// (k - N mod k) buckets get floor(N/k) slots. E.g. N=7, k=3 -> [3,2,2].
|
||
// No other partition of N into k parts is generated (bounds plan count to
|
||
// exactly N, not the full integer-partition combinatorial space).
|
||
//
|
||
// Bucket order within one plan (v1 default): descending by slot count --
|
||
// largest bucket first, so a plan fails fast on its hardest-to-place chunk
|
||
// rather than discovering that late. A simplest-deterministic tie-break,
|
||
// not itself claimed as an optimization.
|
||
//
|
||
// This is deliberately "blind": v1's plan-shape and partition choices are a
|
||
// pure function of maxTasks alone -- they never inspect sched->workers'
|
||
// actual free-slot counts, topology, or load when deciding WHAT to generate
|
||
// or in WHAT order (only try_alloc_on_host/Req 3's host_iter ever look at
|
||
// live worker state, to decide whether a generated plan succeeds). Accepted
|
||
// consequence: a job with a very large maxTasks (e.g. thousands) on a modest
|
||
// cluster generates up to N candidate plans/host-counts in v1, most of which
|
||
// exceed the live worker count and fail immediately once host_iter finds too
|
||
// few candidates -- cheap per attempt, but not free, and not capped by
|
||
// anything cluster-state-aware. Real optimization -- e.g. skipping a
|
||
// host-count already known infeasible from current worker counts, or
|
||
// ordering by observed cluster state -- is explicitly future work, not v1;
|
||
// ordering MAY consult the full
|
||
// primary_resreq already passed to init() (span/same/affinity) without an
|
||
// interface change, matching AC1's existing forward-compatibility note.
|
||
// Deferred to a later enhancement, not this spec: a formal pluggable-strategy
|
||
// API for swapping either policy independently. v1 hardcodes both inside the
|
||
// combinatorial plan iterator; only allocator swap-out (Req 2 AC1's existing
|
||
// "future allocators bring their own iterator" path) is required now.
|
||
```
|
||
|
||
### `xce_host_iter_t` (new, Req 3)
|
||
|
||
```c
|
||
typedef struct xce_host_iter_s xce_host_iter_t;
|
||
|
||
// Correction (found during task 1.1 implementation): next() returns
|
||
// xce_sched_worker_t* (sched/types.h -- the scheduler-internal type
|
||
// carrying resvals/resInUse/host_eval_ctx), NOT xce_worker_t* (xce/types.h
|
||
// -- the wire/config type make_sched_worker() converts FROM). This
|
||
// Composition Algorithm's own pseudocode already uses
|
||
// `xce_sched_worker_t* w = host_iter->next(host_iter)` -- this struct
|
||
// declaration was simply out of sync with it. sched_check_host_satisfy()/
|
||
// sched_meet_host_selection() (resource.c), which any real host_iter
|
||
// implementation must call, both take xce_sched_worker_t*, not
|
||
// xce_worker_t*.
|
||
struct xce_host_iter_s {
|
||
// Initialize the iterator for a given plan and worker set.
|
||
// Returns 0 on success, nonzero on error.
|
||
int (*init)(xce_host_iter_t* iter, xce_alloc_plan_t* plan,
|
||
var workers);
|
||
|
||
// Return the next candidate worker matching the plan's host filter,
|
||
// or NULL when exhausted. The returned worker pointer is valid until
|
||
// the next next() or fini() call.
|
||
xce_sched_worker_t* (*next)(xce_host_iter_t* iter);
|
||
|
||
// Release all resources held by the iterator.
|
||
void (*fini)(xce_host_iter_t* iter);
|
||
|
||
// Private state: current position, filtering state
|
||
void* private;
|
||
};
|
||
```
|
||
|
||
### New types (Req 7)
|
||
|
||
```c
|
||
// Resreq portion of the cache key — pre-computed, shareable across jobs
|
||
// with identical resreqs. Fits in a single cache line (24 bytes).
|
||
typedef struct {
|
||
int min_tasks;
|
||
int max_tasks;
|
||
uint32_t select_node_id; // CSE-interned root; 0 if select is NULL
|
||
uint64_t rusage_hash; // computed at parse time, stable for life of resreq
|
||
} xce_resreq_fingerprint_t;
|
||
|
||
// Full cache key — resreq fingerprint + identity.
|
||
// Identity fields are xstring pointers (not owned, not freed).
|
||
// Pointer equality = content equality for xstring-interned strings.
|
||
typedef struct {
|
||
xce_resreq_fingerprint_t resreq;
|
||
const char* user; // xstring pointer
|
||
const char* queue; // xstring pointer
|
||
const char* project; // xstring pointer
|
||
} xce_cache_key_t;
|
||
|
||
// Correction (this review): this codebase's real kvtab API (kv.h/kv.c) does
|
||
// NOT take a hash/eq pair per lookup call -- a kvtab_t binds ONE fixed
|
||
// kv_key_ops_t (.hash/.assign/.cmp/.destroy) at table-CREATION time
|
||
// (kv_params_t), and every lookup/insert goes through the generic
|
||
// key_find()/key_add()/key_del() using whatever keyops that table was built
|
||
// with. int_key_ops/string_key_ops are the only existing kv_key_ops_t
|
||
// instances -- a composite struct key needs a THIRD one, new to this spec:
|
||
static uint32_t cache_key_hash_(void* k) {
|
||
const xce_cache_key_t* key = (const xce_cache_key_t*)k;
|
||
uint32_t h = 2166136261u; /* FNV-1a 32-bit -- no
|
||
* existing hash-update
|
||
* helper in this codebase;
|
||
* this is new, self-
|
||
* contained code. */
|
||
const unsigned char* p = (const unsigned char*)&key->resreq;
|
||
for (size_t i = 0; i < sizeof(key->resreq); i++) {
|
||
h = (h ^ p[i]) * 16777619u;
|
||
}
|
||
h = (h ^ (uint32_t)(uintptr_t)key->user) * 16777619u;
|
||
h = (h ^ (uint32_t)(uintptr_t)key->queue) * 16777619u;
|
||
h = (h ^ (uint32_t)(uintptr_t)key->project) * 16777619u;
|
||
return h;
|
||
}
|
||
|
||
// cmp -- three-way comparator, 0 == equal (confirmed against kv.c's real
|
||
// int_key_ops.cmp -> cmp_int_key: returns -1/0/1, and kvtab_get() checks
|
||
// `cmp(...) == 0` for a match). Correction (design-phase discovery): an
|
||
// earlier version of this function returned a BOOLEAN (1 for match, via
|
||
// &&-chained == comparisons) -- exactly inverted from the real contract.
|
||
// As originally written, every lookup would have been a false miss,
|
||
// silently disabling the entire cache (Req 7) without failing any test
|
||
// that didn't specifically assert a hit. memcmp() on the resreq portion
|
||
// already returns a 3-way result; the identity pointers just need the
|
||
// same shape.
|
||
static int cache_key_cmp_(void* a, void* b) {
|
||
const xce_cache_key_t* ka = (const xce_cache_key_t*)a;
|
||
const xce_cache_key_t* kb = (const xce_cache_key_t*)b;
|
||
int rc = memcmp(&ka->resreq, &kb->resreq, sizeof(ka->resreq));
|
||
if (rc != 0) {
|
||
return rc;
|
||
}
|
||
if (ka->user != kb->user) {
|
||
return (ka->user < kb->user) ? -1 : 1;
|
||
}
|
||
if (ka->queue != kb->queue) {
|
||
return (ka->queue < kb->queue) ? -1 : 1;
|
||
}
|
||
if (ka->project != kb->project) {
|
||
return (ka->project < kb->project) ? -1 : 1;
|
||
}
|
||
return 0;
|
||
}
|
||
|
||
// assign -- kvtab keys are heap-owned by the table (see string_key_ops'
|
||
// strdup-on-insert precedent); *dest must be a fresh heap copy of the
|
||
// stack-constructed key passed at the lookup site, not an alias of it.
|
||
static int cache_key_assign_(void** dest, void* src) {
|
||
xce_cache_key_t* copy = (xce_cache_key_t*)malloc(sizeof(xce_cache_key_t));
|
||
if (copy == NULL) {
|
||
return -ENOMEM;
|
||
}
|
||
memcpy(copy, src, sizeof(xce_cache_key_t));
|
||
*dest = copy;
|
||
return 0;
|
||
}
|
||
|
||
// destroy -- frees only the key struct itself; user/queue/project are
|
||
// borrowed xstring pointers (not owned, not freed here).
|
||
static void cache_key_destroy_(void* k) {
|
||
free(k);
|
||
}
|
||
|
||
static kv_key_ops_t cache_key_ops = {
|
||
.hash = cache_key_hash_,
|
||
.assign = cache_key_assign_,
|
||
.cmp = cache_key_cmp_,
|
||
.destroy = cache_key_destroy_,
|
||
};
|
||
|
||
// Table construction: standard kvtab_type RTTI pattern (see sched->core.plugins/
|
||
// sched->core.resmgr for a live, already-working example of this exact
|
||
// construction call site in this codebase) with
|
||
// kv_params_t{ .nslots = N, .keyops = &cache_key_ops, .datakiller = NULL }
|
||
// (NULL datakiller: a cache "value" is a hit/miss sentinel, nothing to free).
|
||
```
|
||
|
||
### `xce_primary_resreq_t` addition (Req 7)
|
||
|
||
One new field, set once at parse time:
|
||
|
||
```c
|
||
// In xce_primary_resreq_t (resreq.h):
|
||
uint64_t rusage_hash; // set by parse_resreq_primary(), stable for life of resreq
|
||
```
|
||
|
||
### `xce_sched_obj_t` / `xce_primary_schedobj_t` addition (Req 7)
|
||
|
||
Store the resreq portion of the fingerprint on the scheduling object so it is computed once and reused for every cache lookup of that job:
|
||
|
||
```c
|
||
// On xce_primary_schedobj_t (or xce_sched_obj_t):
|
||
xce_resreq_fingerprint_t resreq_fp; // populated on job ingest, read by cache lookup
|
||
```
|
||
|
||
### Lookup site (pseudocode)
|
||
|
||
```c
|
||
// Before allocation: construct key on stack, single hash + lookup via the
|
||
// real kvtab API (key_find/key_add -- NOT a free-function kvtab_lookup()
|
||
// taking hash/eq params, which doesn't exist in this codebase).
|
||
xce_cache_key_t key = {
|
||
.resreq = obj->u.primary->resreq_fp,
|
||
.user = obj->user,
|
||
.queue = obj->queue,
|
||
.project = obj->project,
|
||
};
|
||
void* unused = NULL;
|
||
if (key_find(sched->unsatisfiable_cache, &key, &unused) > 0) {
|
||
pend_job(obj, original_reason);
|
||
return;
|
||
}
|
||
// on exhaustion (Req 7 AC 3): key_add(sched->unsatisfiable_cache, &key, SENTINEL)
|
||
```
|
||
|
||
### Key design decisions
|
||
|
||
- **struct-based**: hash/cmp/assign/destroy live together in one `kv_key_ops_t`, matching this codebase's existing `int_key_ops`/`string_key_ops` idiom rather than an ad-hoc per-call hash/eq pair.
|
||
- **`memcmp` on resreq portion**: all fields are fixed-size integers/IDs — no pointers, no deep compare.
|
||
- **pointer equality on identity**: xstring guarantees identical content → identical pointer.
|
||
- **select via `node_id`**: CSE intern table deduplicates identical select expressions. O(1).
|
||
- **rusages via pre-computed hash**: set once at parse time, stable for life of resreq. O(1).
|
||
- **span/same/affinity deferred**: currently empty structs; add to `xce_resreq_fingerprint_t` when they gain fields.
|
||
|
||
### `xce_sched_ctx_t` addition (allocation mode, Req 9 AC 5)
|
||
|
||
A mode field that controls which resource pool the allocator draws from. Defaults to `XCE_ALLOC_NORMAL`. Consumed by `try_alloc_on_host()` and any resource-availability check — applies to all resource types (slots, shared resources, limit instances, licenses, etc.), not just CPU slots.
|
||
|
||
```c
|
||
// In xce_sched_ctx_t (sched/types.h):
|
||
typedef enum {
|
||
XCE_ALLOC_NORMAL = 0, // only truly free resources
|
||
XCE_ALLOC_PREEMPT, // count preemptible job resources as available
|
||
// Future:
|
||
// XCE_ALLOC_RESERVE, // count reserved-but-idle resources as available
|
||
// XCE_ALLOC_BACKFILL, // count reserved + preemptible resources
|
||
} xce_alloc_mode_t;
|
||
|
||
xce_alloc_mode_t alloc_mode; // XCE_ALLOC_NORMAL by default
|
||
```
|
||
|
||
When `alloc_mode != XCE_ALLOC_NORMAL`, resource-availability checks count the full resource footprint of victim-candidate jobs (not just slots) as borrowable. The allocator logic is otherwise identical — no fork in `try_alloc_on_host()`.
|
||
|
||
### `xce_preempt_victim_iter_t` (new, scheduler-preemption spec)
|
||
|
||
Same lazy-iterator pattern as plan iterator, applied to victim selection:
|
||
|
||
```c
|
||
typedef struct xce_preempt_victim_iter_s xce_preempt_victim_iter_t;
|
||
|
||
struct xce_preempt_victim_iter_s {
|
||
int (*init)(xce_preempt_victim_iter_t*, xce_sched_job_t* job, xce_scheduler_t*);
|
||
xce_preempt_victim_set_t* (*next)(xce_preempt_victim_iter_t*);
|
||
void (*fini)(xce_preempt_victim_iter_t*);
|
||
void* private;
|
||
};
|
||
```
|
||
|
||
The default policy (`lowest_priority_first`) generates victim sets lazily: smallest sufficient set of lowest-priority running jobs from `preemptible_queues` that would free enough resources. Listed here because the preemption flow wraps `allocate_primary()` and consumes `xce_alloc_mode_t` — both defined in this spec.
|
||
|
||
### `parjob_allocator` changes (Req 4)
|
||
|
||
- `my_alloc()` is replaced from a stub (`return 0`) to a real implementation that creates a plan iterator, walks candidate plans, and commits on first success.
|
||
- **Correction (this review): "only `job_alloc_fn` changes" is wrong** -- an earlier pass of this document said so before Req 4 AC 6 was added; `parjob_prepare_allocation()` (the plugin's separate `prepare_allocation_hook`) also changes, per the next bullet. What IS unchanged is the plugin's registration struct itself (`name`/`init_fn`/`fini_fn`/`job_elect_fn`/`data`, `register_parjob_allocator()`) -- only the bodies of its two hook implementations change.
|
||
- **`parjob_prepare_allocation()` (currently also a stub -- see Req 4 AC 6) must be fixed alongside `my_alloc()`**: it is the hook that populates `ctx->alloc_req.host`/`ctx->alloc_req.shared` (via `calc_wanted_host_resvals`/`calc_wanted_global_resvals`, the same calls `seqjob_prepare_allocation()` already makes) for whichever bucket is currently being tried. `xce_sched_ctx_t` already carries `nhosts`/`nslots` fields for exactly this; the per-bucket loop sets `ctx->nslots = bucket_slots[b]` (`ctx->nhosts = 1`) before each host try, and the fixed hook reads them instead of the hardcoded `1, 1` seqjob's version uses. Because `ctx->alloc_req` is a single scratch buffer reset and repopulated per candidate host (`plugin_prepare_allocation()`, `plugin.c:591`, `xvec_idval_fini`+`init`s it before each fan-out), this is safe to recompute per host-try -- see Req 4 AC 6 for the corollary `seqjob_prepare_allocation()` gate this depends on.
|
||
- **Do not bypass `plugin_prepare_allocation()`'s fan-out to precompute a bucket's resource want once and reuse it across every candidate host.** The rusage-derived portion (`calc_wanted_host_resvals`) is host-independent and in principle cacheable per bucket, but the same function's tail call to `sched_fill_limit_requests(sched, &ctx->alloc_req.host, hlimits)` reads `hlimits = zarray_get(&obj->hlimits, ctx->worker->idx)` -- genuinely host-specific per-host limit-tracking state. A full per-bucket bypass would silently stop enforcing per-host limits for jobs allocated through the new iterator path. Caching only the rusage-derived slice (still running the limit-fill per host) is a valid future optimization but is not required for this spec.
|
||
|
||
**Final allocation result -- merged from each claimed bucket, no new type needed.** `xce_job_alloc_t` (`xce/types.h`) already generalizes past one host: `nhosts` + `xce_host_alloc_t** allocs`. Today only `seqjob_allocator` exercises it, always with `nhosts=1` via the variadic `make_host_allocs(1, halloc)`. For a successful multi-bucket plan, `commit_plan_allocation()` builds one `xce_host_alloc_t` per claimed host (each from that bucket's own snapshotted `alloc_req.host`/`uncomit`, captured at the moment `try_alloc_on_host` succeeded for it -- `alloc_req.host` is overwritten on the next bucket's prepare/check cycle, so it cannot be read lazily after the loop moves on), collects them into an `xce_host_alloc_t** hallocs` array of length `nbuckets` (built directly with `calloc`+loop, since `make_host_allocs()`'s variadic signature only fits a compile-time-fixed argument list, not a runtime-sized claimed set), and calls the existing `make_job_alloc(jobid, nbuckets, hallocs)` unchanged. Shared-resource allocation (`allocs_shared`/`nres_shared`) stays job-level and computed once for the whole primary resreq, exactly as it is today -- it does not vary per bucket and is not part of the merge.
|
||
|
||
### `seqjob_allocator` changes (Req 9 AC 1)
|
||
|
||
- Internal rewrap: `my_alloc()` is restructured to use the plan and host iterator interfaces internally as a trivial case (1 host x 1 slot plan, then host iteration matching today's `sched_collect_workers()` + `xlist_find()` logic).
|
||
- Externally observable behavior is unchanged -- same host selection given same state.
|
||
- **`seqjob_prepare_allocation()` gains a `primary->maxTasks == 1` guard** (Req 4 AC 6). This is a no-op for seqjob's own observable behavior -- it never handles `maxTasks > 1` jobs regardless -- but is required for `parjob_allocator` correctness once its own prepare hook stops being a stub; see Req 4 AC 6 for why the unguarded fan-out is unsafe.
|
||
|
||
## Architecture Diagram
|
||
|
||
### Allocation Flow (PRIMARY leaf)
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────┐
|
||
│ ALLOCATE PRIMARY (Req 2+3+4) │
|
||
│ │
|
||
│ plan_iter_init(primary_resreq) │
|
||
│ │ │
|
||
│ ▼ │
|
||
│ ┌──────────────────────┐ │
|
||
│ │ plan_iter->next() │ ←─ generates per-host bucket │
|
||
│ │ returns plan or NULL │ partitions (e.g. [2,1]) │
|
||
│ └──────────┬───────────┘ most-preferred first │
|
||
│ │ │
|
||
│ plan │ NULL (exhausted) │
|
||
│ ▼ ▼ │
|
||
│ ┌──────────────────┐ return ALLOC_FAIL │
|
||
│ │ host_iter_init() │ │
|
||
│ └────────┬─────────┘ │
|
||
│ │ │
|
||
│ ▼ │
|
||
│ ┌──────────────────────┐ │
|
||
│ │ host_iter->next() │ ←─ filters workers by │
|
||
│ │ returns worker or NULL│ select expression + │
|
||
│ └──────────┬───────────┘ slot availability │
|
||
│ │ │
|
||
│ worker │ NULL (no host for plan) │
|
||
│ ▼ ▼ │
|
||
│ ┌──────────────────┐ host_iter->fini() │
|
||
│ │ try_alloc(worker, │ plan_iter->next() (next plan) │
|
||
│ │ plan, ctx) │ │
|
||
│ └────────┬─────────┘ │
|
||
│ │ │
|
||
│ success │ fail │
|
||
│ ▼ ▼ │
|
||
│ ┌────────────┐ host_iter->next() (next worker for plan) │
|
||
│ │ commit │ │
|
||
│ │ return │ │
|
||
│ │ ALLOC_SUCCESS│ │
|
||
│ └────────────┘ │
|
||
└──────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
**Simplified -- see "Composition Algorithm" above for the authoritative version.** This diagram shows one worker being tried per plan, which is exactly right for `seqjob_allocator` (`nbuckets == 1`) but is NOT the full picture for `parjob_allocator`'s multi-bucket plans: a real plan with `nbuckets > 1` requires an inner loop over each bucket, accumulating a distinct claimed host per bucket before the plan counts as satisfied (host-exclusivity, per-bucket resource snapshotting, and rollback-on-partial-failure all live in that inner loop, none of which fits cleanly into this box diagram). Treat this diagram as the single-bucket special case; the pseudocode under "Composition Algorithm" is the one to implement against.
|
||
|
||
### Unsatisfiable Resreq Cache (Req 7)
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────┐
|
||
│ SESSION START (begin_session) │
|
||
│ │
|
||
│ Clear unsatisfiable resreq fingerprint cache │
|
||
│ │
|
||
│ For each job from picker: │
|
||
│ │ │
|
||
│ ▼ │
|
||
│ ┌──────────────────────────────┐ │
|
||
│ │ Lookup fingerprint in cache │ │
|
||
│ │ key = hash(minTasks, │ │
|
||
│ │ maxTasks, select_node_id, │ │
|
||
│ │ rusage_hash, user, queue, │ │
|
||
│ │ project) │ │
|
||
│ └──────────┬───────────────────┘ │
|
||
│ │ │
|
||
│ hit │ miss │
|
||
│ ▼ │ ▼ │
|
||
│ ┌────────┐ │ ┌──────────────────┐ │
|
||
│ │ PEND │ │ │ Run full │ │
|
||
│ │ immedi- │ │ │ allocation │ │
|
||
│ │ ately │ │ │ (Req 2+3+4) │ │
|
||
│ └────────┘ │ └────────┬─────────┘ │
|
||
│ │ │ │
|
||
│ │ success │ fail (exhausted) │
|
||
│ │ ▼ ▼ │
|
||
│ │ ┌────────────┐ ┌──────────────────────┐ │
|
||
│ │ │ DISPATCH │ │ Populate cache with │ │
|
||
│ │ │ (no cache │ │ resreq fingerprint │ │
|
||
│ │ │ entry) │ │ then PEND job │ │
|
||
│ │ └────────────┘ └──────────────────────┘ │
|
||
└──────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## Integration Note: Preemption Interaction
|
||
|
||
Preemption (scheduler-preemption spec) fires when allocation fails for a preemptible job. It follows the same lazy-iterator pattern as allocation, applied to victim selection instead of host selection.
|
||
|
||
### Preemption Iterator
|
||
|
||
Same shape as plan iterator, different domain:
|
||
|
||
```c
|
||
// Preemption victim iterator — lazy generation of victim sets
|
||
typedef struct xce_preempt_victim_iter_s xce_preempt_victim_iter_t;
|
||
|
||
struct xce_preempt_victim_iter_s {
|
||
int (*init)(xce_preempt_victim_iter_t* iter, xce_sched_job_t* job,
|
||
xce_scheduler_t* sched);
|
||
xce_preempt_victim_set_t* (*next)(xce_preempt_victim_iter_t* iter);
|
||
void (*fini)(xce_preempt_victim_iter_t* iter);
|
||
void* private;
|
||
};
|
||
```
|
||
|
||
The default policy (`lowest_priority_first`) generates victim sets in order: smallest sufficient set of lowest-priority running jobs from `preemptible_queues` that would free enough resources for the preempting job.
|
||
|
||
### Allocation Mode
|
||
|
||
A `xce_alloc_mode_t` enum on `xce_sched_ctx_t` controls which resource pool the allocator draws from — applies to all resource types (slots, shared resources, limit instances, licenses), not just CPU slots:
|
||
|
||
```c
|
||
typedef enum {
|
||
XCE_ALLOC_NORMAL = 0, // only truly free resources
|
||
XCE_ALLOC_PREEMPT, // count preemptible job resources as available
|
||
} xce_alloc_mode_t;
|
||
```
|
||
|
||
When `alloc_mode != XCE_ALLOC_NORMAL`, resource-availability checks count the full resource footprint of victim-candidate jobs as borrowable. The allocator logic is otherwise identical — no fork in `try_alloc_on_host()`.
|
||
|
||
### Full Preemption Flow
|
||
|
||
**Cache placement, corrected:** the cache is keyed by `resreq_fp`, which lives on `xce_primary_schedobj_t` -- a PRIMARY-level concept, not a whole-job one (a COMPOUND/ALTER job has no single resreq shape of its own to fingerprint). So the cache check wraps `allocate_primary()` itself, not `allocate_job()`'s top-level dispatch. For a bare PRIMARY job this is the whole story; for a COMPOUND or ALTER job, each nested primary gets its own cache check for free, once for every call `allocate_compound()`/`allocate_alter()` already makes into `allocate_primary()` -- no job-level cache entry is needed or possible.
|
||
|
||
```c
|
||
// Wraps allocate_primary() -- called for EVERY primary reached, whether the
|
||
// job's own resreq IS a primary, or this is one primary nested inside a
|
||
// COMPOUND (allocate_compound()'s loop) or an ALTER's compound (transitively).
|
||
int allocate_primary_with_preempt(sched, primary_resreq, ctx, job) {
|
||
// 1. Cache check (Req 7) -- keyed on THIS primary's resreq_fp, not job-level
|
||
if (cache_lookup(primary_resreq, job)) return PEND;
|
||
|
||
// 2. Normal allocation
|
||
if (allocate_primary(sched, primary_resreq, ctx) > 0) return ALLOC_SUCCESS;
|
||
|
||
// 3. Preemption (if applicable)
|
||
if (job->queue->preemptible_queues == NULL) {
|
||
cache_insert(primary_resreq, job);
|
||
return PEND;
|
||
}
|
||
|
||
victim_iter_init(&viter, job, sched);
|
||
while ((victims = viter.next(&viter)) != NULL) {
|
||
ctx.alloc_mode = XCE_ALLOC_PREEMPT;
|
||
if (allocate_primary(sched, primary_resreq, &ctx) > 0) {
|
||
viter.fini(&viter);
|
||
return commit_preempt(victims);
|
||
}
|
||
ctx.alloc_mode = XCE_ALLOC_NORMAL;
|
||
}
|
||
viter.fini(&viter);
|
||
|
||
cache_insert(primary_resreq, job);
|
||
return PEND;
|
||
}
|
||
```
|
||
|
||
Key properties:
|
||
- v1 cache is preemption-agnostic (populated on allocation failure alone).
|
||
- When preemption ships, it fires between allocation failure and cache population.
|
||
- If preemption succeeds → no cache entry, preempting job is re-pended for next round.
|
||
- If preemption fails/exhausted → cache entry populated, job pended normally.
|
||
- Plan and host iterators are not preemption-aware — they simply report success/failure.
|
||
- `xce_alloc_mode_t alloc_mode` is the only new allocator surface needed.
|
||
|
||
## Implementation Order
|
||
|
||
Scheduler sub-specs are designed for sequential delivery:
|
||
|
||
```
|
||
Phase 1: scheduler-allocation (this spec)
|
||
├── plan_iter + host_iter foundation iterators
|
||
├── seqjob rewrap validates abstraction on simple case
|
||
├── parjob combinatorial real parallel allocation
|
||
└── resreq cache performance optimization
|
||
|
||
Phase 2: scheduler-core
|
||
├── core/driver split extracts xce_sched_core_t from xce_scheduler_t
|
||
├── picker interface xce_job_picker_iface_t + builtin_fcfs
|
||
└── session loop wires core_go() calls allocate_job()
|
||
|
||
Phase 3: scheduler-preemption
|
||
├── victim_iter lazy victim-set generator
|
||
├── XCE_ALLOC_PREEMPT mode tells allocator to count borrowed resources
|
||
└── preempt_policy plugin lowest_priority_first default
|
||
```
|
||
|
||
| Phase | Why there |
|
||
|---|---|
|
||
| Allocation first | Self-contained. Unit-testable with fake workers. No core, no preemption needed. Iterators are the building block everything else sits on. |
|
||
| Core second | Wires allocation into a real scheduling loop. Picker feeds jobs → allocator places them → decisions flow. Without allocation, core has nothing to call. |
|
||
| Preemption last | Wraps allocation. Needs `allocate_primary()` to exist. Needs core session loop to re-pend preempted job. Is a "retry with borrowing" layer on top of normal allocation. |
|