docs(scheduler-allocation): approve requirements, add design and tasks
Extensive requirements review found and fixed a real correctness bug (host_iter shared across plan buckets instead of re-scanned fresh per bucket, which would silently drop solvable multi-host allocations), a self-contradiction around parjob_prepare_allocation's scope, a plan/host-iterator init calling-convention mismatch, and an inverted kvtab .cmp polarity that would have made the unsatisfiable-resreq cache permanently non-functional. brief.md brought back into sync with requirements.md's corrected architecture. design.md and tasks.md generated and validated (kiro-validate-design GO after two fixes; independent task-graph sanity review PASS). Two pre-existing, out-of-boundary gaps found along the way logged as bug-1766/bug-1767. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
This commit is contained in:
parent
c4f2fec0d5
commit
48c789dfcb
7 changed files with 1044 additions and 75 deletions
|
|
@ -12,7 +12,7 @@ This spec closes the remaining two allocation gaps:
|
|||
|
||||
This spec introduces an **iterator-driven allocation architecture** with these components:
|
||||
|
||||
- A lazy `xce_alloc_plan_iter_t` -- candidate `(nhosts, slots_per_host)` plans, most-preferred first
|
||||
- A lazy `xce_alloc_plan_iter_t` -- candidate plans (each a partition of the job's task count into per-host slot buckets, e.g. `[2,1]` for 3 tasks across 2 hosts), most-preferred first
|
||||
- A lazy `xce_host_iter_t` -- candidate hosts for a given plan, filtered by the existing select-expression evaluator
|
||||
- A `parjob_allocator` that walks the plan iterator to do real combinatorial multi-slot allocation
|
||||
- A session-scoped unsatisfiable-resreq fingerprint cache so a job whose resreq no plan can satisfy doesn't re-pay full allocation cost for every later job with the same fingerprint in the same scheduling session
|
||||
|
|
@ -49,8 +49,8 @@ The iterator composition (`xce_alloc_plan_iter_t` + `xce_host_iter_t`) operates
|
|||
│ xce_alloc_plan_iter_t │ Req 2
|
||||
│ │
|
||||
│ init(primary_resreq, sched) │
|
||||
│ next() -> plan │ generates (nhosts, slots_per_host)
|
||||
│ fini() │ candidates, most-preferred first
|
||||
│ next() -> plan │ generates per-host bucket
|
||||
│ fini() │ partitions, most-preferred first
|
||||
└────────────┬───────────────┘
|
||||
│ plan
|
||||
▼
|
||||
|
|
@ -73,21 +73,94 @@ The iterator composition (`xce_alloc_plan_iter_t` + `xce_host_iter_t`) operates
|
|||
|
||||
### Composition Algorithm (All Three Types)
|
||||
|
||||
A plan is a **partition** of the primary's task count into per-host buckets (e.g. for `maxTasks=3`: `[3]`, `[2,1]`, `[1,1,1]`) -- not a single `(nhosts, slots_per_host)` pair, which cannot express uneven splits. `allocate_primary()` must therefore satisfy every bucket in a plan against a **distinct** host before the plan counts as satisfied; satisfying only the first bucket and stopping (as a single-pass `try_alloc_on_host` loop would) is not a valid implementation of Requirement 4 -- it degenerates into the single-host plan and never actually exercises the multi-host case.
|
||||
|
||||
**Host exclusivity within a plan** is enforced by the caller, not `xce_host_iter_t` itself: `allocate_primary()` skips any host already present in `claimed` before calling `try_alloc_on_host` -- Requirement 3's `init(plan, workers)` signature is unchanged, host_iter stays a pure select+availability filter. This distinction matters: a host *offered and rejected* for one bucket (too small for that bucket's slot count) must remain eligible for a later, smaller bucket in the same plan -- only a host *actually claimed* by a bucket is permanently excluded for the rest of that plan attempt.
|
||||
|
||||
**Correction (this review): a fresh `host_iter` per bucket, not one shared across the whole plan.** An earlier version of this pseudocode created one `host_iter` before the bucket loop and let every bucket's `while` pull from the same cursor. Since `next()` is a forward-only cursor with no rewind, that silently breaks the "offered-and-rejected stays eligible for a later bucket" guarantee just stated: a host bucket 0 tries and rejects (too small for it) is gone from the cursor forever, even if it exactly satisfies bucket 1. Concretely, for plan `[2,1]` with host A (1 free slot) and host B (2 free slots): bucket 0 ("need 2") pulls A first, rejects it, pulls B, succeeds -- cursor now exhausted; bucket 1 ("need 1") gets nothing back even though A was sitting there the whole time. The fix: `host_iter_init(iter, plan, sched->workers)` is called fresh at the top of **each** bucket iteration (same plan, same host_filter every time -- only `is_claimed()`'s growing exclusion set differs call to call), and `fini()`'d at the end of that same bucket's search. This is a three-level nested loop (plan → bucket/partition → worker, the third level always starting fresh), not a two-level one with an inner index.
|
||||
|
||||
**Bucket order is not merely a fail-fast heuristic -- for v1's single-resource-dimension model, descending (largest bucket first) can succeed where ascending would fail, with the identical set of available hosts.** Same example: with buckets tried ascending `[1,2]` instead, a fresh scan for "need 1" could greedily claim host B (2 free slots -- trivially satisfies 1, but wastefully overqualified), leaving only A (1 free slot) for "need 2", which fails. Descending order avoids this by construction: the largest ask is checked against the full pool before a smaller ask has a chance to consume a host only the larger one could have used. This is why the v1 default (descending, "Data Structure Changes Summary" below) is a real correctness property here, not just an optimization -- and why generating the reverse order as a *second* candidate plan for the same partition isn't needed on top of it. Which specific host gets picked among *multiple* hosts that satisfy the same bucket (host_iter's own internal candidate order, `HOST_ORDER_ANY`/`HOST_ORDER_MOST_FREE` etc.) is a separate, softer question already deferred as future optimization -- it affects efficiency under adversarial cases, not whether a solvable case gets missed outright the way the shared-cursor bug did.
|
||||
|
||||
**Rollback scope.** `ctx->uncomitted` (`sched_ctx_init`/`sched_ctx_fini`, `schedobj.c`) is per-job-allocation-attempt scratch state, torn down at the end of one job's `sched_make_primary_allocation()`/`sched_make_compound_allocation()` call -- it never touches the real, session-wide `resInUse` books, which are only charged later, separately, once a job actually dispatches (`job->accounts`/`sched_account_jobs()`). So `rollback_plan_allocation()` can never corrupt cluster-wide resource state. It is still required, though: without it, a bucket that succeeded before a later bucket exhausted the host iterator leaves its provisional debit in `ctx->uncomitted.hosts[idx]` for the rest of that job's allocation attempt -- a *later* plan considering the same host would see it as falsely less available, purely from a phantom debit an earlier, abandoned plan left behind, which could wrongly pend a job a later plan could actually have satisfied. The reversal itself has no existing precedent to reuse (`sched_make_compound_allocation()`'s own on-failure path, `scheduler.c:206-238`, does not actually clean up partial state on a mid-loop failure -- a separate, pre-existing gap outside this spec's scope, tracked separately, not modeled here). The underlying subtract primitive already exists (`xvec_op(v1, v2, XVEC_SUB_FLOAT)`, `xvec.c`); it just needs a small new "uncommit" helper wired to it, mirroring the existing `XVEC_ADD_FLOAT` commit call in `sched_try_alloc_on_host_()`. For this to work, each `claimed[]` entry must retain the exact `alloc_req.host` snapshot captured when that bucket succeeded (needed for the merge step too -- see "Final allocation result" below), since `ctx->alloc_req` is reset and repopulated for every subsequent host tried.
|
||||
|
||||
**Allocation convention (this review): iterators are stack-allocated by the caller, then initialized in place** -- matching `sched_ctx_init(&ctx, ...)`'s existing convention in this codebase, NOT a heap-allocating factory. `init()`'s own declared signature (`int (*init)(xce_alloc_plan_iter_t* iter, ...)`, "Data Structure Changes Summary" below) takes the iterator's own address as its first argument for exactly this reason. An earlier version of this pseudocode wrote `plan_iter = plan_iter_init(primary_resreq, sched)` as if it were a 2-arg factory function returning a new iterator -- inconsistent with the 3-arg in-place-init method the struct actually declares. Corrected below.
|
||||
|
||||
```c
|
||||
// === PRIMARY leaf: iterator composition ===
|
||||
// Returns >0 on success, 0 on failure
|
||||
int allocate_primary(sched, primary_resreq, ctx) {
|
||||
plan_iter = plan_iter_init(primary_resreq, sched);
|
||||
xce_alloc_plan_iter_t plan_iter_storage;
|
||||
xce_alloc_plan_iter_t* plan_iter = &plan_iter_storage;
|
||||
if (plan_iter->init(plan_iter, primary_resreq, sched) != 0) {
|
||||
return ALLOC_FAIL; // exact error-propagation contract: design.md
|
||||
}
|
||||
|
||||
while ((plan = plan_iter->next(plan_iter)) != NULL) {
|
||||
host_iter = host_iter_init(plan, sched->workers);
|
||||
while ((worker = host_iter->next(host_iter)) != NULL) {
|
||||
if (try_alloc_on_host(worker, plan, ctx) > 0) {
|
||||
// Correction (kiro-validate-design pass): sized by maxTasks, not an
|
||||
// undefined fixed constant -- nbuckets can never exceed maxTasks
|
||||
// (each bucket needs >=1 slot), so this is the true, tight upper
|
||||
// bound. A fixed compile-time guess risks either overflow for a
|
||||
// large parallel job or wasted stack for a small one.
|
||||
xce_sched_worker_t* claimed[primary_resreq->maxTasks];
|
||||
int nclaimed = 0;
|
||||
int plan_ok = 1;
|
||||
|
||||
// Three levels of iteration: plan (outer while, above) -> bucket/
|
||||
// partition (this for) -> worker (host_iter, fresh per bucket, below).
|
||||
for (int b = 0; b < plan->nbuckets; b++) {
|
||||
// Correction (this review): a FRESH host_iter per bucket, not
|
||||
// one shared across the whole plan -- next() is a forward-only
|
||||
// cursor, so sharing it would drop any host an earlier, larger
|
||||
// bucket tried and rejected, even if it exactly satisfies THIS,
|
||||
// smaller bucket. Same plan (same select-expression host_filter)
|
||||
// every call; only is_claimed()'s growing exclusion set differs.
|
||||
xce_host_iter_t host_iter_storage;
|
||||
xce_host_iter_t* host_iter = &host_iter_storage;
|
||||
xce_sched_worker_t* w;
|
||||
int bucket_ok = 0;
|
||||
|
||||
if (host_iter->init(host_iter, plan, sched->workers) == 0) {
|
||||
while ((w = host_iter->next(host_iter)) != NULL) {
|
||||
if (is_claimed(claimed, nclaimed, w)) {
|
||||
continue;
|
||||
}
|
||||
if (try_alloc_on_host(w, plan->bucket_slots[b], ctx) > 0) {
|
||||
// snapshot ctx->alloc_req.host here (it is reset on
|
||||
// the next plugin_prepare_allocation() cycle) --
|
||||
// needed for both rollback below and the merge step
|
||||
// in commit_plan_allocation().
|
||||
claimed[nclaimed++] = capture_claim(w, ctx);
|
||||
bucket_ok = 1;
|
||||
break;
|
||||
}
|
||||
}
|
||||
host_iter->fini(host_iter);
|
||||
plan_iter->fini(plan_iter);
|
||||
return ALLOC_SUCCESS;
|
||||
}
|
||||
// host_iter->init() failing is treated the same as "no
|
||||
// candidates" for this bucket -- an infrastructure error, not a
|
||||
// resource-availability determination; see design.md for the
|
||||
// exact error-propagation contract.
|
||||
|
||||
if (!bucket_ok) {
|
||||
plan_ok = 0;
|
||||
break; // this bucket has no satisfying host -- abandon the plan
|
||||
}
|
||||
}
|
||||
host_iter->fini(host_iter);
|
||||
|
||||
if (plan_ok) {
|
||||
commit_plan_allocation(claimed, nclaimed, ctx);
|
||||
plan_iter->fini(plan_iter);
|
||||
return ALLOC_SUCCESS;
|
||||
}
|
||||
|
||||
// Partial success within this plan attempt must not leak into the
|
||||
// next plan attempt for the SAME job (ctx->uncomitted persists across
|
||||
// plan attempts until sched_ctx_fini() -- see "Rollback scope" above):
|
||||
// for each claimed host, subtract back exactly the debit
|
||||
// sched_try_alloc_on_host_() added via XVEC_ADD_FLOAT, using the
|
||||
// snapshot captured above (new xvec_op(..., XVEC_SUB_FLOAT)-based
|
||||
// helper -- no existing "uncommit" primitive to call into).
|
||||
rollback_plan_allocation(claimed, nclaimed, ctx);
|
||||
}
|
||||
plan_iter->fini(plan_iter);
|
||||
return ALLOC_FAIL;
|
||||
|
|
@ -95,7 +168,12 @@ int allocate_primary(sched, primary_resreq, ctx) {
|
|||
|
||||
// === COMPOUND: AND of primaries ===
|
||||
// ALL primaries must allocate. On any failure, roll back prior allocations
|
||||
// and return failure. Same logic as today's sched_make_compound_allocation().
|
||||
// and return failure. NOTE: today's sched_make_compound_allocation() does
|
||||
// NOT actually do this rollback (see bug-1766, found during this spec's
|
||||
// review) -- rollback_compound_allocations() below is new/corrective
|
||||
// behavior this spec should introduce, not a port of existing logic. Out of
|
||||
// this spec's own boundary per Req 9 AC2 (COMPOUND/ALTER dispatch
|
||||
// unchanged) unless the user elects to fold bug-1766's fix in here.
|
||||
int allocate_compound(sched, compound_resreq, ctx) {
|
||||
for (int i = 0; i < compound_resreq->primaries.count; i++) {
|
||||
primary = compound_resreq->primaries[i];
|
||||
|
|
@ -137,7 +215,7 @@ int allocate_job(sched, job, ctx) {
|
|||
}
|
||||
```
|
||||
|
||||
**COMPOUND rollback note:** If primaries 1 and 2 of a compound allocate but primary 3 fails, primaries 1 and 2 must be rolled back -- the compound is atomic. Today this is handled by `sched_make_compound_allocation()` tracking allocations in `ctx` and cleaning up on failure. This behavior is unchanged; the iterator interfaces don't alter the rollback contract.
|
||||
**COMPOUND rollback note:** If primaries 1 and 2 of a compound allocate but primary 3 fails, primaries 1 and 2 *should* be rolled back -- the compound is meant to be atomic. **Correction (this review): today's `sched_make_compound_allocation()` does not actually do this** -- see bug-1766. It shares one `ctx` across the whole primaries loop, `break`s on the first failing primary without undoing earlier successes, and its unconditional post-loop `if (ctx.alloc != NULL) MOVE_PTR(obj->alloc, ctx.alloc)` can attach a partial, orphaned allocation to a compound job reported as failed. This spec's iterator rework does not touch that path (Req 9 AC2 keeps COMPOUND/ALTER dispatch unchanged) -- the pre-existing gap is tracked separately in bug-1766, not fixed here, unless scope is deliberately expanded to include it.
|
||||
|
||||
### Who Uses This Composition
|
||||
|
||||
|
|
@ -150,8 +228,8 @@ int allocate_job(sched, job, ctx) {
|
|||
### Key Design Properties
|
||||
|
||||
- **Lazy**: `plan_iter->next()` generates one plan at a time. `host_iter->next()` returns one host at a time. No full materialization.
|
||||
- **Cancel-safe**: both `next()` calls have `pthread_cancel` cancellation points.
|
||||
- **Independent lifecycles**: a `host_iter` is created from a plan, used, and destroyed; the `plan_iter` is advanced to the next plan only after the `host_iter` is done. `fini()` on one does not affect the other.
|
||||
- **Cancel-safe**: both `next()` calls have `pthread_cancel` cancellation points (Req 2 AC 6 / Req 3 AC 6 -- a testable requirement, not just a design intent, since neither loop has a natural cancellation point of its own).
|
||||
- **Independent lifecycles**: a `host_iter` is created from a plan, used, and destroyed -- and, for a multi-bucket plan, this happens **once per bucket**, not once per plan (a fresh instance each time, per Req 3 AC 3/4). The `plan_iter` is advanced to the next plan only after every bucket's `host_iter` for the current plan is done. `fini()` on one does not affect the other.
|
||||
- **COMPOUND/ALTER unchanged**: the composition rules above the PRIMARY leaf (AND for compound, OR for alter) are unchanged from today. The iterator interfaces only change what happens inside `allocate_primary()`.
|
||||
- **Seqjob rewrap**: `seqjob_allocator` is rewrapped onto the iterator interfaces as a trivial case -- its plan iterator always returns the single plan (1 host x 1 slot), and the inner loop is identical to today's first-match logic. This validates the abstraction before applying it to the complex parjob case.
|
||||
- **Pluggable plan generation, shared host iteration**: plan generation is allocator-specific — each `job_alloc_fn` brings its own strategy (trivial, combinatorial, topology-aware, bin-packing) and creates its own `xce_alloc_plan_iter_t` internally. Host iteration is shared infrastructure — `xce_host_iter_t` has one implementation reused by all allocators, parameterized by sort order (`HOST_ORDER_ANY`, `HOST_ORDER_MOST_FREE`, etc.). Swapping allocation strategy = swapping the plugin; no core loop change.
|
||||
|
|
@ -176,7 +254,7 @@ int allocate_job(sched, job, ctx) {
|
|||
|
||||
v1 implements simple combinatorial generation: `1host×N, 2hosts×ceil(N/2), ..., Nhosts×1`. Future allocators read `span`/`same`/`affinity` to constrain the plan space without touching the `init()` signature.
|
||||
|
||||
2. Each call to `next()` shall generate exactly one candidate `(nhosts, slots_per_host, host_filter)` plan without generating or materializing any other plan in the same call.
|
||||
2. Each call to `next()` shall generate exactly one candidate plan -- a partition of `maxTasks` into per-host buckets (`xce_alloc_plan_t.bucket_slots[]`, see "Composition Algorithm" above; NOT a uniform `(nhosts, slots_per_host)` pair, which cannot express uneven splits) -- without generating or materializing any other plan in the same call.
|
||||
|
||||
3. The Scheduler shall order plans returned by successive `next()` calls from most-preferred to least-preferred, per the owning allocator's preference policy.
|
||||
|
||||
|
|
@ -184,9 +262,11 @@ int allocate_job(sched, job, ctx) {
|
|||
|
||||
5. Calling `fini()` shall release all resources held by the iterator, and the interface shall define no other required cleanup step.
|
||||
|
||||
6. `next()` shall call `pthread_testcancel()` (or otherwise provide an equivalent, real cancellation point) at least once per invocation, matching the existing convention `sched_job_picker()`/`sched_allocator()` already establish at the top of their own loops (`scheduler.c`). This is required because, unlike those threads' blocking `cbuf_get()` calls (a POSIX cancellation point via `pthread_cond_wait`), combinatorial plan generation is fast, non-blocking pure computation with no cancellation point of its own -- an implementation satisfying AC 1-5 without this would silently violate the "Cancel-safe" design property stated for this iterator without failing any other criterion.
|
||||
|
||||
### Requirement 3: Lazy host candidate iterator
|
||||
|
||||
**Objective:** As a maintainer of the scheduler's allocator plugins, I want a reusable lazy host-candidate abstraction per allocation plan, so that host filtering reuses the existing select-expression evaluator instead of each allocator reimplementing worker collection and filtering.
|
||||
**Objective:** As a maintainer of the scheduler's allocator plugins, I want a reusable lazy host-candidate abstraction for finding a host satisfying one plan bucket at a time, so that host filtering reuses the existing select-expression evaluator instead of each allocator reimplementing worker collection and filtering. (Correction, this review: NOT "one `xce_host_iter_t` per plan" -- a plan with multiple buckets requires a fresh instance per bucket, per Req 4 AC 2.)
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
|
|
@ -194,37 +274,47 @@ int allocate_job(sched, job, ctx) {
|
|||
|
||||
2. Each call to `next()` shall return the next candidate host satisfying the given plan's host filter (select expression plus resource availability), evaluated against the existing select-expression evaluator without changes to that evaluator.
|
||||
|
||||
3. `next()` shall return NULL once no further candidate host exists for the current plan, and the caller shall not need to call `next()` again after receiving NULL for that plan.
|
||||
3. `next()` shall return NULL once no further candidate host exists for the current bucket's search, and the caller shall not need to call `next()` again on that same `host_iter` instance after receiving NULL. This does NOT mean the caller is done with the plan: a plan with more buckets remaining requires a fresh `host_iter` instance (a new `init()` call) for the next bucket, per Req 4 AC 2 -- exhaustion is scoped to one `host_iter` instance, not to the plan as a whole.
|
||||
|
||||
4. The host iterator shall compose with the plan iterator: a new `xce_host_iter_t` shall be constructible from any plan produced by Requirement 2's `xce_alloc_plan_iter_t.next()`.
|
||||
4. The host iterator shall compose with the plan iterator: a new `xce_host_iter_t` shall be constructible from any plan produced by Requirement 2's `xce_alloc_plan_iter_t.next()` -- and, for a multi-bucket plan, constructible **repeatedly** from the same plan, once per bucket (Req 4 AC 2), each a fully independent instance.
|
||||
|
||||
5. Calling `fini()` shall release all resources held by the host iterator, independent of the plan iterator's lifecycle.
|
||||
|
||||
6. `next()` shall call `pthread_testcancel()` (or otherwise provide an equivalent, real cancellation point) at least once per invocation, for the same reason as Requirement 2 AC 6: scanning/filtering candidate hosts is non-blocking pure computation with no cancellation point of its own, so this must be explicit rather than assumed.
|
||||
|
||||
### Requirement 4: Parallel job combinatorial allocation
|
||||
|
||||
**Objective:** As a cluster operator submitting a job that needs multiple task slots, I want `parjob_allocator` to actually allocate across hosts, so that jobs with `maxTasks > 1` are no longer silently unallocatable.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a job's `xce_primary_resreq_t.maxTasks > 1` reaches `parjob_allocator`'s `job_alloc_fn` (`my_alloc()` in `parjob_allocator.c`), the Scheduler shall generate candidate plans covering at minimum the sequence 1 host x N slots, 2 hosts x ceil(N/2) slots, ..., N hosts x 1 slot, using Requirement 2's plan iterator.
|
||||
1. When a job's `xce_primary_resreq_t.maxTasks > 1` reaches `parjob_allocator`'s `job_alloc_fn` (`my_alloc()` in `parjob_allocator.c`), the Scheduler shall generate candidate plans covering at minimum the host-count sequence `1..N` hosts, where each plan's bucket sizes are a partition of `maxTasks` into that many buckets (e.g. for `N=3`: `[3]`, `[2,1]`, `[1,1,1]` -- balanced ceil/floor split, not a uniform `slots_per_host` that cannot express uneven splits), using Requirement 2's plan iterator.
|
||||
|
||||
2. For each candidate plan, the Scheduler shall use Requirement 3's host iterator to collect hosts with enough available slots to satisfy that plan, and shall attempt allocation before advancing to the next plan.
|
||||
2. For each candidate plan, the Scheduler shall use Requirement 3's host iterator to find, for every bucket in the plan, a distinct host with enough available slots to satisfy that bucket -- accumulating claimed hosts one bucket at a time -- and shall attempt allocation before advancing to the next plan. A plan is satisfied only when every one of its buckets has a claimed host; if any bucket exhausts the host iterator without a match, the plan fails as a whole and any buckets already matched within that same plan attempt are released before advancing to the next plan (see Composition Algorithm's `rollback_plan_allocation`).
|
||||
|
||||
3. When a plan's host collection satisfies the plan's slot requirement, the Scheduler shall commit that allocation and shall not evaluate any remaining plan for that job.
|
||||
**A new `xce_host_iter_t` instance shall be constructed for each bucket** (three levels of iteration: plan -> bucket/partition -> worker, the third always starting fresh) -- never one shared, continuously-advancing instance reused across every bucket of a plan. Reusing one instance is an incorrect implementation: `next()`'s forward-only cursor would permanently drop any host an earlier, larger bucket tried and rejected as too small, even when that same host exactly satisfies a later, smaller bucket -- silently failing plans a correct implementation would have satisfied. Only `is_claimed()` (Criterion 2's own "distinct host" exclusion, applied by the caller across each bucket's fresh scan) may exclude a host across buckets; rejection by an earlier bucket's size check must not.
|
||||
|
||||
3. When every bucket of a plan is satisfied, the Scheduler shall commit that allocation as a single `xce_job_alloc_t` merging one `xce_host_alloc_t` per claimed bucket/host (`nhosts` == the plan's bucket count; see "Final allocation result" under parjob_allocator changes, below) and shall not evaluate any remaining plan for that job.
|
||||
|
||||
4. When no plan among those generated in Criterion 1 can be satisfied, `my_alloc()` shall return the same not-satisfied signal (`0`) that the current stub returns today, so pend-reason handling in `sched_make_primary_alloc_()` is unaffected.
|
||||
|
||||
5. `parjob_allocator`'s `job_alloc_fn` shall no longer unconditionally `return 0` regardless of `ctx` contents -- the current stub behavior -- for any job with `maxTasks > 1`.
|
||||
|
||||
6. Before each candidate host is checked against a plan's bucket, the Scheduler shall set `ctx->nslots` (and `ctx->nhosts`, for the shared/global-resource pass) to that bucket's slot count so that `parjob_prepare_allocation()` computes the per-host resource want via the existing `calc_wanted_host_resvals(..., ctx->nhosts, ctx->nslots)` call -- the same mechanism `seqjob_prepare_allocation()` already uses, currently hardcoded to `1, 1`. Concurrently, `seqjob_prepare_allocation()` shall gain a `primary->maxTasks == 1` guard (it currently has none, unlike `parjob_prepare_allocation()`, which already gates on `maxTasks`). This guard is required, not optional cleanup: `plugin_prepare_allocation()` (`plugin.c:591`) unconditionally calls every registered plugin's `prepare_allocation_hook`, and `xvec_idval_store()` (`xvec.c:247`) appends `(idx, val)` pairs with no dedup by resource idx -- so once `parjob_prepare_allocation()` stores real values, an unguarded `seqjob_prepare_allocation()` would *also* fire for the same `maxTasks > 1` job and append a second, wrong (`nslots=1`) entry for the same resource idx into `ctx->alloc_req`, corrupting the availability check that Criterion 2 depends on. This is a latent defect in the existing fan-out, invisible today only because `parjob_prepare_allocation()` is currently a no-op.
|
||||
|
||||
7. Shared-resource satisfaction (`sched_check_shared_satisfy()`, `ctx->alloc_req.shared`) shall be computed and checked once per primary resreq -- independent of which plan or bucket-shape is currently being tried -- exactly as it already is today for `seqjob_allocator`. Plan/bucket iteration (Criteria 1-3) governs only per-host slot allocation; it does not change when or how often the job-level shared-resource check runs.
|
||||
|
||||
### Requirement 7: Session-level unsatisfiable resreq cache
|
||||
|
||||
**Objective:** As a maintainer of scheduling-loop performance, I want jobs with a resreq the current session already proved unsatisfiable to skip allocation immediately, so that a batch of similarly-shaped unsatisfiable jobs doesn't each re-pay full plan/host iteration cost.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The Scheduler shall maintain a per-session unsatisfiable-resreq fingerprint cache on `xce_scheduler_t`, keyed by `xce_cache_key_t` — a value type composing `xce_resreq_fingerprint_t` (pre-computed `{minTasks, maxTasks, select_node_id, rusage_hash}`) and identity (`{user, queue, project}` as xstring pointers). The resreq fingerprint is stored on the scheduling object (`xce_primary_schedobj_t`) and computed once on job ingest. The full cache key is constructed on the stack at lookup time — a single `kvtab_lookup()` with `cache_key_hash()` and `cache_key_eq()`. Fully O(1): no list walks, no string traversals, no allocations. This captures everything the allocator and limit plugin consume.
|
||||
1. The Scheduler shall maintain a per-session unsatisfiable-resreq fingerprint cache on `xce_scheduler_t`, keyed by `xce_cache_key_t` — a value type composing `xce_resreq_fingerprint_t` (pre-computed `{minTasks, maxTasks, select_node_id, rusage_hash}`) and identity (`{user, queue, project}` as xstring pointers). The resreq fingerprint is stored on the scheduling object (`xce_primary_schedobj_t`) and computed once on job ingest. The full cache key is constructed on the stack at lookup time and looked up via a single `key_find()` call against a `kvtab_t` bound to a new `cache_key_ops` (`kv_key_ops_t`) — this codebase's real hash-table API binds hash/cmp/assign/destroy to the table at creation, not per call (see "New types" below; no free-function `kvtab_lookup()` taking hash/eq params exists). Lookup itself is O(1): no list walks, no string traversals; `key_find()` does not allocate (only `key_add()`, on insert, allocates the heap-owned key copy). This captures everything the allocator and limit plugin consume.
|
||||
|
||||
2. The Scheduler shall clear the unsatisfiable-resreq cache at the start of every scheduling session (`begin_session`).
|
||||
2. The Scheduler shall clear the unsatisfiable-resreq cache at the start of every scheduling session (`begin_session`), by placing the clear inside `core_begin_session_internal()` (`core_session.c`) alongside the existing, unlocked `core_reset_shared_resval()`/`core_reset_workers()` calls -- a core-level function both `driver.c` (normal mode) and `sim_loop.c`/`sim.c` (simulation mode) reach identically through `xce_core_begin_session()`, so no driver-specific clearing code is needed for either mode. No explicit lock is required: this matches the existing, already-safe pattern those two sibling resets use, which relies on `xce_core_begin_session()`'s own documented ordering guarantee -- the driver completes all begin-session reset work strictly before `xce_core_signal_go()` unblocks the picker thread (the one `cbuf_put()` that lets the picker/allocator touch anything for the new session), so the mutex/condvar inside that `cbuf_put()`/`cbuf_get()` pair provides the memory-barrier the reset needs, the same way it already does for worker `resvals`/`sharedInUse`. Neither driver touches the cache object anywhere else -- lookup (AC 4) and insert (AC 3) happen exclusively on the allocator thread, inside `allocate_primary()`.
|
||||
|
||||
**Not part of the snapshot/replay wire format.** The cache is pure in-memory, session-scoped state -- `sched_snapshot_data_t` (`snapshot.c`/`typemap.h`) does not capture it, and none should be added. Since AC 2 unconditionally clears it at the next `begin_session` regardless of what a loaded snapshot contained, any captured cache state would be discarded before a single lookup against it could occur -- persisting it would be dead weight with no observable effect.
|
||||
|
||||
3. When a job's primary resreq's plan iterator (Requirement 2) is exhausted without a successful allocation, the Scheduler shall populate the cache with that resreq's fingerprint before pending the job. In v1, cache population is preemption-agnostic: it fires on allocation failure alone. When the scheduler-preemption spec ships, preemption fires between allocation failure and cache population — if preemption succeeds, no cache entry is created; if preemption fails or is inapplicable, the cache entry is populated as normal. The allocation layer does not call into preemption; the caller (scheduling loop) orchestrates the sequence.
|
||||
|
||||
|
|
@ -232,6 +322,8 @@ int allocate_job(sched, job, ctx) {
|
|||
|
||||
5. A cache hit shall attribute the same pend reason category as the original unsatisfiable determination, so operators are not misled about why a later job was skipped.
|
||||
|
||||
**Note on `resreq_fp` and snapshot restore (informational, not an AC):** `resreq_fp` is computed at job-ingest time (`core_build_schedobj()`/`core_events.c`) but not by `snapshot_load.c`'s independent local `schedobj` reconstruction (scheduler-snapshot-job-allocation spec) -- so a session restored via `xce_scheduler -i -s <replay>` would run with a permanently-cold cache for this Requirement. Confirmed (`xce_sched_core_load_snapshot()`'s 12 call sites are all in `sim.c`; none in `driver.c`) that this can only ever affect offline simulation/investigation sessions, never live production scheduling throughput -- so this is *not* elevated to a formal Acceptance Criterion. Tracked in bug-1767 alongside the broader recurring-duplication pattern in that reconstruction path, should someone want to close it later.
|
||||
|
||||
### Requirement 9: No regression to existing allocators or HA soak
|
||||
|
||||
**Objective:** As a maintainer, I want the iterator-driven rework scoped so that existing single-slot allocation and COMPOUND/ALTER dispatch are unaffected, so that this framework change is safe to ship independently of any new allocator.
|
||||
|
|
@ -246,6 +338,10 @@ int allocate_job(sched, job, ctx) {
|
|||
|
||||
4. **HA soak**: A full HA soak (sequential and chaotic `/xce-test-ha` regression) shall show no new job loss, divergence, or liveness stall attributable to this feature, confirming the scheduler-side change does not perturb controller-side commit/replication behavior it reads state from.
|
||||
|
||||
5. **Allocation-mode extension point**: The Scheduler shall add an `xce_alloc_mode_t` enum (`XCE_ALLOC_NORMAL = 0`, `XCE_ALLOC_PREEMPT`) and an `alloc_mode` field on `xce_sched_ctx_t`, defaulting to `XCE_ALLOC_NORMAL`. This spec's own plan/host iterators and `try_alloc_on_host()` shall not branch on any value other than `XCE_ALLOC_NORMAL` -- implementing what `XCE_ALLOC_PREEMPT` (or any future mode) actually does to resource-availability checks is `scheduler-preemption`'s job, not this spec's. The field exists here, in this spec's own scope, purely so `scheduler-preemption` and later resource-pool modes (e.g. reservation-aware backfill) can extend `xce_sched_ctx_t` and the availability-check call sites without a further interface change to types this spec already owns -- avoiding a sibling spec's first task being "go modify scheduler-allocation's struct." Adding further enum values beyond the two declared here is explicitly out of this spec's scope.
|
||||
|
||||
**Note (informational, not an AC):** Requirement 7's `resreq_fp` field is only ever populated on the live job-ingest path in this spec's v1 -- `snapshot_load.c`'s independent restore path does not compute it (see Requirement 7's note above and bug-1767). Since that path is exclusively simulation/investigation tooling (`sim.c`, never `driver.c`), this does not warrant a dedicated `/xce-audit-invariants scheduler` pass as a formal AC here; ordinary unit-test coverage of Requirement 7's cache hit/miss behavior is sufficient for this spec's scope.
|
||||
|
||||
## Data Structure Changes Summary
|
||||
|
||||
### `xce_alloc_plan_iter_t` (new, Req 2)
|
||||
|
|
@ -270,12 +366,58 @@ struct xce_alloc_plan_iter_s {
|
|||
void* private;
|
||||
};
|
||||
|
||||
// A plan describes how many hosts and slots to attempt
|
||||
// A plan is a partition of maxTasks into per-host buckets, e.g. for
|
||||
// maxTasks=3: [3], [2,1], [1,1,1] -- NOT a uniform (nhosts, slots_per_host)
|
||||
// pair, which cannot express uneven splits.
|
||||
typedef struct {
|
||||
int nhosts; // number of hosts needed
|
||||
int slots_per_host; // slots required on each host
|
||||
int nbuckets; // number of hosts this plan needs (== len(bucket_slots))
|
||||
int* bucket_slots; // partition of maxTasks; bucket_slots[i] = slots
|
||||
// needed on the i-th distinct host claimed for this plan
|
||||
// host_filter: derived from primary_resreq->select, plus slot availability
|
||||
} xce_alloc_plan_t;
|
||||
|
||||
// Generating which host-counts to try, in what order (plan-shape policy) and
|
||||
// how to partition maxTasks for a given host-count (partition policy) are
|
||||
// deliberately left as internal, swappable policy inside the plan iterator's
|
||||
// reference implementation -- not fixed by this interface. v1 ships ONE
|
||||
// concrete, fully-specified, deterministic default of each -- "swappable
|
||||
// later" does not mean "unspecified now":
|
||||
//
|
||||
// Plan-shape order (v1 default): ascending host-count, k = 1, 2, ..., N
|
||||
// (N = maxTasks). Most-consolidated (fewest hosts) tried first, most-spread
|
||||
// (N hosts x 1 slot) tried last -- the brief's original "1hostxN, 2hostsx
|
||||
// ceil(N/2), ..., Nhostsx1" sequence, generalized to real partitions below.
|
||||
//
|
||||
// Partition order (v1 default), for a given host-count k: the canonical
|
||||
// BALANCED split -- (N mod k) buckets get ceil(N/k) slots, the remaining
|
||||
// (k - N mod k) buckets get floor(N/k) slots. E.g. N=7, k=3 -> [3,2,2].
|
||||
// No other partition of N into k parts is generated (bounds plan count to
|
||||
// exactly N, not the full integer-partition combinatorial space).
|
||||
//
|
||||
// Bucket order within one plan (v1 default): descending by slot count --
|
||||
// largest bucket first, so a plan fails fast on its hardest-to-place chunk
|
||||
// rather than discovering that late. A simplest-deterministic tie-break,
|
||||
// not itself claimed as an optimization.
|
||||
//
|
||||
// This is deliberately "blind": v1's plan-shape and partition choices are a
|
||||
// pure function of maxTasks alone -- they never inspect sched->workers'
|
||||
// actual free-slot counts, topology, or load when deciding WHAT to generate
|
||||
// or in WHAT order (only try_alloc_on_host/Req 3's host_iter ever look at
|
||||
// live worker state, to decide whether a generated plan succeeds). Accepted
|
||||
// consequence: a job with a very large maxTasks (e.g. thousands) on a modest
|
||||
// cluster generates up to N candidate plans/host-counts in v1, most of which
|
||||
// exceed the live worker count and fail immediately once host_iter finds too
|
||||
// few candidates -- cheap per attempt, but not free, and not capped by
|
||||
// anything cluster-state-aware. Real optimization -- e.g. skipping a
|
||||
// host-count already known infeasible from current worker counts, or
|
||||
// ordering by observed cluster state -- is explicitly future work, not v1;
|
||||
// ordering MAY consult the full
|
||||
// primary_resreq already passed to init() (span/same/affinity) without an
|
||||
// interface change, matching AC1's existing forward-compatibility note.
|
||||
// Deferred to a later enhancement, not this spec: a formal pluggable-strategy
|
||||
// API for swapping either policy independently. v1 hardcodes both inside the
|
||||
// combinatorial plan iterator; only allocator swap-out (Req 2 AC1's existing
|
||||
// "future allocators bring their own iterator" path) is required now.
|
||||
```
|
||||
|
||||
### `xce_host_iter_t` (new, Req 3)
|
||||
|
|
@ -324,23 +466,90 @@ typedef struct {
|
|||
const char* project; // xstring pointer
|
||||
} xce_cache_key_t;
|
||||
|
||||
// Hash — single call site, no ad-hoc field assembly
|
||||
static inline uint64_t cache_key_hash(const xce_cache_key_t* k) {
|
||||
uint64_t h = FNV64_INIT;
|
||||
hash_update(&h, &k->resreq, sizeof(k->resreq));
|
||||
hash_update_ptr(&h, k->user);
|
||||
hash_update_ptr(&h, k->queue);
|
||||
hash_update_ptr(&h, k->project);
|
||||
// Correction (this review): this codebase's real kvtab API (kv.h/kv.c) does
|
||||
// NOT take a hash/eq pair per lookup call -- a kvtab_t binds ONE fixed
|
||||
// kv_key_ops_t (.hash/.assign/.cmp/.destroy) at table-CREATION time
|
||||
// (kv_params_t), and every lookup/insert goes through the generic
|
||||
// key_find()/key_add()/key_del() using whatever keyops that table was built
|
||||
// with. int_key_ops/string_key_ops are the only existing kv_key_ops_t
|
||||
// instances -- a composite struct key needs a THIRD one, new to this spec:
|
||||
static uint32_t cache_key_hash_(void* k) {
|
||||
const xce_cache_key_t* key = (const xce_cache_key_t*)k;
|
||||
uint32_t h = 2166136261u; /* FNV-1a 32-bit -- no
|
||||
* existing hash-update
|
||||
* helper in this codebase;
|
||||
* this is new, self-
|
||||
* contained code. */
|
||||
const unsigned char* p = (const unsigned char*)&key->resreq;
|
||||
for (size_t i = 0; i < sizeof(key->resreq); i++) {
|
||||
h = (h ^ p[i]) * 16777619u;
|
||||
}
|
||||
h = (h ^ (uint32_t)(uintptr_t)key->user) * 16777619u;
|
||||
h = (h ^ (uint32_t)(uintptr_t)key->queue) * 16777619u;
|
||||
h = (h ^ (uint32_t)(uintptr_t)key->project) * 16777619u;
|
||||
return h;
|
||||
}
|
||||
|
||||
// Equality — memcmp on fixed-size resreq portion + pointer equality on identity
|
||||
static inline bool cache_key_eq(const xce_cache_key_t* a, const xce_cache_key_t* b) {
|
||||
return memcmp(&a->resreq, &b->resreq, sizeof(a->resreq)) == 0
|
||||
&& a->user == b->user
|
||||
&& a->queue == b->queue
|
||||
&& a->project == b->project;
|
||||
// cmp -- three-way comparator, 0 == equal (confirmed against kv.c's real
|
||||
// int_key_ops.cmp -> cmp_int_key: returns -1/0/1, and kvtab_get() checks
|
||||
// `cmp(...) == 0` for a match). Correction (design-phase discovery): an
|
||||
// earlier version of this function returned a BOOLEAN (1 for match, via
|
||||
// &&-chained == comparisons) -- exactly inverted from the real contract.
|
||||
// As originally written, every lookup would have been a false miss,
|
||||
// silently disabling the entire cache (Req 7) without failing any test
|
||||
// that didn't specifically assert a hit. memcmp() on the resreq portion
|
||||
// already returns a 3-way result; the identity pointers just need the
|
||||
// same shape.
|
||||
static int cache_key_cmp_(void* a, void* b) {
|
||||
const xce_cache_key_t* ka = (const xce_cache_key_t*)a;
|
||||
const xce_cache_key_t* kb = (const xce_cache_key_t*)b;
|
||||
int rc = memcmp(&ka->resreq, &kb->resreq, sizeof(ka->resreq));
|
||||
if (rc != 0) {
|
||||
return rc;
|
||||
}
|
||||
if (ka->user != kb->user) {
|
||||
return (ka->user < kb->user) ? -1 : 1;
|
||||
}
|
||||
if (ka->queue != kb->queue) {
|
||||
return (ka->queue < kb->queue) ? -1 : 1;
|
||||
}
|
||||
if (ka->project != kb->project) {
|
||||
return (ka->project < kb->project) ? -1 : 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
// assign -- kvtab keys are heap-owned by the table (see string_key_ops'
|
||||
// strdup-on-insert precedent); *dest must be a fresh heap copy of the
|
||||
// stack-constructed key passed at the lookup site, not an alias of it.
|
||||
static int cache_key_assign_(void** dest, void* src) {
|
||||
xce_cache_key_t* copy = (xce_cache_key_t*)malloc(sizeof(xce_cache_key_t));
|
||||
if (copy == NULL) {
|
||||
return -ENOMEM;
|
||||
}
|
||||
memcpy(copy, src, sizeof(xce_cache_key_t));
|
||||
*dest = copy;
|
||||
return 0;
|
||||
}
|
||||
|
||||
// destroy -- frees only the key struct itself; user/queue/project are
|
||||
// borrowed xstring pointers (not owned, not freed here).
|
||||
static void cache_key_destroy_(void* k) {
|
||||
free(k);
|
||||
}
|
||||
|
||||
static kv_key_ops_t cache_key_ops = {
|
||||
.hash = cache_key_hash_,
|
||||
.assign = cache_key_assign_,
|
||||
.cmp = cache_key_cmp_,
|
||||
.destroy = cache_key_destroy_,
|
||||
};
|
||||
|
||||
// Table construction: standard kvtab_type RTTI pattern (see sched->core.plugins/
|
||||
// sched->core.resmgr for a live, already-working example of this exact
|
||||
// construction call site in this codebase) with
|
||||
// kv_params_t{ .nslots = N, .keyops = &cache_key_ops, .datakiller = NULL }
|
||||
// (NULL datakiller: a cache "value" is a hit/miss sentinel, nothing to free).
|
||||
```
|
||||
|
||||
### `xce_primary_resreq_t` addition (Req 7)
|
||||
|
|
@ -364,29 +573,33 @@ xce_resreq_fingerprint_t resreq_fp; // populated on job ingest, read by cache l
|
|||
### Lookup site (pseudocode)
|
||||
|
||||
```c
|
||||
// Before allocation: construct key on stack, single hash + lookup
|
||||
// Before allocation: construct key on stack, single hash + lookup via the
|
||||
// real kvtab API (key_find/key_add -- NOT a free-function kvtab_lookup()
|
||||
// taking hash/eq params, which doesn't exist in this codebase).
|
||||
xce_cache_key_t key = {
|
||||
.resreq = obj->u.primary->resreq_fp,
|
||||
.user = obj->user,
|
||||
.queue = obj->queue,
|
||||
.project = obj->project,
|
||||
};
|
||||
if (kvtab_lookup(sched->unsatisfiable_cache, &key, cache_key_hash, cache_key_eq)) {
|
||||
void* unused = NULL;
|
||||
if (key_find(sched->unsatisfiable_cache, &key, &unused) > 0) {
|
||||
pend_job(obj, original_reason);
|
||||
return;
|
||||
}
|
||||
// on exhaustion (Req 7 AC 3): key_add(sched->unsatisfiable_cache, &key, SENTINEL)
|
||||
```
|
||||
|
||||
### Key design decisions
|
||||
|
||||
- **struct-based**: hash and equality live next to the type. Single call site, no scattered field assembly.
|
||||
- **struct-based**: hash/cmp/assign/destroy live together in one `kv_key_ops_t`, matching this codebase's existing `int_key_ops`/`string_key_ops` idiom rather than an ad-hoc per-call hash/eq pair.
|
||||
- **`memcmp` on resreq portion**: all fields are fixed-size integers/IDs — no pointers, no deep compare.
|
||||
- **pointer equality on identity**: xstring guarantees identical content → identical pointer.
|
||||
- **select via `node_id`**: CSE intern table deduplicates identical select expressions. O(1).
|
||||
- **rusages via pre-computed hash**: set once at parse time, stable for life of resreq. O(1).
|
||||
- **span/same/affinity deferred**: currently empty structs; add to `xce_resreq_fingerprint_t` when they gain fields.
|
||||
|
||||
### `xce_sched_ctx_t` addition (allocation mode)
|
||||
### `xce_sched_ctx_t` addition (allocation mode, Req 9 AC 5)
|
||||
|
||||
A mode field that controls which resource pool the allocator draws from. Defaults to `XCE_ALLOC_NORMAL`. Consumed by `try_alloc_on_host()` and any resource-availability check — applies to all resource types (slots, shared resources, limit instances, licenses, etc.), not just CPU slots.
|
||||
|
||||
|
|
@ -425,12 +638,17 @@ The default policy (`lowest_priority_first`) generates victim sets lazily: small
|
|||
### `parjob_allocator` changes (Req 4)
|
||||
|
||||
- `my_alloc()` is replaced from a stub (`return 0`) to a real implementation that creates a plan iterator, walks candidate plans, and commits on first success.
|
||||
- The allocator's registration in the plugin framework is unchanged -- only the `job_alloc_fn` implementation changes.
|
||||
- **Correction (this review): "only `job_alloc_fn` changes" is wrong** -- an earlier pass of this document said so before Req 4 AC 6 was added; `parjob_prepare_allocation()` (the plugin's separate `prepare_allocation_hook`) also changes, per the next bullet. What IS unchanged is the plugin's registration struct itself (`name`/`init_fn`/`fini_fn`/`job_elect_fn`/`data`, `register_parjob_allocator()`) -- only the bodies of its two hook implementations change.
|
||||
- **`parjob_prepare_allocation()` (currently also a stub -- see Req 4 AC 6) must be fixed alongside `my_alloc()`**: it is the hook that populates `ctx->alloc_req.host`/`ctx->alloc_req.shared` (via `calc_wanted_host_resvals`/`calc_wanted_global_resvals`, the same calls `seqjob_prepare_allocation()` already makes) for whichever bucket is currently being tried. `xce_sched_ctx_t` already carries `nhosts`/`nslots` fields for exactly this; the per-bucket loop sets `ctx->nslots = bucket_slots[b]` (`ctx->nhosts = 1`) before each host try, and the fixed hook reads them instead of the hardcoded `1, 1` seqjob's version uses. Because `ctx->alloc_req` is a single scratch buffer reset and repopulated per candidate host (`plugin_prepare_allocation()`, `plugin.c:591`, `xvec_idval_fini`+`init`s it before each fan-out), this is safe to recompute per host-try -- see Req 4 AC 6 for the corollary `seqjob_prepare_allocation()` gate this depends on.
|
||||
- **Do not bypass `plugin_prepare_allocation()`'s fan-out to precompute a bucket's resource want once and reuse it across every candidate host.** The rusage-derived portion (`calc_wanted_host_resvals`) is host-independent and in principle cacheable per bucket, but the same function's tail call to `sched_fill_limit_requests(sched, &ctx->alloc_req.host, hlimits)` reads `hlimits = zarray_get(&obj->hlimits, ctx->worker->idx)` -- genuinely host-specific per-host limit-tracking state. A full per-bucket bypass would silently stop enforcing per-host limits for jobs allocated through the new iterator path. Caching only the rusage-derived slice (still running the limit-fill per host) is a valid future optimization but is not required for this spec.
|
||||
|
||||
**Final allocation result -- merged from each claimed bucket, no new type needed.** `xce_job_alloc_t` (`xce/types.h`) already generalizes past one host: `nhosts` + `xce_host_alloc_t** allocs`. Today only `seqjob_allocator` exercises it, always with `nhosts=1` via the variadic `make_host_allocs(1, halloc)`. For a successful multi-bucket plan, `commit_plan_allocation()` builds one `xce_host_alloc_t` per claimed host (each from that bucket's own snapshotted `alloc_req.host`/`uncomit`, captured at the moment `try_alloc_on_host` succeeded for it -- `alloc_req.host` is overwritten on the next bucket's prepare/check cycle, so it cannot be read lazily after the loop moves on), collects them into an `xce_host_alloc_t** hallocs` array of length `nbuckets` (built directly with `calloc`+loop, since `make_host_allocs()`'s variadic signature only fits a compile-time-fixed argument list, not a runtime-sized claimed set), and calls the existing `make_job_alloc(jobid, nbuckets, hallocs)` unchanged. Shared-resource allocation (`allocs_shared`/`nres_shared`) stays job-level and computed once for the whole primary resreq, exactly as it is today -- it does not vary per bucket and is not part of the merge.
|
||||
|
||||
### `seqjob_allocator` changes (Req 9 AC 1)
|
||||
|
||||
- Internal rewrap: `my_alloc()` is restructured to use the plan and host iterator interfaces internally as a trivial case (1 host x 1 slot plan, then host iteration matching today's `sched_collect_workers()` + `xlist_find()` logic).
|
||||
- Externally observable behavior is unchanged -- same host selection given same state.
|
||||
- **`seqjob_prepare_allocation()` gains a `primary->maxTasks == 1` guard** (Req 4 AC 6). This is a no-op for seqjob's own observable behavior -- it never handles `maxTasks > 1` jobs regardless -- but is required for `parjob_allocator` correctness once its own prepare hook stops being a stub; see Req 4 AC 6 for why the unguarded fan-out is unsafe.
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
|
|
@ -444,8 +662,8 @@ The default policy (`lowest_priority_first`) generates victim sets lazily: small
|
|||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────────┐ │
|
||||
│ │ plan_iter->next() │ ←─ generates (nhosts, │
|
||||
│ │ returns plan or NULL │ slots_per_host) candidates │
|
||||
│ │ plan_iter->next() │ ←─ generates per-host bucket │
|
||||
│ │ returns plan or NULL │ partitions (e.g. [2,1]) │
|
||||
│ └──────────┬───────────┘ most-preferred first │
|
||||
│ │ │
|
||||
│ plan │ NULL (exhausted) │
|
||||
|
|
@ -477,6 +695,8 @@ The default policy (`lowest_priority_first`) generates victim sets lazily: small
|
|||
└──────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Simplified -- see "Composition Algorithm" above for the authoritative version.** This diagram shows one worker being tried per plan, which is exactly right for `seqjob_allocator` (`nbuckets == 1`) but is NOT the full picture for `parjob_allocator`'s multi-bucket plans: a real plan with `nbuckets > 1` requires an inner loop over each bucket, accumulating a distinct claimed host per bucket before the plan counts as satisfied (host-exclusivity, per-bucket resource snapshotting, and rollback-on-partial-failure all live in that inner loop, none of which fits cleanly into this box diagram). Treat this diagram as the single-bucket special case; the pseudocode under "Composition Algorithm" is the one to implement against.
|
||||
|
||||
### Unsatisfiable Resreq Cache (Req 7)
|
||||
|
||||
```
|
||||
|
|
@ -552,33 +772,38 @@ When `alloc_mode != XCE_ALLOC_NORMAL`, resource-availability checks count the fu
|
|||
|
||||
### Full Preemption Flow
|
||||
|
||||
**Cache placement, corrected:** the cache is keyed by `resreq_fp`, which lives on `xce_primary_schedobj_t` -- a PRIMARY-level concept, not a whole-job one (a COMPOUND/ALTER job has no single resreq shape of its own to fingerprint). So the cache check wraps `allocate_primary()` itself, not `allocate_job()`'s top-level dispatch. For a bare PRIMARY job this is the whole story; for a COMPOUND or ALTER job, each nested primary gets its own cache check for free, once for every call `allocate_compound()`/`allocate_alter()` already makes into `allocate_primary()` -- no job-level cache entry is needed or possible.
|
||||
|
||||
```c
|
||||
int allocate_job(sched, job) {
|
||||
// 1. Cache check
|
||||
if (cache_lookup(job)) return pend(job);
|
||||
// Wraps allocate_primary() -- called for EVERY primary reached, whether the
|
||||
// job's own resreq IS a primary, or this is one primary nested inside a
|
||||
// COMPOUND (allocate_compound()'s loop) or an ALTER's compound (transitively).
|
||||
int allocate_primary_with_preempt(sched, primary_resreq, ctx, job) {
|
||||
// 1. Cache check (Req 7) -- keyed on THIS primary's resreq_fp, not job-level
|
||||
if (cache_lookup(primary_resreq, job)) return PEND;
|
||||
|
||||
// 2. Normal allocation
|
||||
if (allocate_primary(sched, job->resreq, ctx) > 0) return dispatch(job);
|
||||
if (allocate_primary(sched, primary_resreq, ctx) > 0) return ALLOC_SUCCESS;
|
||||
|
||||
// 3. Preemption (if applicable)
|
||||
if (job->queue->preemptible_queues == NULL) {
|
||||
cache_insert(job);
|
||||
return pend(job);
|
||||
cache_insert(primary_resreq, job);
|
||||
return PEND;
|
||||
}
|
||||
|
||||
victim_iter_init(&viter, job, sched);
|
||||
while ((victims = viter.next(&viter)) != NULL) {
|
||||
ctx.alloc_mode = XCE_ALLOC_PREEMPT;
|
||||
if (allocate_primary(sched, job->resreq, &ctx) > 0) {
|
||||
if (allocate_primary(sched, primary_resreq, &ctx) > 0) {
|
||||
viter.fini(&viter);
|
||||
return commit_preempt(victims, job);
|
||||
return commit_preempt(victims);
|
||||
}
|
||||
ctx.alloc_mode = XCE_ALLOC_NORMAL;
|
||||
}
|
||||
viter.fini(&viter);
|
||||
|
||||
cache_insert(job);
|
||||
return pend(job);
|
||||
cache_insert(primary_resreq, job);
|
||||
return PEND;
|
||||
}
|
||||
```
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue