myxce/.kiro/specs/parjob-placement/requirements.md

Requirements Document

Project Description (Input)

Scope (renamed 2026-10-09 from parjob-capacity-partitioning): complete placement of parallel jobs, LSF-style. Covers:

1. Task range (-n min,max) — place as many tasks as fit, up to maxTasks; if minTasks cannot be placed, pend after one search (feasibility is monotone in the task count for nested per-task requests). bug-1931 delivers a first, min-first + upward-search version on today's balanced plans; this spec owns the final behaviour.
2. Capacity-driven partitioning — plan shapes from the hosts' free capacity (below).
3. span[ptile=n] — exactly n tasks per host (the last host may get fewer, as in LSF).
4. span[hosts=1] — all tasks on one host.
5. span[block=n] — tasks per host a multiple of n.
6. Host-count limit (nhosts) — bounds on how many hosts a job may use (span[hosts=n] style; exact syntax and min/max form to settle in requirements).
7. Cross-cutting — parse span[...] into xce_span_t (today section.peg:55 only captures it as raw spanString, and xce_span_t is an empty struct, resreq.h:233); validation and conflicting options (with each other and with -n min,max); decision-trace and pend-reason wording (e.g. "ptile=4 but no host has 4 free slots"); submission and display in the apiserver, xcecli and job detail.

Out of scope for now: same[...] and affinity[...] (also reserved, unused).

Each span option becomes a constraint on the capacity-driven heuristic (ptile fixes per-host counts, hosts=1 forces k=1, block rounds per-host counts down to multiples of n, nhosts bounds k), which is why they are designed together.

Who has the problem: users submitting parallel (multi-task, maxTasks > 1) jobs, and the operators who must explain why such jobs pend.

Current situation: parjob_plan_iter_next() (src/scheduler/src/plugins/parjob_allocator.c) generates exactly one plan per host count k = 1..maxTasks, always the balanced ceil/floor partition (e.g. N=5, k=2 → only [3,2]). Plan shape is a pure function of maxTasks and never looks at worker state (scheduler-allocation requirements.md, Req 4 AC 1 and "Data Structure Changes Summary", v1 balanced default). Unbalanced splits such as [4,1] or [3,1,1] are never generated, so a job pends even when the cluster's free capacity could hold it (e.g. two hosts with 4 and 1 free slots for a 5-task job). The per-position first-fit host claim in sched_claim_bucket_() (alloc_runner.c) is already complete for balanced plans (nested per-bucket requests, largest bucket first); the coverage gap is only the partition shape. Exhaustive partition enumeration is infeasible (p(100) ≈ 1.9×10⁸).

What should change: derive plan shape from the candidate hosts' free capacity instead of from maxTasks alone — order the candidate hosts (by preference / free slots) and greedily give each host min(what fits, tasks remaining) until every task is placed or the hosts run out. Requirements to settle in discussion:

• keep the fewest-hosts-first preference and bounded cost (~O(hosts) per attempt);
• keep the decision trace meaningful (plan / position / host verdicts, plan counts) and pend reasons recorded once per job (add_job_preason_once, bug-1927);
• relationship to the existing balanced plans (replace, or try balanced first and fall back to capacity-driven);
• this supersedes the "pure function of maxTasks" constraint in the scheduler-allocation spec, which must be amended accordingly;
• multi-dimensional requests (slots + mem etc.): "what fits" per host is the max task count satisfying every resource;
• interaction with snapshot/equivalence tests (test_alloc_equivalence_parjob.c) and the sim decision trace.

Candidate Approaches (discussion notes, 2026-10-09)

Groundwork shared by every heuristic — per-host fit. For each candidate host that passes the existing host-selection pre-filter:

fit(h) = min over requested resources r of floor(free_r(h) / per_task_r)

Fit values only shape plans. Every host still goes through the authoritative per-host check in sched_try_alloc_on_host_(), and the first-fit position filling in sched_claim_bucket_() (alloc_runner.c) stays valid: a plan's parts are still ordered largest first, and a host that fits a larger part also fits a smaller one, so first-fit loses nothing.

Heuristics considered

1. Largest-fit greedy (fewest hosts). Sort hosts by fit, largest first; take hosts until their fits add up to at least N, giving each min(fit, remaining). Always finds the fewest possible hosts (the smallest k whose k largest fits cover N). O(H log H). Produces [4,1] for hosts with fits 4 and 1 and N=5.
2. Balanced split under the caps. Once k is known, spread the N tasks as evenly as each host's fit allows: each host gets min(fit, level), with level chosen so the total is N. Keeps the even split the balanced plans aim for (good for tightly synchronized tasks, e.g. MPI) while respecting capacity.
3. Best-fit to limit fragmentation. Among ways to cover N with k hosts, prefer hosts whose fit is close to the share they receive. Simple version: run greedy, then swap the last chosen host for the smallest-fit host that still covers the remainder. Keeps large free holes for later big jobs, at the cost of a slightly less even split.
4. Today's balanced plans as a fallback. Keep the existing per-k balanced plans as last resorts, so current behaviour is a subset and existing equivalence tests keep passing on clusters where balanced plans already succeed.
5. Locality scoring (later). Rank hosts by rack/switch affinity before applying 1–3; plugs into the span/same/affinity fields the scheduler-allocation spec already reserves.

Rejected: search-based methods (local search, simulated annealing, beam search). The plan generator runs every scheduling cycle for every pending parallel job, so its cost must be small, bounded and deterministic; determinism also matters for the simulator's decision trace and the equivalence tests.

Recommended direction (to confirm)

Plan list, in order:

1. largest-fit greedy plan (fewest hosts);
2. balanced split under the caps on those same k hosts;
3. today's balanced plans as a fallback.

At most about two plans plus the existing N. Configurable: the preference between balance and fewest hosts (or fragmentation), and whether the fallback plans are kept at all.

Points the requirements must settle

• Reading live worker state. The plan generator would read worker state, breaking the "plan shape depends only on maxTasks" rule (scheduler-allocation Req 4 AC 1) — that spec must be amended — and must take the same read lock the host iterator uses.
• Stale fit values. A fit computed at plan time can be out of date by the time a host is checked; the per-host check stays authoritative and a failed plan moves on to the next one.
• Decision trace. The trace should name the heuristic that produced each plan, e.g. plan 1 (min-hosts) [4,1], so pend explanations stay readable.
• Pend reasons. Still recorded once per job (add_job_preason_once, bug-1927).

Task-range search and span constraints (discussion notes, 2026-10-09)

Split (agreed). bug-1931 (e3b87d28) already reuses the plan search for a fixed task count (sched_alloc_count_(), alloc_runner.c) and bisects the range (sched_alloc_range_()). Every span constraint (ptile, hosts=1, block, host-count limit, capacity-driven shapes) lives in the planner for a fixed N; the range search only chooses N. The search needs one contract from the planner: over the job's allowed counts, "fits with N ⇒ fits with every smaller allowed count". Each planned shape keeps it (balanced, capacity-greedy, ptile, hosts=1, nhosts max), provided the allowed counts are right. The range search asks the planner (optional allocator hook, default "every N in [min, max]") for the ordered allowed counts and bisects over that list.

LSF 10.1 reference (../misc/lsf/IBM_Spectrum_LSF_10.1_Administering_LSF_Admin_Guide.pdf, PDF pages):

• span[block=n] (pp. 340-341): only multiples of n are allocated. -n 2,13 → 4, 8 or 12; -n 5, -n 2, -n 3, -n 2,3, -n 9,10 are rejected; the range minimum is silently rounded up (-n 2,8 → 4,8). Packing: as many blocks as fit on a host, then the next; each host checked once (-n 9 block=3, hosts with 6 free → 6 + 3). block cannot be combined with ptile or hosts. Allowed counts: multiples of n.
• span[ptile=n] with a range (p. 339, written for resizable jobs, stated to apply to the first allocation as well): either the maximum (n per host, one host with fewer) or a multiple of n below it (-n 1,14 ptile=4 → 14, or 4, 8, 12). Allowed counts: multiples of n plus max (requirements decide whether to follow this or allow every N).
• span[stripe] / span[stripe=max] (p. 337): even spread over the free candidate hosts, extra tasks to the first hosts; stripe=max caps per-host tasks. Candidate scope item (close to today's balanced plans).
• Host-count limit: LSF has only span[hosts=1] per job; the host limit is the queue's HOSTLIMIT_PER_JOB (p. 663), which pends a job that needs more hosts and rejects a ptile job whose host count is known at submit (the guide's -n 1,4096 ptile=1 example instead says the job is limited to 20 hosts; pick one). A per-job min/max host count would be an XCE extension.
• Submission must reject a task range containing no allowed count.

Where span comes from (/kiro-discovery 2026-10-09): the job's effective resreq produced by job-config-merge: the application profile's resreq (application-profile) overridden section by section by the job's -R; queues carry no resreq. A host-count limit from queue policy is a queue hard limit in that merge.

Status: initialized only; requirements to be discussed with the user before /kiro-spec-requirements.

Requirements