docs: log bug-1931/1932, init parjob-capacity-partitioning spec

- bug-1931: the parjob allocator ignores minTasks (a tasks 2..5 job pends
  although 2 slots are free).
- bug-1932: the unsatisfiable-request cache drops pend reasons on a hit
  and is unused in the simulator.
- .kiro/specs/parjob-capacity-partitioning: initialized spec with the
  candidate heuristics for capacity-driven plan generation.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-09 08:02:00 +00:00
parent a64d0846c7
commit cff3304f5d
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
5 changed files with 270 additions and 0 deletions

View file

@ -0,0 +1,53 @@
# Requirements Document
## Project Description (Input)
**Who has the problem:** users submitting parallel (multi-task, `maxTasks > 1`) jobs, and the operators who must explain why such jobs pend.
**Current situation:** `parjob_plan_iter_next()` (`src/scheduler/src/plugins/parjob_allocator.c`) generates exactly one plan per host count k = 1..maxTasks, always the balanced ceil/floor partition (e.g. N=5, k=2 → only `[3,2]`). Plan shape is a pure function of `maxTasks` and never looks at worker state (`scheduler-allocation` requirements.md, Req 4 AC 1 and "Data Structure Changes Summary", v1 balanced default). Unbalanced splits such as `[4,1]` or `[3,1,1]` are never generated, so a job pends even when the cluster's free capacity could hold it (e.g. two hosts with 4 and 1 free slots for a 5-task job). The per-position first-fit host claim in `sched_claim_bucket_()` (`alloc_runner.c`) is already complete for balanced plans (nested per-bucket requests, largest bucket first); the coverage gap is only the partition shape. Exhaustive partition enumeration is infeasible (p(100) ≈ 1.9×10⁸).
**What should change:** derive plan shape from the candidate hosts' free capacity instead of from `maxTasks` alone — order the candidate hosts (by preference / free slots) and greedily give each host min(what fits, tasks remaining) until every task is placed or the hosts run out. Requirements to settle in discussion:
- keep the fewest-hosts-first preference and bounded cost (~O(hosts) per attempt);
- keep the decision trace meaningful (plan / position / host verdicts, plan counts) and pend reasons recorded once per job (`add_job_preason_once`, bug-1927);
- relationship to the existing balanced plans (replace, or try balanced first and fall back to capacity-driven);
- this supersedes the "pure function of maxTasks" constraint in the `scheduler-allocation` spec, which must be amended accordingly;
- multi-dimensional requests (slots + mem etc.): "what fits" per host is the max task count satisfying every resource;
- interaction with snapshot/equivalence tests (`test_alloc_equivalence_parjob.c`) and the sim decision trace.
## Candidate Approaches (discussion notes, 2026-10-09)
**Groundwork shared by every heuristic — per-host fit.** For each candidate host that passes the existing host-selection pre-filter:
`fit(h) = min over requested resources r of floor(free_r(h) / per_task_r)`
Fit values only shape plans. Every host still goes through the authoritative per-host check in `sched_try_alloc_on_host_()`, and the first-fit position filling in `sched_claim_bucket_()` (`alloc_runner.c`) stays valid: a plan's parts are still ordered largest first, and a host that fits a larger part also fits a smaller one, so first-fit loses nothing.
### Heuristics considered
1. **Largest-fit greedy (fewest hosts).** Sort hosts by fit, largest first; take hosts until their fits add up to at least N, giving each min(fit, remaining). Always finds the fewest possible hosts (the smallest k whose k largest fits cover N). O(H log H). Produces `[4,1]` for hosts with fits 4 and 1 and N=5.
2. **Balanced split under the caps.** Once k is known, spread the N tasks as evenly as each host's fit allows: each host gets min(fit, level), with level chosen so the total is N. Keeps the even split the balanced plans aim for (good for tightly synchronized tasks, e.g. MPI) while respecting capacity.
3. **Best-fit to limit fragmentation.** Among ways to cover N with k hosts, prefer hosts whose fit is close to the share they receive. Simple version: run greedy, then swap the last chosen host for the smallest-fit host that still covers the remainder. Keeps large free holes for later big jobs, at the cost of a slightly less even split.
4. **Today's balanced plans as a fallback.** Keep the existing per-k balanced plans as last resorts, so current behaviour is a subset and existing equivalence tests keep passing on clusters where balanced plans already succeed.
5. **Locality scoring (later).** Rank hosts by rack/switch affinity before applying 1–3; plugs into the `span`/`same`/`affinity` fields the `scheduler-allocation` spec already reserves.
**Rejected: search-based methods** (local search, simulated annealing, beam search). The plan generator runs every scheduling cycle for every pending parallel job, so its cost must be small, bounded and deterministic; determinism also matters for the simulator's decision trace and the equivalence tests.
### Recommended direction (to confirm)
Plan list, in order:
1. largest-fit greedy plan (fewest hosts);
2. balanced split under the caps on those same k hosts;
3. today's balanced plans as a fallback.
At most about two plans plus the existing N. Configurable: the preference between balance and fewest hosts (or fragmentation), and whether the fallback plans are kept at all.
### Points the requirements must settle
- **Reading live worker state.** The plan generator would read worker state, breaking the "plan shape depends only on maxTasks" rule (`scheduler-allocation` Req 4 AC 1) — that spec must be amended — and must take the same read lock the host iterator uses.
- **Stale fit values.** A fit computed at plan time can be out of date by the time a host is checked; the per-host check stays authoritative and a failed plan moves on to the next one.
- **Decision trace.** The trace should name the heuristic that produced each plan, e.g. `plan 1 (min-hosts) [4,1]`, so pend explanations stay readable.
- **Pend reasons.** Still recorded once per job (`add_job_preason_once`, bug-1927).
Status: initialized only; requirements to be discussed with the user before `/kiro-spec-requirements`.
## Requirements
<!-- Will be generated in /kiro-spec-requirements phase -->

View file

@ -0,0 +1,24 @@
{
"feature_name": "parjob-capacity-partitioning",
"created_at": "2026-10-09T07:06:43Z",
"updated_at": "2026-10-09T07:08:41Z",
"language": "en",
"phase": "initialized",
"approvals": {
"requirements": {
"generated": false,
"approved": false
},
"design": {
"generated": false,
"approved": false
},
"tasks": {
"generated": false,
"approved": false
}
},
"ready_for_implementation": false
}