diff --git a/.kiro/specs/parjob-capacity-partitioning/requirements.md b/.kiro/specs/parjob-placement/requirements.md similarity index 77% rename from .kiro/specs/parjob-capacity-partitioning/requirements.md rename to .kiro/specs/parjob-placement/requirements.md index cd2d8486..83d63fb1 100644 --- a/.kiro/specs/parjob-capacity-partitioning/requirements.md +++ b/.kiro/specs/parjob-placement/requirements.md @@ -1,6 +1,19 @@ # Requirements Document ## Project Description (Input) +**Scope (renamed 2026-10-09 from `parjob-capacity-partitioning`):** complete placement of parallel jobs, LSF-style. Covers: +1. **Task range (`-n min,max`)** — place as many tasks as fit, up to `maxTasks`; if `minTasks` cannot be placed, pend after one search (feasibility is monotone in the task count for nested per-task requests). bug-1931 delivers a first, min-first + upward-search version on today's balanced plans; this spec owns the final behaviour. +2. **Capacity-driven partitioning** — plan shapes from the hosts' free capacity (below). +3. **`span[ptile=n]`** — exactly n tasks per host (the last host may get fewer, as in LSF). +4. **`span[hosts=1]`** — all tasks on one host. +5. **`span[block=n]`** — tasks per host a multiple of n. +6. **Host-count limit (nhosts)** — bounds on how many hosts a job may use (`span[hosts=n]` style; exact syntax and min/max form to settle in requirements). +7. **Cross-cutting** — parse `span[...]` into `xce_span_t` (today `section.peg:55` only captures it as raw `spanString`, and `xce_span_t` is an empty struct, `resreq.h:233`); validation and conflicting options (with each other and with `-n min,max`); decision-trace and pend-reason wording (e.g. "ptile=4 but no host has 4 free slots"); submission and display in the apiserver, `xcecli` and job detail. + +Out of scope for now: `same[...]` and `affinity[...]` (also reserved, unused). + +Each span option becomes a constraint on the capacity-driven heuristic (ptile fixes per-host counts, hosts=1 forces k=1, block rounds per-host counts down to multiples of n, nhosts bounds k), which is why they are designed together. + **Who has the problem:** users submitting parallel (multi-task, `maxTasks > 1`) jobs, and the operators who must explain why such jobs pend. **Current situation:** `parjob_plan_iter_next()` (`src/scheduler/src/plugins/parjob_allocator.c`) generates exactly one plan per host count k = 1..maxTasks, always the balanced ceil/floor partition (e.g. N=5, k=2 → only `[3,2]`). Plan shape is a pure function of `maxTasks` and never looks at worker state (`scheduler-allocation` requirements.md, Req 4 AC 1 and "Data Structure Changes Summary", v1 balanced default). Unbalanced splits such as `[4,1]` or `[3,1,1]` are never generated, so a job pends even when the cluster's free capacity could hold it (e.g. two hosts with 4 and 1 free slots for a 5-task job). The per-position first-fit host claim in `sched_claim_bucket_()` (`alloc_runner.c`) is already complete for balanced plans (nested per-bucket requests, largest bucket first); the coverage gap is only the partition shape. Exhaustive partition enumeration is infeasible (p(100) ≈ 1.9×10⁸). diff --git a/.kiro/specs/parjob-capacity-partitioning/spec.json b/.kiro/specs/parjob-placement/spec.json similarity index 80% rename from .kiro/specs/parjob-capacity-partitioning/spec.json rename to .kiro/specs/parjob-placement/spec.json index 7792ebcc..d9b176ce 100644 --- a/.kiro/specs/parjob-capacity-partitioning/spec.json +++ b/.kiro/specs/parjob-placement/spec.json @@ -1,7 +1,7 @@ { - "feature_name": "parjob-capacity-partitioning", + "feature_name": "parjob-placement", "created_at": "2026-10-09T07:06:43Z", - "updated_at": "2026-10-09T07:08:41Z", + "updated_at": "2026-10-09T08:03:23Z", "language": "en", "phase": "initialized", "approvals": { diff --git a/.kiro/specs/pend-reason-push-dedup/requirements.md b/.kiro/specs/pend-reason-push-dedup/requirements.md new file mode 100644 index 00000000..f905181d --- /dev/null +++ b/.kiro/specs/pend-reason-push-dedup/requirements.md @@ -0,0 +1,25 @@ +# Requirements Document + +## Project Description (Input) +**Who has the problem:** operators of an XCE cluster, through the load on the controller and its HA standbys. Every pend-reason push becomes a durable, Raft-replicated log entry. + +**Current situation:** the scheduler publishes every PEND decision in every scheduling session. `xce_sched_driver_on_decision()` (`src/scheduler/src/driver.c:914-919`) wraps the decision and calls `sched_publish_pend_reason()` (`scheduler.c:1913`) without comparing it with what was pushed before. On the controller each push goes through `set_preason()` (`src/controller/job.c`) and `persist_job_hook_fun`'s `HOOK_DETAIL_SETPREASON` arm, which appends a pend-reason WAL record (`src/include/wal_types.h` ~235-265, bug-1692) that is replicated to the standbys. A job that stays pending for hours with the same reason therefore costs one network push plus one durable, replicated WAL write per session, multiplied by every pending job. + +**What should change:** when a job's pend reason in the current session is identical to the one last successfully pushed for it (full structural match, libxce `job_preason_eq()`, `preason.h`), the scheduler skips the push. Only real changes reach the controller. + +Questions to settle in requirements: +- **Baseline:** the last reason the controller *accepted* for the job, not merely the last one computed. A failed or retried push must not update it. +- **Losing the baseline:** after a scheduler restart or failover, or a controller failover or resync, the baseline starts empty, so every pending job is pushed once. +- **Controller-side clears:** `do_clr_preason()` (dispatch, requeue, other state changes) must invalidate the baseline so the next PEND is pushed even if it matches the old reason. +- **Memory:** cost per pending job — keep a deep copy (exact compare) or a hash (cheaper, tiny collision risk). +- **Where the baseline lives:** on the scheduler's job object, or in the driver. +- **Simulator and replay:** the decision stream, the decision trace and `--sim-verify` must still see every PEND decision; only the publish to the controller is skipped. +- **Observability:** count skipped pushes (log or stats). +- **Defense in depth (optional):** a controller-side guard that skips the WAL append when the incoming reason equals the stored one. + +Related: bug-1932 (session-level unsatisfiable-request cache, which skips repeated computation within one session), bug-1927 (pend reasons recorded once per job), spec `libxce-alloc-preason-equality` (`job_preason_eq`). + +Status: initialized only; requirements to be discussed with the user before `/kiro-spec-requirements`. + +## Requirements + diff --git a/.kiro/specs/pend-reason-push-dedup/spec.json b/.kiro/specs/pend-reason-push-dedup/spec.json new file mode 100644 index 00000000..c407f97e --- /dev/null +++ b/.kiro/specs/pend-reason-push-dedup/spec.json @@ -0,0 +1,24 @@ +{ + "feature_name": "pend-reason-push-dedup", + "created_at": "2026-10-09T08:19:27Z", + "updated_at": "2026-10-09T08:19:27Z", + "language": "en", + "phase": "initialized", + "approvals": { + "requirements": { + "generated": false, + "approved": false + }, + "design": { + "generated": false, + "approved": false + }, + "tasks": { + "generated": false, + "approved": false + } + }, + "ready_for_implementation": false +} + + diff --git a/.wolf/bugs/bug-1931.md b/.wolf/bugs/bug-1931.md index 4be417a4..bb7ea56b 100644 --- a/.wolf/bugs/bug-1931.md +++ b/.wolf/bugs/bug-1931.md @@ -64,4 +64,4 @@ TODO: required for fixed/partial ## References - Related: bug-1770 (wired real minTasks/maxTasks into the primary resreq; the allocator never consumed minTasks), bug-1932 (same trace; unsatisfiable cache not used in sim) -- Docs: .kiro/specs/scheduler-allocation/requirements.md (Req 4 AC 1), .kiro/specs/resreq-maxtasks-wiring/, .kiro/specs/parjob-capacity-partitioning/ +- Docs: .kiro/specs/scheduler-allocation/requirements.md (Req 4 AC 1), .kiro/specs/resreq-maxtasks-wiring/, .kiro/specs/parjob-placement/ (owns the full task-range behaviour together with capacity-driven partitioning and span options; this bug fix covers only the min-first + upward search on today's balanced plans)