docs(spec): fairshare belongs to projects; add project-fairshare brief
Fairshare weights and usage history live on the project tree; queues keep priority between queues and only choose fairshare or FCFS ordering. The per-queue `fairshare:` user lists in queues.yaml go away. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
f3d4dba970
commit
16b9059116
5 changed files with 84 additions and 7 deletions
|
|
@ -50,8 +50,8 @@ borrowing threshold instead of a cap; `max` becomes the cap).
|
|||
## Scope
|
||||
- **In**: `max`, borrowing rule, protection of owners, reclaim policy,
|
||||
owned/borrowed accounting, visibility.
|
||||
- **Out**: owned shares and caps (`project-config`); fairshare ordering
|
||||
(queues turn it on; weights from the project tree, later);
|
||||
- **Out**: owned shares and caps (`project-config`); fairshare
|
||||
(`project-fairshare`);
|
||||
accounting/billing history.
|
||||
|
||||
## Boundary Candidates
|
||||
|
|
|
|||
|
|
@ -72,8 +72,8 @@ This feature makes projects first-class cluster configuration entities that cont
|
|||
- job filtering by project.
|
||||
- **Out of scope**:
|
||||
- borrowing idle shares above a project's own (`max`), protecting owners from borrowers, and reclaim by preemption (`project-borrowing`);
|
||||
- fairshare;
|
||||
- any project influence on job ordering or priority (queues own ordering);
|
||||
- fairshare (`project-fairshare`: weights and usage history on the project tree, decided 2026-10-12);
|
||||
- any project influence on job ordering or priority in this feature (fairshare ordering from the project tree comes with `project-fairshare`);
|
||||
- accounting history and billing of finished jobs;
|
||||
- runtime edits of a project's members, admins or shares by project admins (`project-delegated-membership`; in this stage only the cluster administrator changes them, through cluster config);
|
||||
- per-user default projects in `users.yaml`;
|
||||
|
|
|
|||
75
.kiro/specs/project-fairshare/brief.md
Normal file
75
.kiro/specs/project-fairshare/brief.md
Normal file
|
|
@ -0,0 +1,75 @@
|
|||
# Brief: project-fairshare
|
||||
|
||||
(2026-10-12, user decision during the project-config discussion:
|
||||
"fairshare is actually done by projects". Backlog: starts after
|
||||
`project-config`.)
|
||||
|
||||
## Problem
|
||||
Jobs are ordered only by queue priority, then job priority, then queue
|
||||
time (`scheduler/src/plugins/builtin_fcfs.c`, `fcfs_job_cmp`). A project
|
||||
or user that submits a lot gets ahead of everyone else in the same queue,
|
||||
however much it has already used. Owned shares (`project-config`) cap what
|
||||
a project holds at once; they do not decide whose pending job goes first,
|
||||
or make up over time for projects and users that used less.
|
||||
|
||||
## Current State
|
||||
- `queues.yaml` carries LSF-style per-queue user shares
|
||||
(`fairshare: [{user, share}]`, `configs/schema/queues.schema.json`
|
||||
`FairShare`); `xceadmin` maps nothing from them (`xcecommon/mapping.py`,
|
||||
QueueConf has no field) and the scheduler has no fairshare code.
|
||||
- The project tree, owned shares and member quotas come with
|
||||
`project-config`.
|
||||
|
||||
## Desired Outcome (decided: fairshare belongs to projects)
|
||||
- **Entitlement from the project tree.** Each project has a fairshare
|
||||
weight among its siblings; users inside a project share it (equal by
|
||||
default). There are no fairshare lists in queues; the `fairshare:` key
|
||||
is removed from `queues.yaml` and its schema.
|
||||
- **Usage history per project and user**: decayed consumption of
|
||||
allocated resources over time, charged to the job's project, every
|
||||
ancestor and the user within the project.
|
||||
- **Ordering**: pending jobs ranked by the hierarchical fairshare of their
|
||||
(project, user) — siblings compared by usage relative to weight, then
|
||||
down the tree (Slurm "fair tree" precedent), applied across all queues
|
||||
in the same way.
|
||||
- Visibility: each project's and user's weight, usage, and fairshare
|
||||
rank or factor (REST, `xcecli`, REPL), and the fairshare position in
|
||||
pend reasons or job details.
|
||||
- LSF migration: queue `FAIRSHARE` user shares are reported, mapped to
|
||||
project weights where a mapping is obvious.
|
||||
|
||||
## Proposals (for discussion)
|
||||
1. **Weight**: a single `fairshare` number per project (default: equal
|
||||
among siblings), separate from the per-resource `share`, because
|
||||
fairshare needs one weight and shares are per resource. Alternative:
|
||||
derive it from the `slots` share when declared.
|
||||
2. **Usage measure**: a billing-weighted sum of allocated resources ×
|
||||
time (e.g. slots, GPUs and memory with cluster-wide weights), with a
|
||||
cluster-wide half-life for decay.
|
||||
3. **How it combines with the queue**: queues keep their priority between
|
||||
queues (policy); within a queue, jobs are ordered by fairshare, then job
|
||||
priority, then queue time. A queue may choose plain FCFS instead
|
||||
(`order: fcfs`), for example for a debug queue. The queue chooses whether
|
||||
to use fairshare; the project tree supplies all of its data.
|
||||
4. **Same tree, two jobs**: shares cap what a project holds at once;
|
||||
fairshare decides who goes first and evens out use over time.
|
||||
With strict shares, fairshare matters most among users of one project,
|
||||
among children drawing from the same unassigned part, and later with
|
||||
borrowing (`project-borrowing`).
|
||||
|
||||
## Constraints
|
||||
- Usage history is scheduler state that must survive failover and restart:
|
||||
scheduler snapshot and any source it is rebuilt from must carry it
|
||||
completely (`persistence.md`, field completeness); live and simulator
|
||||
identical (driver equivalence).
|
||||
- Ordering is a scheduler plugin concern (`builtin_fcfs.c` is the current
|
||||
orderer); the fairshare ranking must be cheap per round for large
|
||||
pending queues.
|
||||
- Full 3-phase workflow (scheduling policy + persisted state).
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream**: `project-config` (tree, members), scheduler ordering
|
||||
plugins, `job-config-layering.md`.
|
||||
- **Downstream**: `project-borrowing` (who borrows first),
|
||||
`lsf-config-migration` (queue FAIRSHARE), `admin-config-tool` (schema
|
||||
change).
|
||||
|
|
@ -20,9 +20,9 @@ XCE instead gives **every attribute exactly one owning level**.
|
|||
|
||||
| Level | Owns | Never owns |
|
||||
|---|---|---|
|
||||
| **Queue**: scheduling policy | priority and ordering, host partition, dispatch/run windows, preemption policy, per-job ceilings (max run time, max tasks, max hosts, ...) | resource requirement, anything about how a job executes (pre/post-exec, env, launch, limits enforced on the job), aggregate caps on concurrent use |
|
||||
| **Queue**: scheduling policy | priority between queues and the ordering policy (fairshare or FCFS), host partition, dispatch/run windows, preemption policy, per-job ceilings (max run time, max tasks, max hosts, ...) | resource requirement, anything about how a job executes (pre/post-exec, env, launch, limits enforced on the job), aggregate caps on concurrent use |
|
||||
| **Application profile**: what the application needs and how it runs (essentials plus default reservations, 2026-10-12) | resreq (select / rusage / span / same / order), task range (min / max), execution mode and options, launch type, env, **pre/post-exec**, execution limits (mem, runtime, CPU, tasks) | priority, queue choice, shares |
|
||||
| **Project**: resource ownership | membership, owned shares of resources (`share`, `member_quota`), later borrowing and fairshare weights, accounting | how a job runs or is ordered |
|
||||
| **Project**: resource ownership | membership, owned shares of resources (`share`, `member_quota`), **fairshare** (weights and usage history; user decision 2026-10-12), later borrowing, accounting | how a job runs or is ordered |
|
||||
| **Job** | its own values, only for attributes its profile allows | n/a |
|
||||
|
||||
Access to each is separate: queue and profile by RBAC permission, project by
|
||||
|
|
@ -143,7 +143,8 @@ them:
|
|||
| Concept | Question it answers | Owner | Examples |
|
||||
|---|---|---|---|
|
||||
| **Supply** | what a host has | host / resource config | slots per host (LSF `MXJ`), host resources |
|
||||
| **Share** (ownership) | how much a group of jobs owns and may hold at once | **project** | `share`, `member_quota`; later borrowing (`max`), fairshare weight |
|
||||
| **Share** (ownership) | how much a group of jobs owns and may hold at once | **project** | `share`, `member_quota`; later borrowing (`max`) |
|
||||
| **Fairshare** (entitlement over time) | whose pending job goes first, evening out use over time | **project** (weights, usage); the queue only chooses fairshare or FCFS ordering | `project-fairshare` |
|
||||
| **Ceiling** (job shape) | how big one job may be | queue | max tasks, max run time, max hosts per job |
|
||||
| **Execution limit** (enforcement) | what the running job may use | application profile | cgroup memory, runtime, CPU |
|
||||
|
||||
|
|
|
|||
|
|
@ -120,6 +120,7 @@ Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generi
|
|||
## Future Specs / Backlog (dependency order)
|
||||
- [ ] gang-groups — NEW 2026-10-11; **demoted to backlog 2026-10-12** (user decision N24 in `resource-questions.md`). LLM training is one parallel job with topology requirements (PR-10), so it needs no gang. Re-scoped as a **co-start group** of separately managed jobs and services: RL post-training (trainer, rollout engines, reward and reference models), disaggregated serving (prefill and decode pools, router), streaming pipelines. All-or-nothing start, close placement, group failure policy. Revisit when RL or disaggregated serving reaches the roadmap. Brief: `.kiro/specs/gang-groups/brief.md`. Dependencies: resource-instances, resource-claims, service-replica-set.
|
||||
- [ ] project-borrowing — NEW 2026-10-09 as project-guarantee (configuration layering, backlog); renamed and narrowed 2026-10-12: ownership moved into project-config's owned shares. A `max` above a project's share, borrowing idle capacity whose owner is not waiting for it, reclaim through preemption, owned/borrowed accounting. Brief: `.kiro/specs/project-borrowing/brief.md`. Dependencies: project-config, scheduler-preemption.
|
||||
- [ ] project-fairshare — NEW 2026-10-12 (user decision: fairshare is done by projects). Fairshare weights on the project tree (users equal within a project by default), decayed usage history per project and user, hierarchical ranking applied in every queue that orders by fairshare (queues keep priority between queues and may choose FCFS); the per-queue `fairshare:` user lists are removed from `queues.yaml`. Usage history persisted in the scheduler snapshot. Brief: `.kiro/specs/project-fairshare/brief.md`. Dependencies: project-config.
|
||||
- [ ] limits-to-owners — NEW 2026-10-12 (configuration layering, user decisions; backlog, no brief yet). Retire `limits.yaml` (not in production, removed without deprecation): the limit plugin becomes internal, fed only by generated limits. `USERS`/`PER_USER` caps become `member_quota` (root project for a cluster-wide per-user cap), a user group's total becomes a project quota, `HOSTS`/`PER_HOST` slot caps move to host supply in host config (LSF `MXJ`), queue aggregate caps and mixed tuples (user group × queue) are dropped. Touches `xceadmin` mapping and schema (`admin-config-tool`), `ClusterConfig.limits`, `src/configs/limits.yaml`, scheduler sim fixtures and the limit tests. Rules: `job-config-layering.md` "Resource control". Dependencies: project-config.
|
||||
- [ ] job-resize — NEW 2026-10-10 (from long-running services discovery; not on the services path). Grow/shrink a running parallel job's allocation (LSF resizable jobs, `bresize`) for elastic applications. Brief: `.kiro/specs/job-resize/brief.md`. Dependencies: parjob-placement, host-failure-job-recovery.
|
||||
- [ ] project-delegated-membership — NEW 2026-10-10 (from project-config requirements, user decision: future). Project managers edit their project's (and subtree's) members at runtime without an admin config push; needs a durable store that coexists with `xceadmin` config pushes (e.g. a separate delegated member list) and audit. Dependencies: project-config, rbac-persistence.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue