docs(spec): fairshare belongs to projects; add project-fairshare brief

Fairshare weights and usage history live on the project tree; queues keep
priority between queues and only choose fairshare or FCFS ordering. The
per-queue `fairshare:` user lists in queues.yaml go away.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-13 06:50:34 +00:00
parent f3d4dba970
commit 16b9059116
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
5 changed files with 84 additions and 7 deletions

View file

@ -50,8 +50,8 @@ borrowing threshold instead of a cap; `max` becomes the cap).
## Scope
- **In**: `max`, borrowing rule, protection of owners, reclaim policy,
owned/borrowed accounting, visibility.
- **Out**: owned shares and caps (`project-config`); fairshare ordering
(queues turn it on; weights from the project tree, later);
- **Out**: owned shares and caps (`project-config`); fairshare
(`project-fairshare`);
accounting/billing history.
## Boundary Candidates

View file

@ -72,8 +72,8 @@ This feature makes projects first-class cluster configuration entities that cont
- job filtering by project.
- **Out of scope**:
- borrowing idle shares above a project's own (`max`), protecting owners from borrowers, and reclaim by preemption (`project-borrowing`);
- fairshare;
- any project influence on job ordering or priority (queues own ordering);
- fairshare (`project-fairshare`: weights and usage history on the project tree, decided 2026-10-12);
- any project influence on job ordering or priority in this feature (fairshare ordering from the project tree comes with `project-fairshare`);
- accounting history and billing of finished jobs;
- runtime edits of a project's members, admins or shares by project admins (`project-delegated-membership`; in this stage only the cluster administrator changes them, through cluster config);
- per-user default projects in `users.yaml`;

View file

@ -0,0 +1,75 @@
# Brief: project-fairshare
(2026-10-12, user decision during the project-config discussion:
"fairshare is actually done by projects". Backlog: starts after
`project-config`.)
## Problem
Jobs are ordered only by queue priority, then job priority, then queue
time (`scheduler/src/plugins/builtin_fcfs.c`, `fcfs_job_cmp`). A project
or user that submits a lot gets ahead of everyone else in the same queue,
however much it has already used. Owned shares (`project-config`) cap what
a project holds at once; they do not decide whose pending job goes first,
or make up over time for projects and users that used less.
## Current State
- `queues.yaml` carries LSF-style per-queue user shares
(`fairshare: [{user, share}]`, `configs/schema/queues.schema.json`
`FairShare`); `xceadmin` maps nothing from them (`xcecommon/mapping.py`,
QueueConf has no field) and the scheduler has no fairshare code.
- The project tree, owned shares and member quotas come with
`project-config`.
## Desired Outcome (decided: fairshare belongs to projects)
- **Entitlement from the project tree.** Each project has a fairshare
weight among its siblings; users inside a project share it (equal by
default). There are no fairshare lists in queues; the `fairshare:` key
is removed from `queues.yaml` and its schema.
- **Usage history per project and user**: decayed consumption of
allocated resources over time, charged to the job's project, every
ancestor and the user within the project.
- **Ordering**: pending jobs ranked by the hierarchical fairshare of their
(project, user) — siblings compared by usage relative to weight, then
down the tree (Slurm "fair tree" precedent), applied across all queues
in the same way.
- Visibility: each project's and user's weight, usage, and fairshare
rank or factor (REST, `xcecli`, REPL), and the fairshare position in
pend reasons or job details.
- LSF migration: queue `FAIRSHARE` user shares are reported, mapped to
project weights where a mapping is obvious.
## Proposals (for discussion)
1. **Weight**: a single `fairshare` number per project (default: equal
among siblings), separate from the per-resource `share`, because
fairshare needs one weight and shares are per resource. Alternative:
derive it from the `slots` share when declared.
2. **Usage measure**: a billing-weighted sum of allocated resources ×
time (e.g. slots, GPUs and memory with cluster-wide weights), with a
cluster-wide half-life for decay.
3. **How it combines with the queue**: queues keep their priority between
queues (policy); within a queue, jobs are ordered by fairshare, then job
priority, then queue time. A queue may choose plain FCFS instead
(`order: fcfs`), for example for a debug queue. The queue chooses whether
to use fairshare; the project tree supplies all of its data.
4. **Same tree, two jobs**: shares cap what a project holds at once;
fairshare decides who goes first and evens out use over time.
With strict shares, fairshare matters most among users of one project,
among children drawing from the same unassigned part, and later with
borrowing (`project-borrowing`).
## Constraints
- Usage history is scheduler state that must survive failover and restart:
scheduler snapshot and any source it is rebuilt from must carry it
completely (`persistence.md`, field completeness); live and simulator
identical (driver equivalence).
- Ordering is a scheduler plugin concern (`builtin_fcfs.c` is the current
orderer); the fairshare ranking must be cheap per round for large
pending queues.
- Full 3-phase workflow (scheduling policy + persisted state).
## Upstream / Downstream
- **Upstream**: `project-config` (tree, members), scheduler ordering
plugins, `job-config-layering.md`.
- **Downstream**: `project-borrowing` (who borrows first),
`lsf-config-migration` (queue FAIRSHARE), `admin-config-tool` (schema
change).

View file

@ -20,9 +20,9 @@ XCE instead gives **every attribute exactly one owning level**.
| Level | Owns | Never owns |
|---|---|---|
| **Queue**: scheduling policy | priority and ordering, host partition, dispatch/run windows, preemption policy, per-job ceilings (max run time, max tasks, max hosts, ...) | resource requirement, anything about how a job executes (pre/post-exec, env, launch, limits enforced on the job), aggregate caps on concurrent use |
| **Queue**: scheduling policy | priority between queues and the ordering policy (fairshare or FCFS), host partition, dispatch/run windows, preemption policy, per-job ceilings (max run time, max tasks, max hosts, ...) | resource requirement, anything about how a job executes (pre/post-exec, env, launch, limits enforced on the job), aggregate caps on concurrent use |
| **Application profile**: what the application needs and how it runs (essentials plus default reservations, 2026-10-12) | resreq (select / rusage / span / same / order), task range (min / max), execution mode and options, launch type, env, **pre/post-exec**, execution limits (mem, runtime, CPU, tasks) | priority, queue choice, shares |
| **Project**: resource ownership | membership, owned shares of resources (`share`, `member_quota`), later borrowing and fairshare weights, accounting | how a job runs or is ordered |
| **Project**: resource ownership | membership, owned shares of resources (`share`, `member_quota`), **fairshare** (weights and usage history; user decision 2026-10-12), later borrowing, accounting | how a job runs or is ordered |
| **Job** | its own values, only for attributes its profile allows | n/a |
Access to each is separate: queue and profile by RBAC permission, project by
@ -143,7 +143,8 @@ them:
| Concept | Question it answers | Owner | Examples |
|---|---|---|---|
| **Supply** | what a host has | host / resource config | slots per host (LSF `MXJ`), host resources |
| **Share** (ownership) | how much a group of jobs owns and may hold at once | **project** | `share`, `member_quota`; later borrowing (`max`), fairshare weight |
| **Share** (ownership) | how much a group of jobs owns and may hold at once | **project** | `share`, `member_quota`; later borrowing (`max`) |
| **Fairshare** (entitlement over time) | whose pending job goes first, evening out use over time | **project** (weights, usage); the queue only chooses fairshare or FCFS ordering | `project-fairshare` |
| **Ceiling** (job shape) | how big one job may be | queue | max tasks, max run time, max hosts per job |
| **Execution limit** (enforcement) | what the running job may use | application profile | cgroup memory, runtime, CPU |

View file

@ -120,6 +120,7 @@ Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generi
## Future Specs / Backlog (dependency order)
- [ ] gang-groups — NEW 2026-10-11; **demoted to backlog 2026-10-12** (user decision N24 in `resource-questions.md`). LLM training is one parallel job with topology requirements (PR-10), so it needs no gang. Re-scoped as a **co-start group** of separately managed jobs and services: RL post-training (trainer, rollout engines, reward and reference models), disaggregated serving (prefill and decode pools, router), streaming pipelines. All-or-nothing start, close placement, group failure policy. Revisit when RL or disaggregated serving reaches the roadmap. Brief: `.kiro/specs/gang-groups/brief.md`. Dependencies: resource-instances, resource-claims, service-replica-set.
- [ ] project-borrowing — NEW 2026-10-09 as project-guarantee (configuration layering, backlog); renamed and narrowed 2026-10-12: ownership moved into project-config's owned shares. A `max` above a project's share, borrowing idle capacity whose owner is not waiting for it, reclaim through preemption, owned/borrowed accounting. Brief: `.kiro/specs/project-borrowing/brief.md`. Dependencies: project-config, scheduler-preemption.
- [ ] project-fairshare — NEW 2026-10-12 (user decision: fairshare is done by projects). Fairshare weights on the project tree (users equal within a project by default), decayed usage history per project and user, hierarchical ranking applied in every queue that orders by fairshare (queues keep priority between queues and may choose FCFS); the per-queue `fairshare:` user lists are removed from `queues.yaml`. Usage history persisted in the scheduler snapshot. Brief: `.kiro/specs/project-fairshare/brief.md`. Dependencies: project-config.
- [ ] limits-to-owners — NEW 2026-10-12 (configuration layering, user decisions; backlog, no brief yet). Retire `limits.yaml` (not in production, removed without deprecation): the limit plugin becomes internal, fed only by generated limits. `USERS`/`PER_USER` caps become `member_quota` (root project for a cluster-wide per-user cap), a user group's total becomes a project quota, `HOSTS`/`PER_HOST` slot caps move to host supply in host config (LSF `MXJ`), queue aggregate caps and mixed tuples (user group × queue) are dropped. Touches `xceadmin` mapping and schema (`admin-config-tool`), `ClusterConfig.limits`, `src/configs/limits.yaml`, scheduler sim fixtures and the limit tests. Rules: `job-config-layering.md` "Resource control". Dependencies: project-config.
- [ ] job-resize — NEW 2026-10-10 (from long-running services discovery; not on the services path). Grow/shrink a running parallel job's allocation (LSF resizable jobs, `bresize`) for elastic applications. Brief: `.kiro/specs/job-resize/brief.md`. Dependencies: parjob-placement, host-failure-job-recovery.
- [ ] project-delegated-membership — NEW 2026-10-10 (from project-config requirements, user decision: future). Project managers edit their project's (and subtree's) members at runtime without an admin config push; needs a durable store that coexists with `xceadmin` config pushes (e.g. a separate delegated member list) and audit. Dependencies: project-config, rbac-persistence.