From 16b9059116e959e8617af0354944df3971d1dc3b Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 13 Oct 2026 06:50:34 +0000 Subject: [PATCH] docs(spec): fairshare belongs to projects; add project-fairshare brief Fairshare weights and usage history live on the project tree; queues keep priority between queues and only choose fairshare or FCFS ordering. The per-queue `fairshare:` user lists in queues.yaml go away. Co-Authored-By: Claude Opus 5.5 (1M context) --- .kiro/specs/project-borrowing/brief.md | 4 +- .kiro/specs/project-config/requirements.md | 4 +- .kiro/specs/project-fairshare/brief.md | 75 ++++++++++++++++++++++ .kiro/steering/job-config-layering.md | 7 +- .kiro/steering/roadmap.md | 1 + 5 files changed, 84 insertions(+), 7 deletions(-) create mode 100644 .kiro/specs/project-fairshare/brief.md diff --git a/.kiro/specs/project-borrowing/brief.md b/.kiro/specs/project-borrowing/brief.md index 80c68e98..60b81910 100644 --- a/.kiro/specs/project-borrowing/brief.md +++ b/.kiro/specs/project-borrowing/brief.md @@ -50,8 +50,8 @@ borrowing threshold instead of a cap; `max` becomes the cap). ## Scope - **In**: `max`, borrowing rule, protection of owners, reclaim policy, owned/borrowed accounting, visibility. -- **Out**: owned shares and caps (`project-config`); fairshare ordering - (queues turn it on; weights from the project tree, later); +- **Out**: owned shares and caps (`project-config`); fairshare + (`project-fairshare`); accounting/billing history. ## Boundary Candidates diff --git a/.kiro/specs/project-config/requirements.md b/.kiro/specs/project-config/requirements.md index ddffb131..6b25b83e 100644 --- a/.kiro/specs/project-config/requirements.md +++ b/.kiro/specs/project-config/requirements.md @@ -72,8 +72,8 @@ This feature makes projects first-class cluster configuration entities that cont - job filtering by project. - **Out of scope**: - borrowing idle shares above a project's own (`max`), protecting owners from borrowers, and reclaim by preemption (`project-borrowing`); - - fairshare; - - any project influence on job ordering or priority (queues own ordering); + - fairshare (`project-fairshare`: weights and usage history on the project tree, decided 2026-10-12); + - any project influence on job ordering or priority in this feature (fairshare ordering from the project tree comes with `project-fairshare`); - accounting history and billing of finished jobs; - runtime edits of a project's members, admins or shares by project admins (`project-delegated-membership`; in this stage only the cluster administrator changes them, through cluster config); - per-user default projects in `users.yaml`; diff --git a/.kiro/specs/project-fairshare/brief.md b/.kiro/specs/project-fairshare/brief.md new file mode 100644 index 00000000..c856b773 --- /dev/null +++ b/.kiro/specs/project-fairshare/brief.md @@ -0,0 +1,75 @@ +# Brief: project-fairshare + +(2026-10-12, user decision during the project-config discussion: +"fairshare is actually done by projects". Backlog: starts after +`project-config`.) + +## Problem +Jobs are ordered only by queue priority, then job priority, then queue +time (`scheduler/src/plugins/builtin_fcfs.c`, `fcfs_job_cmp`). A project +or user that submits a lot gets ahead of everyone else in the same queue, +however much it has already used. Owned shares (`project-config`) cap what +a project holds at once; they do not decide whose pending job goes first, +or make up over time for projects and users that used less. + +## Current State +- `queues.yaml` carries LSF-style per-queue user shares + (`fairshare: [{user, share}]`, `configs/schema/queues.schema.json` + `FairShare`); `xceadmin` maps nothing from them (`xcecommon/mapping.py`, + QueueConf has no field) and the scheduler has no fairshare code. +- The project tree, owned shares and member quotas come with + `project-config`. + +## Desired Outcome (decided: fairshare belongs to projects) +- **Entitlement from the project tree.** Each project has a fairshare + weight among its siblings; users inside a project share it (equal by + default). There are no fairshare lists in queues; the `fairshare:` key + is removed from `queues.yaml` and its schema. +- **Usage history per project and user**: decayed consumption of + allocated resources over time, charged to the job's project, every + ancestor and the user within the project. +- **Ordering**: pending jobs ranked by the hierarchical fairshare of their + (project, user) — siblings compared by usage relative to weight, then + down the tree (Slurm "fair tree" precedent), applied across all queues + in the same way. +- Visibility: each project's and user's weight, usage, and fairshare + rank or factor (REST, `xcecli`, REPL), and the fairshare position in + pend reasons or job details. +- LSF migration: queue `FAIRSHARE` user shares are reported, mapped to + project weights where a mapping is obvious. + +## Proposals (for discussion) +1. **Weight**: a single `fairshare` number per project (default: equal + among siblings), separate from the per-resource `share`, because + fairshare needs one weight and shares are per resource. Alternative: + derive it from the `slots` share when declared. +2. **Usage measure**: a billing-weighted sum of allocated resources × + time (e.g. slots, GPUs and memory with cluster-wide weights), with a + cluster-wide half-life for decay. +3. **How it combines with the queue**: queues keep their priority between + queues (policy); within a queue, jobs are ordered by fairshare, then job + priority, then queue time. A queue may choose plain FCFS instead + (`order: fcfs`), for example for a debug queue. The queue chooses whether + to use fairshare; the project tree supplies all of its data. +4. **Same tree, two jobs**: shares cap what a project holds at once; + fairshare decides who goes first and evens out use over time. + With strict shares, fairshare matters most among users of one project, + among children drawing from the same unassigned part, and later with + borrowing (`project-borrowing`). + +## Constraints +- Usage history is scheduler state that must survive failover and restart: + scheduler snapshot and any source it is rebuilt from must carry it + completely (`persistence.md`, field completeness); live and simulator + identical (driver equivalence). +- Ordering is a scheduler plugin concern (`builtin_fcfs.c` is the current + orderer); the fairshare ranking must be cheap per round for large + pending queues. +- Full 3-phase workflow (scheduling policy + persisted state). + +## Upstream / Downstream +- **Upstream**: `project-config` (tree, members), scheduler ordering + plugins, `job-config-layering.md`. +- **Downstream**: `project-borrowing` (who borrows first), + `lsf-config-migration` (queue FAIRSHARE), `admin-config-tool` (schema + change). diff --git a/.kiro/steering/job-config-layering.md b/.kiro/steering/job-config-layering.md index 964b6d73..72ed326e 100644 --- a/.kiro/steering/job-config-layering.md +++ b/.kiro/steering/job-config-layering.md @@ -20,9 +20,9 @@ XCE instead gives **every attribute exactly one owning level**. | Level | Owns | Never owns | |---|---|---| -| **Queue**: scheduling policy | priority and ordering, host partition, dispatch/run windows, preemption policy, per-job ceilings (max run time, max tasks, max hosts, ...) | resource requirement, anything about how a job executes (pre/post-exec, env, launch, limits enforced on the job), aggregate caps on concurrent use | +| **Queue**: scheduling policy | priority between queues and the ordering policy (fairshare or FCFS), host partition, dispatch/run windows, preemption policy, per-job ceilings (max run time, max tasks, max hosts, ...) | resource requirement, anything about how a job executes (pre/post-exec, env, launch, limits enforced on the job), aggregate caps on concurrent use | | **Application profile**: what the application needs and how it runs (essentials plus default reservations, 2026-10-12) | resreq (select / rusage / span / same / order), task range (min / max), execution mode and options, launch type, env, **pre/post-exec**, execution limits (mem, runtime, CPU, tasks) | priority, queue choice, shares | -| **Project**: resource ownership | membership, owned shares of resources (`share`, `member_quota`), later borrowing and fairshare weights, accounting | how a job runs or is ordered | +| **Project**: resource ownership | membership, owned shares of resources (`share`, `member_quota`), **fairshare** (weights and usage history; user decision 2026-10-12), later borrowing, accounting | how a job runs or is ordered | | **Job** | its own values, only for attributes its profile allows | n/a | Access to each is separate: queue and profile by RBAC permission, project by @@ -143,7 +143,8 @@ them: | Concept | Question it answers | Owner | Examples | |---|---|---|---| | **Supply** | what a host has | host / resource config | slots per host (LSF `MXJ`), host resources | -| **Share** (ownership) | how much a group of jobs owns and may hold at once | **project** | `share`, `member_quota`; later borrowing (`max`), fairshare weight | +| **Share** (ownership) | how much a group of jobs owns and may hold at once | **project** | `share`, `member_quota`; later borrowing (`max`) | +| **Fairshare** (entitlement over time) | whose pending job goes first, evening out use over time | **project** (weights, usage); the queue only chooses fairshare or FCFS ordering | `project-fairshare` | | **Ceiling** (job shape) | how big one job may be | queue | max tasks, max run time, max hosts per job | | **Execution limit** (enforcement) | what the running job may use | application profile | cgroup memory, runtime, CPU | diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index af5f2bbf..034c3d86 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -120,6 +120,7 @@ Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generi ## Future Specs / Backlog (dependency order) - [ ] gang-groups — NEW 2026-10-11; **demoted to backlog 2026-10-12** (user decision N24 in `resource-questions.md`). LLM training is one parallel job with topology requirements (PR-10), so it needs no gang. Re-scoped as a **co-start group** of separately managed jobs and services: RL post-training (trainer, rollout engines, reward and reference models), disaggregated serving (prefill and decode pools, router), streaming pipelines. All-or-nothing start, close placement, group failure policy. Revisit when RL or disaggregated serving reaches the roadmap. Brief: `.kiro/specs/gang-groups/brief.md`. Dependencies: resource-instances, resource-claims, service-replica-set. - [ ] project-borrowing — NEW 2026-10-09 as project-guarantee (configuration layering, backlog); renamed and narrowed 2026-10-12: ownership moved into project-config's owned shares. A `max` above a project's share, borrowing idle capacity whose owner is not waiting for it, reclaim through preemption, owned/borrowed accounting. Brief: `.kiro/specs/project-borrowing/brief.md`. Dependencies: project-config, scheduler-preemption. +- [ ] project-fairshare — NEW 2026-10-12 (user decision: fairshare is done by projects). Fairshare weights on the project tree (users equal within a project by default), decayed usage history per project and user, hierarchical ranking applied in every queue that orders by fairshare (queues keep priority between queues and may choose FCFS); the per-queue `fairshare:` user lists are removed from `queues.yaml`. Usage history persisted in the scheduler snapshot. Brief: `.kiro/specs/project-fairshare/brief.md`. Dependencies: project-config. - [ ] limits-to-owners — NEW 2026-10-12 (configuration layering, user decisions; backlog, no brief yet). Retire `limits.yaml` (not in production, removed without deprecation): the limit plugin becomes internal, fed only by generated limits. `USERS`/`PER_USER` caps become `member_quota` (root project for a cluster-wide per-user cap), a user group's total becomes a project quota, `HOSTS`/`PER_HOST` slot caps move to host supply in host config (LSF `MXJ`), queue aggregate caps and mixed tuples (user group × queue) are dropped. Touches `xceadmin` mapping and schema (`admin-config-tool`), `ClusterConfig.limits`, `src/configs/limits.yaml`, scheduler sim fixtures and the limit tests. Rules: `job-config-layering.md` "Resource control". Dependencies: project-config. - [ ] job-resize — NEW 2026-10-10 (from long-running services discovery; not on the services path). Grow/shrink a running parallel job's allocation (LSF resizable jobs, `bresize`) for elastic applications. Brief: `.kiro/specs/job-resize/brief.md`. Dependencies: parjob-placement, host-failure-job-recovery. - [ ] project-delegated-membership — NEW 2026-10-10 (from project-config requirements, user decision: future). Project managers edit their project's (and subtree's) members at runtime without an admin config push; needs a durable store that coexists with `xceadmin` config pushes (e.g. a separate delegated member list) and audit. Dependencies: project-config, rbac-persistence.