docs(spec): projects own shares of their parent's resources; project-guarantee becomes project-borrowing
A project's `share` (absolute or a percentage of its parent's) is carved from the parent; siblings never add up to more than the parent. v1 is strict: a project holds at most its share. A project's own jobs and its children without a share draw from its unassigned part. Borrowing idle shares above a project's own (`max`) and reclaim are future work, in the renamed project-borrowing brief. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
136e4a0132
commit
f3d4dba970
GPG key ID:
80533536B0EB651C
7 changed files with 235 additions and 148 deletions
|
|
@ -4,7 +4,7 @@ How a job's settings are split between the job, its application profile, its
|
|||
queue and its project, and how they interact. Decided 2026-10-09
|
||||
(`/kiro-discovery`, configuration layering). Specs that follow this:
|
||||
`job-config-merge`, `application-profile`, `project-config`,
|
||||
`project-guarantee`, `parjob-placement`, `lsf-config-migration`,
|
||||
`project-borrowing`, `parjob-placement`, `lsf-config-migration`,
|
||||
`limits-to-owners`.
|
||||
|
||||
## Why not LSF's rules
|
||||
|
|
@ -21,8 +21,8 @@ XCE instead gives **every attribute exactly one owning level**.
|
|||
| Level | Owns | Never owns |
|
||||
|---|---|---|
|
||||
| **Queue**: scheduling policy | priority and ordering, host partition, dispatch/run windows, preemption policy, per-job ceilings (max run time, max tasks, max hosts, ...) | resource requirement, anything about how a job executes (pre/post-exec, env, launch, limits enforced on the job), aggregate caps on concurrent use |
|
||||
| **Application profile**: what the application needs and how it runs (essentials plus default reservations, 2026-10-12) | resreq (select / rusage / span / same / order), task range (min / max), execution mode and options, launch type, env, **pre/post-exec**, execution limits (mem, runtime, CPU, tasks) | priority, queue choice, quotas |
|
||||
| **Project**: resource ownership | membership, the only caps on concurrent use (`quota`, `member_quota`), later guarantees and fairshare weights, accounting | how a job runs or is ordered |
|
||||
| **Application profile**: what the application needs and how it runs (essentials plus default reservations, 2026-10-12) | resreq (select / rusage / span / same / order), task range (min / max), execution mode and options, launch type, env, **pre/post-exec**, execution limits (mem, runtime, CPU, tasks) | priority, queue choice, shares |
|
||||
| **Project**: resource ownership | membership, owned shares of resources (`share`, `member_quota`), later borrowing and fairshare weights, accounting | how a job runs or is ordered |
|
||||
| **Job** | its own values, only for attributes its profile allows | n/a |
|
||||
|
||||
Access to each is separate: queue and profile by RBAC permission, project by
|
||||
|
|
@ -44,7 +44,7 @@ One line per part; details below.
|
|||
| annotations | the profile's; exported as `XCE_APP_ANN_*`; the job cannot change them |
|
||||
| mode, launch type, job starter (wraps the full job), task starter (wraps every task), hooks `pre/post_job_exec` (first host) and `pre/post_host_exec` (every host), run by the worker | the profile's |
|
||||
| compound / alternative job resreq | only the profile's `select` applies; a profile `rusage` default or `span`/`same`/`order` → rejected (until PR-15) |
|
||||
| queue ceilings, project quotas | check the merged result; never change it |
|
||||
| queue ceilings, project shares | check the merged result; never change it |
|
||||
|
||||
## The only two interactions
|
||||
|
||||
|
|
@ -108,7 +108,7 @@ One line per part; details below.
|
|||
grammar.
|
||||
- Consequence: `rusage` is the one profile part that holds **defaults**; a
|
||||
profile cannot enforce a minimum reservation. Hard protection of shared
|
||||
resources comes from project quotas (see "Resource control").
|
||||
resources comes from project shares (see "Resource control").
|
||||
- Side effects to handle in requirements: measured resources (memory) are
|
||||
charged `max(reserved, actual)` once usage is reported
|
||||
(`resource-accounting.md`), so lowering them mostly self-corrects, but
|
||||
|
|
@ -116,17 +116,17 @@ One line per part; details below.
|
|||
job lowers a profile default, and when a preset restates one (it would
|
||||
pin the job against later profile changes); job details label each
|
||||
`rusage` term as profile default or job override; queue ceilings and
|
||||
project quotas check the merged value and the error names the job's
|
||||
project shares check the merged value and the error names the job's
|
||||
override; a profile `rusage` change changes the content hash, so
|
||||
in-flight dispatches re-pend and pending jobs re-merge even if they
|
||||
override that resource (accepted).
|
||||
|
||||
2. **Ceiling: queue or project vs. the job's effective values.** A queue
|
||||
ceiling or project quota never changes the job's values, it only checks
|
||||
ceiling or project share never changes the job's values, it only checks
|
||||
them:
|
||||
- never satisfiable (runtime above the queue's maximum, task range above
|
||||
its max tasks): rejected at submit, never silently clipped;
|
||||
- not satisfiable right now (project quota in use): the job pends with a
|
||||
- not satisfiable right now (project share in use): the job pends with a
|
||||
reason naming the queue or project.
|
||||
|
||||
There is no third case: no queue resreq, no rusage merge with the queue, no
|
||||
|
|
@ -135,35 +135,36 @@ stages.
|
|||
|
||||
## Resource control
|
||||
User decisions 2026-10-12. The **project tree is the only place where
|
||||
concurrent use of resources is capped**. Four concepts, one owner each; the
|
||||
word "limit" alone is not used for any of them:
|
||||
concurrent use of resources is controlled**, and it is controlled by
|
||||
**ownership**: a project owns a share of its parent's resources. Four
|
||||
concepts, one owner each; the word "limit" alone is not used for any of
|
||||
them:
|
||||
|
||||
| Concept | Question it answers | Owner | Examples |
|
||||
|---|---|---|---|
|
||||
| **Supply** | what a host has | host / resource config | slots per host (LSF `MXJ`), host resources |
|
||||
| **Quota** (consumption) | how much a group of jobs may hold at once | **project** | `quota`, `member_quota`; later guarantee, fairshare weight |
|
||||
| **Share** (ownership) | how much a group of jobs owns and may hold at once | **project** | `share`, `member_quota`; later borrowing (`max`), fairshare weight |
|
||||
| **Ceiling** (job shape) | how big one job may be | queue | max tasks, max run time, max hosts per job |
|
||||
| **Execution limit** (enforcement) | what the running job may use | application profile | cgroup memory, runtime, CPU |
|
||||
|
||||
- **No aggregate caps outside projects.** A queue has per-job ceilings only;
|
||||
a queue that must not flood the cluster gets a host partition, or its users
|
||||
a capped project. User-group × queue caps do not exist.
|
||||
- **The limit plugin is internal.** The scheduler compiles project quotas
|
||||
into limits on every config push (`proj:<path>:<res>` for `quota`,
|
||||
`projm:<path>:<res>` with a per-user consumer for `member_quota`), reusing
|
||||
the vres usage counters, instance carry-over across pushes (bug-1967) and
|
||||
live/sim parity. Admins never write limits.
|
||||
a project with a share. User-group × queue caps do not exist.
|
||||
- **The limit plugin is internal.** The scheduler compiles shares into
|
||||
limits on every config push, reusing the vres usage counters, instance
|
||||
carry-over across pushes (bug-1967) and live/sim parity. Admins never
|
||||
write limits.
|
||||
- **`limits.yaml` is removed** (not in production; no deprecation period, so
|
||||
there is one config path, not two). Its entries map as: `PROJECTS` /
|
||||
`PER_PROJECT` → project `quota`; `USERS` + `PER_USER` → `member_quota` (on
|
||||
`PER_PROJECT` → project `share`; `USERS` + `PER_USER` → `member_quota` (on
|
||||
the root project for a cluster-wide per-user cap); a user group's total →
|
||||
a project for the group; `HOSTS` + `PER_HOST` → host supply; queue
|
||||
aggregates and mixed tuples → dropped. Retirement and the host-supply move
|
||||
are `limits-to-owners`; project quotas are `project-config`.
|
||||
- **Quotas count allocation**, what the scheduler reserved, not measured use.
|
||||
A job without a memory reservation can exceed its project's memory quota
|
||||
are `limits-to-owners`; shares are `project-config`.
|
||||
- **Shares count allocation**, what the scheduler reserved, not measured use.
|
||||
A job without a memory reservation can exceed its project's memory share
|
||||
at run time (`resource-accounting.md` charges hosts `max(reserved,
|
||||
actual)`; quotas do not). Accepted: profile default reservations close the
|
||||
actual)`; shares do not). Accepted: profile default reservations close the
|
||||
gap.
|
||||
|
||||
### The project tree
|
||||
|
|
@ -179,13 +180,49 @@ word "limit" alone is not used for any of them:
|
|||
membership says who may submit. Members are users and user groups (plus
|
||||
`all` and excludes, exclude wins), resolved at submit. A sub-project is
|
||||
never a member, and membership is not inherited either way.
|
||||
- **`quota`** on P: a job dispatches only if it fits P's quota and every
|
||||
ancestor's; P's usage includes all descendants' jobs. Absolute amounts
|
||||
only (no percentages in v1).
|
||||
- **`member_quota`** on P: a cap per user over that user's jobs anywhere in
|
||||
P's subtree, whether or not the user is a direct member of P.
|
||||
- Pend reasons name the project, resource and usage ("project eng quota on
|
||||
gpu full (16/16)"), never an internal limit name.
|
||||
|
||||
### Shares
|
||||
```yaml
|
||||
- path: eng
|
||||
share: { slots: 400, gpu: 16 } # owned: carved from the cluster
|
||||
- path: eng/hpc
|
||||
share: { slots: 60% } # owned: 60% of eng's slots
|
||||
- path: eng/ai
|
||||
share: { slots: 40%, gpu: 16 }
|
||||
- path: eng/tools # no share: draws from eng's unassigned part
|
||||
```
|
||||
- **A child owns part of its parent's share.** Per resource, a share is an
|
||||
absolute amount or a percentage of the parent's effective share; the
|
||||
root's is the cluster's capacity (the static total of the resource over
|
||||
the cluster's hosts), so percentage shares follow the cluster's size.
|
||||
Siblings' shares never add up to more than the parent's: the config tool
|
||||
rejects it (siblings use one form per resource; percentages sum to at
|
||||
most 100%).
|
||||
- **v1 is strict: a project holds at most its share** (its own and its
|
||||
descendants' jobs). Because siblings cannot overlap, a project can always
|
||||
reach its share while the hardware is there; what it does not use stays
|
||||
idle for it. **Borrowing** idle shares up to a `max`, and reclaiming them,
|
||||
is future work (`project-borrowing`).
|
||||
- **Unassigned part.** A project's share minus its children's shares is its
|
||||
unassigned part. The project's own jobs and every child without a share
|
||||
draw from it together, first come, first served, and together hold at
|
||||
most it. A child without a share owns nothing and is capped only by that
|
||||
part. A project may declare a share for a resource only if its parent
|
||||
has one for that resource (or it is top-level), so a share is always
|
||||
carved from something defined.
|
||||
- **No shares, no change.** A resource for which no project declares a
|
||||
share is unassigned at the root: no cap, today's behaviour.
|
||||
- **Shares are per resource** (slots, mem, gpu, licences, ...): a project
|
||||
may own slots but not declare GPUs, and then draws GPUs from its parent's
|
||||
unassigned part.
|
||||
- **`member_quota`** on P: an absolute cap per user over that user's jobs
|
||||
anywhere in P's subtree, whether or not the user is a direct member of P.
|
||||
It caps a user inside what the project owns; it is not a share.
|
||||
- Changing a share never stops running jobs; a project above its new share
|
||||
cannot start more until it is below it.
|
||||
- Pend reasons name the project, resource and usage ("project eng/ai share
|
||||
of gpu full (16/16)", "unassigned part of eng full for slots"), never an
|
||||
internal limit name.
|
||||
|
||||
## Convenience defaults are presets, not a level
|
||||
Apart from default reservations (the profile's `rusage` entries), tuning
|
||||
|
|
|
|||
|
|
@ -73,7 +73,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
|
|||
- [ ] controller-job-cleanup — NEW 2026-09-29 (`/kiro-discovery`, Path C). Primary-only periodic sweep (configurable `job_cleanup_period`, default 300 s) removes DONE/EXIT jobs finished longer than configurable `job_retention` (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever `jobmgr` delete; refcount-safe `xkvmgr` delete), propagated to scheduler and apiserver (`OnJobDeleted`). Retires the node-local snapshot-only `job_should_retain` finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: `.kiro/specs/controller-job-cleanup/brief.md`. Dependencies: none (extends ha-entity-snapshots).
|
||||
- [ ] job-status-detail-display — NEW 2026-09-28 (`/kiro-discovery`, Path E). End users (REST/`xcecli`) see NO pend reason for ANY job today: the apiserver cgo conversion (`backend/convert.go`) never maps `xce_job_t.pend_reasons`, so `pending_reason` is always empty. Maps it to a structured JSON (including `alternatives` from day one) using libxce's code→text mapping, and renders it in `xcecli` job detail. Open for requirements: also map `run_usage` → `JobRusage` (same gap; reserved vs. used for RUN jobs). Brief: `.kiro/specs/job-status-detail-display/brief.md`. Dependencies: none for flat reasons; scheduler-alter-pend-reasons for per-alternative rendering.
|
||||
- [ ] job-config-merge — NEW 2026-10-09 (`/kiro-discovery`, Path E "configuration layering"; ref LSF 10.1 Admin Guide pp. 314-316, 435-437, 561-563, 643, 660). One merge function in `src/lib` turning the job's own values + application profile + queue policy + project into effective values (resreq incl. `span`, task range, limits, pre/post-exec, env). One owner per attribute (user decision; `.kiro/steering/job-config-layering.md`): the job may add to a profile's values but never change or remove them (additive rule, 2026-10-12, replacing *default*/*fixed* markers); queue ceilings and project quotas only check (reject if never satisfiable, pend if full now). **Live** merge (user decision): WAL keeps inputs + names; controller/apiserver validate at submit; scheduler computes effective values and re-evaluates pending jobs on config change. Queue is policy only: queue `res_req` and `pre_exec`/`post_exec` deprecated (pre/post-exec moves to the profile), dropped queue limits (cpulimit/memlimit/runlimit/task limit) wired. Brief: `.kiro/specs/job-config-merge/brief.md`. Dependencies: none.
|
||||
- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (path identity, members, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config, quotas written on the project (`quota`, `member_quota` over the subtree; user decision 2026-10-12, replacing `PROJECTS`/`PER_PROJECT` limits in `limits.yaml`) and compiled into the limit plugin, per-project usage display; `project:submit` RBAC implied by membership, project admins, authorization in the apiserver only. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none to ship; production rejection of direct submits comes from peer-authentication-hardening.
|
||||
- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (path identity, members, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config, owned shares written on the project (`share` of the parent's resources, absolute or percentage, strict in v1; children without a share draw from the parent's unassigned part; `member_quota` over the subtree; user decisions 2026-10-12, replacing `PROJECTS`/`PER_PROJECT` limits in `limits.yaml`) compiled into the limit plugin, per-project usage display; `project:submit` RBAC implied by membership, project admins, authorization in the apiserver only. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none to ship; production rejection of direct submits comes from peer-authentication-hardening.
|
||||
|
||||
- [ ] host-failure-job-recovery — NEW 2026-10-10 (`/kiro-discovery`, "Long-running services" phase 1; also a batch gap: RUN jobs on a dead host stay RUN forever, `controller/sched.c:631` FIXME). After a grace period, jobs on a lost host go to an unknown state, then are requeued (rerunnable, job option or profile default) or end EXIT "host lost"; multi-host jobs resolved as a whole; attempt numbers fence a stale copy when the host returns; manual requeue (REST/xcecli). Brief: `.kiro/specs/host-failure-job-recovery/brief.md`. Dependencies: none.
|
||||
- [ ] service-supervision — NEW 2026-10-10 (Long-running services phase 1). Application profile kind `service`: restart policy (always/on-failure/never, backoff, give-up), liveness and readiness probes (command/TCP/HTTP) run by each replica's own worker (no special replica), in-place restart, change-only health reports, display. Brief: `.kiro/specs/service-supervision/brief.md`. Dependencies: application-profile, host-failure-job-recovery.
|
||||
|
|
@ -119,7 +119,7 @@ Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generi
|
|||
|
||||
## Future Specs / Backlog (dependency order)
|
||||
- [ ] gang-groups — NEW 2026-10-11; **demoted to backlog 2026-10-12** (user decision N24 in `resource-questions.md`). LLM training is one parallel job with topology requirements (PR-10), so it needs no gang. Re-scoped as a **co-start group** of separately managed jobs and services: RL post-training (trainer, rollout engines, reward and reference models), disaggregated serving (prefill and decode pools, router), streaming pipelines. All-or-nothing start, close placement, group failure policy. Revisit when RL or disaggregated serving reaches the roadmap. Brief: `.kiro/specs/gang-groups/brief.md`. Dependencies: resource-instances, resource-claims, service-replica-set.
|
||||
- [ ] project-guarantee — NEW 2026-10-09 (configuration layering, backlog). Project-owned guarantees (resource amount or host pool), borrowing of idle guaranteed capacity, reclaim through preemption. Brief: `.kiro/specs/project-guarantee/brief.md`. Dependencies: project-config, scheduler-preemption.
|
||||
- [ ] project-borrowing — NEW 2026-10-09 as project-guarantee (configuration layering, backlog); renamed and narrowed 2026-10-12: ownership moved into project-config's owned shares. A `max` above a project's share, borrowing idle capacity whose owner is not waiting for it, reclaim through preemption, owned/borrowed accounting. Brief: `.kiro/specs/project-borrowing/brief.md`. Dependencies: project-config, scheduler-preemption.
|
||||
- [ ] limits-to-owners — NEW 2026-10-12 (configuration layering, user decisions; backlog, no brief yet). Retire `limits.yaml` (not in production, removed without deprecation): the limit plugin becomes internal, fed only by generated limits. `USERS`/`PER_USER` caps become `member_quota` (root project for a cluster-wide per-user cap), a user group's total becomes a project quota, `HOSTS`/`PER_HOST` slot caps move to host supply in host config (LSF `MXJ`), queue aggregate caps and mixed tuples (user group × queue) are dropped. Touches `xceadmin` mapping and schema (`admin-config-tool`), `ClusterConfig.limits`, `src/configs/limits.yaml`, scheduler sim fixtures and the limit tests. Rules: `job-config-layering.md` "Resource control". Dependencies: project-config.
|
||||
- [ ] job-resize — NEW 2026-10-10 (from long-running services discovery; not on the services path). Grow/shrink a running parallel job's allocation (LSF resizable jobs, `bresize`) for elastic applications. Brief: `.kiro/specs/job-resize/brief.md`. Dependencies: parjob-placement, host-failure-job-recovery.
|
||||
- [ ] project-delegated-membership — NEW 2026-10-10 (from project-config requirements, user decision: future). Project managers edit their project's (and subtree's) members at runtime without an admin config push; needs a durable store that coexists with `xceadmin` config pushes (e.g. a separate delegated member list) and audit. Dependencies: project-config, rbac-persistence.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue