diff --git a/.kiro/specs/project-borrowing/brief.md b/.kiro/specs/project-borrowing/brief.md new file mode 100644 index 00000000..80c68e98 --- /dev/null +++ b/.kiro/specs/project-borrowing/brief.md @@ -0,0 +1,76 @@ +# Brief: project-borrowing + +(`/kiro-discovery`, 2026-10-09, Path E "configuration layering", as +`project-guarantee`. Renamed and narrowed 2026-10-12 (user decisions): +ownership moved into `project-config` as owned shares, so what remains here +is borrowing and reclaim. Backlog: not started until `project-config` and +`scheduler-preemption` land.) + +## Problem +With `project-config`, a project owns a share of its parent's resources and, +in v1, holds at most that share (`job-config-layering.md` "Shares"). That +keeps every owner's share available to it, but capacity a project owns and +does not use sits idle while other projects' jobs wait. Sites want idle +shares lent out and taken back when the owner needs them. + +## Current State +- Owned shares (absolute or percentage of the parent's), the unassigned + part and `member_quota` are specified in `project-config` (Req 6), compiled + into the internal limit plugin. No project can exceed its share. +- No fairshare: it is in the queue YAML schema only and dropped by + `xceadmin` (`mapping.py:80-97`). +- Preemption is specified but not implemented (`scheduler-preemption`, + requirements generated). + +## Desired Outcome +- **`max`** per project and resource (absolute or percentage of the + parent's share, at least the project's share): the project may hold up to + `max`, using capacity owned by other projects that is idle. Without + `max`, v1's strict rule (hold at most the share) stays. +- **Borrowing from idle capacity only**: a job may borrow capacity whose + owner has no pending job ready to use it, so borrowers do not take + capacity an owner is waiting for (precedents: YARN capacity scheduler + `capacity` / `maximum-capacity`, Kueue `nominalQuota` / + `borrowingLimit`). +- **Reclaim**: when an owner's pending job needs its share back, borrowers + are preempted, requeued or allowed to finish, by policy, through + `scheduler-preemption`. Without preemption, borrowed jobs run to + completion and the owner waits at most one job's run time (bounded by + the queue's run-time ceiling). +- Accounting and display of owned vs borrowed usage per project (REST, + `xcecli`, REPL); pend reasons that say whether a job waits for its own + share or for capacity to borrow. + +## Approach +To be chosen in discovery when the spec starts. Builds on the same project +tree and usage counters as `project-config`; the borrowing check is a new +scheduler step next to the generated share limits (a share becomes a +borrowing threshold instead of a cap; `max` becomes the cap). + +## Scope +- **In**: `max`, borrowing rule, protection of owners, reclaim policy, + owned/borrowed accounting, visibility. +- **Out**: owned shares and caps (`project-config`); fairshare ordering + (queues turn it on; weights from the project tree, later); + accounting/billing history. + +## Boundary Candidates +- `max` config and validation. +- Scheduler accounting of owned vs borrowed usage. +- Reclaim through `scheduler-preemption`. + +## Out of Boundary +- Project definitions, membership and shares (`project-config`). + +## Upstream / Downstream +- **Upstream**: `project-config`, `scheduler-preemption`, `job-config-merge`. +- **Downstream**: `lsf-config-migration` (guarantee pools, LSF + `GuaranteedResourcePool`). + +## Existing Spec Touchpoints +- **Extends**: `project-config` (shares). +- **Adjacent**: `scheduler-preemption`, `resource-distribution-mapping`. + +## Constraints +- Scheduler driver equivalence (live and simulator identical). +- Touches scheduling policy and HA state: full 3-phase workflow. diff --git a/.kiro/specs/project-config/brief.md b/.kiro/specs/project-config/brief.md index 549aa52f..8948b4b1 100644 --- a/.kiro/specs/project-config/brief.md +++ b/.kiro/specs/project-config/brief.md @@ -26,9 +26,9 @@ limits cannot be configured end to end, and nothing shows usage per project. `ugmgr`/`hgmgr`), so project limits cannot be configured to work. - No project fairshare, accounting view or access check exists. -## Desired Outcome (v1 = quota; user decision 2026-10-09) +## Desired Outcome (v1 = owned shares; user decisions 2026-10-09, 2026-10-12) - **Project definitions** in the cluster configuration: path name, - description, members (users and user groups), `quota` and `member_quota`; + description, members (users and user groups), `share` and `member_quota`; plus a cluster default project. (No priority: queues own ordering.) Schema, YAML, JSON schema, `xceadmin` mapping, live-upgrade replication. - **Project selection** at submit: `-P` > the user's default project (if @@ -41,7 +41,7 @@ limits cannot be configured end to end, and nothing shows usage per project. - **Project admins**: a project's `admins` get every job permission and `project:view` over its subtree, implied by config like membership. Only the cluster admin edits projects in this stage (config push). -- **Quotas** (user decision 2026-10-12, replacing "`PROJECTS` / +- **Owned shares** (user decisions 2026-10-12, replacing "`PROJECTS` / `PER_PROJECT` limits in `limits.yaml`"): written on the project itself, rolling up the tree, with pend reasons naming the project; see `job-config-layering.md` "Resource control". @@ -57,43 +57,63 @@ scheduler fills `pgmgr` from them, and the existing limit machinery does the rest. Projects already reach the scheduler on each job, so the WAL needs no new job field. -**Quotas compile to limits** (user decisions 2026-10-12). The project tree -is the only place concurrent use is capped; the limit plugin stays as the -internal mechanism and admins no longer write limits: +**Owned shares compile to limits** (user decisions 2026-10-12). The +project tree is the only place concurrent use is controlled: a project owns +a share of its parent's resources and, in v1, holds at most that share (no +borrowing; `project-borrowing`). The limit plugin stays as the internal +mechanism and admins no longer write limits. Semantics: +`job-config-layering.md` "Shares". ```yaml projects: - path: eng members: [eng-ug] - quota: { slots: 400, gpu: 16, mem: 2T } + share: { slots: 400, gpu: 16, mem: 2T } # carved from the cluster member_quota: { slots: 64 } # per user, over eng's whole subtree admins: [eng-leads] # job control over eng's subtree - path: eng/hpc members: [alice, bob] - quota: { slots: 200 } # must also fit eng + share: { slots: 60% } # 60% of eng's 400 slots + - path: eng/ai + members: [carol] + share: { slots: 40%, gpu: 16 } + - path: eng/tools # no share: draws from eng's unassigned part + members: [eng-ug] ``` -- On each config push the scheduler generates, per project P and resource - r, a limit `proj:

:` (project consumer {P} matching P and its - descendants) and, for `member_quota`, `projm:

:` (same consumer plus - users `all` per user, vres key `limit=projm:...:/u=`, - `lib/xcelimit.c:402`). New in the plugin: descendant matching on the - project consumer. +- On each config push, and when the cluster's capacity changes, the + scheduler generates per project P and resource r: + - a share limit `proj:

:` (project consumer {P} matching P and its + descendants) with P's effective share. A nested percentage is the + product of the percentages down from the root (an absolute amount once + an ancestor is absolute), so it compiles to the + plugin's existing `percent` form, which since bug-1834 is a fraction of + the static total capacity (`limit.c` `update_limit_res_value`); + - an unassigned-part limit `projpool:

:` (consumer {P and its + descendants}, excluding the subtrees of P's children with a share of + r, using the consumer's existing exclude list) with P's share minus + its children's shares; the root gets one when any top-level share of r + exists; + - for `member_quota`, `projm:

:` (same consumer as the share limit + plus users `all` per user, vres key `limit=projm:...:/u=`, + `lib/xcelimit.c:402`). + New in the plugin: descendant matching on the project consumer. - The path contains `/`, which is also the vres key's field separator; the trace splitter (`sched_trace_capture.c:40`, `sti_copy_scope`) would render `p=eng/hpc` as `p=eng,hpc`. Escape the project field in the key or use another internal separator (design). - Project-scoped entries in `limits.yaml` are rejected by `xceadmin` with a - pointer to project quotas; the rest of `limits.yaml` is retired by + pointer to project shares; the rest of `limits.yaml` is retired by `limits-to-owners`. ## Scope -- **In**: project config model and pipeline (`quota`, `member_quota`); - default project resolution; submit access check; `pgmgr` loading; quotas +- **In**: project config model and pipeline (`share`, `member_quota`); + default project resolution; submit access check; `pgmgr` loading; shares compiled to limits and enforced end to end; display; tests (unit, sim, live). -- **Out**: guaranteed shares, owned pools, borrowing and reclaim - (`project-guarantee`); fairshare; accounting history/billing. +- **Out**: borrowing idle shares (`max`), protecting owners from + borrowers and reclaim by preemption (`project-borrowing`); fairshare; + accounting history/billing. ## Boundary Candidates - Config model and pipeline. @@ -108,7 +128,7 @@ projects: - **Upstream**: config pipeline (`admin-config-tool`, `config-upgrade-snapshot`), `peer-authentication-hardening` (authenticated apiserver connections), limit plugin (`plugins/limit.c`, `lib/xcelimit.c`), RBAC (`rbac_check.go`). -- **Downstream**: `job-config-merge` (project level), `project-guarantee`, +- **Downstream**: `job-config-merge` (project level), `project-borrowing`, `limits-to-owners`, `lsf-config-migration`, `rbac-persistence` (if access uses RBAC scopes). ## Existing Spec Touchpoints diff --git a/.kiro/specs/project-config/requirements.md b/.kiro/specs/project-config/requirements.md index 03dd54d7..ddffb131 100644 --- a/.kiro/specs/project-config/requirements.md +++ b/.kiro/specs/project-config/requirements.md @@ -1,9 +1,9 @@ # Requirements Document ## Project Description (Input) -Discovery brief: `.kiro/specs/project-config/brief.md`. Governing rules: `.kiro/steering/job-config-layering.md` (2026-10-09): the project owns resource ownership (membership, quotas, later guarantees, accounting) and never owns how a job runs or is ordered. +Discovery brief: `.kiro/specs/project-config/brief.md`. Governing rules: `.kiro/steering/job-config-layering.md` (2026-10-09): the project owns resource ownership (membership, owned shares, later borrowing, accounting) and never owns how a job runs or is ordered. -**Who has the problem:** cluster admins who need to control which users may charge work to a project and how much of the cluster each project may use; users who need to know which projects they can use and why a job is held by a project quota. +**Who has the problem:** cluster admins who need to control which users may charge work to a project and how much of the cluster each project may use; users who need to know which projects they can use and why a job is held by a project share. **Current situation:** - A project is a free-text label on the job (`-P`). Anyone can name any project. The apiserver only checks its length, and the scheduler maps an empty name to `"default"`. @@ -12,7 +12,7 @@ Discovery brief: `.kiro/specs/project-config/brief.md`. Governing rules: `.kiro/ - `xcecli jobs list -P` is rejected by the apiserver ("unsupported filter"). - Nothing shows a project's usage. -**What should change (v1 = quota):** nested project definitions in cluster config, each with its own quotas; default-project resolution; a `project:submit` check at submit (implied by membership); project quotas enforced end to end and rolling up the tree; project visibility. Guarantees are `project-guarantee`. +**What should change (v1 = owned shares):** nested project definitions in cluster config, each owning a share of its parent's resources; default-project resolution; a `project:submit` check at submit (implied by membership); project shares enforced end to end and rolling up the tree; project visibility. Borrowing is `project-borrowing`. **Decisions (2026-10-10):** - Access is by project membership (2026-10-12: membership implies the RBAC permission `project:submit`; roles may also grant it). @@ -22,20 +22,27 @@ Discovery brief: `.kiro/specs/project-config/brief.md`. Governing rules: `.kiro/ - Projects nest, named by path (e.g. `eng/hpc/cfd`). - A job may be charged to any project in the tree. - Membership is not inherited between parent and child, and a user may be a member of any number of projects. -- Quotas roll up: a job must fit the quotas of its project and of every ancestor. +- Usage rolls up: a job counts against its project and every ancestor. - RBAC interaction: - a new `project` scope for job permissions, so a project manager can view and control the jobs of a project and its whole subtree; - a `project:view` permission. - Runtime (delegated) membership edits are left for later; membership changes only through cluster config. **Decisions (2026-10-12, resource control under the project tree; `job-config-layering.md` "Resource control"):** -- The project tree is the only place concurrent use is capped. Quotas are written on the project (`quota`, `member_quota`), not in `limits.yaml`; the limit plugin stays as the internal mechanism, and the scheduler compiles project quotas into limits. +- The project tree is the only place concurrent use is capped. Quotas are written on the project (`quota`, `member_quota`), not in `limits.yaml`; the limit plugin stays as the internal mechanism, and the scheduler compiles them into limits. (`quota` became `share` the same day, below.) - `limits.yaml` is removed; this feature rejects its project-scoped entries, `limits-to-owners` retires the rest. - `member_quota` caps each user's jobs in the project's whole subtree. -- Quota values are absolute amounts; no percentages in v1. +- (Superseded the same day by "owned shares", below: `quota` became `share`.) - The full path is the project's identity and the only form a submit accepts; there is no last-segment shorthand. - Members are users and user groups (with `all` and excludes); a sub-project is never a member. +**Decisions (2026-10-12, owned shares; `job-config-layering.md` "Shares"):** +- A project owns a `share` of its parent's resources, per resource, as an absolute amount or a percentage of the parent's effective share; the root's share is the cluster's capacity. Siblings' shares never add up to more than their parent's. +- v1 is strict: a project holds at most its share. Borrowing idle shares (a `max` above the share) and reclaiming them is future work (`project-borrowing`). +- A project's own jobs and its children without a share draw from its unassigned part (its share minus its children's shares), first come, first served. +- A project may declare a share for a resource only if its parent has one for that resource, or it is top-level. +- A resource no project declares a share for is not controlled (today's behaviour). + **Decisions (2026-10-12, project RBAC):** - Submitting under a project is an RBAC permission, `project:submit` (scope type `project`), checked like `queue:submit`. - Membership implies the grant: being a member of P gives an implicit `project:submit` on P only (membership is not inherited). Roles may also grant `project:submit` explicitly (e.g. to service accounts); a `project` scope covers the named projects and their subtrees, as for every `project` scope. @@ -45,44 +52,43 @@ Discovery brief: `.kiro/specs/project-config/brief.md`. Governing rules: `.kiro/ ## Introduction This feature makes projects first-class cluster configuration entities that control resource ownership. -- **Definitions:** admins define a tree of projects named by path, each with its own members and quotas, and say which user groups default to which project. +- **Definitions:** admins define a tree of projects named by path, each with its own members and owned shares, and say which user groups default to which project. - **Submit:** the system resolves each job's project, admits only users holding `project:submit` on it (every member does), and records the project on the job. -- **Quotas:** each project's quotas apply to its own and its descendants' jobs and roll up the tree, and a held job's reason names the project. -- **Visibility:** users and admins can see projects and their usage against quotas. +- **Shares:** each project owns part of its parent's resources and holds at most that part with its own and its descendants' jobs; a held job's reason names the project. +- **Visibility:** users and admins can see projects and their usage against their shares. - **RBAC:** submitting under a project is the `project:submit` permission, implied by membership; a project's admins get job permissions over its subtree; roles can grant the same permissions explicitly. - **Compatibility:** a cluster that defines no projects keeps today's behaviour. ## Boundary Context - **In scope**: - - nested project definitions (path name, description, members, admins, default-for groups, `quota`, `member_quota`) and a cluster default project; + - nested project definitions (path name, description, members, admins, default-for groups, `share`, `member_quota`) and a cluster default project; - default-project resolution, the `project:submit` check at submit, and an "open" mode when no projects are defined; - pending-job behaviour when a project or membership changes; - - project quotas compiled into the scheduler's limits, applying end to end and rolling up the tree, with project-naming pend reasons; + - owned shares (absolute or percentage), the unassigned part, and `member_quota`, compiled into the scheduler's limits, applying end to end, with project-naming pend reasons; - rejecting project-scoped entries in `limits.yaml`; - overlapping membership (a user in several projects); - - project listing (as a tree) and detail with usage against quotas; + - project listing (as a tree) and detail with usage against shares; - RBAC: a `project:submit` permission implied by membership; project admins with implied job permissions over the subtree; a `project` scope for job permissions (covering the project's subtree) and a `project:view` permission; - job filtering by project. - **Out of scope**: - - guaranteed shares, owned pools, borrowing, reclaim (`project-guarantee`); + - borrowing idle shares above a project's own (`max`), protecting owners from borrowers, and reclaim by preemption (`project-borrowing`); - fairshare; - any project influence on job ordering or priority (queues own ordering); - accounting history and billing of finished jobs; - - runtime edits of a project's members, admins or quotas by project admins (`project-delegated-membership`; in this stage only the cluster administrator changes them, through cluster config); + - runtime edits of a project's members, admins or shares by project admins (`project-delegated-membership`; in this stage only the cluster administrator changes them, through cluster config); - per-user default projects in `users.yaml`; - membership inherited between parent and child projects; - moving or renaming a project while keeping its jobs' recorded name; - wildcard or pattern matching of member names; - - percentage quotas; - retiring the rest of `limits.yaml` (user, queue and host scopes) and moving per-host slots to host config (`limits-to-owners`); - aggregate caps on queues (dropped by decision 2026-10-12, not deferred). - **Adjacent expectations**: - - Project quotas are written on the project (decision 2026-10-12, reversing "`limits.yaml` stays the place where project quotas are written"). The scheduler's limit plugin enforces them as generated limits; admins never write those limits. + - Project shares are written on the project (decision 2026-10-12, reversing "`limits.yaml` stays the place where project quotas are written"). The scheduler's limit plugin enforces them as generated limits; admins never write those limits. - `peer-authentication-hardening` makes the controller accept user requests in production mode only from an authenticated apiserver connection; Requirement 3.12 relies on it, and until it lands the controller's trust in the apiserver's authorization is a trusted-network assumption. - - Quotas count what the scheduler allocated or reserved, not measured use; a job without a memory reservation can exceed its project's memory quota at run time (accepted; profile default reservations close the gap). + - Shares count what the scheduler allocated or reserved, not measured use; a job without a memory reservation can exceed its project's memory share at run time (accepted; profile default reservations close the gap). - User groups come from the existing user-group configuration. - Grants made through the role API are subject to the existing `rbac-persistence` gap (not durable across apiserver restart) until that spec lands. - - `job-config-merge` treats project quotas as ceilings: they hold or admit a job but never change its values. + - `job-config-merge` treats project shares as ceilings: they hold or admit a job but never change its values. - The defect that resolved limit host and project scopes against each other's group tables (it also broke host-group limits) was fixed separately as bug-1933 (`6ecd418d`); Requirement 6.8 keeps it as a requirement. ## Requirements @@ -91,14 +97,14 @@ This feature makes projects first-class cluster configuration entities that cont **Objective:** As a cluster admin, I want to define projects and their members in the cluster configuration, so that project ownership is explicit and versioned with the rest of the cluster config. #### Acceptance Criteria -1. The configuration tool shall accept project definitions consisting of a path name, an optional description, members (user names, user-group names, or `all`), optional excluded members, optional admins (user names or user-group names), an optional list of user groups for which the project is the default, an optional `quota` and an optional `member_quota`, each a map from resource name to an absolute amount. +1. The configuration tool shall accept project definitions consisting of a path name, an optional description, members (user names, user-group names, or `all`), optional excluded members, optional admins (user names or user-group names), an optional list of user groups for which the project is the default, an optional `share` (a map from resource name to an absolute amount or a percentage of the parent's share) and an optional `member_quota` (a map from resource name to an absolute amount). 2. The configuration tool shall treat a path name of one segment (e.g. `eng`) as a top-level project and a path of several `/`-separated segments (e.g. `eng/hpc/cfd`) as a child of the project named by the path without its last segment. 3. The configuration tool shall accept an optional cluster default project naming one defined project. -4. If a project definition repeats a path name, uses a reserved name, has an empty or malformed path segment, names a parent project that is not defined, names an undefined user group (in members, excluded members or admins), lists a default-for user group that is not covered by the project's members, or has a `quota` or `member_quota` entry naming an undeclared resource or giving a negative, non-numeric or percentage amount, then the configuration tool shall reject the configuration with an error identifying the project and the offending field. +4. If a project definition repeats a path name, uses a reserved name, has an empty or malformed path segment, names a parent project that is not defined, names an undefined user group (in members, excluded members or admins), lists a default-for user group that is not covered by the project's members, has a `share` or `member_quota` entry naming an undeclared resource or giving a negative or non-numeric amount, has a percentage in `member_quota` or a percentage above 100% in `share`, declares a `share` for a resource its parent (when it has one) declares no share for, gives siblings shares of one resource in different forms (absolute and percentage), or gives siblings shares that add up to more than 100% or, when all are absolute and the parent's is absolute, to more than the parent's share, then the configuration tool shall reject the configuration with an error identifying the project and the offending field. 5. If the cluster default project names an undefined project, or a project whose members do not include every user, then the configuration tool shall reject the configuration with an error naming the cluster default project. 6. When a configuration with changed project definitions is applied by live configuration upgrade, the Cluster shall apply the new definitions to all subsequent submissions and scheduling decisions without a restart. -7. The Cluster shall keep project definitions, including their quotas, across controller failover and restart identically to other cluster configuration. -8. If the limit configuration contains a limit scoped to projects (listed or per project), then the configuration tool shall reject the configuration with an error naming the limit and stating that project quotas are written on the project. +7. The Cluster shall keep project definitions, including their shares, across controller failover and restart identically to other cluster configuration. +8. If the limit configuration contains a limit scoped to projects (listed or per project), then the configuration tool shall reject the configuration with an error naming the limit and stating that project shares are written on the project. ### Requirement 2: Project resolution at submit **Objective:** As a user, I want my job to get the right project when I do not name one, so that I do not have to pass `-P` for my usual project. @@ -148,32 +154,37 @@ This feature makes projects first-class cluster configuration entities that cont 4. While a job is running, the Cluster shall not stop, requeue or alter the job because of a change to its project's definition or membership. 5. The Cluster shall allow an admin to kill or modify a job held by Requirement 5 using the existing job controls. -### Requirement 6: Project quotas -**Objective:** As a cluster admin, I want each project's quotas to cap the resources its jobs hold at once, so that a project, or one member of it, cannot use more than its share of the cluster. +### Requirement 6: Owned shares +**Objective:** As a cluster admin, I want each project to own a share of its parent's resources and hold no more than it, so that a project can always get what it owns and no project, or member of one, takes more than its part. #### Acceptance Criteria -1. The Scheduler shall count against a project's `quota` for a resource the amount of that resource allocated to every running job whose recorded project is that project or one of its descendants. -2. The Scheduler shall count against a project's `member_quota` for a resource, separately for each user, the amount of that resource allocated to that user's running jobs whose recorded project is that project or one of its descendants, whether or not the user is a direct member of that project. -3. The Scheduler shall dispatch a job only if, for every resource it is allocated, it fits the `quota` and `member_quota` of its recorded project and of every ancestor of that project. -4. The Scheduler shall count a job only against the quotas of its recorded project and that project's ancestors, never against quotas of other projects its owner belongs to. -5. If dispatching a job would exceed a quota, then the Scheduler shall keep the job pending with a reason naming the project whose quota is full (the job's own project or the ancestor), the resource, whether it is the project's quota or the member quota of the job's owner, and the current usage and quota amount; the reason shall not name an internal limit. -6. When usage under a quota falls below its amount, the Scheduler shall consider jobs held by that quota for scheduling again. -7. When a configuration change adds, changes or removes a quota, the Scheduler shall apply the new amounts to all subsequent scheduling decisions without a restart, keep counting the usage of running jobs against their recorded project and its ancestors, and never stop or alter a running job because a quota now exceeds its usage. -8. The Scheduler shall resolve each generated quota's project and user scopes against project and user names only, so that a project name never matches a host group or user group (the scope separation fixed by bug-1933). -9. The live Scheduler and the scheduler simulator shall make identical quota decisions for the same input. -10. The Scheduler shall show a project path containing `/` unchanged in pend reasons, decision traces and diagnostics. +1. The Scheduler shall compute each project's effective share of a resource as its absolute amount, or as its percentage of its parent's effective share; a top-level project's percentage is of the cluster's capacity of that resource (the static total over the cluster's hosts), so percentage shares change when hosts are added or removed. +2. The Scheduler shall count against a project's share of a resource the amount of that resource allocated to every running job whose recorded project is that project or one of its descendants. +3. The Scheduler shall treat a project's unassigned part of a resource as its effective share minus the effective shares of its children that declare a share of that resource (the cluster's capacity minus the top-level shares at the root), and shall count against it the jobs of the project itself and of its descendants that are not under a child with a share of that resource. +4. The Scheduler shall count against a project's `member_quota` for a resource, separately for each user, the amount of that resource allocated to that user's running jobs whose recorded project is that project or one of its descendants, whether or not the user is a direct member of that project. +5. The Scheduler shall dispatch a job only if, for every resource it is allocated, the job fits the share of every project on its path that declares one, the unassigned part it draws from, and the `member_quota` of its recorded project and of every ancestor. +6. The Scheduler shall not let a project hold more than its effective share by using capacity owned by another project (no borrowing in this feature). +7. While no project declares a share of a resource, the Scheduler shall not restrict that resource by project. +8. The Scheduler shall count a job only against its recorded project and that project's ancestors, never against other projects its owner belongs to. +9. If dispatching a job would exceed a share, an unassigned part or a member quota, then the Scheduler shall keep the job pending with a reason naming the project, the resource, which of the three is full, and its current usage and amount; the reason shall not name an internal limit. +10. When usage falls below a share, unassigned part or member quota, the Scheduler shall consider jobs held by it for scheduling again. +11. When a configuration change or a change in the cluster's hosts changes an effective share, the Scheduler shall apply the new amounts to all subsequent scheduling decisions without a restart, keep counting running jobs against their recorded project and its ancestors, and never stop or alter a running job because a share is now below its usage. +12. Where absolute shares of top-level projects add up to more than the cluster's current capacity of a resource, the Cluster shall report it in project diagnostics and in a configuration-tool warning; the Scheduler shall still apply each share as written. +13. The Scheduler shall resolve each generated limit's project and user scopes against project and user names only, so that a project name never matches a host group or user group (the scope separation fixed by bug-1933). +14. The live Scheduler and the scheduler simulator shall make identical share decisions for the same input. +15. The Scheduler shall show a project path containing `/` unchanged in pend reasons, decision traces and diagnostics. ### Requirement 7: Project visibility -**Objective:** As a user or admin, I want to see projects, their members and their usage against quotas, so that I know which projects I can use and why a job is held. +**Objective:** As a user or admin, I want to see projects, their members and their usage against shares, so that I know which projects I can use and why a job is held. #### Acceptance Criteria 1. When a user lists projects through the REST API or `xcecli`, the Cluster shall return every project the user may submit under (as a member or through a role grant) or administers, with its full path name and description, shown in tree order, and mark the project that would be the user's default, so that a submit client can offer them for choosing the project to charge. 2. Where the requester has the cluster administrator role, or a `project:view` permission covering a project, the Cluster shall include that project in the list with its members, excluded members and default-for user groups. -3. When a user requests a project's details, the Cluster shall show, for the project and for each of its ancestors, every `quota` and `member_quota` entry with the project it is set on, the resource, the amount and the current usage counted against it (including descendants' jobs; for `member_quota`, the requesting user's own usage, and every user's usage to an admin or a holder of `project:view` covering the project). +3. When a user requests a project's details, the Cluster shall show, for the project and for each of its ancestors, the effective share (and the form it was written in), the unassigned part, and every `member_quota` entry, each with the project it is set on, the resource, the amount and the current usage counted against it (including descendants' jobs; for `member_quota`, the requesting user's own usage, and every user's usage to an admin or a holder of `project:view` covering the project). 4. If a user requests the details of a project they are neither a member nor an admin of, and they have neither the cluster administrator role nor a `project:view` permission covering the project, then the Cluster shall reject the request without revealing the project's members or usage. 5. When a user lists jobs filtered by project through the REST API or `xcecli`, the Cluster shall return only jobs with that recorded project, or, when the user asks to include sub-projects, with that project or any descendant, subject to the user's existing job-visibility permissions. 6. While no project is defined, the Cluster shall report that projects are not configured when projects are listed. -7. The scheduler's diagnostic interface shall show, for a named project, the quotas that apply to it (its own and its ancestors') and their current usage. +7. The scheduler's diagnostic interface shall show, for a named project, the shares, unassigned parts and member quotas that apply to it (its own and its ancestors'), their effective amounts and their current usage. ### Requirement 8: Project-scoped access control **Objective:** As a cluster admin, I want project admins and roles to hold rights over the jobs and details of a project and its sub-projects, so that project leads can manage their project's work without being cluster administrators. @@ -189,7 +200,7 @@ This feature makes projects first-class cluster configuration entities that cont 8. The Cluster shall leave the behaviour of existing roles and grants unchanged; the default RBAC configuration shall grant no `project`-scoped permission and no explicit `project:submit` to any role other than the cluster administrator. 9. The Cluster shall accept roles that grant `project:submit` with the scope type `all` or `project`. 10. The Cluster shall treat each admin of a project (a listed user, or a member of a listed user group) as holding every job permission and `project:view` with scope `project` on that project, which covers its subtree, without any role assignment. -11. The Cluster shall not give a project admin `project:submit` on the project, nor any right to change a project's definition, members, admins or quotas, by being its admin. +11. The Cluster shall not give a project admin `project:submit` on the project, nor any right to change a project's definition, members, admins or shares, by being its admin. 12. When a configuration change adds or removes a project admin, the Cluster shall apply the change to all subsequent requests without a restart. 13. When a user lists their projects, the Cluster shall mark the projects the user administers. @@ -200,4 +211,4 @@ This feature makes projects first-class cluster configuration entities that cont 1. When a job recorded before this feature is recovered, the Cluster shall keep its recorded project name, treating an empty name as `default`. 2. The Cluster shall accept a cluster configuration without any project definitions and behave as Requirement 4 describes. 3. When a configuration tool of an earlier version pushes a configuration without project definitions, the Cluster shall treat it as defining no projects. -4. While no project defines a quota, the Scheduler shall dispatch jobs exactly as it would without this feature's quota checks. +4. While no project declares a share or member quota, the Scheduler shall dispatch jobs exactly as it would without this feature's share checks. diff --git a/.kiro/specs/project-guarantee/brief.md b/.kiro/specs/project-guarantee/brief.md deleted file mode 100644 index 9c273aed..00000000 --- a/.kiro/specs/project-guarantee/brief.md +++ /dev/null @@ -1,57 +0,0 @@ -# Brief: project-guarantee - -(`/kiro-discovery`, 2026-10-09, Path E "configuration layering". Backlog: -not started until `project-config` and `scheduler-preemption` land.) - -## Problem -With `project-config`, projects can only be capped. Owners of a project (a -group that funded hardware, a department with a committed share) need a -**guarantee**: a minimum share or a pool of resources they get when they ask -for it, while idle capacity is still lent to others. - -## Current State -- No guarantee, share or pool concept exists; fairshare is in the queue YAML - schema only and dropped by `xceadmin` (`mapping.py:80-97`). -- Preemption is specified but not implemented (`scheduler-preemption`, - requirements generated). - -## Desired Outcome -- A project can own a guarantee: a resource amount (slots, hosts or any - resource) or a pool of hosts. -- Idle guaranteed capacity may be borrowed by other projects' jobs. -- When an owner's pending job needs its guarantee back, borrowers are - reclaimed (preempted, requeued or allowed to finish, by policy). -- Guarantee state and borrowing are visible (REST, `xcecli`, REPL). - -## Approach -Guarantees (and later fairshare weights) sit on the same project-tree nodes -as `quota` / `member_quota` (`job-config-layering.md` "Resource control", -2026-10-12): one tree, one ownership model; the queue only turns fairshare -on. To be chosen in discovery when the spec starts. Candidates: LSF-style -guarantee service classes (`lsb.resources` GuaranteedResourcePool), or -guarantees as reserved virtual resources built on the limit/vres machinery -plus preemption. - -## Scope -- **In**: guarantee model, borrowing, reclaim policy, visibility. -- **Out**: quotas (`project-config`), fairshare ordering, accounting/billing. - -## Boundary Candidates -- Guarantee model and config. -- Scheduler accounting of owned vs borrowed usage. -- Reclaim through `scheduler-preemption`. - -## Out of Boundary -- Project definitions and membership (`project-config`). - -## Upstream / Downstream -- **Upstream**: `project-config`, `scheduler-preemption`, `job-config-merge`. -- **Downstream**: `lsf-config-migration` (guarantee pools). - -## Existing Spec Touchpoints -- **Extends**: none. -- **Adjacent**: `scheduler-preemption`, `resource-distribution-mapping`. - -## Constraints -- Scheduler driver equivalence (live and simulator identical). -- Touches scheduling policy and HA state: full 3-phase workflow. diff --git a/.kiro/specs/resource-claims/brief.md b/.kiro/specs/resource-claims/brief.md index 55c96f77..397bd3e4 100644 --- a/.kiro/specs/resource-claims/brief.md +++ b/.kiro/specs/resource-claims/brief.md @@ -71,7 +71,7 @@ XCE has no such object. File staging (`transfer_files`) is parsed but never perf ## Existing Spec Touchpoints - **Extends:** `job-sandbox-execution` (workspace as a claim); closes bug-1937's deferred staging. -- **Adjacent:** `elastic-hosts`, `project-guarantee` (quota on claims, decision 3). +- **Adjacent:** `elastic-hosts`, `project-config` (shares on claims, decision 3). ## Constraints - **HA/persistence:** full 3-phase spec workflow; a claim that holds data or costs money is durable from v1. diff --git a/.kiro/steering/job-config-layering.md b/.kiro/steering/job-config-layering.md index 52df7a0a..964b6d73 100644 --- a/.kiro/steering/job-config-layering.md +++ b/.kiro/steering/job-config-layering.md @@ -4,7 +4,7 @@ How a job's settings are split between the job, its application profile, its queue and its project, and how they interact. Decided 2026-10-09 (`/kiro-discovery`, configuration layering). Specs that follow this: `job-config-merge`, `application-profile`, `project-config`, -`project-guarantee`, `parjob-placement`, `lsf-config-migration`, +`project-borrowing`, `parjob-placement`, `lsf-config-migration`, `limits-to-owners`. ## Why not LSF's rules @@ -21,8 +21,8 @@ XCE instead gives **every attribute exactly one owning level**. | Level | Owns | Never owns | |---|---|---| | **Queue**: scheduling policy | priority and ordering, host partition, dispatch/run windows, preemption policy, per-job ceilings (max run time, max tasks, max hosts, ...) | resource requirement, anything about how a job executes (pre/post-exec, env, launch, limits enforced on the job), aggregate caps on concurrent use | -| **Application profile**: what the application needs and how it runs (essentials plus default reservations, 2026-10-12) | resreq (select / rusage / span / same / order), task range (min / max), execution mode and options, launch type, env, **pre/post-exec**, execution limits (mem, runtime, CPU, tasks) | priority, queue choice, quotas | -| **Project**: resource ownership | membership, the only caps on concurrent use (`quota`, `member_quota`), later guarantees and fairshare weights, accounting | how a job runs or is ordered | +| **Application profile**: what the application needs and how it runs (essentials plus default reservations, 2026-10-12) | resreq (select / rusage / span / same / order), task range (min / max), execution mode and options, launch type, env, **pre/post-exec**, execution limits (mem, runtime, CPU, tasks) | priority, queue choice, shares | +| **Project**: resource ownership | membership, owned shares of resources (`share`, `member_quota`), later borrowing and fairshare weights, accounting | how a job runs or is ordered | | **Job** | its own values, only for attributes its profile allows | n/a | Access to each is separate: queue and profile by RBAC permission, project by @@ -44,7 +44,7 @@ One line per part; details below. | annotations | the profile's; exported as `XCE_APP_ANN_*`; the job cannot change them | | mode, launch type, job starter (wraps the full job), task starter (wraps every task), hooks `pre/post_job_exec` (first host) and `pre/post_host_exec` (every host), run by the worker | the profile's | | compound / alternative job resreq | only the profile's `select` applies; a profile `rusage` default or `span`/`same`/`order` → rejected (until PR-15) | -| queue ceilings, project quotas | check the merged result; never change it | +| queue ceilings, project shares | check the merged result; never change it | ## The only two interactions @@ -108,7 +108,7 @@ One line per part; details below. grammar. - Consequence: `rusage` is the one profile part that holds **defaults**; a profile cannot enforce a minimum reservation. Hard protection of shared - resources comes from project quotas (see "Resource control"). + resources comes from project shares (see "Resource control"). - Side effects to handle in requirements: measured resources (memory) are charged `max(reserved, actual)` once usage is reported (`resource-accounting.md`), so lowering them mostly self-corrects, but @@ -116,17 +116,17 @@ One line per part; details below. job lowers a profile default, and when a preset restates one (it would pin the job against later profile changes); job details label each `rusage` term as profile default or job override; queue ceilings and - project quotas check the merged value and the error names the job's + project shares check the merged value and the error names the job's override; a profile `rusage` change changes the content hash, so in-flight dispatches re-pend and pending jobs re-merge even if they override that resource (accepted). 2. **Ceiling: queue or project vs. the job's effective values.** A queue - ceiling or project quota never changes the job's values, it only checks + ceiling or project share never changes the job's values, it only checks them: - never satisfiable (runtime above the queue's maximum, task range above its max tasks): rejected at submit, never silently clipped; - - not satisfiable right now (project quota in use): the job pends with a + - not satisfiable right now (project share in use): the job pends with a reason naming the queue or project. There is no third case: no queue resreq, no rusage merge with the queue, no @@ -135,35 +135,36 @@ stages. ## Resource control User decisions 2026-10-12. The **project tree is the only place where -concurrent use of resources is capped**. Four concepts, one owner each; the -word "limit" alone is not used for any of them: +concurrent use of resources is controlled**, and it is controlled by +**ownership**: a project owns a share of its parent's resources. Four +concepts, one owner each; the word "limit" alone is not used for any of +them: | Concept | Question it answers | Owner | Examples | |---|---|---|---| | **Supply** | what a host has | host / resource config | slots per host (LSF `MXJ`), host resources | -| **Quota** (consumption) | how much a group of jobs may hold at once | **project** | `quota`, `member_quota`; later guarantee, fairshare weight | +| **Share** (ownership) | how much a group of jobs owns and may hold at once | **project** | `share`, `member_quota`; later borrowing (`max`), fairshare weight | | **Ceiling** (job shape) | how big one job may be | queue | max tasks, max run time, max hosts per job | | **Execution limit** (enforcement) | what the running job may use | application profile | cgroup memory, runtime, CPU | - **No aggregate caps outside projects.** A queue has per-job ceilings only; a queue that must not flood the cluster gets a host partition, or its users - a capped project. User-group × queue caps do not exist. -- **The limit plugin is internal.** The scheduler compiles project quotas - into limits on every config push (`proj::` for `quota`, - `projm::` with a per-user consumer for `member_quota`), reusing - the vres usage counters, instance carry-over across pushes (bug-1967) and - live/sim parity. Admins never write limits. + a project with a share. User-group × queue caps do not exist. +- **The limit plugin is internal.** The scheduler compiles shares into + limits on every config push, reusing the vres usage counters, instance + carry-over across pushes (bug-1967) and live/sim parity. Admins never + write limits. - **`limits.yaml` is removed** (not in production; no deprecation period, so there is one config path, not two). Its entries map as: `PROJECTS` / - `PER_PROJECT` → project `quota`; `USERS` + `PER_USER` → `member_quota` (on + `PER_PROJECT` → project `share`; `USERS` + `PER_USER` → `member_quota` (on the root project for a cluster-wide per-user cap); a user group's total → a project for the group; `HOSTS` + `PER_HOST` → host supply; queue aggregates and mixed tuples → dropped. Retirement and the host-supply move - are `limits-to-owners`; project quotas are `project-config`. -- **Quotas count allocation**, what the scheduler reserved, not measured use. - A job without a memory reservation can exceed its project's memory quota + are `limits-to-owners`; shares are `project-config`. +- **Shares count allocation**, what the scheduler reserved, not measured use. + A job without a memory reservation can exceed its project's memory share at run time (`resource-accounting.md` charges hosts `max(reserved, - actual)`; quotas do not). Accepted: profile default reservations close the + actual)`; shares do not). Accepted: profile default reservations close the gap. ### The project tree @@ -179,13 +180,49 @@ word "limit" alone is not used for any of them: membership says who may submit. Members are users and user groups (plus `all` and excludes, exclude wins), resolved at submit. A sub-project is never a member, and membership is not inherited either way. -- **`quota`** on P: a job dispatches only if it fits P's quota and every - ancestor's; P's usage includes all descendants' jobs. Absolute amounts - only (no percentages in v1). -- **`member_quota`** on P: a cap per user over that user's jobs anywhere in - P's subtree, whether or not the user is a direct member of P. -- Pend reasons name the project, resource and usage ("project eng quota on - gpu full (16/16)"), never an internal limit name. + +### Shares +```yaml +- path: eng + share: { slots: 400, gpu: 16 } # owned: carved from the cluster +- path: eng/hpc + share: { slots: 60% } # owned: 60% of eng's slots +- path: eng/ai + share: { slots: 40%, gpu: 16 } +- path: eng/tools # no share: draws from eng's unassigned part +``` +- **A child owns part of its parent's share.** Per resource, a share is an + absolute amount or a percentage of the parent's effective share; the + root's is the cluster's capacity (the static total of the resource over + the cluster's hosts), so percentage shares follow the cluster's size. + Siblings' shares never add up to more than the parent's: the config tool + rejects it (siblings use one form per resource; percentages sum to at + most 100%). +- **v1 is strict: a project holds at most its share** (its own and its + descendants' jobs). Because siblings cannot overlap, a project can always + reach its share while the hardware is there; what it does not use stays + idle for it. **Borrowing** idle shares up to a `max`, and reclaiming them, + is future work (`project-borrowing`). +- **Unassigned part.** A project's share minus its children's shares is its + unassigned part. The project's own jobs and every child without a share + draw from it together, first come, first served, and together hold at + most it. A child without a share owns nothing and is capped only by that + part. A project may declare a share for a resource only if its parent + has one for that resource (or it is top-level), so a share is always + carved from something defined. +- **No shares, no change.** A resource for which no project declares a + share is unassigned at the root: no cap, today's behaviour. +- **Shares are per resource** (slots, mem, gpu, licences, ...): a project + may own slots but not declare GPUs, and then draws GPUs from its parent's + unassigned part. +- **`member_quota`** on P: an absolute cap per user over that user's jobs + anywhere in P's subtree, whether or not the user is a direct member of P. + It caps a user inside what the project owns; it is not a share. +- Changing a share never stops running jobs; a project above its new share + cannot start more until it is below it. +- Pend reasons name the project, resource and usage ("project eng/ai share + of gpu full (16/16)", "unassigned part of eng full for slots"), never an + internal limit name. ## Convenience defaults are presets, not a level Apart from default reservations (the profile's `rusage` entries), tuning diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index 5773db4c..af5f2bbf 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -73,7 +73,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han - [ ] controller-job-cleanup — NEW 2026-09-29 (`/kiro-discovery`, Path C). Primary-only periodic sweep (configurable `job_cleanup_period`, default 300 s) removes DONE/EXIT jobs finished longer than configurable `job_retention` (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever `jobmgr` delete; refcount-safe `xkvmgr` delete), propagated to scheduler and apiserver (`OnJobDeleted`). Retires the node-local snapshot-only `job_should_retain` finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: `.kiro/specs/controller-job-cleanup/brief.md`. Dependencies: none (extends ha-entity-snapshots). - [ ] job-status-detail-display — NEW 2026-09-28 (`/kiro-discovery`, Path E). End users (REST/`xcecli`) see NO pend reason for ANY job today: the apiserver cgo conversion (`backend/convert.go`) never maps `xce_job_t.pend_reasons`, so `pending_reason` is always empty. Maps it to a structured JSON (including `alternatives` from day one) using libxce's code→text mapping, and renders it in `xcecli` job detail. Open for requirements: also map `run_usage` → `JobRusage` (same gap; reserved vs. used for RUN jobs). Brief: `.kiro/specs/job-status-detail-display/brief.md`. Dependencies: none for flat reasons; scheduler-alter-pend-reasons for per-alternative rendering. - [ ] job-config-merge — NEW 2026-10-09 (`/kiro-discovery`, Path E "configuration layering"; ref LSF 10.1 Admin Guide pp. 314-316, 435-437, 561-563, 643, 660). One merge function in `src/lib` turning the job's own values + application profile + queue policy + project into effective values (resreq incl. `span`, task range, limits, pre/post-exec, env). One owner per attribute (user decision; `.kiro/steering/job-config-layering.md`): the job may add to a profile's values but never change or remove them (additive rule, 2026-10-12, replacing *default*/*fixed* markers); queue ceilings and project quotas only check (reject if never satisfiable, pend if full now). **Live** merge (user decision): WAL keeps inputs + names; controller/apiserver validate at submit; scheduler computes effective values and re-evaluates pending jobs on config change. Queue is policy only: queue `res_req` and `pre_exec`/`post_exec` deprecated (pre/post-exec moves to the profile), dropped queue limits (cpulimit/memlimit/runlimit/task limit) wired. Brief: `.kiro/specs/job-config-merge/brief.md`. Dependencies: none. -- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (path identity, members, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config, quotas written on the project (`quota`, `member_quota` over the subtree; user decision 2026-10-12, replacing `PROJECTS`/`PER_PROJECT` limits in `limits.yaml`) and compiled into the limit plugin, per-project usage display; `project:submit` RBAC implied by membership, project admins, authorization in the apiserver only. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none to ship; production rejection of direct submits comes from peer-authentication-hardening. +- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (path identity, members, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config, owned shares written on the project (`share` of the parent's resources, absolute or percentage, strict in v1; children without a share draw from the parent's unassigned part; `member_quota` over the subtree; user decisions 2026-10-12, replacing `PROJECTS`/`PER_PROJECT` limits in `limits.yaml`) compiled into the limit plugin, per-project usage display; `project:submit` RBAC implied by membership, project admins, authorization in the apiserver only. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none to ship; production rejection of direct submits comes from peer-authentication-hardening. - [ ] host-failure-job-recovery — NEW 2026-10-10 (`/kiro-discovery`, "Long-running services" phase 1; also a batch gap: RUN jobs on a dead host stay RUN forever, `controller/sched.c:631` FIXME). After a grace period, jobs on a lost host go to an unknown state, then are requeued (rerunnable, job option or profile default) or end EXIT "host lost"; multi-host jobs resolved as a whole; attempt numbers fence a stale copy when the host returns; manual requeue (REST/xcecli). Brief: `.kiro/specs/host-failure-job-recovery/brief.md`. Dependencies: none. - [ ] service-supervision — NEW 2026-10-10 (Long-running services phase 1). Application profile kind `service`: restart policy (always/on-failure/never, backoff, give-up), liveness and readiness probes (command/TCP/HTTP) run by each replica's own worker (no special replica), in-place restart, change-only health reports, display. Brief: `.kiro/specs/service-supervision/brief.md`. Dependencies: application-profile, host-failure-job-recovery. @@ -119,7 +119,7 @@ Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generi ## Future Specs / Backlog (dependency order) - [ ] gang-groups — NEW 2026-10-11; **demoted to backlog 2026-10-12** (user decision N24 in `resource-questions.md`). LLM training is one parallel job with topology requirements (PR-10), so it needs no gang. Re-scoped as a **co-start group** of separately managed jobs and services: RL post-training (trainer, rollout engines, reward and reference models), disaggregated serving (prefill and decode pools, router), streaming pipelines. All-or-nothing start, close placement, group failure policy. Revisit when RL or disaggregated serving reaches the roadmap. Brief: `.kiro/specs/gang-groups/brief.md`. Dependencies: resource-instances, resource-claims, service-replica-set. -- [ ] project-guarantee — NEW 2026-10-09 (configuration layering, backlog). Project-owned guarantees (resource amount or host pool), borrowing of idle guaranteed capacity, reclaim through preemption. Brief: `.kiro/specs/project-guarantee/brief.md`. Dependencies: project-config, scheduler-preemption. +- [ ] project-borrowing — NEW 2026-10-09 as project-guarantee (configuration layering, backlog); renamed and narrowed 2026-10-12: ownership moved into project-config's owned shares. A `max` above a project's share, borrowing idle capacity whose owner is not waiting for it, reclaim through preemption, owned/borrowed accounting. Brief: `.kiro/specs/project-borrowing/brief.md`. Dependencies: project-config, scheduler-preemption. - [ ] limits-to-owners — NEW 2026-10-12 (configuration layering, user decisions; backlog, no brief yet). Retire `limits.yaml` (not in production, removed without deprecation): the limit plugin becomes internal, fed only by generated limits. `USERS`/`PER_USER` caps become `member_quota` (root project for a cluster-wide per-user cap), a user group's total becomes a project quota, `HOSTS`/`PER_HOST` slot caps move to host supply in host config (LSF `MXJ`), queue aggregate caps and mixed tuples (user group × queue) are dropped. Touches `xceadmin` mapping and schema (`admin-config-tool`), `ClusterConfig.limits`, `src/configs/limits.yaml`, scheduler sim fixtures and the limit tests. Rules: `job-config-layering.md` "Resource control". Dependencies: project-config. - [ ] job-resize — NEW 2026-10-10 (from long-running services discovery; not on the services path). Grow/shrink a running parallel job's allocation (LSF resizable jobs, `bresize`) for elastic applications. Brief: `.kiro/specs/job-resize/brief.md`. Dependencies: parjob-placement, host-failure-job-recovery. - [ ] project-delegated-membership — NEW 2026-10-10 (from project-config requirements, user decision: future). Project managers edit their project's (and subtree's) members at runtime without an admin config push; needs a durable store that coexists with `xceadmin` config pushes (e.g. a separate delegated member list) and audit. Dependencies: project-config, rbac-persistence.