From 332f86eaa1513720a0c2a4a1dd6b8e69a8b5d3ef Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 13 Oct 2026 04:07:00 +0000 Subject: [PATCH] docs(spec): resource control under the project tree; retire limits.yaml The project tree is the only place concurrent use is capped: quotas are written on the project (quota, member_quota over the subtree) and compiled into the limit plugin, which becomes internal. Queues keep per-job ceilings only; host slot caps are supply. Project path is the identity and the only submit form; members are users and user groups, never sub-projects. Adds the limits-to-owners backlog item. Co-Authored-By: Claude Opus 5.5 (1M context) --- .kiro/specs/application-profile/brief.md | 6 +- .kiro/specs/project-config/brief.md | 52 ++++++++++++--- .kiro/specs/project-config/requirements.md | 74 ++++++++++++++-------- .kiro/specs/project-config/spec.json | 2 +- .kiro/specs/project-guarantee/brief.md | 5 +- .kiro/steering/job-config-layering.md | 65 +++++++++++++++++-- .kiro/steering/roadmap.md | 5 +- 7 files changed, 161 insertions(+), 48 deletions(-) diff --git a/.kiro/specs/application-profile/brief.md b/.kiro/specs/application-profile/brief.md index fb1251f1..702a5ceb 100644 --- a/.kiro/specs/application-profile/brief.md +++ b/.kiro/specs/application-profile/brief.md @@ -293,8 +293,8 @@ fits XCE's LSF-style job model. (`app-access-control`). - Resource values in a profile describe what the application needs (its essential resource requirement and per-job execution limits). They are - not scheduling policy. Queue-level scheduling limits keep applying on top, - unchanged. + not scheduling policy. Queue per-job ceilings and project quotas check + the merged result on top (no queue aggregate caps, 2026-10-12). - **Live merge** (`job-config-merge`). The job records its profile name and its own values, not the effective values. The scheduler computes the effective values and recomputes them when the profile changes: a pending job @@ -501,7 +501,7 @@ resreq: grammar. - Consequence: `rusage` is the one profile part that holds **defaults**; a profile cannot enforce a minimum reservation. Hard protection of shared - resources comes from `limits.yaml` and project quotas. + resources comes from project quotas (`limits.yaml` is retired, 2026-10-12). - Side effects to handle in requirements: measured resources (memory) are charged `max(reserved, actual)` once usage is reported (`resource-accounting.md`), so lowering them mostly self-corrects, but diff --git a/.kiro/specs/project-config/brief.md b/.kiro/specs/project-config/brief.md index 267395dd..b0d508e3 100644 --- a/.kiro/specs/project-config/brief.md +++ b/.kiro/specs/project-config/brief.md @@ -27,8 +27,9 @@ limits cannot be configured end to end, and nothing shows usage per project. - No project fairshare, accounting view or access check exists. ## Desired Outcome (v1 = quota; user decision 2026-10-09) -- **Project definitions** in the cluster configuration: name, description, - members (users and user groups), priority; plus a cluster default project. +- **Project definitions** in the cluster configuration: path name, + description, members (users and user groups), `quota` and `member_quota`; + plus a cluster default project. (No priority: queues own ordering.) Schema, YAML, JSON schema, `xceadmin` mapping, live-upgrade replication. - **Project selection** at submit: `-P` > the user's default project (if defined) > the cluster default > `"default"` (LSF p107 order, without @@ -37,9 +38,10 @@ limits cannot be configured end to end, and nothing shows usage per project. (apiserver early check, controller authoritative), with a clear error; `cluster_admin` keeps its existing bypass. Whether this is RBAC scope (`project` scope type) or config membership is settled in requirements. -- **Quotas**: the scheduler loads project groups from config so - `PROJECTS` / `PER_PROJECT` limits in `limits.yaml` work end to end - (slots, jobs, mem, any resource), with pend reasons naming the project. +- **Quotas** (user decision 2026-10-12, replacing "`PROJECTS` / + `PER_PROJECT` limits in `limits.yaml`"): written on the project itself, + rolling up the tree, with pend reasons naming the project; see + `job-config-layering.md` "Resource control". - **Visibility**: project shown in job details and listings; per-project usage against its limits (REST, `xcecli`, REPL `info limit`). - **Merge input**: the project is one level for `job-config-merge` (default @@ -52,10 +54,40 @@ scheduler fills `pgmgr` from them, and the existing limit machinery does the rest. Projects already reach the scheduler on each job, so the WAL needs no new job field. +**Quotas compile to limits** (user decisions 2026-10-12). The project tree +is the only place concurrent use is capped; the limit plugin stays as the +internal mechanism and admins no longer write limits: + +```yaml +projects: + - path: eng + members: [eng-ug] + quota: { slots: 400, gpu: 16, mem: 2T } + member_quota: { slots: 64 } # per user, over eng's whole subtree + - path: eng/hpc + members: [alice, bob] + quota: { slots: 200 } # must also fit eng +``` + +- On each config push the scheduler generates, per project P and resource + r, a limit `proj:

:` (project consumer {P} matching P and its + descendants) and, for `member_quota`, `projm:

:` (same consumer plus + users `all` per user, vres key `limit=projm:...:/u=`, + `lib/xcelimit.c:402`). New in the plugin: descendant matching on the + project consumer. +- The path contains `/`, which is also the vres key's field separator; the + trace splitter (`sched_trace_capture.c:40`, `sti_copy_scope`) would render + `p=eng/hpc` as `p=eng,hpc`. Escape the project field in the key or use + another internal separator (design). +- Project-scoped entries in `limits.yaml` are rejected by `xceadmin` with a + pointer to project quotas; the rest of `limits.yaml` is retired by + `limits-to-owners`. + ## Scope -- **In**: project config model and pipeline; default project resolution; - submit access check; `pgmgr` loading; project limits end to end; display; - tests (unit, sim, live). +- **In**: project config model and pipeline (`quota`, `member_quota`); + default project resolution; submit access check; `pgmgr` loading; quotas + compiled to limits and enforced end to end; display; tests (unit, sim, + live). - **Out**: guaranteed shares, owned pools, borrowing and reclaim (`project-guarantee`); fairshare; accounting history/billing. @@ -72,7 +104,7 @@ new job field. - **Upstream**: config pipeline (`admin-config-tool`, `config-upgrade-snapshot`), limit plugin (`plugins/limit.c`, `lib/xcelimit.c`), RBAC (`rbac_check.go`). - **Downstream**: `job-config-merge` (project level), `project-guarantee`, - `lsf-config-migration`, `rbac-persistence` (if access uses RBAC scopes). + `limits-to-owners`, `lsf-config-migration`, `rbac-persistence` (if access uses RBAC scopes). ## Existing Spec Touchpoints - **Extends**: none. @@ -83,3 +115,5 @@ new job field. - Middleware chain unchanged; additive Cap'n Proto; old jobs keep their recorded project name even if the project is later removed from config (requirements decide what a pending job with a removed project does). +- Project paths are identities: a job keeps the path it was submitted + under; usage of a running job is never moved by a tree change. diff --git a/.kiro/specs/project-config/requirements.md b/.kiro/specs/project-config/requirements.md index 20f43eda..27e34ee9 100644 --- a/.kiro/specs/project-config/requirements.md +++ b/.kiro/specs/project-config/requirements.md @@ -3,7 +3,7 @@ ## Project Description (Input) Discovery brief: `.kiro/specs/project-config/brief.md`. Governing rules: `.kiro/steering/job-config-layering.md` (2026-10-09): the project owns resource ownership (membership, quotas, later guarantees, accounting) and never owns how a job runs or is ordered. -**Who has the problem:** cluster admins who need to control which users may charge work to a project and how much of the cluster each project may use; users who need to know which projects they can use and why a job is held by a project limit. +**Who has the problem:** cluster admins who need to control which users may charge work to a project and how much of the cluster each project may use; users who need to know which projects they can use and why a job is held by a project quota. **Current situation:** - A project is a free-text label on the job (`-P`). Anyone can name any project. The apiserver only checks its length, and the scheduler maps an empty name to `"default"`. @@ -12,7 +12,7 @@ Discovery brief: `.kiro/specs/project-config/brief.md`. Governing rules: `.kiro/ - `xcecli jobs list -P` is rejected by the apiserver ("unsupported filter"). - Nothing shows a project's usage. -**What should change (v1 = quota):** nested project definitions in cluster config; default-project resolution; a membership check at submit; project limits working end to end and rolling up the tree; project visibility. Guarantees are `project-guarantee`. +**What should change (v1 = quota):** nested project definitions in cluster config, each with its own quotas; default-project resolution; a membership check at submit; project quotas enforced end to end and rolling up the tree; project visibility. Guarantees are `project-guarantee`. **Decisions (2026-10-10):** - Access is by project membership. @@ -22,29 +22,38 @@ Discovery brief: `.kiro/specs/project-config/brief.md`. Governing rules: `.kiro/ - Projects nest, named by path (e.g. `eng/hpc/cfd`). - A job may be charged to any project in the tree. - Membership is not inherited between parent and child, and a user may be a member of any number of projects. -- Quotas roll up: a job must fit the limits of its project and of every ancestor. +- Quotas roll up: a job must fit the quotas of its project and of every ancestor. - RBAC interaction: - a new `project` scope for job permissions, so a project manager can view and control the jobs of a project and its whole subtree; - a `project:view` permission. - Runtime (delegated) membership edits are left for later; membership changes only through cluster config. +**Decisions (2026-10-12, resource control under the project tree; `job-config-layering.md` "Resource control"):** +- The project tree is the only place concurrent use is capped. Quotas are written on the project (`quota`, `member_quota`), not in `limits.yaml`; the limit plugin stays as the internal mechanism, and the scheduler compiles project quotas into limits. +- `limits.yaml` is removed; this feature rejects its project-scoped entries, `limits-to-owners` retires the rest. +- `member_quota` caps each user's jobs in the project's whole subtree. +- Quota values are absolute amounts; no percentages in v1. +- The full path is the project's identity and the only form a submit accepts; there is no last-segment shorthand. +- Members are users and user groups (with `all` and excludes); a sub-project is never a member. + ## Introduction This feature makes projects first-class cluster configuration entities that control resource ownership. -- **Definitions:** admins define a tree of projects named by path, each with its own members, and say which user groups default to which project. +- **Definitions:** admins define a tree of projects named by path, each with its own members and quotas, and say which user groups default to which project. - **Submit:** the system resolves each job's project, admits only members, and records the project on the job. -- **Quotas:** existing project-scoped limits apply to the right jobs and roll up the tree, and a held job's reason names the project. -- **Visibility:** users and admins can see projects and their usage against limits. +- **Quotas:** each project's quotas apply to its own and its descendants' jobs and roll up the tree, and a held job's reason names the project. +- **Visibility:** users and admins can see projects and their usage against quotas. - **RBAC:** roles can grant job permissions over a project subtree (project managers) and the right to view projects one does not belong to. - **Compatibility:** a cluster that defines no projects keeps today's behaviour. ## Boundary Context - **In scope**: - - nested project definitions (path name, description, members, default-for groups) and a cluster default project; + - nested project definitions (path name, description, members, default-for groups, `quota`, `member_quota`) and a cluster default project; - default-project resolution, membership check at submit, and an "open" mode when no projects are defined; - pending-job behaviour when a project or membership changes; - - project-scoped limits applying end to end and rolling up the tree, with project-naming pend reasons; + - project quotas compiled into the scheduler's limits, applying end to end and rolling up the tree, with project-naming pend reasons; + - rejecting project-scoped entries in `limits.yaml`; - overlapping membership (a user in several projects); - - project listing (as a tree) and detail with usage against limits; + - project listing (as a tree) and detail with usage against quotas; - RBAC: a `project` scope for job permissions (covering the project's subtree) and a `project:view` permission; - job filtering by project. - **Out of scope**: @@ -57,13 +66,17 @@ This feature makes projects first-class cluster configuration entities that cont - per-user default projects in `users.yaml`; - membership inherited between parent and child projects; - moving or renaming a project while keeping its jobs' recorded name; - - wildcard or pattern matching of member names. + - wildcard or pattern matching of member names; + - percentage quotas; + - retiring the rest of `limits.yaml` (user, queue and host scopes) and moving per-host slots to host config (`limits-to-owners`); + - aggregate caps on queues (dropped by decision 2026-10-12, not deferred). - **Adjacent expectations**: - - The existing limit configuration (`limits.yaml`) stays the place where project quotas are written; this feature makes those limits take effect for defined projects. + - Project quotas are written on the project (decision 2026-10-12, reversing "`limits.yaml` stays the place where project quotas are written"). The scheduler's limit plugin enforces them as generated limits; admins never write those limits. + - Quotas count what the scheduler allocated or reserved, not measured use; a job without a memory reservation can exceed its project's memory quota at run time (accepted; profile default reservations close the gap). - User groups come from the existing user-group configuration. - Grants made through the role API are subject to the existing `rbac-persistence` gap (not durable across apiserver restart) until that spec lands. - `job-config-merge` treats project quotas as ceilings: they hold or admit a job but never change its values. - - The defect that resolved limit host and project scopes against each other's group tables (it also broke host-group limits) was fixed separately as bug-1933 (`6ecd418d`); Requirement 6.7 keeps it as a requirement. + - The defect that resolved limit host and project scopes against each other's group tables (it also broke host-group limits) was fixed separately as bug-1933 (`6ecd418d`); Requirement 6.8 keeps it as a requirement. ## Requirements @@ -71,13 +84,14 @@ This feature makes projects first-class cluster configuration entities that cont **Objective:** As a cluster admin, I want to define projects and their members in the cluster configuration, so that project ownership is explicit and versioned with the rest of the cluster config. #### Acceptance Criteria -1. The configuration tool shall accept project definitions consisting of a path name, an optional description, members (user names, user-group names, or `all`), optional excluded members, and an optional list of user groups for which the project is the default. +1. The configuration tool shall accept project definitions consisting of a path name, an optional description, members (user names, user-group names, or `all`), optional excluded members, an optional list of user groups for which the project is the default, an optional `quota` and an optional `member_quota`, each a map from resource name to an absolute amount. 2. The configuration tool shall treat a path name of one segment (e.g. `eng`) as a top-level project and a path of several `/`-separated segments (e.g. `eng/hpc/cfd`) as a child of the project named by the path without its last segment. 3. The configuration tool shall accept an optional cluster default project naming one defined project. -4. If a project definition repeats a path name, uses a reserved name, has an empty or malformed path segment, names a parent project that is not defined, names an undefined user group, or lists a default-for user group that is not covered by the project's members, then the configuration tool shall reject the configuration with an error identifying the project and the offending field. +4. If a project definition repeats a path name, uses a reserved name, has an empty or malformed path segment, names a parent project that is not defined, names an undefined user group, lists a default-for user group that is not covered by the project's members, or has a `quota` or `member_quota` entry naming an undeclared resource or giving a negative, non-numeric or percentage amount, then the configuration tool shall reject the configuration with an error identifying the project and the offending field. 5. If the cluster default project names an undefined project, or a project whose members do not include every user, then the configuration tool shall reject the configuration with an error naming the cluster default project. 6. When a configuration with changed project definitions is applied by live configuration upgrade, the Cluster shall apply the new definitions to all subsequent submissions and scheduling decisions without a restart. -7. The Cluster shall keep project definitions across controller failover and restart identically to other cluster configuration. +7. The Cluster shall keep project definitions, including their quotas, across controller failover and restart identically to other cluster configuration. +8. If the limit configuration contains a limit scoped to projects (listed or per project), then the configuration tool shall reject the configuration with an error naming the limit and stating that project quotas are written on the project. ### Requirement 2: Project resolution at submit **Objective:** As a user, I want my job to get the right project when I do not name one, so that I do not have to pass `-P` for my usual project. @@ -88,6 +102,7 @@ This feature makes projects first-class cluster configuration entities that cont 3. When a job is submitted without a project name and no default-for rule matches the submitter, the Cluster shall use the cluster default project if one is configured, and otherwise the project `default`. 4. The Cluster shall record the resolved project name on the job at submit, show it in the job's details, and keep it unchanged across later configuration changes, failover and restart. 5. The Cluster shall apply the same project resolution to every submit path (REST API, `xcecli`, harness and client library) and to every element of a job array. +6. The Cluster shall identify a project at submit only by its full path name; a name that is only the last segment of a defined project's path (e.g. `cfd` for `eng/hpc/cfd`) is treated as the top-level project of that name. ### Requirement 3: Membership check at submit **Objective:** As a cluster admin, I want only a project's members to submit work under it, so that projects control who may use their resources. @@ -121,29 +136,31 @@ This feature makes projects first-class cluster configuration entities that cont 5. The Cluster shall allow an admin to kill or modify a job held by Requirement 5 using the existing job controls. ### Requirement 6: Project quotas -**Objective:** As a cluster admin, I want limits scoped to projects to cap each project's usage, so that a project cannot use more than its share of the cluster. +**Objective:** As a cluster admin, I want each project's quotas to cap the resources its jobs hold at once, so that a project, or one member of it, cannot use more than its share of the cluster. #### Acceptance Criteria -1. When a limit is scoped to listed projects, the Scheduler shall count against that limit exactly the jobs whose recorded project is a listed project or a descendant of one (or any project, for `all`), and is neither an excluded project nor a descendant of one. -2. When a limit is scoped per project, the Scheduler shall apply the limit value separately to each matching project, counting that project's own jobs and the jobs of its descendants. -3. The Scheduler shall dispatch a job only if it fits every limit that applies to the job's recorded project or to any of its ancestors. -4. The Scheduler shall count a job only against limits matched through its recorded project and that project's ancestors, never against limits of other projects its owner belongs to. -5. If dispatching a job would exceed a project-scoped limit, then the Scheduler shall keep the job pending with a reason naming the limit and the project whose limit is full (the job's own project or the ancestor). -6. When usage under a project-scoped limit falls below its value, the Scheduler shall consider jobs held by that limit for scheduling again. -7. The Scheduler shall resolve each limit's user, queue, project and host scopes against the corresponding kind of names only, so that a project name never matches a host group and a host-group name in a host scope matches the hosts of that group. -8. The live Scheduler and the scheduler simulator shall make identical project-limit decisions for the same input. +1. The Scheduler shall count against a project's `quota` for a resource the amount of that resource allocated to every running job whose recorded project is that project or one of its descendants. +2. The Scheduler shall count against a project's `member_quota` for a resource, separately for each user, the amount of that resource allocated to that user's running jobs whose recorded project is that project or one of its descendants, whether or not the user is a direct member of that project. +3. The Scheduler shall dispatch a job only if, for every resource it is allocated, it fits the `quota` and `member_quota` of its recorded project and of every ancestor of that project. +4. The Scheduler shall count a job only against the quotas of its recorded project and that project's ancestors, never against quotas of other projects its owner belongs to. +5. If dispatching a job would exceed a quota, then the Scheduler shall keep the job pending with a reason naming the project whose quota is full (the job's own project or the ancestor), the resource, whether it is the project's quota or the member quota of the job's owner, and the current usage and quota amount; the reason shall not name an internal limit. +6. When usage under a quota falls below its amount, the Scheduler shall consider jobs held by that quota for scheduling again. +7. When a configuration change adds, changes or removes a quota, the Scheduler shall apply the new amounts to all subsequent scheduling decisions without a restart, keep counting the usage of running jobs against their recorded project and its ancestors, and never stop or alter a running job because a quota now exceeds its usage. +8. The Scheduler shall resolve each generated quota's project and user scopes against project and user names only, so that a project name never matches a host group or user group (the scope separation fixed by bug-1933). +9. The live Scheduler and the scheduler simulator shall make identical quota decisions for the same input. +10. The Scheduler shall show a project path containing `/` unchanged in pend reasons, decision traces and diagnostics. ### Requirement 7: Project visibility -**Objective:** As a user or admin, I want to see projects, their members and their usage against limits, so that I know which projects I can use and why a job is held. +**Objective:** As a user or admin, I want to see projects, their members and their usage against quotas, so that I know which projects I can use and why a job is held. #### Acceptance Criteria -1. When a user lists projects through the REST API or `xcecli`, the Cluster shall return every project the user is a member of, with its path name and description, shown in tree order, and mark the project that would be the user's default. +1. When a user lists projects through the REST API or `xcecli`, the Cluster shall return every project the user is a member of, with its full path name and description, shown in tree order, and mark the project that would be the user's default, so that a submit client can offer them for choosing the project to charge. 2. Where the requester has the cluster administrator role, or a `project:view` permission covering a project, the Cluster shall include that project in the list with its members, excluded members and default-for user groups. -3. When a user requests a project's details, the Cluster shall show, for each limit that applies to the project or to any of its ancestors, the limit name, the project the limit is matched through, the limited resource, the limit value and the current usage counted against it (including descendants' jobs). +3. When a user requests a project's details, the Cluster shall show, for the project and for each of its ancestors, every `quota` and `member_quota` entry with the project it is set on, the resource, the amount and the current usage counted against it (including descendants' jobs; for `member_quota`, the requesting user's own usage, and every user's usage to an admin or a holder of `project:view` covering the project). 4. If a user requests the details of a project they are not a member of, and they have neither the cluster administrator role nor a `project:view` permission covering the project, then the Cluster shall reject the request without revealing the project's members or usage. 5. When a user lists jobs filtered by project through the REST API or `xcecli`, the Cluster shall return only jobs with that recorded project, or, when the user asks to include sub-projects, with that project or any descendant, subject to the user's existing job-visibility permissions. 6. While no project is defined, the Cluster shall report that projects are not configured when projects are listed. -7. The scheduler's diagnostic interface shall show, for a named project, the limits that apply to it and its current usage. +7. The scheduler's diagnostic interface shall show, for a named project, the quotas that apply to it (its own and its ancestors') and their current usage. ### Requirement 8: Project-scoped access control **Objective:** As a cluster admin, I want to grant a project manager rights over the jobs and details of a project and its sub-projects, so that project leads can manage their project's work without being cluster administrators. @@ -165,3 +182,4 @@ This feature makes projects first-class cluster configuration entities that cont 1. When a job recorded before this feature is recovered, the Cluster shall keep its recorded project name, treating an empty name as `default`. 2. The Cluster shall accept a cluster configuration without any project definitions and behave as Requirement 4 describes. 3. When a configuration tool of an earlier version pushes a configuration without project definitions, the Cluster shall treat it as defining no projects. +4. While no project defines a quota, the Scheduler shall dispatch jobs exactly as it would without this feature's quota checks. diff --git a/.kiro/specs/project-config/spec.json b/.kiro/specs/project-config/spec.json index 78addea7..9f1baa54 100644 --- a/.kiro/specs/project-config/spec.json +++ b/.kiro/specs/project-config/spec.json @@ -1,7 +1,7 @@ { "feature_name": "project-config", "created_at": "2026-10-10T03:25:03Z", - "updated_at": "2026-10-10T03:48:51Z", + "updated_at": "2026-10-13T04:06:36Z", "language": "en", "phase": "requirements-generated", "approvals": { diff --git a/.kiro/specs/project-guarantee/brief.md b/.kiro/specs/project-guarantee/brief.md index c974a444..9c273aed 100644 --- a/.kiro/specs/project-guarantee/brief.md +++ b/.kiro/specs/project-guarantee/brief.md @@ -24,7 +24,10 @@ for it, while idle capacity is still lent to others. - Guarantee state and borrowing are visible (REST, `xcecli`, REPL). ## Approach -To be chosen in discovery when the spec starts. Candidates: LSF-style +Guarantees (and later fairshare weights) sit on the same project-tree nodes +as `quota` / `member_quota` (`job-config-layering.md` "Resource control", +2026-10-12): one tree, one ownership model; the queue only turns fairshare +on. To be chosen in discovery when the spec starts. Candidates: LSF-style guarantee service classes (`lsb.resources` GuaranteedResourcePool), or guarantees as reserved virtual resources built on the limit/vres machinery plus preemption. diff --git a/.kiro/steering/job-config-layering.md b/.kiro/steering/job-config-layering.md index 0c3bbeb9..52df7a0a 100644 --- a/.kiro/steering/job-config-layering.md +++ b/.kiro/steering/job-config-layering.md @@ -4,7 +4,8 @@ How a job's settings are split between the job, its application profile, its queue and its project, and how they interact. Decided 2026-10-09 (`/kiro-discovery`, configuration layering). Specs that follow this: `job-config-merge`, `application-profile`, `project-config`, -`project-guarantee`, `parjob-placement`, `lsf-config-migration`. +`project-guarantee`, `parjob-placement`, `lsf-config-migration`, +`limits-to-owners`. ## Why not LSF's rules LSF lets the queue, the application profile and the job set the same @@ -19,9 +20,9 @@ XCE instead gives **every attribute exactly one owning level**. | Level | Owns | Never owns | |---|---|---| -| **Queue**: scheduling policy | priority and ordering, host partition, dispatch/run windows, preemption policy, policy ceilings (max run time, max tasks, max hosts, ...) | resource requirement, anything about how a job executes (pre/post-exec, env, launch, limits enforced on the job) | +| **Queue**: scheduling policy | priority and ordering, host partition, dispatch/run windows, preemption policy, per-job ceilings (max run time, max tasks, max hosts, ...) | resource requirement, anything about how a job executes (pre/post-exec, env, launch, limits enforced on the job), aggregate caps on concurrent use | | **Application profile**: what the application needs and how it runs (essentials plus default reservations, 2026-10-12) | resreq (select / rusage / span / same / order), task range (min / max), execution mode and options, launch type, env, **pre/post-exec**, execution limits (mem, runtime, CPU, tasks) | priority, queue choice, quotas | -| **Project**: resource ownership | membership, quotas / limits, later guarantees, accounting | how a job runs or is ordered | +| **Project**: resource ownership | membership, the only caps on concurrent use (`quota`, `member_quota`), later guarantees and fairshare weights, accounting | how a job runs or is ordered | | **Job** | its own values, only for attributes its profile allows | n/a | Access to each is separate: queue and profile by RBAC permission, project by @@ -107,7 +108,7 @@ One line per part; details below. grammar. - Consequence: `rusage` is the one profile part that holds **defaults**; a profile cannot enforce a minimum reservation. Hard protection of shared - resources comes from `limits.yaml` and project quotas. + resources comes from project quotas (see "Resource control"). - Side effects to handle in requirements: measured resources (memory) are charged `max(reserved, actual)` once usage is reported (`resource-accounting.md`), so lowering them mostly self-corrects, but @@ -132,6 +133,60 @@ There is no third case: no queue resreq, no rusage merge with the queue, no ordering of queue against profile settings, no compound/alternative merge stages. +## Resource control +User decisions 2026-10-12. The **project tree is the only place where +concurrent use of resources is capped**. Four concepts, one owner each; the +word "limit" alone is not used for any of them: + +| Concept | Question it answers | Owner | Examples | +|---|---|---|---| +| **Supply** | what a host has | host / resource config | slots per host (LSF `MXJ`), host resources | +| **Quota** (consumption) | how much a group of jobs may hold at once | **project** | `quota`, `member_quota`; later guarantee, fairshare weight | +| **Ceiling** (job shape) | how big one job may be | queue | max tasks, max run time, max hosts per job | +| **Execution limit** (enforcement) | what the running job may use | application profile | cgroup memory, runtime, CPU | + +- **No aggregate caps outside projects.** A queue has per-job ceilings only; + a queue that must not flood the cluster gets a host partition, or its users + a capped project. User-group × queue caps do not exist. +- **The limit plugin is internal.** The scheduler compiles project quotas + into limits on every config push (`proj::` for `quota`, + `projm::` with a per-user consumer for `member_quota`), reusing + the vres usage counters, instance carry-over across pushes (bug-1967) and + live/sim parity. Admins never write limits. +- **`limits.yaml` is removed** (not in production; no deprecation period, so + there is one config path, not two). Its entries map as: `PROJECTS` / + `PER_PROJECT` → project `quota`; `USERS` + `PER_USER` → `member_quota` (on + the root project for a cluster-wide per-user cap); a user group's total → + a project for the group; `HOSTS` + `PER_HOST` → host supply; queue + aggregates and mixed tuples → dropped. Retirement and the host-supply move + are `limits-to-owners`; project quotas are `project-config`. +- **Quotas count allocation**, what the scheduler reserved, not measured use. + A job without a memory reservation can exceed its project's memory quota + at run time (`resource-accounting.md` charges hosts `max(reserved, + actual)`; quotas do not). Accepted: profile default reservations close the + gap. + +### The project tree +- **Identity is the full path** (`eng/hpc/cfd`), unique among siblings only. + The job records the path at submit and is charged to that path's + ancestors for as long as it runs; a later tree change never moves a + running job's usage. Moving a subtree renames its projects (rare, + visible). Config is a flat list of full paths, not nested YAML. +- **Submit names the full path** (`xcecli -P eng/hpc/cfd`, REST + `project_name`, any UI); no last-segment shorthand. Clients offer the + user's projects (the project listing) instead. +- **Two relations, kept apart:** the tree says where usage rolls up; + membership says who may submit. Members are users and user groups (plus + `all` and excludes, exclude wins), resolved at submit. A sub-project is + never a member, and membership is not inherited either way. +- **`quota`** on P: a job dispatches only if it fits P's quota and every + ancestor's; P's usage includes all descendants' jobs. Absolute amounts + only (no percentages in v1). +- **`member_quota`** on P: a cap per user over that user's jobs anywhere in + P's subtree, whether or not the user is a direct member of P. +- Pend reasons name the project, resource and usage ("project eng quota on + gpu full (16/16)"), never an internal limit name. + ## Convenience defaults are presets, not a level Apart from default reservations (the profile's `rusage` entries), tuning that most jobs of an application want but none must have @@ -183,3 +238,5 @@ is today. - LSF migration (`lsf-config-migration`): an LSF queue `RES_REQ` that only selects hosts becomes a queue host partition; anything else moves into a profile or is reported. LSF queue `PRE_EXEC`/`POST_EXEC` move into profiles. + LSF `Limit` sections are split by consumer as in "Resource control"; queue + and mixed entries are reported, not migrated. diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index ea02a3ee..d6da1a54 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -73,7 +73,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han - [ ] controller-job-cleanup — NEW 2026-09-29 (`/kiro-discovery`, Path C). Primary-only periodic sweep (configurable `job_cleanup_period`, default 300 s) removes DONE/EXIT jobs finished longer than configurable `job_retention` (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever `jobmgr` delete; refcount-safe `xkvmgr` delete), propagated to scheduler and apiserver (`OnJobDeleted`). Retires the node-local snapshot-only `job_should_retain` finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: `.kiro/specs/controller-job-cleanup/brief.md`. Dependencies: none (extends ha-entity-snapshots). - [ ] job-status-detail-display — NEW 2026-09-28 (`/kiro-discovery`, Path E). End users (REST/`xcecli`) see NO pend reason for ANY job today: the apiserver cgo conversion (`backend/convert.go`) never maps `xce_job_t.pend_reasons`, so `pending_reason` is always empty. Maps it to a structured JSON (including `alternatives` from day one) using libxce's code→text mapping, and renders it in `xcecli` job detail. Open for requirements: also map `run_usage` → `JobRusage` (same gap; reserved vs. used for RUN jobs). Brief: `.kiro/specs/job-status-detail-display/brief.md`. Dependencies: none for flat reasons; scheduler-alter-pend-reasons for per-alternative rendering. - [ ] job-config-merge — NEW 2026-10-09 (`/kiro-discovery`, Path E "configuration layering"; ref LSF 10.1 Admin Guide pp. 314-316, 435-437, 561-563, 643, 660). One merge function in `src/lib` turning the job's own values + application profile + queue policy + project into effective values (resreq incl. `span`, task range, limits, pre/post-exec, env). One owner per attribute (user decision; `.kiro/steering/job-config-layering.md`): the job may add to a profile's values but never change or remove them (additive rule, 2026-10-12, replacing *default*/*fixed* markers); queue ceilings and project quotas only check (reject if never satisfiable, pend if full now). **Live** merge (user decision): WAL keeps inputs + names; controller/apiserver validate at submit; scheduler computes effective values and re-evaluates pending jobs on config change. Queue is policy only: queue `res_req` and `pre_exec`/`post_exec` deprecated (pre/post-exec moves to the profile), dropped queue limits (cpulimit/memlimit/runlimit/task limit) wired. Brief: `.kiro/specs/job-config-merge/brief.md`. Dependencies: none. -- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (members, priority, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config so `PROJECTS`/`PER_PROJECT` limits work end to end, per-project usage display. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none. +- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (path identity, members, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config, quotas written on the project (`quota`, `member_quota` over the subtree; user decision 2026-10-12, replacing `PROJECTS`/`PER_PROJECT` limits in `limits.yaml`) and compiled into the limit plugin, per-project usage display. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none. - [ ] host-failure-job-recovery — NEW 2026-10-10 (`/kiro-discovery`, "Long-running services" phase 1; also a batch gap: RUN jobs on a dead host stay RUN forever, `controller/sched.c:631` FIXME). After a grace period, jobs on a lost host go to an unknown state, then are requeued (rerunnable, job option or profile default) or end EXIT "host lost"; multi-host jobs resolved as a whole; attempt numbers fence a stale copy when the host returns; manual requeue (REST/xcecli). Brief: `.kiro/specs/host-failure-job-recovery/brief.md`. Dependencies: none. - [ ] service-supervision — NEW 2026-10-10 (Long-running services phase 1). Application profile kind `service`: restart policy (always/on-failure/never, backoff, give-up), liveness and readiness probes (command/TCP/HTTP) run by each replica's own worker (no special replica), in-place restart, change-only health reports, display. Brief: `.kiro/specs/service-supervision/brief.md`. Dependencies: application-profile, host-failure-job-recovery. @@ -120,6 +120,7 @@ Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generi ## Future Specs / Backlog (dependency order) - [ ] gang-groups — NEW 2026-10-11; **demoted to backlog 2026-10-12** (user decision N24 in `resource-questions.md`). LLM training is one parallel job with topology requirements (PR-10), so it needs no gang. Re-scoped as a **co-start group** of separately managed jobs and services: RL post-training (trainer, rollout engines, reward and reference models), disaggregated serving (prefill and decode pools, router), streaming pipelines. All-or-nothing start, close placement, group failure policy. Revisit when RL or disaggregated serving reaches the roadmap. Brief: `.kiro/specs/gang-groups/brief.md`. Dependencies: resource-instances, resource-claims, service-replica-set. - [ ] project-guarantee — NEW 2026-10-09 (configuration layering, backlog). Project-owned guarantees (resource amount or host pool), borrowing of idle guaranteed capacity, reclaim through preemption. Brief: `.kiro/specs/project-guarantee/brief.md`. Dependencies: project-config, scheduler-preemption. +- [ ] limits-to-owners — NEW 2026-10-12 (configuration layering, user decisions; backlog, no brief yet). Retire `limits.yaml` (not in production, removed without deprecation): the limit plugin becomes internal, fed only by generated limits. `USERS`/`PER_USER` caps become `member_quota` (root project for a cluster-wide per-user cap), a user group's total becomes a project quota, `HOSTS`/`PER_HOST` slot caps move to host supply in host config (LSF `MXJ`), queue aggregate caps and mixed tuples (user group × queue) are dropped. Touches `xceadmin` mapping and schema (`admin-config-tool`), `ClusterConfig.limits`, `src/configs/limits.yaml`, scheduler sim fixtures and the limit tests. Rules: `job-config-layering.md` "Resource control". Dependencies: project-config. - [ ] job-resize — NEW 2026-10-10 (from long-running services discovery; not on the services path). Grow/shrink a running parallel job's allocation (LSF resizable jobs, `bresize`) for elastic applications. Brief: `.kiro/specs/job-resize/brief.md`. Dependencies: parjob-placement, host-failure-job-recovery. - [ ] project-delegated-membership — NEW 2026-10-10 (from project-config requirements, user decision: future). Project managers edit their project's (and subtree's) members at runtime without an admin config push; needs a durable store that coexists with `xceadmin` config pushes (e.g. a separate delegated member list) and audit. Dependencies: project-config, rbac-persistence. - [ ] app-access-control — NEW 2026-10-12 (deferred from application-profile, user decision; backlog, design kept in the application-profile brief). Per-profile RBAC: `app:use` / `app:view` with an `app` scope type (profile names or `all`), checks on job create, restart and listing, `rbac.yaml` defaults, the current default profile always usable with `queue:submit`, aliases checked against the target profile, role API support, `cluster_admin` bypass only. Dependencies: application-profile; rbac-persistence for durable API-made grants. @@ -130,7 +131,7 @@ Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generi - [ ] job-requeue — NEW 2026-10-12 (from application-profile discussion, user decision; backlog, no brief yet). Automatic requeue on given exit values (LSF `REQUEUE_EXIT_VALUES`) and success exit values (`SUCCESS_EXIT_VALUES`), with a requeue loop limit; wires the commented-out `requeue` job action (`apiserver/controllers/jobs.go`) and the REST `requeue_error_values` / `rerunable` options that are silently dropped today (bug-1970). Adds the matching application-profile attributes (a profile attribute ships with its job feature). Later, same principle: checkpoint settings and warning-before-kill each get their own spec. Dependencies: application-profile. - [ ] xcecli-presets — NEW 2026-10-12 (from application-profile discussion, user decisions). Client-side presets replace convenience defaults in profiles: a preset is a set of job options (`--app`, `--queue`, `-R`, `-n`, env, limits) expanded by the client, a flag replacing the preset's value for that option; one file per preset (`~/.config/xce/presets//.yaml`, same layout in the apiserver's presets dir), env capture by name (`pass:`; `XCE_*` and secret-looking names excluded); the apiserver only shares admin-deployed presets read-only (`GET /xce/v1/presets[/{ns}/{name}]`, files in `/config/presets/` or `presets.dir`, re-read on change; not in ClusterConfig), and `preset pull` copies one locally so server-side removal never affects users; on a name collision pull refuses unless it is an update of the same server preset (`--as` / `--force` with `.bak`); submit uses local presets only (`preset diff`, `pull --force` user-initiated); user-to-user sharing by `preset export`/`import`; presets carry no privilege and are not a configuration level; no preset field on the server: the client records preset name + hash in the job's `exdata.meta` (passed through uninterpreted; worker skips `meta`), and `preset report` filters jobs on it client-side; the portal does the same expansion. Brief: `.kiro/specs/xcecli-presets/brief.md`. Dependencies: application-profile. - [ ] profile-usage-insight — NEW 2026-10-12 (from application-profile discussion, backlog; no brief yet). Per-profile report over recent jobs comparing the profile's defaults with actual use (requested vs. peak memory, CPU efficiency, OOM kills, pend time, runtime by variant) and suggesting better values for site presets (or a profile essential), e.g. "preset openfoam rusage mem 4000, p95 peak 1600 → suggest 2000" (Kubernetes VPA recommender precedent). **Advisory only** (user decision): the admin applies changes to site presets or `applications.yaml`; nothing tunes itself. Optional user hint ("used 1.4G of 4G requested"). Dependencies: application-profile, run usage in job records. -- [ ] lsf-config-migration — NEW 2026-10-09 (user idea, backlog; no brief yet). A tool (likely an `xceadmin` subcommand) that converts LSF configuration (`lsb.queues`, `lsb.applications`, `lsb.resources` Limit sections, `lsb.users`/`lsb.hosts` groups, `lsb.params` defaults, project settings) into XCE `configs/*.yaml`, with a report of what does not map: queue `RES_REQ` (XCE queues are policy only; move into a profile or drop), LSF's full per-section merge rules vs XCE's simplified ones, compound/alternative resreqs, unsupported parameters. Dependencies: job-config-merge, application-profile, project-config, parjob-placement (span), admin-config-tool. Promote via `/kiro-discovery` when those land. +- [ ] lsf-config-migration — NEW 2026-10-09 (user idea, backlog; no brief yet). A tool (likely an `xceadmin` subcommand) that converts LSF configuration (`lsb.queues`, `lsb.applications`, `lsb.resources` Limit sections (split by consumer into project quotas, member quotas and host supply; queue and mixed entries reported), `lsb.users`/`lsb.hosts` groups, `lsb.params` defaults, project settings) into XCE `configs/*.yaml`, with a report of what does not map: queue `RES_REQ` (XCE queues are policy only; move into a profile or drop), LSF's full per-section merge rules vs XCE's simplified ones, compound/alternative resreqs, unsupported parameters. Dependencies: job-config-merge, application-profile, project-config, parjob-placement (span), admin-config-tool. Promote via `/kiro-discovery` when those land. - [ ] **xce-harness-component** — **PROMOTED to spec 2026-09-13** (`/kiro-discovery`; brief at `.kiro/specs/xce-harness-component/brief.md`). Elevate `xce_test` (the REPL-driven wire-protocol test client used by HA soak / multi-host / resource-value verification) out of `src/controller/testsuite/` into its own first-class `src/harness/` component — own `meson.build`, `CLAUDE.md`, and unit-test binary (carved out of `controller_unit_test`, which today recompiles ~14 of xce_test's own files a second time paired with white-box tests) — fully decoupled from `/xce-build controller` (today's implicit "controller always builds xce_test" coupling, `build-deployment.md:24`). Verified finding: none of xce_test's 15 source files include any controller-private header — only libxce public headers + generated capnp schema — so this is a pure relocation + build-graph split, not a dependency untangling problem, and no xce_test behavior changes. Touches `/xce-test-ha`, `/xce-deploy-ha`, `/xce-test-multihost`, `/xce-test-resources` (each must build `harness` explicitly once decoupled) and is adjacent to (not overlapping) the three existing behavior specs `xce-test-cli-ux`/`xce-test-e2e-primitives`/`xce-test-rc-file`, whose file-path references go stale after the move. Next: `/kiro-spec-init xce-harness-component`. - [x] config-upgrade-snapshot — **descoped from `ha-term-safety` R8 on 2026-06-21 (requirements generated; `.kiro/specs/config-upgrade-snapshot/`).** Closes **bug-857**: a configuration upgrade can leave members with DIVERGENT materialized state because today it does config-push + restart + per-node WAL replay under the new config (`config_repl.c` `do_upgrade_config_with_bytes` ~1001-1163), with no snapshot pinning an identical post-upgrade state (replay is config-dependent — `lost_and_found` re-homes orphaned entities). Approach: an upgrade materializes a **full entity-bearing state snapshot at the config-change marker's `(term,lsn)` boundary**, embeds the new config, and installs it on every member via the `snapshot_transfer_s` transport — no per-node re-derivation; supersedes `ha-term-safety` R7's interim config-push byte delivery for upgrades. **Former HARD dependency `entity-bearing-snapshots` is CLEARED** — it shipped as spec `ha-entity-snapshots` (**IMPLEMENTATION-COMPLETE, validate-impl GO 2026-07-18**; see Future Specs/Backlog below), and `ha-term-safety` (the config-change marker it anchors on) is also implemented. `config-upgrade-snapshot`'s own `spec.json` now has `blocked_on: []`. Verify by reproducing bug-857 (upgrade during chaotic multi-kill, diff materialized state). **Ready to promote to design.**