lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(spec): project shares at a location; pool capacity, defaults, visibility

Browse source 
A share may be owned at a location (host group) as well as anywhere.
Per resource, located host groups and the rest of the cluster form
disjoint pools; usage is charged to the host's pool and every pool follows
the share rules on its own, so located ownership is real. Pool capacity
counts registered hosts including down and closed ones. No-default
submits list the user's projects; a per-user default project is a
client-side xcecli setting; members see their ancestors' usage, not
siblings' details.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-13 07:09:03 +00:00
parent e02310c02d
commit d1be84cd66
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
5 changed files with 91 additions and 31 deletions
Download patch file Download diff file Expand all files Collapse all files

3
.kiro/specs/project-borrowing/brief.md
View file

@ -37,6 +37,9 @@ shares lent out and taken back when the owner needs them.
`scheduler-preemption`. Without preemption, borrowed jobs run to
completion and the owner waits at most one job's run time (bounded by
the queue's run-time ceiling).
- **Located shares are the main case** (2026-10-13): capacity a project
owns at a location (e.g. chem's 16 GPUs on `hg-a100`, or whole condo
nodes) is lent to others while chem does not use it.
- Accounting and display of owned vs borrowed usage per project (REST,
`xcecli`, REPL); pend reasons that say whether a job waits for its own
share or for capacity to borrow.

12
.kiro/specs/project-config/brief.md
View file

@ -80,6 +80,12 @@ projects:
aliases: [eng/ml] # renamed from eng/ml; its jobs follow
- path: eng/tools # no share: draws from eng's unassigned part
members: [eng-ug]
- path: chem
members: [chem-ug]
share:
slots: 200 # in the rest of the cluster
at:
hg-a100: { gpu: 16, slots: 50% } # owned on these hosts
```
- On each config push, and when the cluster's capacity changes, the
@ -95,6 +101,12 @@ projects:
r, using the consumer's existing exclude list) with P's share minus
its children's shares; the root gets one when any top-level share of r
exists;
- a located share or unassigned part adds the hosts consumer (the
location's host group, not per host) to the same limits; the
rest-of-cluster pool's limits use the hosts consumer `all` minus every
located group of that resource (exclude list). The plugin already
combines project and host scopes (`is_host_based_limit`); pool
capacity is the static total over the pool's registered hosts;
- for `member_quota`, `projm:<P>:<r>` (same consumer as the share limit
plus users `all` per user, vres key `limit=projm:...:<r>/u=<user>`,
`lib/xcelimit.c:402`).

61
.kiro/specs/project-config/requirements.md
View file

@ -43,6 +43,13 @@ Discovery brief: `.kiro/specs/project-config/brief.md`. Governing rules: `.kiro/
- A project may declare a share for a resource only if its parent has one for that resource, or it is top-level.
- A resource no project declares a share for is not controlled (today's behaviour).
**Decisions (2026-10-13, locations and visibility):**
- A share may be owned at a location: `share: { at: { <host group>: { <resource>: amount } } }`, alongside amounts anywhere. Per resource, every host group named as a location forms its own pool, and all other hosts form the rest-of-cluster pool; pools are disjoint, a job's usage of a resource on a host is charged to that host's pool, and every pool follows the share rules on its own. An unlocated share therefore means "in the rest of the cluster".
- A pool's capacity is the static total of the resource over its registered hosts, including hosts that are down or closed and excluding decommissioned ones; elastic hosts count once they have joined.
- A submit with no project and no applicable default is rejected (as today) with an error listing the user's projects.
- A per-user default project is a client-side setting (`xcecli` config or a preset, `xcecli-presets`), not server state.
- Members of a project see the shares and usage of the project and all its ancestors, but not the members or per-user usage of sibling projects.
**Decisions (2026-10-13, open items):**
- `jobs` is a built-in resource for `share` and `member_quota`: each running job (each array element) uses 1, whatever its task count; absolute amounts only. Caps on pending jobs are a separate submit-time feature (`pending-job-limits`).
- Jobs may be charged to inner projects. Their own jobs draw from the project's unassigned part; the configuration tool warns when that part is empty for a project with members, and such jobs pend with a reason saying so. A leaf-only option (`accept_jobs: false`) may come later.
@ -72,6 +79,7 @@ This feature makes projects first-class cluster configuration entities that cont
- pending-job behaviour when a project or membership changes;
- owned shares (absolute or percentage), the unassigned part, and `member_quota`, compiled into the scheduler's limits, applying end to end, with project-naming pend reasons;
- a built-in `jobs` count for shares and member quotas;
- shares owned at a location (a host group), with disjoint per-resource pools;
- project aliases (renames that keep jobs);
- rejecting project-scoped entries in `limits.yaml`;
- overlapping membership (a user in several projects);
@ -87,7 +95,7 @@ This feature makes projects first-class cluster configuration entities that cont
- changing a job's project after submit (`job-modify`);
- charging jobs to leaf projects only (`accept_jobs: false`, later);
- runtime edits of a project's members, admins or shares by project admins (`project-delegated-membership`; in this stage only the cluster administrator changes them, through cluster config);
- per-user default projects in `users.yaml`;
- per-user default projects in server configuration (a client-side `xcecli` default or preset instead, `xcecli-presets`);
- membership inherited between parent and child projects;
- wildcard or pattern matching of member names;
- retiring the rest of `limits.yaml` (user, queue and host scopes) and moving per-host slots to host config (`limits-to-owners`);
@ -99,7 +107,7 @@ This feature makes projects first-class cluster configuration entities that cont
- User groups come from the existing user-group configuration.
- Grants made through the role API are subject to the existing `rbac-persistence` gap (not durable across apiserver restart) until that spec lands.
- `job-config-merge` treats project shares as ceilings: they hold or admit a job but never change its values.
- The defect that resolved limit host and project scopes against each other's group tables (it also broke host-group limits) was fixed separately as bug-1933 (`6ecd418d`); Requirement 6.8 keeps it as a requirement.
- The defect that resolved limit host and project scopes against each other's group tables (it also broke host-group limits) was fixed separately as bug-1933 (`6ecd418d`); Requirement 6.17 keeps it as a requirement.
## Requirements
@ -107,10 +115,10 @@ This feature makes projects first-class cluster configuration entities that cont
**Objective:** As a cluster admin, I want to define projects and their members in the cluster configuration, so that project ownership is explicit and versioned with the rest of the cluster config.
#### Acceptance Criteria
1. The configuration tool shall accept project definitions consisting of a path name, an optional description, members (user names, user-group names, or `all`), optional excluded members, optional admins (user names or user-group names), an optional list of user groups for which the project is the default, an optional `share` (a map from resource name to an absolute amount or a percentage of the parent's share), an optional `member_quota` (a map from resource name to an absolute amount), and optional `aliases` (earlier path names of the project). The resource names include the built-in `jobs`.
1. The configuration tool shall accept project definitions consisting of a path name, an optional description, members (user names, user-group names, or `all`), optional excluded members, optional admins (user names or user-group names), an optional list of user groups for which the project is the default, an optional `share` (a map from resource name to an absolute amount or a percentage of the parent's share, plus an optional `at` map from host-group name to such a resource map for shares owned at that location), an optional `member_quota` (a map from resource name to an absolute amount), and optional `aliases` (earlier path names of the project). The resource names include the built-in `jobs`.
2. The configuration tool shall treat a path name of one segment (e.g. `eng`) as a top-level project and a path of several `/`-separated segments (e.g. `eng/hpc/cfd`) as a child of the project named by the path without its last segment.
3. The configuration tool shall accept an optional cluster default project naming one defined project.
4. If a project definition repeats a path name, uses a reserved name, has an empty or malformed path segment, names a parent project that is not defined, names an undefined user group (in members, excluded members or admins), lists a default-for user group that is not covered by the project's members, has a `share` or `member_quota` entry naming an undeclared resource or giving a negative or non-numeric amount, has a percentage in `member_quota`, a percentage above 100% in `share`, or a percentage for `jobs`, declares a `share` for a resource its parent (when it has one) declares no share for, gives siblings shares of one resource in different forms (absolute and percentage), or gives siblings shares that add up to more than 100% or, when all are absolute and the parent's is absolute, to more than the parent's share, then the configuration tool shall reject the configuration with an error identifying the project and the offending field.
4. If a project definition repeats a path name, uses a reserved name, has an empty or malformed path segment, names a parent project that is not defined, names an undefined user group (in members, excluded members or admins), lists a default-for user group that is not covered by the project's members, has a `share` or `member_quota` entry naming an undeclared resource or giving a negative or non-numeric amount, has a percentage in `member_quota`, a percentage above 100% in `share`, or a percentage for `jobs`, names `jobs` under `at` or an undefined host group as a location, names for one resource two locations whose host groups share a host, declares a `share` for a resource in a pool (a location, or the rest of the cluster) in which its parent (when it has one) declares no share of that resource, gives siblings shares of one resource in one pool in different forms (absolute and percentage), or gives siblings shares in one pool that add up to more than 100% or, when all are absolute and the parent's is absolute, to more than the parent's share, then the configuration tool shall reject the configuration with an error identifying the project and the offending field.
5. If the cluster default project names an undefined project, or a project whose members do not include every user, then the configuration tool shall reject the configuration with an error naming the cluster default project.
6. When a configuration with changed project definitions is applied by live configuration upgrade, the Cluster shall apply the new definitions to all subsequent submissions and scheduling decisions without a restart.
7. The Cluster shall keep project definitions, including their shares, across controller failover and restart identically to other cluster configuration.
@ -124,7 +132,7 @@ This feature makes projects first-class cluster configuration entities that cont
#### Acceptance Criteria
1. When a job is submitted with a project path name, the Cluster shall use that project for the job, whether it is a top-level, inner or leaf project.
2. When a job is submitted without a project name and at least one defined project lists a user group the submitter belongs to as default-for, the Cluster shall use the first such project in configuration order.
3. When a job is submitted without a project name and no default-for rule matches the submitter, the Cluster shall use the cluster default project if one is configured, and otherwise the project `default`.
3. When a job is submitted without a project name and no default-for rule matches the submitter, the Cluster shall use the cluster default project if one is configured, and otherwise the project `default`; if, while projects are defined, that project is not defined, the Cluster shall reject the submission with an error stating that no project was given and no default applies, and listing the projects the user may submit under.
4. The Cluster shall record the resolved project name on the job at submit, show it in the job's details, and keep it unchanged across later configuration changes, failover and restart.
5. The Cluster shall apply the same project resolution to every submission through the REST API (`xcecli` and any other client of the API) and to every element of a job array.
6. The Cluster shall identify a project at submit only by its full path name; a name that is only the last segment of a defined project's path (e.g. `cfd` for `eng/hpc/cfd`) is treated as the top-level project of that name.
@ -174,24 +182,28 @@ This feature makes projects first-class cluster configuration entities that cont
**Objective:** As a cluster admin, I want each project to own a share of its parent's resources and hold no more than it, so that a project can always get what it owns and no project, or member of one, takes more than its part.
#### Acceptance Criteria
1. The Scheduler shall compute each project's effective share of a resource as its absolute amount, or as its percentage of its parent's effective share; a top-level project's percentage is of the cluster's capacity of that resource (the static total over the cluster's hosts), so percentage shares change when hosts are added or removed.
2. The Scheduler shall count against a project's share of a resource the amount of that resource allocated to every running job whose recorded project is that project or one of its descendants.
3. The Scheduler shall treat a project's unassigned part of a resource as its effective share minus the effective shares of its children that declare a share of that resource (the cluster's capacity minus the top-level shares at the root), and shall count against it the jobs of the project itself and of its descendants that are not under a child with a share of that resource.
4. The Scheduler shall count against a project's `member_quota` for a resource, separately for each user, the amount of that resource allocated to that user's running jobs whose recorded project is that project or one of its descendants, whether or not the user is a direct member of that project.
5. The Scheduler shall dispatch a job only if, for every resource it is allocated, the job fits the share of every project on its path that declares one, the unassigned part it draws from, and the `member_quota` of its recorded project and of every ancestor.
6. The Scheduler shall not let a project hold more than its effective share by using capacity owned by another project (no borrowing in this feature).
7. While no project declares a share of a resource, the Scheduler shall not restrict that resource by project.
8. The Scheduler shall count a job only against its recorded project and that project's ancestors, never against other projects its owner belongs to.
9. If dispatching a job would exceed a share, an unassigned part or a member quota, then the Scheduler shall keep the job pending with a reason naming the project, the resource, which of the three is full, and its current usage and amount; the reason shall not name an internal limit.
10. When usage falls below a share, unassigned part or member quota, the Scheduler shall consider jobs held by it for scheduling again.
11. When a configuration change or a change in the cluster's hosts changes an effective share, the Scheduler shall apply the new amounts to all subsequent scheduling decisions without a restart, keep counting running jobs against their recorded project and its ancestors, and never stop or alter a running job because a share is now below its usage.
12. Where absolute shares of top-level projects add up to more than the cluster's current capacity of a resource, the Cluster shall report it in project diagnostics and in a configuration-tool warning; the Scheduler shall still apply each share as written.
13. The Scheduler shall resolve each generated limit's project and user scopes against project and user names only, so that a project name never matches a host group or user group (the scope separation fixed by bug-1933).
14. The live Scheduler and the scheduler simulator shall make identical share decisions for the same input.
15. The Scheduler shall show a project path containing `/` unchanged in pend reasons, decision traces and diagnostics.
16. The Scheduler shall count, for the built-in resource `jobs`, one unit per running job (one per running array element), regardless of its task count, against the same shares, unassigned parts and member quotas as any other resource.
17. When a job's recorded project resolves through an alias, the Scheduler shall count the job, running or pending, against the target project and its ancestors; when an alias is added for running jobs, their usage moves to the target's path on the next scheduling round.
18. If a job charged directly to a project cannot get a resource because that project's unassigned part of it is empty, then the Scheduler shall keep the job pending with a reason stating that the project's children own its whole share of that resource.
1. The Scheduler shall divide the hosts, separately for each resource, into disjoint pools: one per host group named as a location in any share of that resource, and the rest of the cluster (all other hosts); all of Requirement 6 applies to each pool of each resource on its own, an unlocated share being a share in the rest-of-cluster pool.
2. The Scheduler shall take a pool's capacity of a resource as the static total of that resource over the pool's registered hosts, including hosts that are down or closed and excluding decommissioned ones, so capacity changes only when hosts join, are added to or removed from a location's host group, or are decommissioned.
3. The Scheduler shall compute each project's effective share of a resource in a pool as its absolute amount, or as its percentage of its parent's effective share in that pool; a top-level project's percentage is of the pool's capacity.
4. The Scheduler shall charge a job's allocation of a resource on a host to the pool that host belongs to for that resource.
5. If a host joining the cluster belongs to two location host groups named for the same resource, then the Scheduler shall place it in the first such location in configuration order and log a warning naming the host, the resource and both groups.
6. The Scheduler shall count against a project's share of a resource the amount of that resource allocated to every running job whose recorded project is that project or one of its descendants.
7. The Scheduler shall treat a project's unassigned part of a resource as its effective share minus the effective shares of its children that declare a share of that resource (the pool's capacity minus the top-level shares at the root), and shall count against it the jobs of the project itself and of its descendants that are not under a child with a share of that resource.
8. The Scheduler shall count against a project's `member_quota` for a resource, separately for each user, the amount of that resource allocated to that user's running jobs whose recorded project is that project or one of its descendants, whether or not the user is a direct member of that project.
9. The Scheduler shall dispatch a job only if, for every resource it is allocated, the job fits the share of every project on its path that declares one, the unassigned part it draws from, and the `member_quota` of its recorded project and of every ancestor.
10. The Scheduler shall not let a project hold more than its effective share by using capacity owned by another project (no borrowing in this feature).
11. While no project declares a share of a resource, the Scheduler shall not restrict that resource by project.
12. The Scheduler shall count a job only against its recorded project and that project's ancestors, never against other projects its owner belongs to.
13. If dispatching a job would exceed a share, an unassigned part or a member quota, then the Scheduler shall keep the job pending with a reason naming the project, the resource, the location (for a located pool), which of the three is full, and its current usage and amount (e.g. "project chem/md share of gpu at hg-a100 full (12/12)", or for a project owning nothing there, "no unowned gpu at hg-a100"); the reason shall not name an internal limit.
14. When usage falls below a share, unassigned part or member quota, the Scheduler shall consider jobs held by it for scheduling again.
15. When a configuration change or a change in the cluster's hosts changes an effective share, the Scheduler shall apply the new amounts to all subsequent scheduling decisions without a restart, keep counting running jobs against their recorded project and its ancestors, and never stop or alter a running job because a share is now below its usage.
16. Where absolute shares of top-level projects in a pool add up to more than that pool's current capacity of a resource (including when a new location takes hosts out of the rest-of-cluster pool), the Cluster shall report it in project diagnostics and in a configuration-tool warning; the Scheduler shall still apply each share as written.
17. The Scheduler shall resolve each generated limit's project and user scopes against project and user names only, so that a project name never matches a host group or user group (the scope separation fixed by bug-1933).
18. The live Scheduler and the scheduler simulator shall make identical share decisions for the same input.
19. The Scheduler shall show a project path containing `/` unchanged in pend reasons, decision traces and diagnostics.
20. The Scheduler shall count, for the built-in resource `jobs`, one unit per running job (one per running array element), regardless of its task count, against the same shares, unassigned parts and member quotas as any other resource.
21. When a job's recorded project resolves through an alias, the Scheduler shall count the job, running or pending, against the target project and its ancestors; when an alias is added for running jobs, their usage moves to the target's path on the next scheduling round.
22. If a job charged directly to a project cannot get a resource because that project's unassigned part of it is empty, then the Scheduler shall keep the job pending with a reason stating that the project's children own its whole share of that resource.
### Requirement 7: Project visibility
**Objective:** As a user or admin, I want to see projects, their members and their usage against shares, so that I know which projects I can use and why a job is held.
@ -199,12 +211,13 @@ This feature makes projects first-class cluster configuration entities that cont
#### Acceptance Criteria
1. When a user lists projects through the REST API or `xcecli`, the Cluster shall return every project the user may submit under (as a member or through a role grant) or administers, with its full path name and description, shown in tree order, and mark the project that would be the user's default, so that a submit client can offer them for choosing the project to charge.
2. Where the requester has the cluster administrator role, or a `project:view` permission covering a project, the Cluster shall include that project in the list with its members, excluded members and default-for user groups.
3. When a user requests a project's details, the Cluster shall show, for the project and for each of its ancestors, the effective share (and the form it was written in), the unassigned part, and every `member_quota` entry, each with the project it is set on, the resource, the amount and the current usage counted against it (including descendants' jobs; for `member_quota`, the requesting user's own usage, and every user's usage to an admin or a holder of `project:view` covering the project).
3. When a user requests a project's details, the Cluster shall show, for the project and for each of its ancestors, per pool (the rest of the cluster and each location), the effective share (and the form it was written in), the unassigned part, and every `member_quota` entry, each with the project it is set on, the resource, the amount and the current usage counted against it (including descendants' jobs; for `member_quota`, the requesting user's own usage, and every user's usage to an admin or a holder of `project:view` covering the project).
4. If a user requests the details of a project they are neither a member nor an admin of, and they have neither the cluster administrator role nor a `project:view` permission covering the project, then the Cluster shall reject the request without revealing the project's members or usage.
5. When a user lists jobs filtered by project through the REST API or `xcecli`, the Cluster shall return only jobs with that recorded project, or, when the user asks to include sub-projects, with that project or any descendant, subject to the user's existing job-visibility permissions.
6. While no project is defined, the Cluster shall report that projects are not configured when projects are listed.
7. The scheduler's diagnostic interface shall show, for a named project, the shares, unassigned parts and member quotas that apply to it (its own and its ancestors'), their effective amounts and their current usage.
8. Where a job's recorded project resolves through an alias, the Cluster shall show both in the job's details (e.g. `research/hpc (recorded as eng/hpc)`), list the alias in the project's details, and include such jobs when jobs are filtered by the target project.
9. A member or admin of a project shall see the shares and usage of that project and of all its ancestors, but neither the members nor the per-user usage of projects that are not its own or its ancestors, unless they hold `project:view` covering them or the cluster administrator role.
### Requirement 8: Project-scoped access control
**Objective:** As a cluster admin, I want project admins and roles to hold rights over the jobs and details of a project and its sub-projects, so that project leads can manage their project's work without being cluster administrators.

42
.kiro/steering/job-config-layering.md
View file

@ -195,8 +195,8 @@ them:
```
- **A child owns part of its parent's share.** Per resource, a share is an
absolute amount or a percentage of the parent's effective share; the
root's is the cluster's capacity (the static total of the resource over
the cluster's hosts), so percentage shares follow the cluster's size.
root's is the pool's capacity (below), so percentage shares follow the
cluster's size.
Siblings' shares never add up to more than the parent's: the config tool
rejects it (siblings use one form per resource; percentages sum to at
most 100%).
@ -212,6 +212,31 @@ them:
part. A project may declare a share for a resource only if its parent
has one for that resource (or it is top-level), so a share is always
carved from something defined.
- **Shares at a location** (user decision 2026-10-13). A project can own an
amount anywhere, an amount at a location (a host group), or both:
```yaml
- path: chem
share:
slots: 200 # in the rest of the cluster
at:
hg-a100: { gpu: 16, slots: 50% }
- path: chem/md
share: { at: { hg-a100: { gpu: 75% } } } # 12 of chem's a100 GPUs
```
**Locations are disjoint pools:** per resource, each host group named as
a location is its own pool and all other hosts form the rest-of-cluster
pool; a job's usage on a host is charged to that host's pool, and every
rule in this section applies to each pool on its own. So an unlocated
share means "in the rest of the cluster", and another project's jobs on
`hg-a100` draw from the root's unassigned part there (16 − 16 = 0):
located ownership is real. A host may be in only one location per
resource (config check; a joining host matching two goes to the first in
config order, with a warning). `jobs` has no location. Owning whole hosts
is 100% of every resource at a location.
- **Pool capacity** is the static total of the resource over the pool's
registered hosts, including down and closed hosts, excluding
decommissioned ones; elastic hosts count once joined. A heartbeat loss
never moves shares.
- **No shares, no change.** A resource for which no project declares a
share is unassigned at the root: no cap, today's behaviour.
- **Shares are per resource** (slots, mem, gpu, licences, ...): a project
@ -240,9 +265,16 @@ them:
changes project.
- Changing a share never stops running jobs; a project above its new share
cannot start more until it is below it.
- Pend reasons name the project, resource and usage ("project eng/ai share
of gpu full (16/16)", "unassigned part of eng full for slots"), never an
internal limit name.
- Pend reasons name the project, resource, location and usage ("project
eng/ai share of gpu full (16/16)", "project chem/md share of gpu at
hg-a100 full (12/12)", "no unowned gpu at hg-a100"), never an internal
limit name.
- **Visibility:** members see their project's and all ancestors' shares and
usage (what their pend reasons refer to), not siblings' members or
per-user usage.
- **Defaults:** no project given and no default applies → rejected with
the user's projects listed. A per-user default project is a client-side
`xcecli` setting or preset, not server state.
## Convenience defaults are presets, not a level
Apart from default reservations (the profile's `rusage` entries), tuning

4
.kiro/steering/roadmap.md
View file

@ -73,7 +73,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
- [ ] controller-job-cleanup — NEW 2026-09-29 (`/kiro-discovery`, Path C). Primary-only periodic sweep (configurable `job_cleanup_period`, default 300 s) removes DONE/EXIT jobs finished longer than configurable `job_retention` (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever `jobmgr` delete; refcount-safe `xkvmgr` delete), propagated to scheduler and apiserver (`OnJobDeleted`). Retires the node-local snapshot-only `job_should_retain` finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: `.kiro/specs/controller-job-cleanup/brief.md`. Dependencies: none (extends ha-entity-snapshots). Once project-accounting exists, a job is removed only after its accounting record is durable (2026-10-13).
- [ ] job-status-detail-display — NEW 2026-09-28 (`/kiro-discovery`, Path E). End users (REST/`xcecli`) see NO pend reason for ANY job today: the apiserver cgo conversion (`backend/convert.go`) never maps `xce_job_t.pend_reasons`, so `pending_reason` is always empty. Maps it to a structured JSON (including `alternatives` from day one) using libxce's code→text mapping, and renders it in `xcecli` job detail. Open for requirements: also map `run_usage` → `JobRusage` (same gap; reserved vs. used for RUN jobs). Brief: `.kiro/specs/job-status-detail-display/brief.md`. Dependencies: none for flat reasons; scheduler-alter-pend-reasons for per-alternative rendering.
- [ ] job-config-merge — NEW 2026-10-09 (`/kiro-discovery`, Path E "configuration layering"; ref LSF 10.1 Admin Guide pp. 314-316, 435-437, 561-563, 643, 660). One merge function in `src/lib` turning the job's own values + application profile + queue policy + project into effective values (resreq incl. `span`, task range, limits, pre/post-exec, env). One owner per attribute (user decision; `.kiro/steering/job-config-layering.md`): the job may add to a profile's values but never change or remove them (additive rule, 2026-10-12, replacing *default*/*fixed* markers); queue ceilings and project quotas only check (reject if never satisfiable, pend if full now). **Live** merge (user decision): WAL keeps inputs + names; controller/apiserver validate at submit; scheduler computes effective values and re-evaluates pending jobs on config change. Queue is policy only: queue `res_req` and `pre_exec`/`post_exec` deprecated (pre/post-exec moves to the profile), dropped queue limits (cpulimit/memlimit/runlimit/task limit) wired. Brief: `.kiro/specs/job-config-merge/brief.md`. Dependencies: none.
- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (path identity, members, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config, owned shares written on the project (`share` of the parent's resources, absolute or percentage, strict in v1; children without a share draw from the parent's unassigned part; `member_quota` over the subtree; user decisions 2026-10-12, replacing `PROJECTS`/`PER_PROJECT` limits in `limits.yaml`) compiled into the limit plugin, a built-in `jobs` count (job limits per project and user), project aliases for renames, charging inner projects, per-project usage display; `project:submit` RBAC implied by membership, project admins, authorization in the apiserver only. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none to ship; production rejection of direct submits comes from peer-authentication-hardening.
- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (path identity, members, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config, owned shares written on the project (`share` of the parent's resources, absolute or percentage, strict in v1; children without a share draw from the parent's unassigned part; `member_quota` over the subtree; user decisions 2026-10-12, replacing `PROJECTS`/`PER_PROJECT` limits in `limits.yaml`) compiled into the limit plugin, a built-in `jobs` count (job limits per project and user), project aliases for renames, charging inner projects, shares owned at a location (host group; disjoint per-resource pools), per-project usage display; `project:submit` RBAC implied by membership, project admins, authorization in the apiserver only. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none to ship; production rejection of direct submits comes from peer-authentication-hardening.
- [ ] host-failure-job-recovery — NEW 2026-10-10 (`/kiro-discovery`, "Long-running services" phase 1; also a batch gap: RUN jobs on a dead host stay RUN forever, `controller/sched.c:631` FIXME). After a grace period, jobs on a lost host go to an unknown state, then are requeued (rerunnable, job option or profile default) or end EXIT "host lost"; multi-host jobs resolved as a whole; attempt numbers fence a stale copy when the host returns; manual requeue (REST/xcecli). Brief: `.kiro/specs/host-failure-job-recovery/brief.md`. Dependencies: none.
- [ ] service-supervision — NEW 2026-10-10 (Long-running services phase 1). Application profile kind `service`: restart policy (always/on-failure/never, backoff, give-up), liveness and readiness probes (command/TCP/HTTP) run by each replica's own worker (no special replica), in-place restart, change-only health reports, display. Brief: `.kiro/specs/service-supervision/brief.md`. Dependencies: application-profile, host-failure-job-recovery.
@ -132,7 +132,7 @@ Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generi
- [ ] job-labels — NEW 2026-10-12 (from application-profile discussion; backlog, no brief yet). General Kubernetes-style job labels: `--label key=value` (repeatable, limited charset and size), an additive submit-data field through WAL and snapshot (field completeness), server-side filtering (`jobs list -l k=v`) and summaries grouped by label (runtime, peak memory, efficiency); metadata only, never used for placement. Reuses or replaces the REST `tags` field that is silently dropped today (bug-1970). Not needed for presets (those use `exdata.meta`); candidates: team-wide comparison, services labelling replicas. Dependencies: none.
- [ ] app-submit-hook — NEW 2026-10-12 (from application-profile discussion, user decision: backlog only). A profile-named program that inspects or adjusts each submission (LSF `esub` style). Cannot run in the controller (no-fork, non-blocking rules); in the apiserver it adds latency and a failure mode to every submit and bypasses the merge rules, so decide client (xcecli) vs apiserver first. Application-profile `params`, annotations, the job starter and pre-exec cover most needs meanwhile. Dependencies: application-profile.
- [ ] job-requeue — NEW 2026-10-12 (from application-profile discussion, user decision; backlog, no brief yet). Automatic requeue on given exit values (LSF `REQUEUE_EXIT_VALUES`) and success exit values (`SUCCESS_EXIT_VALUES`), with a requeue loop limit; wires the commented-out `requeue` job action (`apiserver/controllers/jobs.go`) and the REST `requeue_error_values` / `rerunable` options that are silently dropped today (bug-1970). Adds the matching application-profile attributes (a profile attribute ships with its job feature). Later, same principle: checkpoint settings and warning-before-kill each get their own spec. Dependencies: application-profile.
- [ ] xcecli-presets — NEW 2026-10-12 (from application-profile discussion, user decisions). Client-side presets replace convenience defaults in profiles: a preset is a set of job options (`--app`, `--queue`, `-R`, `-n`, env, limits) expanded by the client, a flag replacing the preset's value for that option; one file per preset (`~/.config/xce/presets/<ns>/<name>.yaml`, same layout in the apiserver's presets dir), env capture by name (`pass:`; `XCE_*` and secret-looking names excluded); the apiserver only shares admin-deployed presets read-only (`GET /xce/v1/presets[/{ns}/{name}]`, files in `<TopDir>/config/presets/` or `presets.dir`, re-read on change; not in ClusterConfig), and `preset pull` copies one locally so server-side removal never affects users; on a name collision pull refuses unless it is an update of the same server preset (`--as` / `--force` with `.bak`); submit uses local presets only (`preset diff`, `pull --force` user-initiated); user-to-user sharing by `preset export`/`import`; presets carry no privilege and are not a configuration level; no preset field on the server: the client records preset name + hash in the job's `exdata.meta` (passed through uninterpreted; worker skips `meta`), and `preset report` filters jobs on it client-side; the portal does the same expansion. Brief: `.kiro/specs/xcecli-presets/brief.md`. Dependencies: application-profile.
- [ ] xcecli-presets — NEW 2026-10-12 (from application-profile discussion, user decisions). Client-side presets replace convenience defaults in profiles: a preset is a set of job options (`--app`, `--queue`, `-R`, `-n`, env, limits) expanded by the client, a flag replacing the preset's value for that option; one file per preset (`~/.config/xce/presets/<ns>/<name>.yaml`, same layout in the apiserver's presets dir), env capture by name (`pass:`; `XCE_*` and secret-looking names excluded); the apiserver only shares admin-deployed presets read-only (`GET /xce/v1/presets[/{ns}/{name}]`, files in `<TopDir>/config/presets/` or `presets.dir`, re-read on change; not in ClusterConfig), and `preset pull` copies one locally so server-side removal never affects users; on a name collision pull refuses unless it is an update of the same server preset (`--as` / `--force` with `.bak`); submit uses local presets only (`preset diff`, `pull --force` user-initiated); user-to-user sharing by `preset export`/`import`; presets carry no privilege and are not a configuration level; no preset field on the server: the client records preset name + hash in the job's `exdata.meta` (passed through uninterpreted; worker skips `meta`), and `preset report` filters jobs on it client-side; the portal does the same expansion. Brief: `.kiro/specs/xcecli-presets/brief.md`. Dependencies: application-profile. Also a per-user default project (user decision 2026-10-13: client-side `default_project` in the `xcecli` config or a preset setting `-P`; no server state).
- [ ] profile-usage-insight — NEW 2026-10-12 (from application-profile discussion, backlog; no brief yet). Per-profile report over recent jobs comparing the profile's defaults with actual use (requested vs. peak memory, CPU efficiency, OOM kills, pend time, runtime by variant) and suggesting better values for site presets (or a profile essential), e.g. "preset openfoam rusage mem 4000, p95 peak 1600 → suggest 2000" (Kubernetes VPA recommender precedent). **Advisory only** (user decision): the admin applies changes to site presets or `applications.yaml`; nothing tunes itself. Optional user hint ("used 1.4G of 4G requested"). Dependencies: application-profile, run usage in job records.
- [ ] lsf-config-migration — NEW 2026-10-09 (user idea, backlog; no brief yet). A tool (likely an `xceadmin` subcommand) that converts LSF configuration (`lsb.queues`, `lsb.applications`, `lsb.resources` Limit sections (split by consumer into project quotas, member quotas and host supply; queue and mixed entries reported), `lsb.users`/`lsb.hosts` groups, `lsb.params` defaults, project settings) into XCE `configs/*.yaml`, with a report of what does not map: queue `RES_REQ` (XCE queues are policy only; move into a profile or drop), LSF's full per-section merge rules vs XCE's simplified ones, compound/alternative resreqs, unsupported parameters. Dependencies: job-config-merge, application-profile, project-config, parjob-placement (span), admin-config-tool. Promote via `/kiro-discovery` when those land.