From e02310c02db42cee1fdc92704e711e9c14953603 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 13 Oct 2026 07:03:20 +0000 Subject: [PATCH] docs(spec): project open items: jobs count, inner projects, aliases, accounting - `jobs` is a built-in resource for shares and member quotas: the job limits per project and user; pending caps go to pending-job-limits. - Jobs may be charged to inner projects, with a warning and pend reason when their unassigned part is empty. - Project aliases keep jobs through renames; running jobs follow them. - Changing a pending job's project is part of job-modify. - New project-accounting brief: durable finished-job records, outside the WAL, before controller-job-cleanup drops a job. Co-Authored-By: Claude Opus 5.5 (1M context) --- .kiro/specs/project-accounting/brief.md | 53 ++++++++++++++++++++++ .kiro/specs/project-config/brief.md | 10 ++-- .kiro/specs/project-config/requirements.md | 29 ++++++++++-- .kiro/specs/project-fairshare/brief.md | 5 ++ .kiro/steering/job-config-layering.md | 21 ++++++++- .kiro/steering/roadmap.md | 8 ++-- 6 files changed, 115 insertions(+), 11 deletions(-) create mode 100644 .kiro/specs/project-accounting/brief.md diff --git a/.kiro/specs/project-accounting/brief.md b/.kiro/specs/project-accounting/brief.md new file mode 100644 index 00000000..cf41b1cd --- /dev/null +++ b/.kiro/specs/project-accounting/brief.md @@ -0,0 +1,53 @@ +# Brief: project-accounting + +(2026-10-13, user decision during the project-config discussion: the next +project feature after `project-config`. Backlog.) + +## Problem +Project owners and finance ask first "how much did `eng/ai` use last month, +and by whom?". XCE cannot answer: `project-config` shows only current usage +against shares, and finished jobs are not kept. + +## Current State +- Finished jobs live in the controller only until `controller-job-cleanup` + (backlog) removes them after `job_retention` (default 1 h); the WAL and + snapshots hold live state, not history. +- Run usage is reported per job (`runUsage`, bug-1736), shown in + `xcecli jobs list -l` while the job is kept. +- `project-fairshare` needs decayed usage per project and user; it has no + source to rebuild that history from after a lost snapshot. + +## Desired Outcome +- **One durable record per finished job**, per requeue and per array + element: job id, owner, project path as recorded and as resolved + (aliases), queue, application profile, submit/start/end times, exit + status, allocated resources per host, measured usage. +- Written **before** the controller may remove the job; + `controller-job-cleanup` waits for it. +- **A history store outside the replicated WAL**, sized for months or years. +- **Reports**: usage by project, subtree, user, period and resource + (REST, `xcecli projects usage eng --since 2026-10-01 --by user`), shown + under the project's current path with old (alias) paths noted. +- **One source for two consumers**: fairshare usage history can be rebuilt + from these records plus the running jobs. +- Access: the same rules as project details (members see their projects, + project admins their subtree, `project:view`, cluster admin). + +## Design questions +1. **Store**: an append-only log written by the primary controller + (rotated files, like Slurm without slurmdbd), or an accounting service + or database fed by the controller (like `slurmdbd`)? +2. **HA**: who writes after a failover, and how a record is never lost + (job removed first) nor written twice (exactly-once by job id + attempt). +3. **Async I/O**: record writes must not block the controller's I/O or + delaymgr paths (slow lane or a separate process). +4. **Retention and export** (CSV/JSON) for billing systems. + +## Upstream / Downstream +- **Upstream**: `project-config`, `controller-job-cleanup`, job run usage. +- **Downstream**: `project-fairshare` (rebuild source), + `profile-usage-insight`, billing exports. + +## Constraints +- Persistence and HA: full 3-phase workflow; field completeness of the + record (`persistence.md`). diff --git a/.kiro/specs/project-config/brief.md b/.kiro/specs/project-config/brief.md index 8948b4b1..11312b5f 100644 --- a/.kiro/specs/project-config/brief.md +++ b/.kiro/specs/project-config/brief.md @@ -68,8 +68,8 @@ mechanism and admins no longer write limits. Semantics: projects: - path: eng members: [eng-ug] - share: { slots: 400, gpu: 16, mem: 2T } # carved from the cluster - member_quota: { slots: 64 } # per user, over eng's whole subtree + share: { slots: 400, gpu: 16, mem: 2T, jobs: 200 } # carved from the cluster + member_quota: { slots: 64, jobs: 20 } # per user, over eng's whole subtree admins: [eng-leads] # job control over eng's subtree - path: eng/hpc members: [alice, bob] @@ -77,6 +77,7 @@ projects: - path: eng/ai members: [carol] share: { slots: 40%, gpu: 16 } + aliases: [eng/ml] # renamed from eng/ml; its jobs follow - path: eng/tools # no share: draws from eng's unassigned part members: [eng-ug] ``` @@ -97,7 +98,10 @@ projects: - for `member_quota`, `projm:

:` (same consumer as the share limit plus users `all` per user, vres key `limit=projm:...:/u=`, `lib/xcelimit.c:402`). - New in the plugin: descendant matching on the project consumer. + New in the plugin: descendant matching on the project consumer, and a + built-in `jobs` resource (one per running job or array element, not per + task) the limits can count; aliases are resolved before matching, so a + job recorded under an alias matches its target's limits. - The path contains `/`, which is also the vres key's field separator; the trace splitter (`sched_trace_capture.c:40`, `sti_copy_scope`) would render `p=eng/hpc` as `p=eng,hpc`. Escape the project field in the key or use diff --git a/.kiro/specs/project-config/requirements.md b/.kiro/specs/project-config/requirements.md index 6b25b83e..c50e70c6 100644 --- a/.kiro/specs/project-config/requirements.md +++ b/.kiro/specs/project-config/requirements.md @@ -43,6 +43,12 @@ Discovery brief: `.kiro/specs/project-config/brief.md`. Governing rules: `.kiro/ - A project may declare a share for a resource only if its parent has one for that resource, or it is top-level. - A resource no project declares a share for is not controlled (today's behaviour). +**Decisions (2026-10-13, open items):** +- `jobs` is a built-in resource for `share` and `member_quota`: each running job (each array element) uses 1, whatever its task count; absolute amounts only. Caps on pending jobs are a separate submit-time feature (`pending-job-limits`). +- Jobs may be charged to inner projects. Their own jobs draw from the project's unassigned part; the configuration tool warns when that part is empty for a project with members, and such jobs pend with a reason saying so. A leaf-only option (`accept_jobs: false`) may come later. +- Projects may list `aliases` (old paths), as application profiles do: a rename never strands jobs. Pending and running jobs recorded under an alias belong to the target; their usage follows it. +- Changing a pending job's project belongs to `job-modify` (pending jobs only, checked against the owner's `project:submit`). Accounting history of finished jobs is `project-accounting`. + **Decisions (2026-10-12, project RBAC):** - Submitting under a project is an RBAC permission, `project:submit` (scope type `project`), checked like `queue:submit`. - Membership implies the grant: being a member of P gives an implicit `project:submit` on P only (membership is not inherited). Roles may also grant `project:submit` explicitly (e.g. to service accounts); a `project` scope covers the named projects and their subtrees, as for every `project` scope. @@ -65,6 +71,8 @@ This feature makes projects first-class cluster configuration entities that cont - default-project resolution, the `project:submit` check at submit, and an "open" mode when no projects are defined; - pending-job behaviour when a project or membership changes; - owned shares (absolute or percentage), the unassigned part, and `member_quota`, compiled into the scheduler's limits, applying end to end, with project-naming pend reasons; + - a built-in `jobs` count for shares and member quotas; + - project aliases (renames that keep jobs); - rejecting project-scoped entries in `limits.yaml`; - overlapping membership (a user in several projects); - project listing (as a tree) and detail with usage against shares; @@ -74,11 +82,13 @@ This feature makes projects first-class cluster configuration entities that cont - borrowing idle shares above a project's own (`max`), protecting owners from borrowers, and reclaim by preemption (`project-borrowing`); - fairshare (`project-fairshare`: weights and usage history on the project tree, decided 2026-10-12); - any project influence on job ordering or priority in this feature (fairshare ordering from the project tree comes with `project-fairshare`); - - accounting history and billing of finished jobs; + - accounting history and billing of finished jobs (`project-accounting`); + - caps on pending jobs (`pending-job-limits`); + - changing a job's project after submit (`job-modify`); + - charging jobs to leaf projects only (`accept_jobs: false`, later); - runtime edits of a project's members, admins or shares by project admins (`project-delegated-membership`; in this stage only the cluster administrator changes them, through cluster config); - per-user default projects in `users.yaml`; - membership inherited between parent and child projects; - - moving or renaming a project while keeping its jobs' recorded name; - wildcard or pattern matching of member names; - retiring the rest of `limits.yaml` (user, queue and host scopes) and moving per-host slots to host config (`limits-to-owners`); - aggregate caps on queues (dropped by decision 2026-10-12, not deferred). @@ -97,14 +107,16 @@ This feature makes projects first-class cluster configuration entities that cont **Objective:** As a cluster admin, I want to define projects and their members in the cluster configuration, so that project ownership is explicit and versioned with the rest of the cluster config. #### Acceptance Criteria -1. The configuration tool shall accept project definitions consisting of a path name, an optional description, members (user names, user-group names, or `all`), optional excluded members, optional admins (user names or user-group names), an optional list of user groups for which the project is the default, an optional `share` (a map from resource name to an absolute amount or a percentage of the parent's share) and an optional `member_quota` (a map from resource name to an absolute amount). +1. The configuration tool shall accept project definitions consisting of a path name, an optional description, members (user names, user-group names, or `all`), optional excluded members, optional admins (user names or user-group names), an optional list of user groups for which the project is the default, an optional `share` (a map from resource name to an absolute amount or a percentage of the parent's share), an optional `member_quota` (a map from resource name to an absolute amount), and optional `aliases` (earlier path names of the project). The resource names include the built-in `jobs`. 2. The configuration tool shall treat a path name of one segment (e.g. `eng`) as a top-level project and a path of several `/`-separated segments (e.g. `eng/hpc/cfd`) as a child of the project named by the path without its last segment. 3. The configuration tool shall accept an optional cluster default project naming one defined project. -4. If a project definition repeats a path name, uses a reserved name, has an empty or malformed path segment, names a parent project that is not defined, names an undefined user group (in members, excluded members or admins), lists a default-for user group that is not covered by the project's members, has a `share` or `member_quota` entry naming an undeclared resource or giving a negative or non-numeric amount, has a percentage in `member_quota` or a percentage above 100% in `share`, declares a `share` for a resource its parent (when it has one) declares no share for, gives siblings shares of one resource in different forms (absolute and percentage), or gives siblings shares that add up to more than 100% or, when all are absolute and the parent's is absolute, to more than the parent's share, then the configuration tool shall reject the configuration with an error identifying the project and the offending field. +4. If a project definition repeats a path name, uses a reserved name, has an empty or malformed path segment, names a parent project that is not defined, names an undefined user group (in members, excluded members or admins), lists a default-for user group that is not covered by the project's members, has a `share` or `member_quota` entry naming an undeclared resource or giving a negative or non-numeric amount, has a percentage in `member_quota`, a percentage above 100% in `share`, or a percentage for `jobs`, declares a `share` for a resource its parent (when it has one) declares no share for, gives siblings shares of one resource in different forms (absolute and percentage), or gives siblings shares that add up to more than 100% or, when all are absolute and the parent's is absolute, to more than the parent's share, then the configuration tool shall reject the configuration with an error identifying the project and the offending field. 5. If the cluster default project names an undefined project, or a project whose members do not include every user, then the configuration tool shall reject the configuration with an error naming the cluster default project. 6. When a configuration with changed project definitions is applied by live configuration upgrade, the Cluster shall apply the new definitions to all subsequent submissions and scheduling decisions without a restart. 7. The Cluster shall keep project definitions, including their shares, across controller failover and restart identically to other cluster configuration. 8. If the limit configuration contains a limit scoped to projects (listed or per project), then the configuration tool shall reject the configuration with an error naming the limit and stating that project shares are written on the project. +9. If an alias is malformed as a path, equals a defined project's path or another alias, names an alias (a chain), or is used as the cluster default project, a default-for target or a parent path, then the configuration tool shall reject the configuration with an error naming the project and the alias. +10. Where a project with members has children whose shares of a resource add up to its whole share, so that its unassigned part of that resource is empty, the configuration tool shall warn that jobs charged directly to that project can never get that resource. ### Requirement 2: Project resolution at submit **Objective:** As a user, I want my job to get the right project when I do not name one, so that I do not have to pass `-P` for my usual project. @@ -116,6 +128,8 @@ This feature makes projects first-class cluster configuration entities that cont 4. The Cluster shall record the resolved project name on the job at submit, show it in the job's details, and keep it unchanged across later configuration changes, failover and restart. 5. The Cluster shall apply the same project resolution to every submission through the REST API (`xcecli` and any other client of the API) and to every element of a job array. 6. The Cluster shall identify a project at submit only by its full path name; a name that is only the last segment of a defined project's path (e.g. `cfd` for `eng/hpc/cfd`) is treated as the top-level project of that name. +7. When a job is submitted under a path that is an alias, the Cluster shall resolve it to the alias's target project, check `project:submit` against the target, and record the target's path on the job. +8. When a path below an alias is named (e.g. `eng/hpc/cfd` while `eng/hpc` is an alias of `research/hpc`), the Cluster shall resolve it to the corresponding path below the target (`research/hpc/cfd`), and treat it as undefined if that project does not exist. ### Requirement 3: Submit authorization **Objective:** As a cluster admin, I want only users allowed on a project to submit work under it, so that projects control who may use their resources. @@ -153,6 +167,8 @@ This feature makes projects first-class cluster configuration entities that cont 3. When a later configuration change makes a held job's project defined again, or restores its owner's membership, the Scheduler shall consider the job for scheduling again without user action. 4. While a job is running, the Cluster shall not stop, requeue or alter the job because of a change to its project's definition or membership. 5. The Cluster shall allow an admin to kill or modify a job held by Requirement 5 using the existing job controls. +6. The Cluster shall resolve a job's recorded project through aliases (Requirement 2.7, 2.8) wherever it is used, keeping the recorded path unchanged on the job; a pending job recorded under an alias is scheduled against the target project. +7. When an alias is removed while jobs still record it, the Cluster shall treat those jobs as it treats jobs whose project is no longer defined (Requirement 5.1), and the live configuration upgrade shall warn listing the pending jobs that still use the alias. ### Requirement 6: Owned shares **Objective:** As a cluster admin, I want each project to own a share of its parent's resources and hold no more than it, so that a project can always get what it owns and no project, or member of one, takes more than its part. @@ -173,6 +189,9 @@ This feature makes projects first-class cluster configuration entities that cont 13. The Scheduler shall resolve each generated limit's project and user scopes against project and user names only, so that a project name never matches a host group or user group (the scope separation fixed by bug-1933). 14. The live Scheduler and the scheduler simulator shall make identical share decisions for the same input. 15. The Scheduler shall show a project path containing `/` unchanged in pend reasons, decision traces and diagnostics. +16. The Scheduler shall count, for the built-in resource `jobs`, one unit per running job (one per running array element), regardless of its task count, against the same shares, unassigned parts and member quotas as any other resource. +17. When a job's recorded project resolves through an alias, the Scheduler shall count the job, running or pending, against the target project and its ancestors; when an alias is added for running jobs, their usage moves to the target's path on the next scheduling round. +18. If a job charged directly to a project cannot get a resource because that project's unassigned part of it is empty, then the Scheduler shall keep the job pending with a reason stating that the project's children own its whole share of that resource. ### Requirement 7: Project visibility **Objective:** As a user or admin, I want to see projects, their members and their usage against shares, so that I know which projects I can use and why a job is held. @@ -185,6 +204,7 @@ This feature makes projects first-class cluster configuration entities that cont 5. When a user lists jobs filtered by project through the REST API or `xcecli`, the Cluster shall return only jobs with that recorded project, or, when the user asks to include sub-projects, with that project or any descendant, subject to the user's existing job-visibility permissions. 6. While no project is defined, the Cluster shall report that projects are not configured when projects are listed. 7. The scheduler's diagnostic interface shall show, for a named project, the shares, unassigned parts and member quotas that apply to it (its own and its ancestors'), their effective amounts and their current usage. +8. Where a job's recorded project resolves through an alias, the Cluster shall show both in the job's details (e.g. `research/hpc (recorded as eng/hpc)`), list the alias in the project's details, and include such jobs when jobs are filtered by the target project. ### Requirement 8: Project-scoped access control **Objective:** As a cluster admin, I want project admins and roles to hold rights over the jobs and details of a project and its sub-projects, so that project leads can manage their project's work without being cluster administrators. @@ -203,6 +223,7 @@ This feature makes projects first-class cluster configuration entities that cont 11. The Cluster shall not give a project admin `project:submit` on the project, nor any right to change a project's definition, members, admins or shares, by being its admin. 12. When a configuration change adds or removes a project admin, the Cluster shall apply the change to all subsequent requests without a restart. 13. When a user lists their projects, the Cluster shall mark the projects the user administers. +14. The Cluster shall treat a `project` scope value that names an alias as covering the alias's target project. ### Requirement 9: Compatibility and durability **Objective:** As an operator, I want projects added without data migration or downtime, so that existing jobs and records keep working. diff --git a/.kiro/specs/project-fairshare/brief.md b/.kiro/specs/project-fairshare/brief.md index 1461aa58..5c477589 100644 --- a/.kiro/specs/project-fairshare/brief.md +++ b/.kiro/specs/project-fairshare/brief.md @@ -56,6 +56,11 @@ or make up over time for projects and users that used less. With strict shares, fairshare matters most among users of one project, among children drawing from the same unassigned part, and later with borrowing (`project-borrowing`). +5. **Jobs charged to an inner project** (2026-10-13): the project's own + jobs compete with its children as one more pseudo-child, with a + child's default weight (Slurm precedent). +6. **Project aliases**: usage history recorded under an alias belongs to + the target, like the jobs themselves. ## Constraints - Usage history is scheduler state that must survive failover and restart: diff --git a/.kiro/steering/job-config-layering.md b/.kiro/steering/job-config-layering.md index 72ed326e..1b41514e 100644 --- a/.kiro/steering/job-config-layering.md +++ b/.kiro/steering/job-config-layering.md @@ -160,7 +160,8 @@ them: `PER_PROJECT` → project `share`; `USERS` + `PER_USER` → `member_quota` (on the root project for a cluster-wide per-user cap); a user group's total → a project for the group; `HOSTS` + `PER_HOST` → host supply; queue - aggregates and mixed tuples → dropped. Retirement and the host-supply move + aggregates and mixed tuples → dropped; `JOBS` caps → the `jobs` + resource in a `share` or `member_quota`. Retirement and the host-supply move are `limits-to-owners`; shares are `project-config`. - **Shares count allocation**, what the scheduler reserved, not measured use. A job without a memory reservation can exceed its project's memory share @@ -219,6 +220,24 @@ them: - **`member_quota`** on P: an absolute cap per user over that user's jobs anywhere in P's subtree, whether or not the user is a direct member of P. It caps a user inside what the project owns; it is not a share. +- **Job limits are the `jobs` resource** (user decision 2026-10-13): a + built-in resource each running job (each array element) uses one of, + whatever its task count; absolute amounts only. A project's running-job + limit is `share: { jobs: N }`; a user's is `member_quota: { jobs: N }` + (on the root project for a cluster-wide per-user job limit). Caps on + *pending* jobs are a submit-time check (`pending-job-limits`). +- **Inner projects take jobs too.** A job charged to `eng` draws from + `eng`'s unassigned part; when its children own the whole share of a + resource, the config tool warns and such jobs pend with a reason saying + so. Leaf-only charging (`accept_jobs: false`) may come later. +- **Aliases** (user decision 2026-10-13, as for application profiles): a + renamed project lists its old paths, `aliases: [eng/hpc]`. Jobs recorded + under an alias, pending or running, belong to the target and count + against it and its ancestors; new submits naming the alias record the + target; paths below an alias map to the same paths below the target. No + chains, no clashes; remove the alias when no job uses it. This is an + explicit admin move, so it is the one case where a running job's usage + changes project. - Changing a share never stops running jobs; a project above its new share cannot start more until it is below it. - Pend reasons name the project, resource and usage ("project eng/ai share diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index 034c3d86..c8273161 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -70,10 +70,10 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han - [ ] job-time-window — NEW 2026-09-29 (`/kiro-discovery`, Path C). Job begin time (scheduler gate in `sched_job_is_ready()` via `xce_core_now`, pend reason BEGIN_TIME_NOT_REACHED) and end time (WORKER enforces running jobs: helper SIGTERM→grace→SIGKILL, exit reason END_TIME_REACHED persisted, unit RuntimeMaxUSec backstop; PEND jobs whose end time can't be met stay PEND with END_TIME_UNMEETABLE until user kills/modifies — no controller sweep, primary not involved). Adds runtime estimate (`-We/--estimated_run_time`, scheduling hint only, never enforced; estimate > run limit rejected); unmeetable rule `now + (estimate ‖ run_limit ‖ 0) > end`. Times/estimate modifiable only while PEND (new `xce jobs modify` + harness `jobmod`). Both xcecli and harness `jobsub` gain all options (`-b`/`-t`/`-W`/estimate). Also fixes the silent drop of xcecli `-b`/`-t`/`-W` and REST `run_time_estimation` (apiserver `modelJobToSubmit` never sets them nor `exdata`). Brief: `.kiro/specs/job-time-window/brief.md`. Dependencies: none (new reason codes need text in job-status-detail-display). - [ ] job-dependency — NEW 2026-09-29 (`/kiro-discovery`, Path D; ref LSF 10.1 Admin Guide "Job dependency and job priority" pp. 459-465 — dependency only, priority deferred). `-w` expressions by job ID: done/ended/exit(id[,op code])/started/post_done/post_err (PDONE/PERR) + bare id, `&& || !` (). Shared libxce grammar (Go via cgo); apiserver early validation (syntax, parent exists, submitter has `job:view`); controller authoritative re-validation, stores expression (additive JobSubmitData field) + in-memory reverse index rebuilt on replay (implements controller-job-cleanup's `job_has_dependents()`); scheduler evaluates in `sched_job_is_ready()` (pend reasons DEP_NOT_SATISFIED / DEP_NEVER_SATISFIED; orphans stay PEND). Fixes scheduler exit-code field-completeness gap (xce_sched_job_t/SchedJob/snapshot lack exit code). xcecli `-w`, `modify -w/-wn`; harness `jobsub -w`/`jobmod`. Brief: `.kiro/specs/job-dependency/brief.md`. Dependencies: job-time-window (shared gate + modify path) — coordinate. - [ ] job-dependency-insight — NEW 2026-09-29. `bjdepinfo`-style query (parents/children/-p blocking/-r levels/-l verdicts, RBAC-filtered) via REST/xcecli/harness, dependency info in job details, optional orphan termination (per-job `-ti` + cluster grace period, default off; EXIT reason ORPHAN; failover-safe persisted grace start; minimal primary periodic work). Brief: `.kiro/specs/job-dependency-insight/brief.md`. Dependencies: job-dependency, job-time-window (exit-reason field). -- [ ] controller-job-cleanup — NEW 2026-09-29 (`/kiro-discovery`, Path C). Primary-only periodic sweep (configurable `job_cleanup_period`, default 300 s) removes DONE/EXIT jobs finished longer than configurable `job_retention` (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever `jobmgr` delete; refcount-safe `xkvmgr` delete), propagated to scheduler and apiserver (`OnJobDeleted`). Retires the node-local snapshot-only `job_should_retain` finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: `.kiro/specs/controller-job-cleanup/brief.md`. Dependencies: none (extends ha-entity-snapshots). +- [ ] controller-job-cleanup — NEW 2026-09-29 (`/kiro-discovery`, Path C). Primary-only periodic sweep (configurable `job_cleanup_period`, default 300 s) removes DONE/EXIT jobs finished longer than configurable `job_retention` (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever `jobmgr` delete; refcount-safe `xkvmgr` delete), propagated to scheduler and apiserver (`OnJobDeleted`). Retires the node-local snapshot-only `job_should_retain` finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: `.kiro/specs/controller-job-cleanup/brief.md`. Dependencies: none (extends ha-entity-snapshots). Once project-accounting exists, a job is removed only after its accounting record is durable (2026-10-13). - [ ] job-status-detail-display — NEW 2026-09-28 (`/kiro-discovery`, Path E). End users (REST/`xcecli`) see NO pend reason for ANY job today: the apiserver cgo conversion (`backend/convert.go`) never maps `xce_job_t.pend_reasons`, so `pending_reason` is always empty. Maps it to a structured JSON (including `alternatives` from day one) using libxce's code→text mapping, and renders it in `xcecli` job detail. Open for requirements: also map `run_usage` → `JobRusage` (same gap; reserved vs. used for RUN jobs). Brief: `.kiro/specs/job-status-detail-display/brief.md`. Dependencies: none for flat reasons; scheduler-alter-pend-reasons for per-alternative rendering. - [ ] job-config-merge — NEW 2026-10-09 (`/kiro-discovery`, Path E "configuration layering"; ref LSF 10.1 Admin Guide pp. 314-316, 435-437, 561-563, 643, 660). One merge function in `src/lib` turning the job's own values + application profile + queue policy + project into effective values (resreq incl. `span`, task range, limits, pre/post-exec, env). One owner per attribute (user decision; `.kiro/steering/job-config-layering.md`): the job may add to a profile's values but never change or remove them (additive rule, 2026-10-12, replacing *default*/*fixed* markers); queue ceilings and project quotas only check (reject if never satisfiable, pend if full now). **Live** merge (user decision): WAL keeps inputs + names; controller/apiserver validate at submit; scheduler computes effective values and re-evaluates pending jobs on config change. Queue is policy only: queue `res_req` and `pre_exec`/`post_exec` deprecated (pre/post-exec moves to the profile), dropped queue limits (cpulimit/memlimit/runlimit/task limit) wired. Brief: `.kiro/specs/job-config-merge/brief.md`. Dependencies: none. -- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (path identity, members, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config, owned shares written on the project (`share` of the parent's resources, absolute or percentage, strict in v1; children without a share draw from the parent's unassigned part; `member_quota` over the subtree; user decisions 2026-10-12, replacing `PROJECTS`/`PER_PROJECT` limits in `limits.yaml`) compiled into the limit plugin, per-project usage display; `project:submit` RBAC implied by membership, project admins, authorization in the apiserver only. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none to ship; production rejection of direct submits comes from peer-authentication-hardening. +- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (path identity, members, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config, owned shares written on the project (`share` of the parent's resources, absolute or percentage, strict in v1; children without a share draw from the parent's unassigned part; `member_quota` over the subtree; user decisions 2026-10-12, replacing `PROJECTS`/`PER_PROJECT` limits in `limits.yaml`) compiled into the limit plugin, a built-in `jobs` count (job limits per project and user), project aliases for renames, charging inner projects, per-project usage display; `project:submit` RBAC implied by membership, project admins, authorization in the apiserver only. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none to ship; production rejection of direct submits comes from peer-authentication-hardening. - [ ] host-failure-job-recovery — NEW 2026-10-10 (`/kiro-discovery`, "Long-running services" phase 1; also a batch gap: RUN jobs on a dead host stay RUN forever, `controller/sched.c:631` FIXME). After a grace period, jobs on a lost host go to an unknown state, then are requeued (rerunnable, job option or profile default) or end EXIT "host lost"; multi-host jobs resolved as a whole; attempt numbers fence a stale copy when the host returns; manual requeue (REST/xcecli). Brief: `.kiro/specs/host-failure-job-recovery/brief.md`. Dependencies: none. - [ ] service-supervision — NEW 2026-10-10 (Long-running services phase 1). Application profile kind `service`: restart policy (always/on-failure/never, backoff, give-up), liveness and readiness probes (command/TCP/HTTP) run by each replica's own worker (no special replica), in-place restart, change-only health reports, display. Brief: `.kiro/specs/service-supervision/brief.md`. Dependencies: application-profile, host-failure-job-recovery. @@ -119,13 +119,15 @@ Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generi ## Future Specs / Backlog (dependency order) - [ ] gang-groups — NEW 2026-10-11; **demoted to backlog 2026-10-12** (user decision N24 in `resource-questions.md`). LLM training is one parallel job with topology requirements (PR-10), so it needs no gang. Re-scoped as a **co-start group** of separately managed jobs and services: RL post-training (trainer, rollout engines, reward and reference models), disaggregated serving (prefill and decode pools, router), streaming pipelines. All-or-nothing start, close placement, group failure policy. Revisit when RL or disaggregated serving reaches the roadmap. Brief: `.kiro/specs/gang-groups/brief.md`. Dependencies: resource-instances, resource-claims, service-replica-set. +- [ ] project-accounting — NEW 2026-10-13 (user decision; next project feature after project-config). One durable record per finished job, requeue and array element (owner, project path as recorded and resolved, queue, profile, times, exit, allocation per host, measured usage), written before `controller-job-cleanup` may drop the job; a history store outside the replicated WAL (controller-written log vs. a separate accounting service: first design decision, with HA writer-after-failover and no-duplicate rules); usage reports by project, subtree, user, period and resource (REST, `xcecli projects usage`). Also the source `project-fairshare` can rebuild its usage history from. Brief: `.kiro/specs/project-accounting/brief.md`. Dependencies: project-config. +- [ ] pending-job-limits — NEW 2026-10-13 (user decision; backlog, no brief yet). Caps on pending jobs per project and per user (LSF `MAX_PEND_JOBS`), checked at submit (rejected with a clear error, never held); an array of N counts N; pending counts that survive failover. Separate from the running-job `jobs` resource in project shares. Dependencies: project-config. - [ ] project-borrowing — NEW 2026-10-09 as project-guarantee (configuration layering, backlog); renamed and narrowed 2026-10-12: ownership moved into project-config's owned shares. A `max` above a project's share, borrowing idle capacity whose owner is not waiting for it, reclaim through preemption, owned/borrowed accounting. Brief: `.kiro/specs/project-borrowing/brief.md`. Dependencies: project-config, scheduler-preemption. - [ ] project-fairshare — NEW 2026-10-12 (user decision: fairshare is done by projects). Fairshare weights on the project tree (users equal within a project by default), decayed usage history per project and user, hierarchical ranking applied in every queue that orders by fairshare (queues keep priority between queues and may choose FCFS); the per-queue `fairshare:` user lists are removed from `queues.yaml`. Usage history persisted in the scheduler snapshot. Brief: `.kiro/specs/project-fairshare/brief.md`. Dependencies: project-config. - [ ] limits-to-owners — NEW 2026-10-12 (configuration layering, user decisions; backlog, no brief yet). Retire `limits.yaml` (not in production, removed without deprecation): the limit plugin becomes internal, fed only by generated limits. `USERS`/`PER_USER` caps become `member_quota` (root project for a cluster-wide per-user cap), a user group's total becomes a project quota, `HOSTS`/`PER_HOST` slot caps move to host supply in host config (LSF `MXJ`), queue aggregate caps and mixed tuples (user group × queue) are dropped. Touches `xceadmin` mapping and schema (`admin-config-tool`), `ClusterConfig.limits`, `src/configs/limits.yaml`, scheduler sim fixtures and the limit tests. Rules: `job-config-layering.md` "Resource control". Dependencies: project-config. - [ ] job-resize — NEW 2026-10-10 (from long-running services discovery; not on the services path). Grow/shrink a running parallel job's allocation (LSF resizable jobs, `bresize`) for elastic applications. Brief: `.kiro/specs/job-resize/brief.md`. Dependencies: parjob-placement, host-failure-job-recovery. - [ ] project-delegated-membership — NEW 2026-10-10 (from project-config requirements, user decision: future). Project managers edit their project's (and subtree's) members at runtime without an admin config push; needs a durable store that coexists with `xceadmin` config pushes (e.g. a separate delegated member list) and audit. Dependencies: project-config, rbac-persistence. - [ ] app-access-control — NEW 2026-10-12 (deferred from application-profile, user decision; backlog, design kept in the application-profile brief). Per-profile RBAC: `app:use` / `app:view` with an `app` scope type (profile names or `all`), checks on job create, restart and listing, `rbac.yaml` defaults, the current default profile always usable with `queue:submit`, aliases checked against the target profile, role API support, `cluster_admin` bypass only. Dependencies: application-profile; rbac-persistence for durable API-made grants. -- [ ] job-modify — NEW 2026-10-12 (from application-profile discussion, user decision; backlog, no brief yet). Modify a pending job (LSF `bmod`): apiserver action (today `"modify"` is commented out, `controllers/jobs.go:2446`; `JOB_MODIFY` only an enum), controller opcode, WAL fact with snapshot field-completeness, scheduler re-merge, xcecli. First attributes: `--app` (owner with `app:use` on the new profile; admins in bulk, e.g. `--app-from `), `-R`, `-n`, queue; later begin/end time (`job-time-window`), dependencies. Dependencies: application-profile, job-config-merge. +- [ ] job-modify — NEW 2026-10-12 (from application-profile discussion, user decision; backlog, no brief yet). Modify a pending job (LSF `bmod`): apiserver action (today `"modify"` is commented out, `controllers/jobs.go:2446`; `JOB_MODIFY` only an enum), controller opcode, WAL fact with snapshot field-completeness, scheduler re-merge, xcecli. First attributes: `--app` (owner with `app:use` on the new profile; admins in bulk, e.g. `--app-from `), `-R`, `-n`, queue, project (`-P`, user decision 2026-10-13: pending jobs only, never running ones; the owner must hold `project:submit` on the new project whoever runs the modify; owners move between their projects, project admins out of their subtree, the cluster admin anywhere; the job's authorization basis is updated); later begin/end time (`job-time-window`), dependencies. Dependencies: application-profile, job-config-merge, project-config (for `-P`). - [ ] config-history-audit — NEW 2026-10-12 (from application-profile discussion, user decision: needed, roadmap). Keep past cluster-config versions (including `applications.yaml` profiles) with who changed what and when: `xceadmin config history` / `diff`, look up a profile by its content hash (the application-profile hash recorded on dispatch records), audit trail of live upgrades. Per-job display of the values a run used is already covered by the application-profile dispatch record; this adds admin history and audit. Covers the whole cluster config, not only profiles (the controller already tracks `configVersion`/`configChecksum`). Dependencies: application-profile, admin-config-tool. - [ ] job-labels — NEW 2026-10-12 (from application-profile discussion; backlog, no brief yet). General Kubernetes-style job labels: `--label key=value` (repeatable, limited charset and size), an additive submit-data field through WAL and snapshot (field completeness), server-side filtering (`jobs list -l k=v`) and summaries grouped by label (runtime, peak memory, efficiency); metadata only, never used for placement. Reuses or replaces the REST `tags` field that is silently dropped today (bug-1970). Not needed for presets (those use `exdata.meta`); candidates: team-wide comparison, services labelling replicas. Dependencies: none. - [ ] app-submit-hook — NEW 2026-10-12 (from application-profile discussion, user decision: backlog only). A profile-named program that inspects or adjusts each submission (LSF `esub` style). Cannot run in the controller (no-fork, non-blocking rules); in the apiserver it adds latency and a failure mode to every submit and bypasses the merge rules, so decide client (xcecli) vs apiserver first. Application-profile `params`, annotations, the job starter and pre-exec cover most needs meanwhile. Dependencies: application-profile.