docs(spec): resource control under the project tree; retire limits.yaml

The project tree is the only place concurrent use is capped: quotas are
written on the project (quota, member_quota over the subtree) and compiled
into the limit plugin, which becomes internal. Queues keep per-job
ceilings only; host slot caps are supply. Project path is the identity
and the only submit form; members are users and user groups, never
sub-projects. Adds the limits-to-owners backlog item.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.kiro/steering/job-config-layering.md

@@ -4,7 +4,8 @@ How a job's settings are split between the job, its application profile, its
 queue and its project, and how they interact. Decided 2026-10-09
 (`/kiro-discovery`, configuration layering). Specs that follow this:
 `job-config-merge`, `application-profile`, `project-config`,
-`project-guarantee`, `parjob-placement`, `lsf-config-migration`.
+`project-guarantee`, `parjob-placement`, `lsf-config-migration`,
+`limits-to-owners`.
 
 ## Why not LSF's rules
 LSF lets the queue, the application profile and the job set the same
@@ -19,9 +20,9 @@ XCE instead gives **every attribute exactly one owning level**.
 
 | Level | Owns | Never owns |
 |---|---|---|
-| **Queue**: scheduling policy | priority and ordering, host partition, dispatch/run windows, preemption policy, policy ceilings (max run time, max tasks, max hosts, ...) | resource requirement, anything about how a job executes (pre/post-exec, env, launch, limits enforced on the job) |
+| **Queue**: scheduling policy | priority and ordering, host partition, dispatch/run windows, preemption policy, per-job ceilings (max run time, max tasks, max hosts, ...) | resource requirement, anything about how a job executes (pre/post-exec, env, launch, limits enforced on the job), aggregate caps on concurrent use |
 | **Application profile**: what the application needs and how it runs (essentials plus default reservations, 2026-10-12) | resreq (select / rusage / span / same / order), task range (min / max), execution mode and options, launch type, env, **pre/post-exec**, execution limits (mem, runtime, CPU, tasks) | priority, queue choice, quotas |
-| **Project**: resource ownership | membership, quotas / limits, later guarantees, accounting | how a job runs or is ordered |
+| **Project**: resource ownership | membership, the only caps on concurrent use (`quota`, `member_quota`), later guarantees and fairshare weights, accounting | how a job runs or is ordered |
 | **Job** | its own values, only for attributes its profile allows | n/a |
 
 Access to each is separate: queue and profile by RBAC permission, project by
@@ -107,7 +108,7 @@ One line per part; details below.
      grammar.
    - Consequence: `rusage` is the one profile part that holds **defaults**; a
      profile cannot enforce a minimum reservation. Hard protection of shared
-     resources comes from `limits.yaml` and project quotas.
+     resources comes from project quotas (see "Resource control").
    - Side effects to handle in requirements: measured resources (memory) are
      charged `max(reserved, actual)` once usage is reported
      (`resource-accounting.md`), so lowering them mostly self-corrects, but
@@ -132,6 +133,60 @@ There is no third case: no queue resreq, no rusage merge with the queue, no
 ordering of queue against profile settings, no compound/alternative merge
 stages.
 
+## Resource control
+User decisions 2026-10-12. The **project tree is the only place where
+concurrent use of resources is capped**. Four concepts, one owner each; the
+word "limit" alone is not used for any of them:
+
+| Concept | Question it answers | Owner | Examples |
+|---|---|---|---|
+| **Supply** | what a host has | host / resource config | slots per host (LSF `MXJ`), host resources |
+| **Quota** (consumption) | how much a group of jobs may hold at once | **project** | `quota`, `member_quota`; later guarantee, fairshare weight |
+| **Ceiling** (job shape) | how big one job may be | queue | max tasks, max run time, max hosts per job |
+| **Execution limit** (enforcement) | what the running job may use | application profile | cgroup memory, runtime, CPU |
+
+- **No aggregate caps outside projects.** A queue has per-job ceilings only;
+  a queue that must not flood the cluster gets a host partition, or its users
+  a capped project. User-group × queue caps do not exist.
+- **The limit plugin is internal.** The scheduler compiles project quotas
+  into limits on every config push (`proj:<path>:<res>` for `quota`,
+  `projm:<path>:<res>` with a per-user consumer for `member_quota`), reusing
+  the vres usage counters, instance carry-over across pushes (bug-1967) and
+  live/sim parity. Admins never write limits.
+- **`limits.yaml` is removed** (not in production; no deprecation period, so
+  there is one config path, not two). Its entries map as: `PROJECTS` /
+  `PER_PROJECT` → project `quota`; `USERS` + `PER_USER` → `member_quota` (on
+  the root project for a cluster-wide per-user cap); a user group's total →
+  a project for the group; `HOSTS` + `PER_HOST` → host supply; queue
+  aggregates and mixed tuples → dropped. Retirement and the host-supply move
+  are `limits-to-owners`; project quotas are `project-config`.
+- **Quotas count allocation**, what the scheduler reserved, not measured use.
+  A job without a memory reservation can exceed its project's memory quota
+  at run time (`resource-accounting.md` charges hosts `max(reserved,
+  actual)`; quotas do not). Accepted: profile default reservations close the
+  gap.
+
+### The project tree
+- **Identity is the full path** (`eng/hpc/cfd`), unique among siblings only.
+  The job records the path at submit and is charged to that path's
+  ancestors for as long as it runs; a later tree change never moves a
+  running job's usage. Moving a subtree renames its projects (rare,
+  visible). Config is a flat list of full paths, not nested YAML.
+- **Submit names the full path** (`xcecli -P eng/hpc/cfd`, REST
+  `project_name`, any UI); no last-segment shorthand. Clients offer the
+  user's projects (the project listing) instead.
+- **Two relations, kept apart:** the tree says where usage rolls up;
+  membership says who may submit. Members are users and user groups (plus
+  `all` and excludes, exclude wins), resolved at submit. A sub-project is
+  never a member, and membership is not inherited either way.
+- **`quota`** on P: a job dispatches only if it fits P's quota and every
+  ancestor's; P's usage includes all descendants' jobs. Absolute amounts
+  only (no percentages in v1).
+- **`member_quota`** on P: a cap per user over that user's jobs anywhere in
+  P's subtree, whether or not the user is a direct member of P.
+- Pend reasons name the project, resource and usage ("project eng quota on
+  gpu full (16/16)"), never an internal limit name.
+
 ## Convenience defaults are presets, not a level
 Apart from default reservations (the profile's `rusage` entries), tuning
 that most jobs of an application want but none must have
@@ -183,3 +238,5 @@ is today.
 - LSF migration (`lsf-config-migration`): an LSF queue `RES_REQ` that only
   selects hosts becomes a queue host partition; anything else moves into a
   profile or is reported. LSF queue `PRE_EXEC`/`POST_EXEC` move into profiles.
+  LSF `Limit` sections are split by consumer as in "Resource control"; queue
+  and mixed entries are reported, not migrated.

.kiro/steering/roadmap.md

@@ -73,7 +73,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
 - [ ] controller-job-cleanup — NEW 2026-09-29 (`/kiro-discovery`, Path C). Primary-only periodic sweep (configurable `job_cleanup_period`, default 300 s) removes DONE/EXIT jobs finished longer than configurable `job_retention` (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever `jobmgr` delete; refcount-safe `xkvmgr` delete), propagated to scheduler and apiserver (`OnJobDeleted`). Retires the node-local snapshot-only `job_should_retain` finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: `.kiro/specs/controller-job-cleanup/brief.md`. Dependencies: none (extends ha-entity-snapshots).
 - [ ] job-status-detail-display — NEW 2026-09-28 (`/kiro-discovery`, Path E). End users (REST/`xcecli`) see NO pend reason for ANY job today: the apiserver cgo conversion (`backend/convert.go`) never maps `xce_job_t.pend_reasons`, so `pending_reason` is always empty. Maps it to a structured JSON (including `alternatives` from day one) using libxce's code→text mapping, and renders it in `xcecli` job detail. Open for requirements: also map `run_usage` → `JobRusage` (same gap; reserved vs. used for RUN jobs). Brief: `.kiro/specs/job-status-detail-display/brief.md`. Dependencies: none for flat reasons; scheduler-alter-pend-reasons for per-alternative rendering.
 - [ ] job-config-merge — NEW 2026-10-09 (`/kiro-discovery`, Path E "configuration layering"; ref LSF 10.1 Admin Guide pp. 314-316, 435-437, 561-563, 643, 660). One merge function in `src/lib` turning the job's own values + application profile + queue policy + project into effective values (resreq incl. `span`, task range, limits, pre/post-exec, env). One owner per attribute (user decision; `.kiro/steering/job-config-layering.md`): the job may add to a profile's values but never change or remove them (additive rule, 2026-10-12, replacing *default*/*fixed* markers); queue ceilings and project quotas only check (reject if never satisfiable, pend if full now). **Live** merge (user decision): WAL keeps inputs + names; controller/apiserver validate at submit; scheduler computes effective values and re-evaluates pending jobs on config change. Queue is policy only: queue `res_req` and `pre_exec`/`post_exec` deprecated (pre/post-exec moves to the profile), dropped queue limits (cpulimit/memlimit/runlimit/task limit) wired. Brief: `.kiro/specs/job-config-merge/brief.md`. Dependencies: none.
-- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (members, priority, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config so `PROJECTS`/`PER_PROJECT` limits work end to end, per-project usage display. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none.
+- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (path identity, members, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config, quotas written on the project (`quota`, `member_quota` over the subtree; user decision 2026-10-12, replacing `PROJECTS`/`PER_PROJECT` limits in `limits.yaml`) and compiled into the limit plugin, per-project usage display. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none.
 
 - [ ] host-failure-job-recovery — NEW 2026-10-10 (`/kiro-discovery`, "Long-running services" phase 1; also a batch gap: RUN jobs on a dead host stay RUN forever, `controller/sched.c:631` FIXME). After a grace period, jobs on a lost host go to an unknown state, then are requeued (rerunnable, job option or profile default) or end EXIT "host lost"; multi-host jobs resolved as a whole; attempt numbers fence a stale copy when the host returns; manual requeue (REST/xcecli). Brief: `.kiro/specs/host-failure-job-recovery/brief.md`. Dependencies: none.
 - [ ] service-supervision — NEW 2026-10-10 (Long-running services phase 1). Application profile kind `service`: restart policy (always/on-failure/never, backoff, give-up), liveness and readiness probes (command/TCP/HTTP) run by each replica's own worker (no special replica), in-place restart, change-only health reports, display. Brief: `.kiro/specs/service-supervision/brief.md`. Dependencies: application-profile, host-failure-job-recovery.
@@ -120,6 +120,7 @@ Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generi
 ## Future Specs / Backlog (dependency order)
 - [ ] gang-groups — NEW 2026-10-11; **demoted to backlog 2026-10-12** (user decision N24 in `resource-questions.md`). LLM training is one parallel job with topology requirements (PR-10), so it needs no gang. Re-scoped as a **co-start group** of separately managed jobs and services: RL post-training (trainer, rollout engines, reward and reference models), disaggregated serving (prefill and decode pools, router), streaming pipelines. All-or-nothing start, close placement, group failure policy. Revisit when RL or disaggregated serving reaches the roadmap. Brief: `.kiro/specs/gang-groups/brief.md`. Dependencies: resource-instances, resource-claims, service-replica-set.
 - [ ] project-guarantee — NEW 2026-10-09 (configuration layering, backlog). Project-owned guarantees (resource amount or host pool), borrowing of idle guaranteed capacity, reclaim through preemption. Brief: `.kiro/specs/project-guarantee/brief.md`. Dependencies: project-config, scheduler-preemption.
+- [ ] limits-to-owners — NEW 2026-10-12 (configuration layering, user decisions; backlog, no brief yet). Retire `limits.yaml` (not in production, removed without deprecation): the limit plugin becomes internal, fed only by generated limits. `USERS`/`PER_USER` caps become `member_quota` (root project for a cluster-wide per-user cap), a user group's total becomes a project quota, `HOSTS`/`PER_HOST` slot caps move to host supply in host config (LSF `MXJ`), queue aggregate caps and mixed tuples (user group × queue) are dropped. Touches `xceadmin` mapping and schema (`admin-config-tool`), `ClusterConfig.limits`, `src/configs/limits.yaml`, scheduler sim fixtures and the limit tests. Rules: `job-config-layering.md` "Resource control". Dependencies: project-config.
 - [ ] job-resize — NEW 2026-10-10 (from long-running services discovery; not on the services path). Grow/shrink a running parallel job's allocation (LSF resizable jobs, `bresize`) for elastic applications. Brief: `.kiro/specs/job-resize/brief.md`. Dependencies: parjob-placement, host-failure-job-recovery.
 - [ ] project-delegated-membership — NEW 2026-10-10 (from project-config requirements, user decision: future). Project managers edit their project's (and subtree's) members at runtime without an admin config push; needs a durable store that coexists with `xceadmin` config pushes (e.g. a separate delegated member list) and audit. Dependencies: project-config, rbac-persistence.
 - [ ] app-access-control — NEW 2026-10-12 (deferred from application-profile, user decision; backlog, design kept in the application-profile brief). Per-profile RBAC: `app:use` / `app:view` with an `app` scope type (profile names or `all`), checks on job create, restart and listing, `rbac.yaml` defaults, the current default profile always usable with `queue:submit`, aliases checked against the target profile, role API support, `cluster_admin` bypass only. Dependencies: application-profile; rbac-persistence for durable API-made grants.
@@ -130,7 +131,7 @@ Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generi
 - [ ] job-requeue — NEW 2026-10-12 (from application-profile discussion, user decision; backlog, no brief yet). Automatic requeue on given exit values (LSF `REQUEUE_EXIT_VALUES`) and success exit values (`SUCCESS_EXIT_VALUES`), with a requeue loop limit; wires the commented-out `requeue` job action (`apiserver/controllers/jobs.go`) and the REST `requeue_error_values` / `rerunable` options that are silently dropped today (bug-1970). Adds the matching application-profile attributes (a profile attribute ships with its job feature). Later, same principle: checkpoint settings and warning-before-kill each get their own spec. Dependencies: application-profile.
 - [ ] xcecli-presets — NEW 2026-10-12 (from application-profile discussion, user decisions). Client-side presets replace convenience defaults in profiles: a preset is a set of job options (`--app`, `--queue`, `-R`, `-n`, env, limits) expanded by the client, a flag replacing the preset's value for that option; one file per preset (`~/.config/xce/presets/<ns>/<name>.yaml`, same layout in the apiserver's presets dir), env capture by name (`pass:`; `XCE_*` and secret-looking names excluded); the apiserver only shares admin-deployed presets read-only (`GET /xce/v1/presets[/{ns}/{name}]`, files in `<TopDir>/config/presets/` or `presets.dir`, re-read on change; not in ClusterConfig), and `preset pull` copies one locally so server-side removal never affects users; on a name collision pull refuses unless it is an update of the same server preset (`--as` / `--force` with `.bak`); submit uses local presets only (`preset diff`, `pull --force` user-initiated); user-to-user sharing by `preset export`/`import`; presets carry no privilege and are not a configuration level; no preset field on the server: the client records preset name + hash in the job's `exdata.meta` (passed through uninterpreted; worker skips `meta`), and `preset report` filters jobs on it client-side; the portal does the same expansion. Brief: `.kiro/specs/xcecli-presets/brief.md`. Dependencies: application-profile.
 - [ ] profile-usage-insight — NEW 2026-10-12 (from application-profile discussion, backlog; no brief yet). Per-profile report over recent jobs comparing the profile's defaults with actual use (requested vs. peak memory, CPU efficiency, OOM kills, pend time, runtime by variant) and suggesting better values for site presets (or a profile essential), e.g. "preset openfoam rusage mem 4000, p95 peak 1600 → suggest 2000" (Kubernetes VPA recommender precedent). **Advisory only** (user decision): the admin applies changes to site presets or `applications.yaml`; nothing tunes itself. Optional user hint ("used 1.4G of 4G requested"). Dependencies: application-profile, run usage in job records.
-- [ ] lsf-config-migration — NEW 2026-10-09 (user idea, backlog; no brief yet). A tool (likely an `xceadmin` subcommand) that converts LSF configuration (`lsb.queues`, `lsb.applications`, `lsb.resources` Limit sections, `lsb.users`/`lsb.hosts` groups, `lsb.params` defaults, project settings) into XCE `configs/*.yaml`, with a report of what does not map: queue `RES_REQ` (XCE queues are policy only; move into a profile or drop), LSF's full per-section merge rules vs XCE's simplified ones, compound/alternative resreqs, unsupported parameters. Dependencies: job-config-merge, application-profile, project-config, parjob-placement (span), admin-config-tool. Promote via `/kiro-discovery` when those land.
+- [ ] lsf-config-migration — NEW 2026-10-09 (user idea, backlog; no brief yet). A tool (likely an `xceadmin` subcommand) that converts LSF configuration (`lsb.queues`, `lsb.applications`, `lsb.resources` Limit sections (split by consumer into project quotas, member quotas and host supply; queue and mixed entries reported), `lsb.users`/`lsb.hosts` groups, `lsb.params` defaults, project settings) into XCE `configs/*.yaml`, with a report of what does not map: queue `RES_REQ` (XCE queues are policy only; move into a profile or drop), LSF's full per-section merge rules vs XCE's simplified ones, compound/alternative resreqs, unsupported parameters. Dependencies: job-config-merge, application-profile, project-config, parjob-placement (span), admin-config-tool. Promote via `/kiro-discovery` when those land.
 
 - [ ] **xce-harness-component** — **PROMOTED to spec 2026-09-13** (`/kiro-discovery`; brief at `.kiro/specs/xce-harness-component/brief.md`). Elevate `xce_test` (the REPL-driven wire-protocol test client used by HA soak / multi-host / resource-value verification) out of `src/controller/testsuite/` into its own first-class `src/harness/` component — own `meson.build`, `CLAUDE.md`, and unit-test binary (carved out of `controller_unit_test`, which today recompiles ~14 of xce_test's own files a second time paired with white-box tests) — fully decoupled from `/xce-build controller` (today's implicit "controller always builds xce_test" coupling, `build-deployment.md:24`). Verified finding: none of xce_test's 15 source files include any controller-private header — only libxce public headers + generated capnp schema — so this is a pure relocation + build-graph split, not a dependency untangling problem, and no xce_test behavior changes. Touches `/xce-test-ha`, `/xce-deploy-ha`, `/xce-test-multihost`, `/xce-test-resources` (each must build `harness` explicitly once decoupled) and is adjacent to (not overlapping) the three existing behavior specs `xce-test-cli-ux`/`xce-test-e2e-primitives`/`xce-test-rc-file`, whose file-path references go stale after the move. Next: `/kiro-spec-init xce-harness-component`.
 - [x] config-upgrade-snapshot — **descoped from `ha-term-safety` R8 on 2026-06-21 (requirements generated; `.kiro/specs/config-upgrade-snapshot/`).** Closes **bug-857**: a configuration upgrade can leave members with DIVERGENT materialized state because today it does config-push + restart + per-node WAL replay under the new config (`config_repl.c` `do_upgrade_config_with_bytes` ~1001-1163), with no snapshot pinning an identical post-upgrade state (replay is config-dependent — `lost_and_found` re-homes orphaned entities). Approach: an upgrade materializes a **full entity-bearing state snapshot at the config-change marker's `(term,lsn)` boundary**, embeds the new config, and installs it on every member via the `snapshot_transfer_s` transport — no per-node re-derivation; supersedes `ha-term-safety` R7's interim config-push byte delivery for upgrades. **Former HARD dependency `entity-bearing-snapshots` is CLEARED** — it shipped as spec `ha-entity-snapshots` (**IMPLEMENTATION-COMPLETE, validate-impl GO 2026-07-18**; see Future Specs/Backlog below), and `ha-term-safety` (the config-change marker it anchors on) is also implemented. `config-upgrade-snapshot`'s own `spec.json` now has `blocked_on: []`. Verify by reproducing bug-857 (upgrade during chaotic multi-kill, diff materialized state). **Ready to promote to design.**