From 1f9aaae050f62804f8db497d0b7529bc8ce8eb28 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 13 Oct 2026 02:49:53 +0000 Subject: [PATCH] docs(spec): structured profile resreq; per: optional (default task); merge rules at a glance - profile resreq split into select / rusage entries / span / same / order; rusage entries carry optional per: (default task, today's PER_DEFAULT behaviour) and optional default - job's rusage term wins whole for a resource in both; unit must match; no modifier inheritance - compound/alternative: only the profile select applies; any profile rusage default or span/same/order rejects until PR-15 - job-config-layering: merge rules at a glance table; profile = essentials plus default reservations Co-Authored-By: Claude Opus 5.5 (1M context) --- .kiro/specs/application-profile/brief.md | 117 +++++++++++-------- .kiro/specs/job-config-merge/brief.md | 8 +- .kiro/specs/job-config-merge/requirements.md | 2 +- .kiro/steering/job-config-layering.md | 99 +++++++++++----- 4 files changed, 145 insertions(+), 81 deletions(-) diff --git a/.kiro/specs/application-profile/brief.md b/.kiro/specs/application-profile/brief.md index 8c6df0cf..1afbf727 100644 --- a/.kiro/specs/application-profile/brief.md +++ b/.kiro/specs/application-profile/brief.md @@ -78,8 +78,12 @@ consumed by `service-supervision` (worker probes and restarts), - **Additive rule:** the job may add to the profile's values, never change or remove them. A conflict is rejected at submit with an error naming the profile value, never resolved silently. - - **Convenience defaults** (tuning such as `rusage[mem=...]`, - `span[ptile=...]`, task count, extra env) go into **presets**, handled + - **Exception: default reservations.** The profile's structured + `rusage` entries hold per-resource defaults the job may override (see + "Essentials and the additive rule"), so REST/portal users without + presets still reserve sensibly. + - **Other convenience defaults** (tuning such as `span[ptile=...]`, task + count, extra env, different `rusage` values) go into **presets**, handled fully by the client (`xcecli-presets`). The server only shares admin-deployed presets read-only; presets never reach the controller, WAL or scheduler and carry no privilege. @@ -316,7 +320,11 @@ fits XCE's LSF-style job model. description: "OpenFOAM, MPI" mode: direct launch_type: openmpi - resreq: "select[ib] rusage[license_foam=1]" # essentials only + resreq: + select: "ib" # binding + rusage: + - { name: license_foam, per: job, default: 1 } # default reservation + - { name: mem, default: 4000 } # per task by default tasks: { min: 32, max: 512 } limits: { runtime: 24h, memory: 128G } # maximums env: { FOAM_INST_DIR: /opt/openfoam } @@ -330,8 +338,9 @@ fits XCE's LSF-style job model. job_pre_post_exec: forbid ``` - Tuning such as `rusage[mem=4000] span[ptile=32]` and a task count of 64 - is not in the profile; the admin publishes it as a site preset named + Default reservations live in the profile's `rusage` entries (the job may + override them); other tuning such as `span[ptile=32]` and a task count of + 64 is not in the profile: the admin publishes it as a site preset named `openfoam`. ### Essentials and the additive rule @@ -341,7 +350,7 @@ How each profile attribute combines with the job's own value (owned by | Profile attribute | Holds | The job may | |---|---|---| | resreq `select` | required terms (`ib`, GPU model, OS) | add terms (ANDed) | -| resreq `rusage` | per-resource defaults (`license=1`, `mem=4000/task`) | add resources; override a profile resource's value (job wins, see below) | +| resreq `rusage` | entries with optional `per:` and `default` | add resources; override a profile default (job's term wins; unit must match, see below) | | resreq `span` / `same` / `order` | only if essential (e.g. a code that runs on one host only) | set them if the profile does not; otherwise rejected | | task range | `min` / `max` | choose within it; no `-n` means `min` | | env | variables the application needs | add others; not redefine these | @@ -349,36 +358,57 @@ How each profile attribute combines with the job's own value (owned by | limits | maximums | ask for less (checked like a ceiling) | | pre/post-exec | outer layer, always runs | add its own inner layer if the profile allows | -**`rusage`: union, the job's value wins** (user decision 2026-10-12). The -effective `rusage[]` is the union of the profile's and the job's resources; -for a resource in both, the job's value wins, in either direction. -- The **per-type** (`/task`, `/job`, `/host`) is the profile's: a job term - without one inherits it (`mem=4000/task` + `mem=16000` gives - `mem=16000/task`); a job term with a *different* per-type is rejected at - submit ("resource mem is per task in profile openfoam"), never silently - reinterpreted (16000/job read as per task would multiply by the task - count). -- Other modifiers: a `duration` / `decay` the job writes replaces the - profile's; one it omits is inherited. -- `name=0` means "reserve none" (the only way to drop a profile resource); - requirements must confirm the scheduler treats a 0 reservation that way. -- Consequence: `rusage` is the one profile section that is a per-resource - **default**, not a binding essential; a profile cannot enforce a minimum - reservation. Hard protection of shared resources comes from `limits.yaml` - and project quotas; later, a "shared/counted" marker on the resource in - `resources.yaml` could forbid lowering below the profile value (licences), - if under-reservation proves a problem. +**Structured profile resreq; `rusage` defaults, the job's term wins** (user +decisions 2026-10-12). A profile's resreq is not one string but separate +parts, so each part's rule is visible from its shape: + +```yaml +resreq: + select: "ib && os == rhel9" # binding: ANDed with the job's select + rusage: # defaults: the job may override per resource + - { name: license_foam, per: job, default: 1 } + - { name: mem, default: 4000 } # per: optional, defaults to task + - { name: gpu } # unit only: nothing reserved by default + span: "ptile=32" # binding if set + same: [switch] + order: "r15s" +``` + +| Part | Merge with the job's `-R` (an ordinary resreq string, parsed into terms) | +|---|---| +| `select` | ANDed with the job's | +| `rusage` entry with `default` | reserved at that value unless the job names the resource; then the job's term wins **whole** (value and `duration`/`decay`), in either direction | +| `rusage` entry without `default` | nothing reserved unless the job asks; it only fixes the unit | +| `per:` (optional, default `task`) | the unit for that resource: a job term without a per-type gets it; a job term with a **different** per-type is rejected at submit ("resource mem is per task in profile openfoam"), never silently reinterpreted (16000/job read as per task multiplies by the task count) | +| a resource only the job names | the job's term as written | +| `span` / `same` / `order` | the profile's if set (binding), otherwise the job's | + +- `per:` defaults to `task`, today's behaviour: `PER_DEFAULT` takes the + caller's per-type (`lib/resreq.c:842`) and the controller passes + `PER_TASK` (`controller/job.c:3909`); `resources.yaml` has no per-type + field. A per-resource default there may come later. +- `name=0` in the job means "reserve none" (the only way to drop a profile + default); requirements must confirm the scheduler treats a 0 reservation + that way. +- `xceadmin` validates each `rusage` entry against `resources.yaml` (name + exists) instead of parsing a string; later per-resource fields + (`duration`, a `min:` floor for counted resources such as licences, if + under-reservation proves a problem) slot into the entry without new + grammar. +- Consequence: `rusage` is the one profile part that holds **defaults**; a + profile cannot enforce a minimum reservation. Hard protection of shared + resources comes from `limits.yaml` and project quotas. - Side effects to handle in requirements: measured resources (memory) are charged `max(reserved, actual)` once usage is reported (`resource-accounting.md`), so lowering them mostly self-corrects, but counted resources (licences) do not; xcecli and `--dry-run` warn when a - job lowers a profile value, and when a preset restates a profile value - (it would pin the job against later profile changes); job details label - each term's origin (profile or job); queue ceilings and project quotas - check the merged value, and the error names the job's override; a - profile `rusage` change changes the content hash, so in-flight dispatches - re-pend and pending jobs re-merge even if they override that resource - (accepted). + job lowers a profile default, and when a preset restates one (it would + pin the job against later profile changes); job details label each + `rusage` term as profile default or job override; queue ceilings and + project quotas check the merged value and the error names the job's + override; a profile `rusage` change changes the content hash, so + in-flight dispatches re-pend and pending jobs re-merge even if they + override that resource (accepted). - A job with no `-R` gets the profile's essentials only; the scheduler's existing empty-resreq default (`type == any`) still applies to what neither sets. @@ -395,20 +425,15 @@ for a resource in both, the job's value wins, in either direction. profile sets them they bind; otherwise the service sets them. - Other later sections (I/O claims, relationship templates, `container:`) state in their own specs how they follow the additive rule. -- **Compound and alternative resreqs** (decided 2026-10-12, user). Parts - are unnamed today, so a profile term can apply only where its meaning per - part is clear: - - the profile's `select` is ANDed into every part (and every `||` - alternative): a host property the application needs applies to every - host it runs on; - - a profile `rusage` resource that is **per task** is merged into every - part with the `rusage` rule above; - - a profile `rusage` resource that is per job, per host or `PER_DEFAULT`, - and a profile `span` / `same` / `order`, make a compound or alternative - job resreq **rejected** with an error naming the section, until PR-15 - named parts let a profile target a part by role; - - admin guidance (not enforceable): no part-specific property (e.g. a GPU - model only one role needs) in a profile's `select`. +- **Compound and alternative resreqs** (decided 2026-10-12, user, + simplified). Parts are unnamed today: only the profile's `select` applies, + ANDed into every part and every `||` alternative. If the profile has any + `rusage` entry with a `default`, or sets `span` / `same` / `order`, a + compound or alternative job resreq is **rejected** with an error naming + the part, until PR-15 named parts let a profile target a part by role. + `rusage` entries without a default only fix a unit and do not block. + Admin guidance (not enforceable): no part-specific property (e.g. a GPU + model only one role needs) in a profile's `select`. ### Environment of the outer layer and reserved names Decided 2026-10-12 (user), replacing the idea of an env denylist: diff --git a/.kiro/specs/job-config-merge/brief.md b/.kiro/specs/job-config-merge/brief.md index 37b4b441..a0b809d8 100644 --- a/.kiro/specs/job-config-merge/brief.md +++ b/.kiro/specs/job-config-merge/brief.md @@ -53,9 +53,11 @@ apiserver and display would disagree about what a job actually asked for. - **Addition (job vs. profile)** (user decision 2026-10-12, replacing the *default*/*fixed* markers): a profile holds only essentials and every profile value binds, except `rusage`: union of profile and job - resources, the job's value wins for a resource in both, the profile's - per-type applies (an explicitly different per-type is rejected), `0` - means none; otherwise the job may add (more `select` terms, `span`/`same`/`order` the profile does not + resources, the job's term wins whole for a resource in both, the profile + entry's `per:` (optional, default `task`) fixes the unit (an explicitly + different per-type is rejected), `0` means none; profile resreq is + structured (`select`, `rusage` entries, `span`, `same`, `order`), see the + at-a-glance table in `job-config-layering.md`; otherwise the job may add (more `select` terms, `span`/`same`/`order` the profile does not set, env variables, lower limits, an inner pre/post-exec if allowed) but never change or remove them; conflicts are rejected at submit. Table: `job-config-layering.md`. This replaces LSF's per-section merge diff --git a/.kiro/specs/job-config-merge/requirements.md b/.kiro/specs/job-config-merge/requirements.md index 5db371bf..72f78efc 100644 --- a/.kiro/specs/job-config-merge/requirements.md +++ b/.kiro/specs/job-config-merge/requirements.md @@ -14,7 +14,7 @@ Discovery brief: `.kiro/specs/job-config-merge/brief.md`. Governing rules: `.kir **What should change:** - One merge function in `src/lib`, shared by controller, scheduler and (through cgo) apiserver. Its inputs are the job's own values, the application profile, the queue's policy and the project; its output is the job's effective values: resreq (select / rusage / span / same / order), task range, execution limits, pre/post-exec, env. - **One owner per attribute.** The queue owns scheduling policy, the application profile owns what the application needs and how it runs, and the project owns resource ownership. Only two interactions exist: - - **Addition** (2026-10-12, replacing *default*/*fixed* markers): a profile holds only essentials and every profile value binds, except `rusage`: the union of profile and job resources, the job's value winning for a resource in both, with the profile's per-type (`/task`, `/job`, `/host`; an explicitly different one is rejected) and `0` meaning none. Otherwise the job may add to them (more `select` terms, `span`/`same`/`order` the profile does not set, env variables, lower limits) but never change or remove them; a conflict is rejected at submit. Convenience defaults are client-side presets and never reach the merge. + - **Addition** (2026-10-12, replacing *default*/*fixed* markers): a profile holds only essentials and every profile value binds, except `rusage`: the union of profile and job resources, the job's term winning whole for a resource in both, with the unit from the profile entry's optional `per:` (default `task`; an explicitly different per-type is rejected) and `0` meaning none. The profile resreq is structured (`select`, `rusage` entries with `per:`/`default`, `span`, `same`, `order`); rules at a glance: `job-config-layering.md`. Otherwise the job may add to them (more `select` terms, `span`/`same`/`order` the profile does not set, env variables, lower limits) but never change or remove them; a conflict is rejected at submit. Convenience defaults are client-side presets and never reach the merge. - **Ceiling:** queue ceilings and project quotas never change a value. A value that can never be satisfied is rejected at submit; a quota that is only full right now makes the job pend with a reason naming the queue or project. - **Live merge.** The WAL keeps only the job's own values plus its queue, profile and project names. - The controller (authoritative) and the apiserver (early) run the merge at submit to validate. diff --git a/.kiro/steering/job-config-layering.md b/.kiro/steering/job-config-layering.md index 036ba14c..3f98dc78 100644 --- a/.kiro/steering/job-config-layering.md +++ b/.kiro/steering/job-config-layering.md @@ -20,7 +20,7 @@ XCE instead gives **every attribute exactly one owning level**. | Level | Owns | Never owns | |---|---|---| | **Queue**: scheduling policy | priority and ordering, host partition, dispatch/run windows, preemption policy, policy ceilings (max run time, max tasks, max hosts, ...) | resource requirement, anything about how a job executes (pre/post-exec, env, launch, limits enforced on the job) | -| **Application profile**: what the application needs and how it runs (essentials only, 2026-10-12) | resreq (select / rusage / span / same / order), task range (min / default / max), execution mode and options, launch type, env, **pre/post-exec**, execution limits (mem, runtime, CPU, tasks) | priority, queue choice, quotas | +| **Application profile**: what the application needs and how it runs (essentials plus default reservations, 2026-10-12) | resreq (select / rusage / span / same / order), task range (min / default / max), execution mode and options, launch type, env, **pre/post-exec**, execution limits (mem, runtime, CPU, tasks) | priority, queue choice, quotas | | **Project**: resource ownership | membership, quotas / limits, later guarantees, accounting | how a job runs or is ordered | | **Job** | its own values, only for attributes its profile allows | n/a | @@ -28,6 +28,21 @@ Access to each is separate: queue and profile by RBAC permission, project by membership (`project-config`). Any allowed queue combines with any allowed profile and project. +## Merge rules at a glance +One line per part; details below. + +| Part | Rule | +|---|---| +| `select` | profile and job ANDed | +| `rusage` | union; the job's term wins for a resource in both; the unit (`per:`, default `task`) is the profile's and must match | +| `span` / `same` / `order` | the profile's if set, otherwise the job's | +| task range | the job chooses within the profile's `min` / `max` | +| limits | the profile's are maximums; the job may lower | +| env | the job adds; cannot redefine the profile's | +| mode, launch type, job starter, outer pre/post-exec | the profile's | +| compound / alternative job resreq | only the profile's `select` applies; a profile `rusage` default or `span`/`same`/`order` → rejected (until PR-15) | +| queue ceilings, project quotas | check the merged result; never change it | + ## The only two interactions 1. **Addition: job vs. application profile** (user decision 2026-10-12, @@ -40,7 +55,7 @@ profile and project. | Profile attribute | The job may | |---|---| | resreq `select` (required terms) | add terms (ANDed) | - | resreq `rusage` (per-resource defaults) | add resources; override a profile value (job wins; per-type is the profile's) | + | resreq `rusage` (entries: optional `per:`, `default`) | add resources; override a profile default (job's term wins; unit must match) | | resreq `span` / `same` / `order` | set them if the profile does not | | task range `min` / `max` | choose within it; no `-n` means `min` | | env | add variables; not redefine the profile's | @@ -50,36 +65,57 @@ profile and project. | `service:` probes, restart policy, ports / replica range / spread, rollout | nothing / choose within / set if the profile does not | The one exception to "every profile value binds": - **`rusage`: union, the job's value wins** (user decision 2026-10-12). The - effective `rusage[]` is the union of the profile's and the job's resources; - for a resource in both, the job's value wins, in either direction. - - The **per-type** (`/task`, `/job`, `/host`) is the profile's: a job term - without one inherits it (`mem=4000/task` + `mem=16000` gives - `mem=16000/task`); a job term with a *different* per-type is rejected at - submit ("resource mem is per task in profile openfoam"), never silently - reinterpreted (16000/job read as per task would multiply by the task - count). - - Other modifiers: a `duration` / `decay` the job writes replaces the - profile's; one it omits is inherited. - - `name=0` means "reserve none" (the only way to drop a profile resource); - requirements must confirm the scheduler treats a 0 reservation that way. - - Consequence: `rusage` is the one profile section that is a per-resource - **default**, not a binding essential; a profile cannot enforce a minimum - reservation. Hard protection of shared resources comes from `limits.yaml` - and project quotas; later, a "shared/counted" marker on the resource in - `resources.yaml` could forbid lowering below the profile value (licences), - if under-reservation proves a problem. + **Structured profile resreq; `rusage` defaults, the job's term wins** (user + decisions 2026-10-12). A profile's resreq is not one string but separate + parts, so each part's rule is visible from its shape: + + ```yaml + resreq: + select: "ib && os == rhel9" # binding: ANDed with the job's select + rusage: # defaults: the job may override per resource + - { name: license_foam, per: job, default: 1 } + - { name: mem, default: 4000 } # per: optional, defaults to task + - { name: gpu } # unit only: nothing reserved by default + span: "ptile=32" # binding if set + same: [switch] + order: "r15s" + ``` + + | Part | Merge with the job's `-R` (an ordinary resreq string, parsed into terms) | + |---|---| + | `select` | ANDed with the job's | + | `rusage` entry with `default` | reserved at that value unless the job names the resource; then the job's term wins **whole** (value and `duration`/`decay`), in either direction | + | `rusage` entry without `default` | nothing reserved unless the job asks; it only fixes the unit | + | `per:` (optional, default `task`) | the unit for that resource: a job term without a per-type gets it; a job term with a **different** per-type is rejected at submit ("resource mem is per task in profile openfoam"), never silently reinterpreted (16000/job read as per task multiplies by the task count) | + | a resource only the job names | the job's term as written | + | `span` / `same` / `order` | the profile's if set (binding), otherwise the job's | + + - `per:` defaults to `task`, today's behaviour: `PER_DEFAULT` takes the + caller's per-type (`lib/resreq.c:842`) and the controller passes + `PER_TASK` (`controller/job.c:3909`); `resources.yaml` has no per-type + field. A per-resource default there may come later. + - `name=0` in the job means "reserve none" (the only way to drop a profile + default); requirements must confirm the scheduler treats a 0 reservation + that way. + - `xceadmin` validates each `rusage` entry against `resources.yaml` (name + exists) instead of parsing a string; later per-resource fields + (`duration`, a `min:` floor for counted resources such as licences, if + under-reservation proves a problem) slot into the entry without new + grammar. + - Consequence: `rusage` is the one profile part that holds **defaults**; a + profile cannot enforce a minimum reservation. Hard protection of shared + resources comes from `limits.yaml` and project quotas. - Side effects to handle in requirements: measured resources (memory) are charged `max(reserved, actual)` once usage is reported (`resource-accounting.md`), so lowering them mostly self-corrects, but counted resources (licences) do not; xcecli and `--dry-run` warn when a - job lowers a profile value, and when a preset restates a profile value - (it would pin the job against later profile changes); job details label - each term's origin (profile or job); queue ceilings and project quotas - check the merged value, and the error names the job's override; a - profile `rusage` change changes the content hash, so in-flight dispatches - re-pend and pending jobs re-merge even if they override that resource - (accepted). + job lowers a profile default, and when a preset restates one (it would + pin the job against later profile changes); job details label each + `rusage` term as profile default or job override; queue ceilings and + project quotas check the merged value and the error names the job's + override; a profile `rusage` change changes the content hash, so + in-flight dispatches re-pend and pending jobs re-merge even if they + override that resource (accepted). 2. **Ceiling: queue or project vs. the job's effective values.** A queue ceiling or project quota never changes the job's values, it only checks @@ -94,9 +130,10 @@ ordering of queue against profile settings, no compound/alternative merge stages. ## Convenience defaults are presets, not a level -Tuning that most jobs of an application want but none must have -(`rusage[mem=...]`, `span[ptile=...]`, a task count, extra env) is not in -the profile. It goes into **presets**, handled fully by the client +Apart from default reservations (the profile's `rusage` entries), tuning +that most jobs of an application want but none must have +(`span[ptile=...]`, a task count, extra env, different `rusage` values) is +not in the profile. It goes into **presets**, handled fully by the client (`xcecli-presets`): a preset expands into job options before submit, so the server only ever sees the job's own values. Presets are therefore not a configuration level, own no attribute and carry no privilege. The apiserver