lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(spec): structured profile resreq; per: optional (default task); merge rules at a glance

Browse source 
- profile resreq split into select / rusage entries / span / same / order;
  rusage entries carry optional per: (default task, today's PER_DEFAULT
  behaviour) and optional default
- job's rusage term wins whole for a resource in both; unit must match;
  no modifier inheritance
- compound/alternative: only the profile select applies; any profile
  rusage default or span/same/order rejects until PR-15
- job-config-layering: merge rules at a glance table; profile = essentials
  plus default reservations

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-13 02:49:53 +00:00
parent 8f3a811a43
commit 1f9aaae050
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
4 changed files with 145 additions and 81 deletions
Download patch file Download diff file Expand all files Collapse all files

117
.kiro/specs/application-profile/brief.md
View file

@ -78,8 +78,12 @@ consumed by `service-supervision` (worker probes and restarts),
- **Additive rule:** the job may add to the profile's values, never change - **Additive rule:** the job may add to the profile's values, never change
or remove them. A conflict is rejected at submit with an error naming the or remove them. A conflict is rejected at submit with an error naming the
profile value, never resolved silently. profile value, never resolved silently.
- **Convenience defaults** (tuning such as `rusage[mem=...]`, - **Exception: default reservations.** The profile's structured
`span[ptile=...]`, task count, extra env) go into **presets**, handled `rusage` entries hold per-resource defaults the job may override (see
"Essentials and the additive rule"), so REST/portal users without
presets still reserve sensibly.
- **Other convenience defaults** (tuning such as `span[ptile=...]`, task
count, extra env, different `rusage` values) go into **presets**, handled
fully by the client (`xcecli-presets`). The server only shares fully by the client (`xcecli-presets`). The server only shares
admin-deployed presets read-only; presets never reach the controller, admin-deployed presets read-only; presets never reach the controller,
WAL or scheduler and carry no privilege. WAL or scheduler and carry no privilege.
@ -316,7 +320,11 @@ fits XCE's LSF-style job model.
description: "OpenFOAM, MPI" description: "OpenFOAM, MPI"
mode: direct mode: direct
launch_type: openmpi launch_type: openmpi
resreq: "select[ib] rusage[license_foam=1]" # essentials only resreq:
select: "ib" # binding
rusage:
- { name: license_foam, per: job, default: 1 } # default reservation
- { name: mem, default: 4000 } # per task by default
tasks: { min: 32, max: 512 } tasks: { min: 32, max: 512 }
limits: { runtime: 24h, memory: 128G } # maximums limits: { runtime: 24h, memory: 128G } # maximums
env: { FOAM_INST_DIR: /opt/openfoam } env: { FOAM_INST_DIR: /opt/openfoam }
@ -330,8 +338,9 @@ fits XCE's LSF-style job model.
job_pre_post_exec: forbid job_pre_post_exec: forbid
``` ```
Tuning such as `rusage[mem=4000] span[ptile=32]` and a task count of 64 Default reservations live in the profile's `rusage` entries (the job may
is not in the profile; the admin publishes it as a site preset named override them); other tuning such as `span[ptile=32]` and a task count of
64 is not in the profile: the admin publishes it as a site preset named
`openfoam`. `openfoam`.
### Essentials and the additive rule ### Essentials and the additive rule
@ -341,7 +350,7 @@ How each profile attribute combines with the job's own value (owned by
| Profile attribute | Holds | The job may | | Profile attribute | Holds | The job may |
|---|---|---| |---|---|---|
| resreq `select` | required terms (`ib`, GPU model, OS) | add terms (ANDed) | | resreq `select` | required terms (`ib`, GPU model, OS) | add terms (ANDed) |
| resreq `rusage` | per-resource defaults (`license=1`, `mem=4000/task`) | add resources; override a profile resource's value (job wins, see below) | | resreq `rusage` | entries with optional `per:` and `default` | add resources; override a profile default (job's term wins; unit must match, see below) |
| resreq `span` / `same` / `order` | only if essential (e.g. a code that runs on one host only) | set them if the profile does not; otherwise rejected | | resreq `span` / `same` / `order` | only if essential (e.g. a code that runs on one host only) | set them if the profile does not; otherwise rejected |
| task range | `min` / `max` | choose within it; no `-n` means `min` | | task range | `min` / `max` | choose within it; no `-n` means `min` |
| env | variables the application needs | add others; not redefine these | | env | variables the application needs | add others; not redefine these |
@ -349,36 +358,57 @@ How each profile attribute combines with the job's own value (owned by
| limits | maximums | ask for less (checked like a ceiling) | | limits | maximums | ask for less (checked like a ceiling) |
| pre/post-exec | outer layer, always runs | add its own inner layer if the profile allows | | pre/post-exec | outer layer, always runs | add its own inner layer if the profile allows |
**`rusage`: union, the job's value wins** (user decision 2026-10-12). The **Structured profile resreq; `rusage` defaults, the job's term wins** (user
effective `rusage[]` is the union of the profile's and the job's resources; decisions 2026-10-12). A profile's resreq is not one string but separate
for a resource in both, the job's value wins, in either direction. parts, so each part's rule is visible from its shape:
- The **per-type** (`/task`, `/job`, `/host`) is the profile's: a job term
without one inherits it (`mem=4000/task` + `mem=16000` gives ```yaml
`mem=16000/task`); a job term with a *different* per-type is rejected at resreq:
submit ("resource mem is per task in profile openfoam"), never silently select: "ib && os == rhel9" # binding: ANDed with the job's select
reinterpreted (16000/job read as per task would multiply by the task rusage: # defaults: the job may override per resource
count). - { name: license_foam, per: job, default: 1 }
- Other modifiers: a `duration` / `decay` the job writes replaces the - { name: mem, default: 4000 } # per: optional, defaults to task
profile's; one it omits is inherited. - { name: gpu } # unit only: nothing reserved by default
- `name=0` means "reserve none" (the only way to drop a profile resource); span: "ptile=32" # binding if set
requirements must confirm the scheduler treats a 0 reservation that way. same: [switch]
- Consequence: `rusage` is the one profile section that is a per-resource order: "r15s"
**default**, not a binding essential; a profile cannot enforce a minimum ```
reservation. Hard protection of shared resources comes from `limits.yaml`
and project quotas; later, a "shared/counted" marker on the resource in | Part | Merge with the job's `-R` (an ordinary resreq string, parsed into terms) |
`resources.yaml` could forbid lowering below the profile value (licences), |---|---|
if under-reservation proves a problem. | `select` | ANDed with the job's |
| `rusage` entry with `default` | reserved at that value unless the job names the resource; then the job's term wins **whole** (value and `duration`/`decay`), in either direction |
| `rusage` entry without `default` | nothing reserved unless the job asks; it only fixes the unit |
| `per:` (optional, default `task`) | the unit for that resource: a job term without a per-type gets it; a job term with a **different** per-type is rejected at submit ("resource mem is per task in profile openfoam"), never silently reinterpreted (16000/job read as per task multiplies by the task count) |
| a resource only the job names | the job's term as written |
| `span` / `same` / `order` | the profile's if set (binding), otherwise the job's |
- `per:` defaults to `task`, today's behaviour: `PER_DEFAULT` takes the
caller's per-type (`lib/resreq.c:842`) and the controller passes
`PER_TASK` (`controller/job.c:3909`); `resources.yaml` has no per-type
field. A per-resource default there may come later.
- `name=0` in the job means "reserve none" (the only way to drop a profile
default); requirements must confirm the scheduler treats a 0 reservation
that way.
- `xceadmin` validates each `rusage` entry against `resources.yaml` (name
exists) instead of parsing a string; later per-resource fields
(`duration`, a `min:` floor for counted resources such as licences, if
under-reservation proves a problem) slot into the entry without new
grammar.
- Consequence: `rusage` is the one profile part that holds **defaults**; a
profile cannot enforce a minimum reservation. Hard protection of shared
resources comes from `limits.yaml` and project quotas.
- Side effects to handle in requirements: measured resources (memory) are - Side effects to handle in requirements: measured resources (memory) are
charged `max(reserved, actual)` once usage is reported charged `max(reserved, actual)` once usage is reported
(`resource-accounting.md`), so lowering them mostly self-corrects, but (`resource-accounting.md`), so lowering them mostly self-corrects, but
counted resources (licences) do not; xcecli and `--dry-run` warn when a counted resources (licences) do not; xcecli and `--dry-run` warn when a
job lowers a profile value, and when a preset restates a profile value job lowers a profile default, and when a preset restates one (it would
(it would pin the job against later profile changes); job details label pin the job against later profile changes); job details label each
each term's origin (profile or job); queue ceilings and project quotas `rusage` term as profile default or job override; queue ceilings and
check the merged value, and the error names the job's override; a project quotas check the merged value and the error names the job's
profile `rusage` change changes the content hash, so in-flight dispatches override; a profile `rusage` change changes the content hash, so
re-pend and pending jobs re-merge even if they override that resource in-flight dispatches re-pend and pending jobs re-merge even if they
(accepted). override that resource (accepted).
- A job with no `-R` gets the profile's essentials only; the scheduler's - A job with no `-R` gets the profile's essentials only; the scheduler's
existing empty-resreq default (`type == any`) still applies to what existing empty-resreq default (`type == any`) still applies to what
neither sets. neither sets.
@ -395,20 +425,15 @@ for a resource in both, the job's value wins, in either direction.
profile sets them they bind; otherwise the service sets them. profile sets them they bind; otherwise the service sets them.
- Other later sections (I/O claims, relationship templates, `container:`) - Other later sections (I/O claims, relationship templates, `container:`)
state in their own specs how they follow the additive rule. state in their own specs how they follow the additive rule.
- **Compound and alternative resreqs** (decided 2026-10-12, user). Parts - **Compound and alternative resreqs** (decided 2026-10-12, user,
are unnamed today, so a profile term can apply only where its meaning per simplified). Parts are unnamed today: only the profile's `select` applies,
part is clear: ANDed into every part and every `||` alternative. If the profile has any
- the profile's `select` is ANDed into every part (and every `||` `rusage` entry with a `default`, or sets `span` / `same` / `order`, a
alternative): a host property the application needs applies to every compound or alternative job resreq is **rejected** with an error naming
host it runs on; the part, until PR-15 named parts let a profile target a part by role.
- a profile `rusage` resource that is **per task** is merged into every `rusage` entries without a default only fix a unit and do not block.
part with the `rusage` rule above; Admin guidance (not enforceable): no part-specific property (e.g. a GPU
- a profile `rusage` resource that is per job, per host or `PER_DEFAULT`, model only one role needs) in a profile's `select`.
and a profile `span` / `same` / `order`, make a compound or alternative
job resreq **rejected** with an error naming the section, until PR-15
named parts let a profile target a part by role;
- admin guidance (not enforceable): no part-specific property (e.g. a GPU
model only one role needs) in a profile's `select`.
### Environment of the outer layer and reserved names ### Environment of the outer layer and reserved names
Decided 2026-10-12 (user), replacing the idea of an env denylist: Decided 2026-10-12 (user), replacing the idea of an env denylist:

8
.kiro/specs/job-config-merge/brief.md
View file

@ -53,9 +53,11 @@ apiserver and display would disagree about what a job actually asked for.
- **Addition (job vs. profile)** (user decision 2026-10-12, replacing the - **Addition (job vs. profile)** (user decision 2026-10-12, replacing the
*default*/*fixed* markers): a profile holds only essentials and every *default*/*fixed* markers): a profile holds only essentials and every
profile value binds, except `rusage`: union of profile and job profile value binds, except `rusage`: union of profile and job
resources, the job's value wins for a resource in both, the profile's resources, the job's term wins whole for a resource in both, the profile
per-type applies (an explicitly different per-type is rejected), `0` entry's `per:` (optional, default `task`) fixes the unit (an explicitly
means none; otherwise the job may add (more `select` terms, `span`/`same`/`order` the profile does not different per-type is rejected), `0` means none; profile resreq is
structured (`select`, `rusage` entries, `span`, `same`, `order`), see the
at-a-glance table in `job-config-layering.md`; otherwise the job may add (more `select` terms, `span`/`same`/`order` the profile does not
set, env variables, lower limits, an inner pre/post-exec if allowed) set, env variables, lower limits, an inner pre/post-exec if allowed)
but never change or remove them; conflicts are rejected at submit. but never change or remove them; conflicts are rejected at submit.
Table: `job-config-layering.md`. This replaces LSF's per-section merge Table: `job-config-layering.md`. This replaces LSF's per-section merge

2
.kiro/specs/job-config-merge/requirements.md
View file

@ -14,7 +14,7 @@ Discovery brief: `.kiro/specs/job-config-merge/brief.md`. Governing rules: `.kir
**What should change:** **What should change:**
- One merge function in `src/lib`, shared by controller, scheduler and (through cgo) apiserver. Its inputs are the job's own values, the application profile, the queue's policy and the project; its output is the job's effective values: resreq (select / rusage / span / same / order), task range, execution limits, pre/post-exec, env. - One merge function in `src/lib`, shared by controller, scheduler and (through cgo) apiserver. Its inputs are the job's own values, the application profile, the queue's policy and the project; its output is the job's effective values: resreq (select / rusage / span / same / order), task range, execution limits, pre/post-exec, env.
- **One owner per attribute.** The queue owns scheduling policy, the application profile owns what the application needs and how it runs, and the project owns resource ownership. Only two interactions exist: - **One owner per attribute.** The queue owns scheduling policy, the application profile owns what the application needs and how it runs, and the project owns resource ownership. Only two interactions exist:
- **Addition** (2026-10-12, replacing *default*/*fixed* markers): a profile holds only essentials and every profile value binds, except `rusage`: the union of profile and job resources, the job's value winning for a resource in both, with the profile's per-type (`/task`, `/job`, `/host`; an explicitly different one is rejected) and `0` meaning none. Otherwise the job may add to them (more `select` terms, `span`/`same`/`order` the profile does not set, env variables, lower limits) but never change or remove them; a conflict is rejected at submit. Convenience defaults are client-side presets and never reach the merge. - **Addition** (2026-10-12, replacing *default*/*fixed* markers): a profile holds only essentials and every profile value binds, except `rusage`: the union of profile and job resources, the job's term winning whole for a resource in both, with the unit from the profile entry's optional `per:` (default `task`; an explicitly different per-type is rejected) and `0` meaning none. The profile resreq is structured (`select`, `rusage` entries with `per:`/`default`, `span`, `same`, `order`); rules at a glance: `job-config-layering.md`. Otherwise the job may add to them (more `select` terms, `span`/`same`/`order` the profile does not set, env variables, lower limits) but never change or remove them; a conflict is rejected at submit. Convenience defaults are client-side presets and never reach the merge.
- **Ceiling:** queue ceilings and project quotas never change a value. A value that can never be satisfied is rejected at submit; a quota that is only full right now makes the job pend with a reason naming the queue or project. - **Ceiling:** queue ceilings and project quotas never change a value. A value that can never be satisfied is rejected at submit; a quota that is only full right now makes the job pend with a reason naming the queue or project.
- **Live merge.** The WAL keeps only the job's own values plus its queue, profile and project names. - **Live merge.** The WAL keeps only the job's own values plus its queue, profile and project names.
- The controller (authoritative) and the apiserver (early) run the merge at submit to validate. - The controller (authoritative) and the apiserver (early) run the merge at submit to validate.

99
.kiro/steering/job-config-layering.md
View file

@ -20,7 +20,7 @@ XCE instead gives **every attribute exactly one owning level**.
| Level | Owns | Never owns | | Level | Owns | Never owns |
|---|---|---| |---|---|---|
| **Queue**: scheduling policy | priority and ordering, host partition, dispatch/run windows, preemption policy, policy ceilings (max run time, max tasks, max hosts, ...) | resource requirement, anything about how a job executes (pre/post-exec, env, launch, limits enforced on the job) | | **Queue**: scheduling policy | priority and ordering, host partition, dispatch/run windows, preemption policy, policy ceilings (max run time, max tasks, max hosts, ...) | resource requirement, anything about how a job executes (pre/post-exec, env, launch, limits enforced on the job) |
| **Application profile**: what the application needs and how it runs (essentials only, 2026-10-12) | resreq (select / rusage / span / same / order), task range (min / default / max), execution mode and options, launch type, env, **pre/post-exec**, execution limits (mem, runtime, CPU, tasks) | priority, queue choice, quotas | | **Application profile**: what the application needs and how it runs (essentials plus default reservations, 2026-10-12) | resreq (select / rusage / span / same / order), task range (min / default / max), execution mode and options, launch type, env, **pre/post-exec**, execution limits (mem, runtime, CPU, tasks) | priority, queue choice, quotas |
| **Project**: resource ownership | membership, quotas / limits, later guarantees, accounting | how a job runs or is ordered | | **Project**: resource ownership | membership, quotas / limits, later guarantees, accounting | how a job runs or is ordered |
| **Job** | its own values, only for attributes its profile allows | n/a | | **Job** | its own values, only for attributes its profile allows | n/a |
@ -28,6 +28,21 @@ Access to each is separate: queue and profile by RBAC permission, project by
membership (`project-config`). Any allowed queue combines with any allowed membership (`project-config`). Any allowed queue combines with any allowed
profile and project. profile and project.
## Merge rules at a glance
One line per part; details below.
| Part | Rule |
|---|---|
| `select` | profile and job ANDed |
| `rusage` | union; the job's term wins for a resource in both; the unit (`per:`, default `task`) is the profile's and must match |
| `span` / `same` / `order` | the profile's if set, otherwise the job's |
| task range | the job chooses within the profile's `min` / `max` |
| limits | the profile's are maximums; the job may lower |
| env | the job adds; cannot redefine the profile's |
| mode, launch type, job starter, outer pre/post-exec | the profile's |
| compound / alternative job resreq | only the profile's `select` applies; a profile `rusage` default or `span`/`same`/`order` → rejected (until PR-15) |
| queue ceilings, project quotas | check the merged result; never change it |
## The only two interactions ## The only two interactions
1. **Addition: job vs. application profile** (user decision 2026-10-12, 1. **Addition: job vs. application profile** (user decision 2026-10-12,
@ -40,7 +55,7 @@ profile and project.
| Profile attribute | The job may | | Profile attribute | The job may |
|---|---| |---|---|
| resreq `select` (required terms) | add terms (ANDed) | | resreq `select` (required terms) | add terms (ANDed) |
| resreq `rusage` (per-resource defaults) | add resources; override a profile value (job wins; per-type is the profile's) | | resreq `rusage` (entries: optional `per:`, `default`) | add resources; override a profile default (job's term wins; unit must match) |
| resreq `span` / `same` / `order` | set them if the profile does not | | resreq `span` / `same` / `order` | set them if the profile does not |
| task range `min` / `max` | choose within it; no `-n` means `min` | | task range `min` / `max` | choose within it; no `-n` means `min` |
| env | add variables; not redefine the profile's | | env | add variables; not redefine the profile's |
@ -50,36 +65,57 @@ profile and project.
| `service:` probes, restart policy, ports / replica range / spread, rollout | nothing / choose within / set if the profile does not | | `service:` probes, restart policy, ports / replica range / spread, rollout | nothing / choose within / set if the profile does not |
The one exception to "every profile value binds": The one exception to "every profile value binds":
**`rusage`: union, the job's value wins** (user decision 2026-10-12). The **Structured profile resreq; `rusage` defaults, the job's term wins** (user
effective `rusage[]` is the union of the profile's and the job's resources; decisions 2026-10-12). A profile's resreq is not one string but separate
for a resource in both, the job's value wins, in either direction. parts, so each part's rule is visible from its shape:
- The **per-type** (`/task`, `/job`, `/host`) is the profile's: a job term
without one inherits it (`mem=4000/task` + `mem=16000` gives ```yaml
`mem=16000/task`); a job term with a *different* per-type is rejected at resreq:
submit ("resource mem is per task in profile openfoam"), never silently select: "ib && os == rhel9" # binding: ANDed with the job's select
reinterpreted (16000/job read as per task would multiply by the task rusage: # defaults: the job may override per resource
count). - { name: license_foam, per: job, default: 1 }
- Other modifiers: a `duration` / `decay` the job writes replaces the - { name: mem, default: 4000 } # per: optional, defaults to task
profile's; one it omits is inherited. - { name: gpu } # unit only: nothing reserved by default
- `name=0` means "reserve none" (the only way to drop a profile resource); span: "ptile=32" # binding if set
requirements must confirm the scheduler treats a 0 reservation that way. same: [switch]
- Consequence: `rusage` is the one profile section that is a per-resource order: "r15s"
**default**, not a binding essential; a profile cannot enforce a minimum ```
reservation. Hard protection of shared resources comes from `limits.yaml`
and project quotas; later, a "shared/counted" marker on the resource in | Part | Merge with the job's `-R` (an ordinary resreq string, parsed into terms) |
`resources.yaml` could forbid lowering below the profile value (licences), |---|---|
if under-reservation proves a problem. | `select` | ANDed with the job's |
| `rusage` entry with `default` | reserved at that value unless the job names the resource; then the job's term wins **whole** (value and `duration`/`decay`), in either direction |
| `rusage` entry without `default` | nothing reserved unless the job asks; it only fixes the unit |
| `per:` (optional, default `task`) | the unit for that resource: a job term without a per-type gets it; a job term with a **different** per-type is rejected at submit ("resource mem is per task in profile openfoam"), never silently reinterpreted (16000/job read as per task multiplies by the task count) |
| a resource only the job names | the job's term as written |
| `span` / `same` / `order` | the profile's if set (binding), otherwise the job's |
- `per:` defaults to `task`, today's behaviour: `PER_DEFAULT` takes the
caller's per-type (`lib/resreq.c:842`) and the controller passes
`PER_TASK` (`controller/job.c:3909`); `resources.yaml` has no per-type
field. A per-resource default there may come later.
- `name=0` in the job means "reserve none" (the only way to drop a profile
default); requirements must confirm the scheduler treats a 0 reservation
that way.
- `xceadmin` validates each `rusage` entry against `resources.yaml` (name
exists) instead of parsing a string; later per-resource fields
(`duration`, a `min:` floor for counted resources such as licences, if
under-reservation proves a problem) slot into the entry without new
grammar.
- Consequence: `rusage` is the one profile part that holds **defaults**; a
profile cannot enforce a minimum reservation. Hard protection of shared
resources comes from `limits.yaml` and project quotas.
- Side effects to handle in requirements: measured resources (memory) are - Side effects to handle in requirements: measured resources (memory) are
charged `max(reserved, actual)` once usage is reported charged `max(reserved, actual)` once usage is reported
(`resource-accounting.md`), so lowering them mostly self-corrects, but (`resource-accounting.md`), so lowering them mostly self-corrects, but
counted resources (licences) do not; xcecli and `--dry-run` warn when a counted resources (licences) do not; xcecli and `--dry-run` warn when a
job lowers a profile value, and when a preset restates a profile value job lowers a profile default, and when a preset restates one (it would
(it would pin the job against later profile changes); job details label pin the job against later profile changes); job details label each
each term's origin (profile or job); queue ceilings and project quotas `rusage` term as profile default or job override; queue ceilings and
check the merged value, and the error names the job's override; a project quotas check the merged value and the error names the job's
profile `rusage` change changes the content hash, so in-flight dispatches override; a profile `rusage` change changes the content hash, so
re-pend and pending jobs re-merge even if they override that resource in-flight dispatches re-pend and pending jobs re-merge even if they
(accepted). override that resource (accepted).
2. **Ceiling: queue or project vs. the job's effective values.** A queue 2. **Ceiling: queue or project vs. the job's effective values.** A queue
ceiling or project quota never changes the job's values, it only checks ceiling or project quota never changes the job's values, it only checks
@ -94,9 +130,10 @@ ordering of queue against profile settings, no compound/alternative merge
stages. stages.
## Convenience defaults are presets, not a level ## Convenience defaults are presets, not a level
Tuning that most jobs of an application want but none must have Apart from default reservations (the profile's `rusage` entries), tuning
(`rusage[mem=...]`, `span[ptile=...]`, a task count, extra env) is not in that most jobs of an application want but none must have
the profile. It goes into **presets**, handled fully by the client (`span[ptile=...]`, a task count, extra env, different `rusage` values) is
not in the profile. It goes into **presets**, handled fully by the client
(`xcecli-presets`): a preset expands into job options before submit, so the (`xcecli-presets`): a preset expands into job options before submit, so the
server only ever sees the job's own values. Presets are therefore not a server only ever sees the job's own values. Presets are therefore not a
configuration level, own no attribute and carry no privilege. The apiserver configuration level, own no attribute and carry no privilege. The apiserver