docs(spec): application profiles hold essentials only; additive rule; client-side presets
- profiles hold only what every job of the application must have; every profile value binds and a job may add to it, never change or remove it (replaces the default/fixed markers of 2026-10-09) - convenience defaults move to client-side presets (new xcecli-presets brief): option-level expansion on the client, the apiserver only shares admin-deployed presets read-only, pull copies, export/import - service section stays in the profile: probes, restart policy, ports bind; replica count within the profile range; spread/rollout if unset - job-config-layering, job-config-merge brief/requirements, roadmap and resource-questions updated to the additive rule - app:override and profile inheritance deferred; recommendations target site presets Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
3fa9361ad8
commit
4536d36f47
GPG key ID:
80533536B0EB651C
7 changed files with 267 additions and 115 deletions
|
|
@ -14,11 +14,10 @@ decisions):**
|
|||
(min / default / max, LSF TASKLIMIT), **project** = resource ownership
|
||||
(`project-config`).
|
||||
- Every attribute has one owning level (`.kiro/steering/job-config-layering.md`).
|
||||
How the levels interact is owned by **`job-config-merge`**: each profile
|
||||
attribute is *default* (the job may replace it) or *fixed* (the job may
|
||||
not; for `select` it may only add ANDed terms); queue ceilings and project
|
||||
quotas only check, never change, the job's values. This answers the open
|
||||
question "how resource defaults combine with a job's own resreq" below.
|
||||
How the levels interact is owned by **`job-config-merge`**; queue
|
||||
ceilings and project quotas only check, never change, the job's values.
|
||||
(The job-vs-profile rule of this update, *default*/*fixed* markers, is
|
||||
replaced by the additive rule of 2026-10-12 below.)
|
||||
- **Pre/post-exec moves here from the queue** (queue `pre_exec`/`post_exec`
|
||||
deprecated). The profile's pre/post-exec is admin-defined and always runs
|
||||
as the outer layer; the job's own pre/post-exec is the inner layer, allowed
|
||||
|
|
@ -38,8 +37,9 @@ decisions):**
|
|||
**Update 2026-10-10 (`/kiro-discovery`, long-running services):** a profile
|
||||
may declare **kind `service`** with a service section: restart policy
|
||||
(always / on-failure / never, backoff, give-up), liveness and readiness
|
||||
probes (command / TCP / HTTP), named ports, default replica count, spread
|
||||
rule and rollout settings. The section is defined with this spec and
|
||||
probes (command / TCP / HTTP), named ports, replica range, spread rule and
|
||||
rollout settings (how these follow the additive rule: "Essentials and the
|
||||
additive rule" below). The section is defined with this spec and
|
||||
consumed by `service-supervision` (worker probes and restarts),
|
||||
`service-replica-set` (replicas, scaling, rolling update) and
|
||||
`service-endpoint` (ports, endpoints).
|
||||
|
|
@ -53,23 +53,38 @@ consumed by `service-supervision` (worker probes and restarts),
|
|||
that is not defined is a configuration error everywhere (`xceadmin`, live
|
||||
upgrade, controller start), never a silent fallback. See "Default profile"
|
||||
below.
|
||||
- **This spec owns the profile's attribute schema**: which attributes exist
|
||||
and how each is marked *default* or *fixed*. `job-config-merge` consumes
|
||||
that schema and owns only the merge and live re-evaluation.
|
||||
- **Purpose: convenience first, enforcement where needed** (user, 2026-10-12).
|
||||
A profile saves users from repeating the same resreq and other settings on
|
||||
every job; most values are defaults the job may change. Only what the
|
||||
application cannot run without is fixed. Consequences:
|
||||
- Profile defaults will sometimes not fit, and users will tune them
|
||||
(sizing, placement). The profile gives room to try: default terms, with
|
||||
admin-set ranges for numeric values.
|
||||
- **This spec owns the profile's attribute schema** (which attributes exist
|
||||
and what each holds). `job-config-merge` consumes that schema and owns the
|
||||
merge and live re-evaluation.
|
||||
- **A profile holds only essentials; convenience lives in client-side
|
||||
presets** (user decision, 2026-10-12). "The profile holds what the job must
|
||||
have; the preset holds what the user would like." A profile value is
|
||||
something the application needs or the admin enforces, and it binds every
|
||||
job of that profile. The *default*/*fixed* markers of 2026-10-09 are
|
||||
dropped: there are no markers, and no per-term markers or ranges. See
|
||||
"Essentials and the additive rule" below.
|
||||
- **Additive rule:** the job may add to the profile's values, never change
|
||||
or remove them. A conflict is rejected at submit with an error naming the
|
||||
profile value, never resolved silently.
|
||||
- **Convenience defaults** (tuning such as `rusage[mem=...]`,
|
||||
`span[ptile=...]`, task count, extra env) go into **presets**, handled
|
||||
fully by the client (`xcecli-presets`). The server only shares
|
||||
admin-deployed presets read-only; presets never reach the controller,
|
||||
WAL or scheduler and carry no privilege.
|
||||
- **Rule of thumb for admins:** something every job of the application must
|
||||
have (required host properties, licenses, launch type, mode,
|
||||
pre/post-exec, limits) goes into the profile; an optional variant or
|
||||
tuning shortcut goes into a site preset, typically one with the profile's
|
||||
name.
|
||||
- **Recommendations are advisory** (user decision): usage-based
|
||||
suggestions for a profile's defaults (requested vs. used, OOM kills,
|
||||
runtime by variant) are reported to the admin, who changes
|
||||
`applications.yaml`; a profile never tunes itself. Backlog:
|
||||
`profile-usage-insight`.
|
||||
suggestions (requested vs. used, OOM kills, runtime by variant) are
|
||||
reported to the admin, who changes site presets (or a profile, for an
|
||||
essential); nothing tunes itself. Backlog: `profile-usage-insight`.
|
||||
- Job details show the effective resreq with each term's origin (profile
|
||||
or job), so users see what they are tuning.
|
||||
or job).
|
||||
- Deferred: a privileged `app:override` for one-off exceptions (an admin
|
||||
can modify the pending job instead; recurring variants are presets), and
|
||||
profile inheritance (`base:`), no longer needed for variants.
|
||||
- **Later sections are added by their consumers**: `service:`
|
||||
(`service-supervision`), I/O claims (resource-model PR-6), relationship
|
||||
and compound templates (PR-14/15), `container:` (R6). This spec only makes
|
||||
|
|
@ -82,7 +97,7 @@ scripts, vendor binaries, MPI solvers. Each needs a different way of running:
|
|||
identity (and later in a container or vm);
|
||||
- a particular launch type (serial, `openmpi`, `mpich`);
|
||||
- particular environment and pre/post-exec steps;
|
||||
- sensible resource defaults and limits.
|
||||
- resource requirements it cannot run without, and limits.
|
||||
|
||||
Today every one of these must be set per job by hand, and several (execution
|
||||
mode, sandbox options) cannot be set at all. Many users log in through the
|
||||
|
|
@ -130,21 +145,24 @@ fits XCE's LSF-style job model.
|
|||
first version contains the following (user decision; the format must be
|
||||
extensible for "maybe others"):
|
||||
1. **Execution mode and its options.** `direct` or `sandbox` now, with room for
|
||||
`container`/`vm`. Sandbox options include allowed and default input/output
|
||||
mounts and isolation extras.
|
||||
`container`/`vm`. Sandbox options include the allowed input/output
|
||||
prefixes and isolation extras.
|
||||
2. **Launch type.** Serial, `openmpi` or `mpich`.
|
||||
3. **Environment and pre/post-exec.** Default environment variables and
|
||||
pre/post-exec commands.
|
||||
4. **Resource requirement and limits.** The job's resource requirement
|
||||
(select / rusage / `span[...]` / same / order), its task range (min /
|
||||
default / max), and per-job execution limits (memory, runtime, CPU,
|
||||
tasks), mapped onto the existing worker cgroup limits.
|
||||
3. **Environment and pre/post-exec.** Environment variables the application
|
||||
needs, and the outer pre/post-exec commands.
|
||||
4. **Essential resource requirement and limits.** The resreq terms the
|
||||
application cannot run without (select / rusage / `span[...]` / same /
|
||||
order), its task range (min / max), and per-job execution limits as
|
||||
maximums (memory, runtime, CPU, tasks), mapped onto the existing worker
|
||||
cgroup limits.
|
||||
All of these are essentials under the additive rule; none is a
|
||||
convenience default.
|
||||
- **Users pick a profile at submit** (REST, `xcecli jobs submit --app <name>`,
|
||||
harness, libxce). A **cluster default profile** applies when none is given.
|
||||
- **Limited overrides.** A job may override only the attributes its profile
|
||||
marks as overridable (for example its own input/output directories within the
|
||||
allowed prefixes, or env additions). Anything else is rejected with a clear
|
||||
error.
|
||||
- **Jobs add, never override.** A job may add to its profile's values (more
|
||||
`select` terms, other resources, its own env variables, its own I/O
|
||||
directories within the allowed prefixes, lower limits) but not change or
|
||||
remove any. Anything else is rejected with a clear error.
|
||||
- **Access by RBAC permission.** No new access mechanism; this follows the
|
||||
existing named-instance pattern (user decision, 2026-09-29: fold profile RBAC
|
||||
into this spec). Today, `queue:submit` scoped to `type: queue`,
|
||||
|
|
@ -176,9 +194,9 @@ fits XCE's LSF-style job model.
|
|||
fairshare or preemption).
|
||||
- Any combination of an allowed queue and an allowed profile is valid. Each is
|
||||
governed by its own access rules; for profiles, that is RBAC permission.
|
||||
- Resource defaults in a profile describe what the application needs (its
|
||||
default resource requirement and per-job execution limits). They are not
|
||||
scheduling policy. Queue-level scheduling limits keep applying on top,
|
||||
- Resource values in a profile describe what the application needs (its
|
||||
essential resource requirement and per-job execution limits). They are
|
||||
not scheduling policy. Queue-level scheduling limits keep applying on top,
|
||||
unchanged.
|
||||
- **Live merge** (`job-config-merge`). The job records its profile name and
|
||||
its own values, not the effective values. The scheduler computes the
|
||||
|
|
@ -186,8 +204,8 @@ fits XCE's LSF-style job model.
|
|||
that no longer satisfies its profile pends with a reason, and running jobs
|
||||
are unaffected. Job details show the profile name and the effective mode.
|
||||
- **The apiserver never checks node accounts** (user decision, 2026-09-29).
|
||||
- Submit-time validation covers profile existence, RBAC permission, overrides
|
||||
and paths only.
|
||||
- Submit-time validation covers profile existence, RBAC permission, the
|
||||
additive rule and paths only.
|
||||
- The execution account is the worker's concern. If a `direct` job's execution
|
||||
user doesn't exist on the node, the worker fails the job at launch with a
|
||||
clear, user-visible reason, instead of today's opaque systemd failure.
|
||||
|
|
@ -203,7 +221,7 @@ fits XCE's LSF-style job model.
|
|||
"Profile configuration" below), with xceadmin validation and live-upgrade
|
||||
replication.
|
||||
- **The controller validates the profile at submit**: the named profile
|
||||
exists, and the job overrides only attributes the profile marks *default*.
|
||||
exists, and the job only adds to the profile's values (additive rule).
|
||||
This makes it authoritative for every submit path (REST, libxce, harness).
|
||||
It stores the profile name and the job's own values; the effective values
|
||||
come from the live merge (`job-config-merge`).
|
||||
|
|
@ -220,8 +238,8 @@ fits XCE's LSF-style job model.
|
|||
two checks. If the profile has gone by then, the job is rejected, never run
|
||||
with a fallback profile.
|
||||
- **The apiserver additionally enforces the RBAC permission** for the profile
|
||||
(middleware chain unchanged; a `RequirePermission`-style check) and validates
|
||||
overrides early (`preCheckJobSubmitRequest`).
|
||||
(middleware chain unchanged; a `RequirePermission`-style check) and checks
|
||||
the additive rule early (`preCheckJobSubmitRequest`).
|
||||
- **The job's own values stay in its existing fields** (`launch_type`,
|
||||
pre/post exec, exdata env and limits, resreq). New job fields are added only
|
||||
for the profile name and the job's own I/O directories. All wire changes
|
||||
|
|
@ -254,9 +272,9 @@ fits XCE's LSF-style job model.
|
|||
at the rate of queues, as occasional admin edits (frequency rule).
|
||||
- **The file is optional.** Without it there is one built-in profile
|
||||
`default` (mode `direct`, no attributes), which is today's behaviour.
|
||||
- **Every attribute carries its override marker** (`default` or `fixed`,
|
||||
`job-config-layering.md`). Profile pre/post-exec is always fixed; whether
|
||||
the job's own pre/post-exec is allowed is a profile setting.
|
||||
- **No override markers.** Every value in a profile is an essential and
|
||||
binds the job (additive rule). Whether the job's own pre/post-exec is
|
||||
allowed is a profile setting.
|
||||
- **Open to new sections.** Later specs add top-level sections to a profile
|
||||
(`service:`, I/O claims, relationship templates, `container:`); the schema
|
||||
must allow that without reshaping existing sections.
|
||||
|
|
@ -268,14 +286,10 @@ fits XCE's LSF-style job model.
|
|||
description: "OpenFOAM, MPI"
|
||||
mode: direct
|
||||
launch_type: openmpi
|
||||
resreq: { value: "select[ib] span[ptile=32] rusage[mem=4000]", override: default }
|
||||
tasks: { min: 32, default: 64, max: 512 }
|
||||
limits:
|
||||
runtime: { value: 24h, override: fixed }
|
||||
memory: { value: 128G, override: fixed }
|
||||
env:
|
||||
vars: { FOAM_INST_DIR: /opt/openfoam }
|
||||
job_env: allow
|
||||
resreq: "select[ib] rusage[license_foam=1]" # essentials only
|
||||
tasks: { min: 32, max: 512 }
|
||||
limits: { runtime: 24h, memory: 128G } # maximums
|
||||
env: { FOAM_INST_DIR: /opt/openfoam }
|
||||
pre_exec: /opt/xce/app/openfoam/pre.sh
|
||||
post_exec: /opt/xce/app/openfoam/post.sh
|
||||
job_pre_post_exec: allow
|
||||
|
|
@ -286,6 +300,60 @@ fits XCE's LSF-style job model.
|
|||
job_pre_post_exec: forbid
|
||||
```
|
||||
|
||||
Tuning such as `rusage[mem=4000] span[ptile=32]` and a task count of 64
|
||||
is not in the profile; the admin publishes it as a site preset named
|
||||
`openfoam`.
|
||||
|
||||
### Essentials and the additive rule
|
||||
How each profile attribute combines with the job's own value (owned by
|
||||
`job-config-merge`, which implements it):
|
||||
|
||||
| Profile attribute | Holds | The job may |
|
||||
|---|---|---|
|
||||
| resreq `select` | required terms (`ib`, GPU model, OS) | add terms (ANDed) |
|
||||
| resreq `rusage` | required consumptions (`license=1`) | add other resources; not change these |
|
||||
| resreq `span` / `same` / `order` | only if essential (e.g. a code that runs on one host only) | set them if the profile does not; otherwise rejected |
|
||||
| task range | `min` / `max` | choose within it; no `-n` means `min` |
|
||||
| env | variables the application needs | add others; not redefine these |
|
||||
| launch type, mode, sandbox options | fixed values | nothing |
|
||||
| limits | maximums | ask for less (checked like a ceiling) |
|
||||
| pre/post-exec | outer layer, always runs | add its own inner layer if the profile allows |
|
||||
|
||||
- A job with no `-R` gets the profile's essentials only; the scheduler's
|
||||
existing empty-resreq default (`type == any`) still applies to what
|
||||
neither sets.
|
||||
- **The `service:` section is configured in the profile** (user,
|
||||
2026-10-12; defined with this spec, consumed by `service-supervision`,
|
||||
`service-replica-set`, `service-endpoint`) and follows the same rule:
|
||||
- restart policy (always / on-failure / never, backoff, give-up),
|
||||
liveness and readiness probes, and named ports are the application's
|
||||
definition: profile values, the service cannot change them;
|
||||
- replicas work like the task range: the profile may set `min` / `max`,
|
||||
and the service chooses its count within them (from its own spec or a
|
||||
preset);
|
||||
- spread rule and rollout settings (max unavailable / surge): if the
|
||||
profile sets them they bind; otherwise the service sets them.
|
||||
- Other later sections (I/O claims, relationship templates, `container:`)
|
||||
state in their own specs how they follow the additive rule.
|
||||
- Compound or alternative resreqs: a compound job resreq under a profile
|
||||
whose resreq sets terms is rejected until PR-15 named parts define
|
||||
per-part merging (leaning; requirements decide).
|
||||
|
||||
### Presets (client side, `xcecli-presets`)
|
||||
Not part of this spec, recorded here because they replace profile defaults:
|
||||
- A preset is a set of job options (`--app`, `--queue`, `-R`, `-n`, env,
|
||||
limits). The client expands it; a command-line flag replaces the preset's
|
||||
value for that option as a whole. The submit carries only the expanded
|
||||
values.
|
||||
- Personal presets live in `~/.config/xce/presets.yaml`.
|
||||
- The apiserver **only shares**: admin-deployed preset files served
|
||||
read-only (`GET /api/v1/presets`, `GET /api/v1/presets/{name}`). Pulling
|
||||
copies a preset into the local file, so a server preset being removed
|
||||
never affects users. `preset diff` / `pull --force` are user-initiated.
|
||||
- User-to-user sharing by `preset export` / `import` (file, mail, git).
|
||||
- Presets carry no privilege: `app:use`, `queue:submit`, the profile's
|
||||
essentials, queue ceilings and project quotas still apply at submit.
|
||||
|
||||
### Default profile
|
||||
- **Set by `default_application: <name>` in `xce.yaml`**, a cluster
|
||||
parameter (as LSF keeps `DEFAULT_APPLICATION` in `lsb.params`), not a flag
|
||||
|
|
@ -321,9 +389,9 @@ fits XCE's LSF-style job model.
|
|||
|
||||
## Scope
|
||||
- **In**:
|
||||
- The profile schema (attributes and override markers) and
|
||||
`applications.yaml`, and the default profile (`default_application`).
|
||||
- Submit-time validation of the profile and the job's overrides.
|
||||
- The profile schema (essentials, additive rule) and `applications.yaml`,
|
||||
and the default profile (`default_application`).
|
||||
- Submit-time validation of the profile and the additive rule.
|
||||
- RBAC permission for profile use.
|
||||
- Submit surfaces: REST, `xcecli --app`, harness, libxce.
|
||||
- The profile name and the job's own values persisted on the job.
|
||||
|
|
@ -332,9 +400,10 @@ fits XCE's LSF-style job model.
|
|||
- A worker launch error when a `direct` exec user is missing.
|
||||
- Profile name and mode shown in job details.
|
||||
- Listing available profiles (REST and xcecli).
|
||||
- Profile `--dry-run` support: the effective values a job would get
|
||||
(shared with `job-config-merge`).
|
||||
- Tests.
|
||||
- **Open for requirements**:
|
||||
- The exact per-attribute override rules.
|
||||
- Which extra attributes go into v1 beyond the four ("maybe others"). Examples:
|
||||
requeue exit values, runtime estimate, job starter or wrapper, output path
|
||||
template, checkpoint settings.
|
||||
|
|
@ -350,35 +419,14 @@ fits XCE's LSF-style job model.
|
|||
- Removing a non-default profile that pending jobs still name (leaning:
|
||||
xceadmin warns, the jobs pend with "profile X not defined", running
|
||||
jobs continue).
|
||||
- Limits a job may lower but not raise: either a third "bounded" marker,
|
||||
or the profile value is a fixed maximum and the job's own value a
|
||||
request checked against it like a ceiling (keeps the two-interaction
|
||||
rule of `job-config-layering.md`, which should then say so).
|
||||
- Env merge: per-variable default/fixed, plus a denylist for sandbox
|
||||
profiles (e.g. `LD_PRELOAD`)?
|
||||
- Resreq override granularity (leaning, follows from the convenience
|
||||
purpose): markers per term instead of per section. `select` has
|
||||
*required* terms (always ANDed) and *default* terms (the job's select
|
||||
replaces only these); `rusage` is per resource (fixed, or default with
|
||||
optional min/max); `span`/`same`/`order` default unless marked. An
|
||||
unmarked term is *default*. Amends `job-config-layering.md` once
|
||||
decided. Also settles per-resource `rusage` merging and the "limits a
|
||||
job may lower" question (default with a range is still only a check).
|
||||
- Privileged override for one-off exceptions: an `app:override`
|
||||
permission per profile (fixed terms treated as default for that job),
|
||||
recorded on the job (WAL/snapshot) and shown in job details; queue
|
||||
ceilings and project quotas still apply. v1 or later?
|
||||
- Profile inheritance (`base: <profile>`) for recurring variants such as
|
||||
"openfoam, large memory": v1 or later (leaning later).
|
||||
- Personal presets for a user's tuned settings: xcecli only (leaning),
|
||||
or server-side for the portal (a per-user layer, which conflicts with
|
||||
one owner per attribute)?
|
||||
- Job labels (`--label key=value`) and a summary grouped by label, to
|
||||
compare tuning variants: here, `job-status-detail-display`, or its own
|
||||
spec?
|
||||
- Compound or alternative resreqs: reject a compound job resreq under a
|
||||
profile with fixed resreq terms until PR-15 named parts define
|
||||
per-part merging (leaning).
|
||||
- A denylist of env variables a sandbox profile forbids jobs to add
|
||||
(e.g. `LD_PRELOAD`)?
|
||||
- Job labels (`--label key=value`, e.g. the preset name) and a summary
|
||||
grouped by label, to compare tuning variants: here,
|
||||
`job-status-detail-display`, or `xcecli-presets`?
|
||||
- Settled 2026-10-12: override granularity (no markers; additive rule),
|
||||
limits a job may lower (profile limits are maximums), personal presets
|
||||
(client side), `app:override` and inheritance (deferred).
|
||||
- Splitting the spec: execution mode, I/O directories, capability
|
||||
advertisement and placement as a separate spec (or folded into
|
||||
`job-sandbox-execution`), leaving this spec the profile core.
|
||||
|
|
@ -390,7 +438,7 @@ fits XCE's LSF-style job model.
|
|||
|
||||
## Boundary Candidates
|
||||
- The profile model and config (schema, validation, live upgrade).
|
||||
- Submit-time validation of the profile and overrides (controller).
|
||||
- Submit-time validation of the profile and the additive rule (controller).
|
||||
- Access control (RBAC permission) and early validation (apiserver).
|
||||
- Submit and listing surfaces (REST, xcecli, harness, libxce).
|
||||
- Persisting the profile name and the job's own values (wire, WAL submit
|
||||
|
|
@ -400,7 +448,10 @@ fits XCE's LSF-style job model.
|
|||
## Out of Boundary
|
||||
- Worker execution backends and sandbox mechanics.
|
||||
- Output file location and retrieval inside the sandbox.
|
||||
- Scheduler policy beyond capability filtering and resreq defaults.
|
||||
- Scheduler policy beyond capability filtering and the profile's essential
|
||||
resreq.
|
||||
- Presets (`xcecli-presets`): client side; the read-only preset listing on
|
||||
the apiserver belongs to that spec.
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream**:
|
||||
|
|
@ -424,7 +475,9 @@ fits XCE's LSF-style job model.
|
|||
- **Adjacent**:
|
||||
- `job-status-detail-display`
|
||||
- `controller-job-cleanup`
|
||||
- `resreq-maxtasks-wiring` (resreq defaults)
|
||||
- `resreq-maxtasks-wiring` (task range)
|
||||
- `xcecli-presets` (convenience defaults, client side)
|
||||
- `profile-usage-insight` (advisory recommendations)
|
||||
- `xce-node-manager`
|
||||
|
||||
## Constraints
|
||||
|
|
@ -436,7 +489,7 @@ fits XCE's LSF-style job model.
|
|||
the job's own values (including I/O directories) round trip through the
|
||||
submit facts and snapshots. Run
|
||||
`/xce-audit-invariants`.
|
||||
- Input validation (`security.md`): overrides and paths are validated
|
||||
- Input validation (`security.md`): the job's additions and paths are validated
|
||||
server-side. No path traversal, and mount prefixes are restricted by the
|
||||
profile.
|
||||
- Size: this is likely 20 or more tasks. If requirements confirm that, split
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue