docs(spec): application profiles hold essentials only; additive rule; client-side presets
- profiles hold only what every job of the application must have; every profile value binds and a job may add to it, never change or remove it (replaces the default/fixed markers of 2026-10-09) - convenience defaults move to client-side presets (new xcecli-presets brief): option-level expansion on the client, the apiserver only shares admin-deployed presets read-only, pull copies, export/import - service section stays in the profile: probes, restart policy, ports bind; replica count within the profile range; spread/rollout if unset - job-config-layering, job-config-merge brief/requirements, roadmap and resource-questions updated to the additive rule - app:override and profile inheritance deferred; recommendations target site presets Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
3fa9361ad8
commit
4536d36f47
GPG key ID:
80533536B0EB651C
7 changed files with 267 additions and 115 deletions
|
|
@ -14,11 +14,10 @@ decisions):**
|
|||
(min / default / max, LSF TASKLIMIT), **project** = resource ownership
|
||||
(`project-config`).
|
||||
- Every attribute has one owning level (`.kiro/steering/job-config-layering.md`).
|
||||
How the levels interact is owned by **`job-config-merge`**: each profile
|
||||
attribute is *default* (the job may replace it) or *fixed* (the job may
|
||||
not; for `select` it may only add ANDed terms); queue ceilings and project
|
||||
quotas only check, never change, the job's values. This answers the open
|
||||
question "how resource defaults combine with a job's own resreq" below.
|
||||
How the levels interact is owned by **`job-config-merge`**; queue
|
||||
ceilings and project quotas only check, never change, the job's values.
|
||||
(The job-vs-profile rule of this update, *default*/*fixed* markers, is
|
||||
replaced by the additive rule of 2026-10-12 below.)
|
||||
- **Pre/post-exec moves here from the queue** (queue `pre_exec`/`post_exec`
|
||||
deprecated). The profile's pre/post-exec is admin-defined and always runs
|
||||
as the outer layer; the job's own pre/post-exec is the inner layer, allowed
|
||||
|
|
@ -38,8 +37,9 @@ decisions):**
|
|||
**Update 2026-10-10 (`/kiro-discovery`, long-running services):** a profile
|
||||
may declare **kind `service`** with a service section: restart policy
|
||||
(always / on-failure / never, backoff, give-up), liveness and readiness
|
||||
probes (command / TCP / HTTP), named ports, default replica count, spread
|
||||
rule and rollout settings. The section is defined with this spec and
|
||||
probes (command / TCP / HTTP), named ports, replica range, spread rule and
|
||||
rollout settings (how these follow the additive rule: "Essentials and the
|
||||
additive rule" below). The section is defined with this spec and
|
||||
consumed by `service-supervision` (worker probes and restarts),
|
||||
`service-replica-set` (replicas, scaling, rolling update) and
|
||||
`service-endpoint` (ports, endpoints).
|
||||
|
|
@ -53,23 +53,38 @@ consumed by `service-supervision` (worker probes and restarts),
|
|||
that is not defined is a configuration error everywhere (`xceadmin`, live
|
||||
upgrade, controller start), never a silent fallback. See "Default profile"
|
||||
below.
|
||||
- **This spec owns the profile's attribute schema**: which attributes exist
|
||||
and how each is marked *default* or *fixed*. `job-config-merge` consumes
|
||||
that schema and owns only the merge and live re-evaluation.
|
||||
- **Purpose: convenience first, enforcement where needed** (user, 2026-10-12).
|
||||
A profile saves users from repeating the same resreq and other settings on
|
||||
every job; most values are defaults the job may change. Only what the
|
||||
application cannot run without is fixed. Consequences:
|
||||
- Profile defaults will sometimes not fit, and users will tune them
|
||||
(sizing, placement). The profile gives room to try: default terms, with
|
||||
admin-set ranges for numeric values.
|
||||
- **This spec owns the profile's attribute schema** (which attributes exist
|
||||
and what each holds). `job-config-merge` consumes that schema and owns the
|
||||
merge and live re-evaluation.
|
||||
- **A profile holds only essentials; convenience lives in client-side
|
||||
presets** (user decision, 2026-10-12). "The profile holds what the job must
|
||||
have; the preset holds what the user would like." A profile value is
|
||||
something the application needs or the admin enforces, and it binds every
|
||||
job of that profile. The *default*/*fixed* markers of 2026-10-09 are
|
||||
dropped: there are no markers, and no per-term markers or ranges. See
|
||||
"Essentials and the additive rule" below.
|
||||
- **Additive rule:** the job may add to the profile's values, never change
|
||||
or remove them. A conflict is rejected at submit with an error naming the
|
||||
profile value, never resolved silently.
|
||||
- **Convenience defaults** (tuning such as `rusage[mem=...]`,
|
||||
`span[ptile=...]`, task count, extra env) go into **presets**, handled
|
||||
fully by the client (`xcecli-presets`). The server only shares
|
||||
admin-deployed presets read-only; presets never reach the controller,
|
||||
WAL or scheduler and carry no privilege.
|
||||
- **Rule of thumb for admins:** something every job of the application must
|
||||
have (required host properties, licenses, launch type, mode,
|
||||
pre/post-exec, limits) goes into the profile; an optional variant or
|
||||
tuning shortcut goes into a site preset, typically one with the profile's
|
||||
name.
|
||||
- **Recommendations are advisory** (user decision): usage-based
|
||||
suggestions for a profile's defaults (requested vs. used, OOM kills,
|
||||
runtime by variant) are reported to the admin, who changes
|
||||
`applications.yaml`; a profile never tunes itself. Backlog:
|
||||
`profile-usage-insight`.
|
||||
suggestions (requested vs. used, OOM kills, runtime by variant) are
|
||||
reported to the admin, who changes site presets (or a profile, for an
|
||||
essential); nothing tunes itself. Backlog: `profile-usage-insight`.
|
||||
- Job details show the effective resreq with each term's origin (profile
|
||||
or job), so users see what they are tuning.
|
||||
or job).
|
||||
- Deferred: a privileged `app:override` for one-off exceptions (an admin
|
||||
can modify the pending job instead; recurring variants are presets), and
|
||||
profile inheritance (`base:`), no longer needed for variants.
|
||||
- **Later sections are added by their consumers**: `service:`
|
||||
(`service-supervision`), I/O claims (resource-model PR-6), relationship
|
||||
and compound templates (PR-14/15), `container:` (R6). This spec only makes
|
||||
|
|
@ -82,7 +97,7 @@ scripts, vendor binaries, MPI solvers. Each needs a different way of running:
|
|||
identity (and later in a container or vm);
|
||||
- a particular launch type (serial, `openmpi`, `mpich`);
|
||||
- particular environment and pre/post-exec steps;
|
||||
- sensible resource defaults and limits.
|
||||
- resource requirements it cannot run without, and limits.
|
||||
|
||||
Today every one of these must be set per job by hand, and several (execution
|
||||
mode, sandbox options) cannot be set at all. Many users log in through the
|
||||
|
|
@ -130,21 +145,24 @@ fits XCE's LSF-style job model.
|
|||
first version contains the following (user decision; the format must be
|
||||
extensible for "maybe others"):
|
||||
1. **Execution mode and its options.** `direct` or `sandbox` now, with room for
|
||||
`container`/`vm`. Sandbox options include allowed and default input/output
|
||||
mounts and isolation extras.
|
||||
`container`/`vm`. Sandbox options include the allowed input/output
|
||||
prefixes and isolation extras.
|
||||
2. **Launch type.** Serial, `openmpi` or `mpich`.
|
||||
3. **Environment and pre/post-exec.** Default environment variables and
|
||||
pre/post-exec commands.
|
||||
4. **Resource requirement and limits.** The job's resource requirement
|
||||
(select / rusage / `span[...]` / same / order), its task range (min /
|
||||
default / max), and per-job execution limits (memory, runtime, CPU,
|
||||
tasks), mapped onto the existing worker cgroup limits.
|
||||
3. **Environment and pre/post-exec.** Environment variables the application
|
||||
needs, and the outer pre/post-exec commands.
|
||||
4. **Essential resource requirement and limits.** The resreq terms the
|
||||
application cannot run without (select / rusage / `span[...]` / same /
|
||||
order), its task range (min / max), and per-job execution limits as
|
||||
maximums (memory, runtime, CPU, tasks), mapped onto the existing worker
|
||||
cgroup limits.
|
||||
All of these are essentials under the additive rule; none is a
|
||||
convenience default.
|
||||
- **Users pick a profile at submit** (REST, `xcecli jobs submit --app <name>`,
|
||||
harness, libxce). A **cluster default profile** applies when none is given.
|
||||
- **Limited overrides.** A job may override only the attributes its profile
|
||||
marks as overridable (for example its own input/output directories within the
|
||||
allowed prefixes, or env additions). Anything else is rejected with a clear
|
||||
error.
|
||||
- **Jobs add, never override.** A job may add to its profile's values (more
|
||||
`select` terms, other resources, its own env variables, its own I/O
|
||||
directories within the allowed prefixes, lower limits) but not change or
|
||||
remove any. Anything else is rejected with a clear error.
|
||||
- **Access by RBAC permission.** No new access mechanism; this follows the
|
||||
existing named-instance pattern (user decision, 2026-09-29: fold profile RBAC
|
||||
into this spec). Today, `queue:submit` scoped to `type: queue`,
|
||||
|
|
@ -176,9 +194,9 @@ fits XCE's LSF-style job model.
|
|||
fairshare or preemption).
|
||||
- Any combination of an allowed queue and an allowed profile is valid. Each is
|
||||
governed by its own access rules; for profiles, that is RBAC permission.
|
||||
- Resource defaults in a profile describe what the application needs (its
|
||||
default resource requirement and per-job execution limits). They are not
|
||||
scheduling policy. Queue-level scheduling limits keep applying on top,
|
||||
- Resource values in a profile describe what the application needs (its
|
||||
essential resource requirement and per-job execution limits). They are
|
||||
not scheduling policy. Queue-level scheduling limits keep applying on top,
|
||||
unchanged.
|
||||
- **Live merge** (`job-config-merge`). The job records its profile name and
|
||||
its own values, not the effective values. The scheduler computes the
|
||||
|
|
@ -186,8 +204,8 @@ fits XCE's LSF-style job model.
|
|||
that no longer satisfies its profile pends with a reason, and running jobs
|
||||
are unaffected. Job details show the profile name and the effective mode.
|
||||
- **The apiserver never checks node accounts** (user decision, 2026-09-29).
|
||||
- Submit-time validation covers profile existence, RBAC permission, overrides
|
||||
and paths only.
|
||||
- Submit-time validation covers profile existence, RBAC permission, the
|
||||
additive rule and paths only.
|
||||
- The execution account is the worker's concern. If a `direct` job's execution
|
||||
user doesn't exist on the node, the worker fails the job at launch with a
|
||||
clear, user-visible reason, instead of today's opaque systemd failure.
|
||||
|
|
@ -203,7 +221,7 @@ fits XCE's LSF-style job model.
|
|||
"Profile configuration" below), with xceadmin validation and live-upgrade
|
||||
replication.
|
||||
- **The controller validates the profile at submit**: the named profile
|
||||
exists, and the job overrides only attributes the profile marks *default*.
|
||||
exists, and the job only adds to the profile's values (additive rule).
|
||||
This makes it authoritative for every submit path (REST, libxce, harness).
|
||||
It stores the profile name and the job's own values; the effective values
|
||||
come from the live merge (`job-config-merge`).
|
||||
|
|
@ -220,8 +238,8 @@ fits XCE's LSF-style job model.
|
|||
two checks. If the profile has gone by then, the job is rejected, never run
|
||||
with a fallback profile.
|
||||
- **The apiserver additionally enforces the RBAC permission** for the profile
|
||||
(middleware chain unchanged; a `RequirePermission`-style check) and validates
|
||||
overrides early (`preCheckJobSubmitRequest`).
|
||||
(middleware chain unchanged; a `RequirePermission`-style check) and checks
|
||||
the additive rule early (`preCheckJobSubmitRequest`).
|
||||
- **The job's own values stay in its existing fields** (`launch_type`,
|
||||
pre/post exec, exdata env and limits, resreq). New job fields are added only
|
||||
for the profile name and the job's own I/O directories. All wire changes
|
||||
|
|
@ -254,9 +272,9 @@ fits XCE's LSF-style job model.
|
|||
at the rate of queues, as occasional admin edits (frequency rule).
|
||||
- **The file is optional.** Without it there is one built-in profile
|
||||
`default` (mode `direct`, no attributes), which is today's behaviour.
|
||||
- **Every attribute carries its override marker** (`default` or `fixed`,
|
||||
`job-config-layering.md`). Profile pre/post-exec is always fixed; whether
|
||||
the job's own pre/post-exec is allowed is a profile setting.
|
||||
- **No override markers.** Every value in a profile is an essential and
|
||||
binds the job (additive rule). Whether the job's own pre/post-exec is
|
||||
allowed is a profile setting.
|
||||
- **Open to new sections.** Later specs add top-level sections to a profile
|
||||
(`service:`, I/O claims, relationship templates, `container:`); the schema
|
||||
must allow that without reshaping existing sections.
|
||||
|
|
@ -268,14 +286,10 @@ fits XCE's LSF-style job model.
|
|||
description: "OpenFOAM, MPI"
|
||||
mode: direct
|
||||
launch_type: openmpi
|
||||
resreq: { value: "select[ib] span[ptile=32] rusage[mem=4000]", override: default }
|
||||
tasks: { min: 32, default: 64, max: 512 }
|
||||
limits:
|
||||
runtime: { value: 24h, override: fixed }
|
||||
memory: { value: 128G, override: fixed }
|
||||
env:
|
||||
vars: { FOAM_INST_DIR: /opt/openfoam }
|
||||
job_env: allow
|
||||
resreq: "select[ib] rusage[license_foam=1]" # essentials only
|
||||
tasks: { min: 32, max: 512 }
|
||||
limits: { runtime: 24h, memory: 128G } # maximums
|
||||
env: { FOAM_INST_DIR: /opt/openfoam }
|
||||
pre_exec: /opt/xce/app/openfoam/pre.sh
|
||||
post_exec: /opt/xce/app/openfoam/post.sh
|
||||
job_pre_post_exec: allow
|
||||
|
|
@ -286,6 +300,60 @@ fits XCE's LSF-style job model.
|
|||
job_pre_post_exec: forbid
|
||||
```
|
||||
|
||||
Tuning such as `rusage[mem=4000] span[ptile=32]` and a task count of 64
|
||||
is not in the profile; the admin publishes it as a site preset named
|
||||
`openfoam`.
|
||||
|
||||
### Essentials and the additive rule
|
||||
How each profile attribute combines with the job's own value (owned by
|
||||
`job-config-merge`, which implements it):
|
||||
|
||||
| Profile attribute | Holds | The job may |
|
||||
|---|---|---|
|
||||
| resreq `select` | required terms (`ib`, GPU model, OS) | add terms (ANDed) |
|
||||
| resreq `rusage` | required consumptions (`license=1`) | add other resources; not change these |
|
||||
| resreq `span` / `same` / `order` | only if essential (e.g. a code that runs on one host only) | set them if the profile does not; otherwise rejected |
|
||||
| task range | `min` / `max` | choose within it; no `-n` means `min` |
|
||||
| env | variables the application needs | add others; not redefine these |
|
||||
| launch type, mode, sandbox options | fixed values | nothing |
|
||||
| limits | maximums | ask for less (checked like a ceiling) |
|
||||
| pre/post-exec | outer layer, always runs | add its own inner layer if the profile allows |
|
||||
|
||||
- A job with no `-R` gets the profile's essentials only; the scheduler's
|
||||
existing empty-resreq default (`type == any`) still applies to what
|
||||
neither sets.
|
||||
- **The `service:` section is configured in the profile** (user,
|
||||
2026-10-12; defined with this spec, consumed by `service-supervision`,
|
||||
`service-replica-set`, `service-endpoint`) and follows the same rule:
|
||||
- restart policy (always / on-failure / never, backoff, give-up),
|
||||
liveness and readiness probes, and named ports are the application's
|
||||
definition: profile values, the service cannot change them;
|
||||
- replicas work like the task range: the profile may set `min` / `max`,
|
||||
and the service chooses its count within them (from its own spec or a
|
||||
preset);
|
||||
- spread rule and rollout settings (max unavailable / surge): if the
|
||||
profile sets them they bind; otherwise the service sets them.
|
||||
- Other later sections (I/O claims, relationship templates, `container:`)
|
||||
state in their own specs how they follow the additive rule.
|
||||
- Compound or alternative resreqs: a compound job resreq under a profile
|
||||
whose resreq sets terms is rejected until PR-15 named parts define
|
||||
per-part merging (leaning; requirements decide).
|
||||
|
||||
### Presets (client side, `xcecli-presets`)
|
||||
Not part of this spec, recorded here because they replace profile defaults:
|
||||
- A preset is a set of job options (`--app`, `--queue`, `-R`, `-n`, env,
|
||||
limits). The client expands it; a command-line flag replaces the preset's
|
||||
value for that option as a whole. The submit carries only the expanded
|
||||
values.
|
||||
- Personal presets live in `~/.config/xce/presets.yaml`.
|
||||
- The apiserver **only shares**: admin-deployed preset files served
|
||||
read-only (`GET /api/v1/presets`, `GET /api/v1/presets/{name}`). Pulling
|
||||
copies a preset into the local file, so a server preset being removed
|
||||
never affects users. `preset diff` / `pull --force` are user-initiated.
|
||||
- User-to-user sharing by `preset export` / `import` (file, mail, git).
|
||||
- Presets carry no privilege: `app:use`, `queue:submit`, the profile's
|
||||
essentials, queue ceilings and project quotas still apply at submit.
|
||||
|
||||
### Default profile
|
||||
- **Set by `default_application: <name>` in `xce.yaml`**, a cluster
|
||||
parameter (as LSF keeps `DEFAULT_APPLICATION` in `lsb.params`), not a flag
|
||||
|
|
@ -321,9 +389,9 @@ fits XCE's LSF-style job model.
|
|||
|
||||
## Scope
|
||||
- **In**:
|
||||
- The profile schema (attributes and override markers) and
|
||||
`applications.yaml`, and the default profile (`default_application`).
|
||||
- Submit-time validation of the profile and the job's overrides.
|
||||
- The profile schema (essentials, additive rule) and `applications.yaml`,
|
||||
and the default profile (`default_application`).
|
||||
- Submit-time validation of the profile and the additive rule.
|
||||
- RBAC permission for profile use.
|
||||
- Submit surfaces: REST, `xcecli --app`, harness, libxce.
|
||||
- The profile name and the job's own values persisted on the job.
|
||||
|
|
@ -332,9 +400,10 @@ fits XCE's LSF-style job model.
|
|||
- A worker launch error when a `direct` exec user is missing.
|
||||
- Profile name and mode shown in job details.
|
||||
- Listing available profiles (REST and xcecli).
|
||||
- Profile `--dry-run` support: the effective values a job would get
|
||||
(shared with `job-config-merge`).
|
||||
- Tests.
|
||||
- **Open for requirements**:
|
||||
- The exact per-attribute override rules.
|
||||
- Which extra attributes go into v1 beyond the four ("maybe others"). Examples:
|
||||
requeue exit values, runtime estimate, job starter or wrapper, output path
|
||||
template, checkpoint settings.
|
||||
|
|
@ -350,35 +419,14 @@ fits XCE's LSF-style job model.
|
|||
- Removing a non-default profile that pending jobs still name (leaning:
|
||||
xceadmin warns, the jobs pend with "profile X not defined", running
|
||||
jobs continue).
|
||||
- Limits a job may lower but not raise: either a third "bounded" marker,
|
||||
or the profile value is a fixed maximum and the job's own value a
|
||||
request checked against it like a ceiling (keeps the two-interaction
|
||||
rule of `job-config-layering.md`, which should then say so).
|
||||
- Env merge: per-variable default/fixed, plus a denylist for sandbox
|
||||
profiles (e.g. `LD_PRELOAD`)?
|
||||
- Resreq override granularity (leaning, follows from the convenience
|
||||
purpose): markers per term instead of per section. `select` has
|
||||
*required* terms (always ANDed) and *default* terms (the job's select
|
||||
replaces only these); `rusage` is per resource (fixed, or default with
|
||||
optional min/max); `span`/`same`/`order` default unless marked. An
|
||||
unmarked term is *default*. Amends `job-config-layering.md` once
|
||||
decided. Also settles per-resource `rusage` merging and the "limits a
|
||||
job may lower" question (default with a range is still only a check).
|
||||
- Privileged override for one-off exceptions: an `app:override`
|
||||
permission per profile (fixed terms treated as default for that job),
|
||||
recorded on the job (WAL/snapshot) and shown in job details; queue
|
||||
ceilings and project quotas still apply. v1 or later?
|
||||
- Profile inheritance (`base: <profile>`) for recurring variants such as
|
||||
"openfoam, large memory": v1 or later (leaning later).
|
||||
- Personal presets for a user's tuned settings: xcecli only (leaning),
|
||||
or server-side for the portal (a per-user layer, which conflicts with
|
||||
one owner per attribute)?
|
||||
- Job labels (`--label key=value`) and a summary grouped by label, to
|
||||
compare tuning variants: here, `job-status-detail-display`, or its own
|
||||
spec?
|
||||
- Compound or alternative resreqs: reject a compound job resreq under a
|
||||
profile with fixed resreq terms until PR-15 named parts define
|
||||
per-part merging (leaning).
|
||||
- A denylist of env variables a sandbox profile forbids jobs to add
|
||||
(e.g. `LD_PRELOAD`)?
|
||||
- Job labels (`--label key=value`, e.g. the preset name) and a summary
|
||||
grouped by label, to compare tuning variants: here,
|
||||
`job-status-detail-display`, or `xcecli-presets`?
|
||||
- Settled 2026-10-12: override granularity (no markers; additive rule),
|
||||
limits a job may lower (profile limits are maximums), personal presets
|
||||
(client side), `app:override` and inheritance (deferred).
|
||||
- Splitting the spec: execution mode, I/O directories, capability
|
||||
advertisement and placement as a separate spec (or folded into
|
||||
`job-sandbox-execution`), leaving this spec the profile core.
|
||||
|
|
@ -390,7 +438,7 @@ fits XCE's LSF-style job model.
|
|||
|
||||
## Boundary Candidates
|
||||
- The profile model and config (schema, validation, live upgrade).
|
||||
- Submit-time validation of the profile and overrides (controller).
|
||||
- Submit-time validation of the profile and the additive rule (controller).
|
||||
- Access control (RBAC permission) and early validation (apiserver).
|
||||
- Submit and listing surfaces (REST, xcecli, harness, libxce).
|
||||
- Persisting the profile name and the job's own values (wire, WAL submit
|
||||
|
|
@ -400,7 +448,10 @@ fits XCE's LSF-style job model.
|
|||
## Out of Boundary
|
||||
- Worker execution backends and sandbox mechanics.
|
||||
- Output file location and retrieval inside the sandbox.
|
||||
- Scheduler policy beyond capability filtering and resreq defaults.
|
||||
- Scheduler policy beyond capability filtering and the profile's essential
|
||||
resreq.
|
||||
- Presets (`xcecli-presets`): client side; the read-only preset listing on
|
||||
the apiserver belongs to that spec.
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream**:
|
||||
|
|
@ -424,7 +475,9 @@ fits XCE's LSF-style job model.
|
|||
- **Adjacent**:
|
||||
- `job-status-detail-display`
|
||||
- `controller-job-cleanup`
|
||||
- `resreq-maxtasks-wiring` (resreq defaults)
|
||||
- `resreq-maxtasks-wiring` (task range)
|
||||
- `xcecli-presets` (convenience defaults, client side)
|
||||
- `profile-usage-insight` (advisory recommendations)
|
||||
- `xce-node-manager`
|
||||
|
||||
## Constraints
|
||||
|
|
@ -436,7 +489,7 @@ fits XCE's LSF-style job model.
|
|||
the job's own values (including I/O directories) round trip through the
|
||||
submit facts and snapshots. Run
|
||||
`/xce-audit-invariants`.
|
||||
- Input validation (`security.md`): overrides and paths are validated
|
||||
- Input validation (`security.md`): the job's additions and paths are validated
|
||||
server-side. No path traversal, and mount prefixes are restricted by the
|
||||
profile.
|
||||
- Size: this is likely 20 or more tasks. If requirements confirm that, split
|
||||
|
|
|
|||
|
|
@ -50,16 +50,21 @@ apiserver and display would disagree about what a job actually asked for.
|
|||
has exactly one owning level (queue = scheduling policy, application
|
||||
profile = what the application needs and how it runs, project = resource
|
||||
ownership), so only two interactions exist:
|
||||
- **Override (job vs. profile):** each profile attribute is *default* (the
|
||||
job's value replaces it) or *fixed* (the job cannot change it; for
|
||||
`select` the job may only add ANDed terms). This replaces LSF's
|
||||
per-section merge table.
|
||||
- **Addition (job vs. profile)** (user decision 2026-10-12, replacing the
|
||||
*default*/*fixed* markers): a profile holds only essentials and every
|
||||
profile value binds; the job may add to them (more `select` terms,
|
||||
other `rusage` resources, `span`/`same`/`order` the profile does not
|
||||
set, env variables, lower limits, an inner pre/post-exec if allowed)
|
||||
but never change or remove them; conflicts are rejected at submit.
|
||||
Table: `job-config-layering.md`. This replaces LSF's per-section merge
|
||||
table. Convenience defaults are client-side presets (`xcecli-presets`)
|
||||
and never reach the merge.
|
||||
- **Ceiling (queue or project vs. effective values):** never changes a
|
||||
value; a never-satisfiable value (runtime above the queue maximum, task
|
||||
range above the queue's max tasks) is rejected at submit, a quota that is
|
||||
only full now makes the job pend with a reason naming the queue/project.
|
||||
- The profile's task limits (min/default/max) bound the job's `-n`; with no
|
||||
`-n`, the profile default applies.
|
||||
- The profile's task range (min/max) bounds the job's `-n`; with no `-n`,
|
||||
`min` applies.
|
||||
- Where this diverges from LSF, the requirements list the divergence
|
||||
explicitly (input for `lsf-config-migration`).
|
||||
- **Live merge (user decision 2026-10-09).** The WAL keeps the job's own
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@ Discovery brief: `.kiro/specs/job-config-merge/brief.md`. Governing rules: `.kir
|
|||
**What should change:**
|
||||
- One merge function in `src/lib`, shared by controller, scheduler and (through cgo) apiserver. Its inputs are the job's own values, the application profile, the queue's policy and the project; its output is the job's effective values: resreq (select / rusage / span / same / order), task range, execution limits, pre/post-exec, env.
|
||||
- **One owner per attribute.** The queue owns scheduling policy, the application profile owns what the application needs and how it runs, and the project owns resource ownership. Only two interactions exist:
|
||||
- **Override:** the job may override a profile attribute marked *default*. It may not change one marked *fixed*, except `select`, where it may add ANDed terms.
|
||||
- **Addition** (2026-10-12, replacing *default*/*fixed* markers): a profile holds only essentials and every profile value binds. The job may add to them (more `select` terms, other `rusage` resources, `span`/`same`/`order` the profile does not set, env variables, lower limits) but never change or remove them; a conflict is rejected at submit. Convenience defaults are client-side presets and never reach the merge.
|
||||
- **Ceiling:** queue ceilings and project quotas never change a value. A value that can never be satisfied is rejected at submit; a quota that is only full right now makes the job pend with a reason naming the queue or project.
|
||||
- **Live merge.** The WAL keeps only the job's own values plus its queue, profile and project names.
|
||||
- The controller (authoritative) and the apiserver (early) run the merge at submit to validate.
|
||||
|
|
|
|||
68
.kiro/specs/xcecli-presets/brief.md
Normal file
68
.kiro/specs/xcecli-presets/brief.md
Normal file
|
|
@ -0,0 +1,68 @@
|
|||
# Brief: xcecli-presets
|
||||
|
||||
(From the `application-profile` discussion, 2026-10-12; user decisions.
|
||||
Backlog.)
|
||||
|
||||
## Problem
|
||||
Application profiles hold only essentials, what every job of an application
|
||||
must have (`job-config-layering.md`, additive rule). Users still repeat the
|
||||
same tuning on every job: `rusage[mem=...]`, `span[ptile=...]`, a task
|
||||
count, env, the queue. They also tune these over time (sizing, placement)
|
||||
and want to keep and share what they find. Putting such defaults in the
|
||||
profile made the server-side rules complex (override markers per section
|
||||
and per term, ranges, privileged overrides) and would add a per-user
|
||||
configuration level.
|
||||
|
||||
## Desired Outcome
|
||||
- **A preset is a set of job options** (`--app`, `--queue`, `-R`, `-n`,
|
||||
env, limits, ...), named and stored by the client. `xcecli jobs submit
|
||||
--preset <name>` expands it; any command-line flag replaces the preset's
|
||||
value for that option as a whole (no resreq parsing). The submit carries
|
||||
only the expanded values.
|
||||
- **Handled fully on the client.** Presets never reach the controller, WAL
|
||||
or scheduler, own no attribute, are not a configuration level and carry
|
||||
no privilege: `app:use`, `queue:submit`, the profile's essentials, queue
|
||||
ceilings and project quotas still apply at submit.
|
||||
- **Personal presets** live in `~/.config/xce/presets.yaml`; users change
|
||||
them on the fly (`xcecli preset set / edit / show / list / delete`).
|
||||
- **The server only shares.** The apiserver serves admin-deployed preset
|
||||
files read-only: `GET /api/v1/presets` (list) and
|
||||
`GET /api/v1/presets/{name}` (fetch). Team presets are namespaced
|
||||
(`<team>/<name>`). No database, no HA; with several apiservers the files
|
||||
are deployed to each, like `rbac.yaml`. Every authenticated user may see
|
||||
them.
|
||||
- **Pull copies.** `xcecli preset pull <name>` writes a complete, standalone
|
||||
local preset with an informational `origin` (server name, content hash,
|
||||
pull date). Removing or changing a server preset never affects local
|
||||
copies. `xcecli preset diff <name>` compares with the server (reports when
|
||||
it is gone); `preset pull --force` refreshes. Both are user-initiated.
|
||||
- **User-to-user sharing** without the server: `preset export` / `import`
|
||||
(file, mail, git).
|
||||
- **Site presets complement profiles.** Admins typically publish a preset
|
||||
with the profile's name holding its usual tuning; clients may point to it
|
||||
("server preset `openfoam` available") but never apply it automatically.
|
||||
- **Safety and visibility:** a warning on export/import when env variable
|
||||
names look like secrets; `xcecli jobs submit --dry-run` shows the expanded
|
||||
options and the effective values after the profile merge
|
||||
(`job-config-merge`); optionally the preset name is recorded as a job
|
||||
label (`preset=<name>`) for comparing variants.
|
||||
- **Portal:** the same expansion (a small option-level merge) and the same
|
||||
read-only listing; its personal presets are stored by the portal itself
|
||||
and are not shared with xcecli in the first version.
|
||||
|
||||
## Scope
|
||||
- **In:** preset format, xcecli commands, expansion and precedence, the
|
||||
read-only apiserver listing/fetch and its file directory, pull/diff,
|
||||
export/import, secret warning, dry-run display, tests.
|
||||
- **Out:** writable server-side shared presets (would need owners,
|
||||
visibility and durable storage; revisit only if the portal needs it, after
|
||||
`rbac-persistence`), automatic application of presets, any server-side
|
||||
preset expansion.
|
||||
- **Open:** whether the preset-name label lives here or in
|
||||
`job-status-detail-display` (job labels); the portal's preset storage.
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream:** `application-profile` (profiles and `--app`),
|
||||
`job-config-merge` (dry-run effective values).
|
||||
- **Downstream:** `profile-usage-insight` (advisory suggestions for site
|
||||
presets).
|
||||
|
|
@ -20,7 +20,7 @@ XCE instead gives **every attribute exactly one owning level**.
|
|||
| Level | Owns | Never owns |
|
||||
|---|---|---|
|
||||
| **Queue**: scheduling policy | priority and ordering, host partition, dispatch/run windows, preemption policy, policy ceilings (max run time, max tasks, max hosts, ...) | resource requirement, anything about how a job executes (pre/post-exec, env, launch, limits enforced on the job) |
|
||||
| **Application profile**: what the application needs and how it runs | resreq (select / rusage / span / same / order), task range (min / default / max), execution mode and options, launch type, env, **pre/post-exec**, execution limits (mem, runtime, CPU, tasks) | priority, queue choice, quotas |
|
||||
| **Application profile**: what the application needs and how it runs (essentials only, 2026-10-12) | resreq (select / rusage / span / same / order), task range (min / default / max), execution mode and options, launch type, env, **pre/post-exec**, execution limits (mem, runtime, CPU, tasks) | priority, queue choice, quotas |
|
||||
| **Project**: resource ownership | membership, quotas / limits, later guarantees, accounting | how a job runs or is ordered |
|
||||
| **Job** | its own values, only for attributes its profile allows | n/a |
|
||||
|
||||
|
|
@ -30,11 +30,24 @@ profile and project.
|
|||
|
||||
## The only two interactions
|
||||
|
||||
1. **Override: job vs. application profile.** Each profile attribute is
|
||||
either
|
||||
- **default**: the job's value replaces it, or
|
||||
- **fixed**: the job cannot change it. Exception: for `select`, the job may
|
||||
add terms (narrow the host set, ANDed) but never drop the profile's.
|
||||
1. **Addition: job vs. application profile** (user decision 2026-10-12,
|
||||
replacing the *default*/*fixed* markers of 2026-10-09). A profile holds
|
||||
only essentials, what every job of the application must have, and every
|
||||
profile value binds. The job may **add** to the profile's values, never
|
||||
change or remove them; a conflict is rejected at submit, never resolved
|
||||
silently. There are no override markers.
|
||||
|
||||
| Profile attribute | The job may |
|
||||
|---|---|
|
||||
| resreq `select` (required terms) | add terms (ANDed) |
|
||||
| resreq `rusage` (required consumptions) | add other resources; not change these |
|
||||
| resreq `span` / `same` / `order` | set them if the profile does not |
|
||||
| task range `min` / `max` | choose within it; no `-n` means `min` |
|
||||
| env | add variables; not redefine the profile's |
|
||||
| launch type, mode, mode options | nothing |
|
||||
| limits (maximums) | ask for less |
|
||||
| pre/post-exec (outer layer) | add an inner layer if the profile allows |
|
||||
| `service:` probes, restart policy, ports / replica range / spread, rollout | nothing / choose within / set if the profile does not |
|
||||
2. **Ceiling: queue or project vs. the job's effective values.** A queue
|
||||
ceiling or project quota never changes the job's values, it only checks
|
||||
them:
|
||||
|
|
@ -43,8 +56,20 @@ profile and project.
|
|||
- not satisfiable right now (project quota in use): the job pends with a
|
||||
reason naming the queue or project.
|
||||
|
||||
There is no third case: no queue resreq, no rusage merge, no ordering of
|
||||
queue against profile settings, no compound/alternative merge stages.
|
||||
There is no third case: no queue resreq, no rusage merge with the queue, no
|
||||
ordering of queue against profile settings, no compound/alternative merge
|
||||
stages.
|
||||
|
||||
## Convenience defaults are presets, not a level
|
||||
Tuning that most jobs of an application want but none must have
|
||||
(`rusage[mem=...]`, `span[ptile=...]`, a task count, extra env) is not in
|
||||
the profile. It goes into **presets**, handled fully by the client
|
||||
(`xcecli-presets`): a preset expands into job options before submit, so the
|
||||
server only ever sees the job's own values. Presets are therefore not a
|
||||
configuration level, own no attribute and carry no privilege. The apiserver
|
||||
only shares admin-deployed presets read-only. Rule of thumb: what every job
|
||||
must have goes into the profile; an optional variant goes into a site
|
||||
preset.
|
||||
|
||||
## Pre/post-exec
|
||||
- Moves from the queue to the application profile. Queue `pre_exec` /
|
||||
|
|
|
|||
|
|
@ -285,7 +285,7 @@ The proposals below are many (PR-1 to PR-15, about 50 questions). They do not ha
|
|||
- The scheduler sees only placement relations, compiled into filters and scores; the rest passes through. The hot path stays cheap and deterministic.
|
||||
- Validation at submit: contradictions, ordering cycles, unknown roles, impossible scopes.
|
||||
- A pending job names the relation that blocks it (`scheduler-alter-pend-reasons`).
|
||||
- Configuration layering: profile relations are profile-owned, and a job overrides only overridable ones (`job-config-layering.md`).
|
||||
- Configuration layering: profile relations are profile-owned and bind; a job may only add relations (additive rule, `job-config-layering.md`, 2026-10-12).
|
||||
- The schema is versioned, and new kinds are additive.
|
||||
- **Would become** a spec, e.g. `task-relationship-spec` (the canonical jobspec), after N31.
|
||||
- **Open:** see N41–N44.
|
||||
|
|
@ -305,7 +305,7 @@ The proposals below are many (PR-1 to PR-15, about 50 questions). They do not ha
|
|||
6. **Sidecar roles:** by default attached to the parent's tasks, sharing their allocation and hosts, with no separate part. Optionally a separate part with a derived count (N48).
|
||||
7. **Task ranges** (`-n min,max`, `parjob-placement` range search): ranges per role, or only on primary jobs (N49).
|
||||
8. **Allocation record:** each host allocation records its role (part index). WAL and snapshot stay field-complete. Dispatch gives each task its role, rank and hostfile (N31); `info job` shows allocations per role.
|
||||
9. **Configuration layering:** the profile owns the compound template and relationships; the job overrides only counts or ranges marked overridable.
|
||||
9. **Configuration layering:** the profile owns the compound template and relationships; the job may only choose counts within the profile's ranges and add relations (additive rule, 2026-10-12).
|
||||
10. **Resize** (N33): per-role resize re-checks the role's relationships.
|
||||
11. **Pend reasons and trace:** per part and per relationship (e.g. "decode: no switch with 3 free 8-GPU hosts near prefill"); trace capture per part; deterministic for sim/replay.
|
||||
12. **Accounting:** actual-usage reconciliation (`max(reserved, actual)`) is single-host only (`resource-accounting.md`); multi-role multi-host jobs inherit that gap. Record it explicitly (N50).
|
||||
|
|
|
|||
|
|
@ -63,7 +63,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
|
|||
- [x] ha-commit-durability — closes bug-661 (acknowledgment did not imply durability; committed jobs lost across failover). Makes acknowledged ⇒ durable on a majority: the primary counts its own copy toward the commit majority only at its durable LSN (`majority_candidate_locked` caps self at `persist_durable_lsn`, not the appended tip); `fsync_worker` repurposed timer→demand-driven group-commit flusher (one `fdatasync` over pending appends, advance `durable_lsn`, wake commit-eval) — absorbs the former "WAL group-commit batching" backlog item (see below); recovery integrity verdict (clean vs corruption-truncated WAL) + durable-backed freshness floor (candidate advertises bare `persist_durable_floor`, voter floor = `max(durable_floor, learned_committed)`) + pre-promotion eligibility fence. Establishes the **controllers-never-self-terminate** invariant: a recovery-truncated/behind member that cannot catch up is a LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — never `exit`/`abort`), self-heals on peer return (see `ha-async-io.md` process-lifecycle invariant + `src/controller/CLAUDE.md`). Dependencies: ha-commit-engine (consumes commit-position semantics), ha-wal-replication (standby ACK/catch-up), ha-commit-barrier-rollout (reply gate, unchanged). (specs done 2026-06-18; **IMPLEMENTED 2026-06-18**: 17/17 tasks (1.1–5.3), validate-impl GO, controller suite 0 failures under ASAN; deterministic ack-before-fdatasync E2E RED→GREEN + rolling-SIGKILL soak zero acked-job loss. Verified by tests + Protocol Verification Checklist, NOT by raft.tla (which assumes a durable log). Fixed bug-760 (adjacent `test_fact_hook_append` raced the async flusher — test made durable-deterministic via `persist_flush`). raft.tla durability-model extension deferred unless explicitly opted in.)
|
||||
- [x] scheduler-allocator-runner — NEW 2026-09-28 (`/kiro-discovery`, Path C). The framework owns `job_alloc_fn` through a new `xce_sched_allocator_iface_t` (accepts guard + plan iterator) and a single `sched_allocator_run()`; both `my_alloc()`s are deleted (seqjob/parjob), and `alloc_iter.h` `init` is retyped to `xce_sched_core_t*`. Behaviour must stay identical (bug-1882/1767/1768). Closes `issues/new_requirements.md` item 4. Brief: `.kiro/specs/scheduler-allocator-runner/brief.md`. Dependencies: scheduler-allocation, scheduler-core (both done). **IMPLEMENTED 2026-09-30** (11/11 tasks, e2266582..28cff179; validate-impl GO; scheduler ASAN suite 96/96): runner in `src/scheduler/src/alloc_runner.c`; decided along the way: the runner clears `alloc_req.host` before every candidate so every short host gets pend reasons (both allocators), and fixes bug-1893 (shared reasons were never recorded). Follow-ups: publish pend reasons only when changed; sim-mode parjob allocation parity fixture; bug-1892; bug-1897.
|
||||
- [ ] scheduler-alter-pend-reasons — NEW 2026-09-28 (`/kiro-discovery`, Path E). An ALTER-resreq job keeps a pend reason for EVERY alternative instead of only the last (`scheduler.c:424` FIXME). Wire shape (user decision): additive `alternatives @3: List(JobPendReason)` on `JobPendReason`, per-alternative attribution; carried end-to-end through controller job record, WAL preason fact, recordings and `job_preason_eq`. Also fixes the confirmed `obj->reason` overwrite leak (`scheduler.c:388`). Brief: `.kiro/specs/scheduler-alter-pend-reasons/brief.md`. Dependencies: libxce-alloc-preason-equality (HARD: extends its `job_preason_eq`); sequence after scheduler-allocator-runner (its "no pend-reason content change" rule).
|
||||
- [ ] application-profile — NEW 2026-09-29 (`/kiro-discovery`, Job Sandbox Execution initiative, Path D; reshaped from `job-execution-mode` by user direction: execution mode is a job attribute determined by the APPLICATION). Admin-defined named application profiles (LSF `-app` precedent) in cluster config: execution mode + options (`direct`|`sandbox`, room for `container`/`vm`), launch type, env + pre/post exec, resource defaults/limits (extensible). Users pick a profile at submit (`--app`), cluster default profile otherwise; queues (scheduling policy) and profiles (application config) are orthogonal — neither restricts the other; only profile-marked overridable attributes may be overridden; access by RBAC permission (new `app:use`/`app:view` with `app` domain type, folded into this spec); apiserver checks profile existence; controller validates at submit and persists the profile name + the job's own values (live merge via job-config-merge; wire additive, WAL/snapshot field-completeness); own config file `applications.yaml`, cluster default `default_application` in `xce.yaml` (undefined name = config error, never a fallback; updated 2026-10-12). apiserver never checks node accounts; worker fails a `direct` job with a clear reason if the exec user is missing. Worker capability advertisement + scheduler placement. Brief: `.kiro/specs/application-profile/brief.md`. Dependencies: none.
|
||||
- [ ] application-profile — NEW 2026-09-29 (`/kiro-discovery`, Job Sandbox Execution initiative, Path D; reshaped from `job-execution-mode` by user direction: execution mode is a job attribute determined by the APPLICATION). Admin-defined named application profiles (LSF `-app` precedent) in cluster config: execution mode + options (`direct`|`sandbox`, room for `container`/`vm`), launch type, env + pre/post exec, resource defaults/limits (extensible). Users pick a profile at submit (`--app`), cluster default profile otherwise; queues (scheduling policy) and profiles (application config) are orthogonal — neither restricts the other; a profile holds only essentials and the job may add to them but never change them (additive rule, 2026-10-12; convenience defaults are client-side presets, `xcecli-presets`); access by RBAC permission (new `app:use`/`app:view` with `app` domain type, folded into this spec); apiserver checks profile existence; controller validates at submit and persists the profile name + the job's own values (live merge via job-config-merge; wire additive, WAL/snapshot field-completeness); own config file `applications.yaml`, cluster default `default_application` in `xce.yaml` (undefined name = config error, never a fallback; updated 2026-10-12). apiserver never checks node accounts; worker fails a `direct` job with a clear reason if the exec user is missing. Worker capability advertisement + scheduler placement. Brief: `.kiro/specs/application-profile/brief.md`. Dependencies: none.
|
||||
- [ ] job-sandbox-execution — NEW 2026-09-29 (Job Sandbox Execution). Worker: pluggable execution-backend interface (`direct` refactored + `systemd-sandbox`; non-systemd backends later). Sandbox = existing transient unit + `DynamicUser` with fresh per-job name (never the submitter's), ProtectSystem=strict/ProtectHome/PrivateTmp/NoNewPrivileges, Bind(ReadOnly)Paths for job I/O dirs or worker-created temp workspace (handed to the uid via SO_PEERCRED), `StandardOutput=file:` output, helper auth by unit/cgroup, MANDATORY post-job cleanup/ownership normalization (dynamic uid reuse), systemd ≥ 242 + nss-systemd capability probe; single-node incl. single-node MPI. Viability-checked 2026-09-29. Brief: `.kiro/specs/job-sandbox-execution/brief.md`. Dependencies: application-profile; remote-task-root fix (Direct-Implementation).
|
||||
- [ ] job-sandbox-multinode — NEW 2026-09-29 (Job Sandbox Execution). Multi-node sandbox jobs: cluster-consistent ephemeral uid (expected: controller-allocated, WAL-replicated uid lease from a reserved range), worker-driven remote/MPI launch (no ssh; PMIx/prted via worker), shared-FS ownership normalization, one unit per job per node. Brief: `.kiro/specs/job-sandbox-multinode/brief.md`. Dependencies: job-sandbox-execution, application-profile.
|
||||
- [ ] rbac-persistence — NEW 2026-09-29 (`/kiro-discovery`, raised during application-profile discovery). Closes the deferred `security.md` gap: runtime RBAC mutations (roles, permissions, user-role assignments, Logon auto-assignment) are in-memory only and silently revert to `rbac.yaml` on apiserver restart — including revocations. Durable-or-rejected mutations, audit trail, defaults-vs-runtime reconciliation, multi-apiserver consistency; storage approach (rbac.yaml write-through / controller-replicated entity / embedded store / read-only) decided in design. Brief: `.kiro/specs/rbac-persistence/brief.md`. Dependencies: none (independent of the sandbox initiative; application-profile's API-made `app:use` grants rely on it for durability).
|
||||
|
|
@ -72,7 +72,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
|
|||
- [ ] job-dependency-insight — NEW 2026-09-29. `bjdepinfo`-style query (parents/children/-p blocking/-r levels/-l verdicts, RBAC-filtered) via REST/xcecli/harness, dependency info in job details, optional orphan termination (per-job `-ti` + cluster grace period, default off; EXIT reason ORPHAN; failover-safe persisted grace start; minimal primary periodic work). Brief: `.kiro/specs/job-dependency-insight/brief.md`. Dependencies: job-dependency, job-time-window (exit-reason field).
|
||||
- [ ] controller-job-cleanup — NEW 2026-09-29 (`/kiro-discovery`, Path C). Primary-only periodic sweep (configurable `job_cleanup_period`, default 300 s) removes DONE/EXIT jobs finished longer than configurable `job_retention` (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever `jobmgr` delete; refcount-safe `xkvmgr` delete), propagated to scheduler and apiserver (`OnJobDeleted`). Retires the node-local snapshot-only `job_should_retain` finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: `.kiro/specs/controller-job-cleanup/brief.md`. Dependencies: none (extends ha-entity-snapshots).
|
||||
- [ ] job-status-detail-display — NEW 2026-09-28 (`/kiro-discovery`, Path E). End users (REST/`xcecli`) see NO pend reason for ANY job today: the apiserver cgo conversion (`backend/convert.go`) never maps `xce_job_t.pend_reasons`, so `pending_reason` is always empty. Maps it to a structured JSON (including `alternatives` from day one) using libxce's code→text mapping, and renders it in `xcecli` job detail. Open for requirements: also map `run_usage` → `JobRusage` (same gap; reserved vs. used for RUN jobs). Brief: `.kiro/specs/job-status-detail-display/brief.md`. Dependencies: none for flat reasons; scheduler-alter-pend-reasons for per-alternative rendering.
|
||||
- [ ] job-config-merge — NEW 2026-10-09 (`/kiro-discovery`, Path E "configuration layering"; ref LSF 10.1 Admin Guide pp. 314-316, 435-437, 561-563, 643, 660). One merge function in `src/lib` turning the job's own values + application profile + queue policy + project into effective values (resreq incl. `span`, task range, limits, pre/post-exec, env). One owner per attribute (user decision; `.kiro/steering/job-config-layering.md`): job overrides a profile attribute only if it is *default* (for `select` only by adding ANDed terms); queue ceilings and project quotas only check (reject if never satisfiable, pend if full now). **Live** merge (user decision): WAL keeps inputs + names; controller/apiserver validate at submit; scheduler computes effective values and re-evaluates pending jobs on config change. Queue is policy only: queue `res_req` and `pre_exec`/`post_exec` deprecated (pre/post-exec moves to the profile), dropped queue limits (cpulimit/memlimit/runlimit/task limit) wired. Brief: `.kiro/specs/job-config-merge/brief.md`. Dependencies: none.
|
||||
- [ ] job-config-merge — NEW 2026-10-09 (`/kiro-discovery`, Path E "configuration layering"; ref LSF 10.1 Admin Guide pp. 314-316, 435-437, 561-563, 643, 660). One merge function in `src/lib` turning the job's own values + application profile + queue policy + project into effective values (resreq incl. `span`, task range, limits, pre/post-exec, env). One owner per attribute (user decision; `.kiro/steering/job-config-layering.md`): the job may add to a profile's values but never change or remove them (additive rule, 2026-10-12, replacing *default*/*fixed* markers); queue ceilings and project quotas only check (reject if never satisfiable, pend if full now). **Live** merge (user decision): WAL keeps inputs + names; controller/apiserver validate at submit; scheduler computes effective values and re-evaluates pending jobs on config change. Queue is policy only: queue `res_req` and `pre_exec`/`post_exec` deprecated (pre/post-exec moves to the profile), dropped queue limits (cpulimit/memlimit/runlimit/task limit) wired. Brief: `.kiro/specs/job-config-merge/brief.md`. Dependencies: none.
|
||||
- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (members, priority, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config so `PROJECTS`/`PER_PROJECT` limits work end to end, per-project usage display. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none.
|
||||
|
||||
- [ ] host-failure-job-recovery — NEW 2026-10-10 (`/kiro-discovery`, "Long-running services" phase 1; also a batch gap: RUN jobs on a dead host stay RUN forever, `controller/sched.c:631` FIXME). After a grace period, jobs on a lost host go to an unknown state, then are requeued (rerunnable, job option or profile default) or end EXIT "host lost"; multi-host jobs resolved as a whole; attempt numbers fence a stale copy when the host returns; manual requeue (REST/xcecli). Brief: `.kiro/specs/host-failure-job-recovery/brief.md`. Dependencies: none.
|
||||
|
|
@ -93,7 +93,7 @@ Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generi
|
|||
- [ ] resource-distribution-mapping — phase 1: shared pools (option 2) plus the resource type descriptor fields (`scope`, `unit`, `reporter`, `layer`, `instances`, `provider`); option 3 moves to `custom-resource-reporting` unless the user keeps it here (decision 1).
|
||||
- [ ] service-endpoint — ports as identified instances of `resource-instances` instead of a separate port model.
|
||||
- [ ] job-sandbox-execution — the per-job workspace becomes the first template claim of `resource-claims`; publish (mounts) through the claim's worker step. Where data staging lives is decision 9.
|
||||
- [ ] application-profile — default claims (model weights, checkpoint) and GPU instance requests in profiles; `container` execution mode decides volume publishing (decision 6).
|
||||
- [ ] application-profile — claims the application needs (model weights, checkpoint) and GPU instance requests in profiles; `container` execution mode decides volume publishing (decision 6).
|
||||
- [ ] job-resize — re-bind a job's standalone claims to the new host set.
|
||||
- [ ] service-replica-set — replicas share one read-many weights claim; rollout attaches version N+1 while N stays attached; per-replica template claims.
|
||||
- [ ] resource-provisioner — its provider contract, lease, epoch and orphan machinery are shared with storage/claim providers.
|
||||
|
|
@ -107,11 +107,11 @@ Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generi
|
|||
- [ ] xce-node-manager — revise requirements (approved 2026-09-03) for the cross-spec interactions recorded 2026-10-10 in its requirements.md: shared supervision/probe module with service-supervision, a lease-based `provisioner` role, worker stop on decommission and role declaration for provisioned hosts, grace period vs worker restart, future service proxy under `reverse-proxy`.
|
||||
|
||||
## Existing Spec Updates (2026-10-10 long-running services discovery)
|
||||
- [ ] application-profile — profile kind `service` section: restart policy, probes, ports, default replicas, spread rule, rollout settings (consumed by service-supervision, service-replica-set, service-endpoint).
|
||||
- [ ] job-config-merge — service attributes follow the ownership rule (profile owns them; the job may override only `default` ones, e.g. replica count).
|
||||
- [ ] application-profile — profile kind `service` section: restart policy, probes, ports, replica range, spread rule, rollout settings, configured in the profile (user, 2026-10-12) (consumed by service-supervision, service-replica-set, service-endpoint).
|
||||
- [ ] job-config-merge — service attributes follow the ownership rule (additive rule, 2026-10-12: probes, restart policy and ports bind; the service chooses its replica count within the profile's range and sets spread/rollout only where the profile does not).
|
||||
|
||||
## Existing Spec Updates (2026-10-09 configuration layering discovery)
|
||||
- [ ] application-profile — the profile carries the job's resource requirement (select/rusage/`span`/same) and task limits (min/default/max); "resolved and frozen at submit" replaced by the live merge of `job-config-merge` (brief updated). Dependencies: job-config-merge.
|
||||
- [ ] application-profile — the profile carries the job's essential resource requirement (select/rusage/`span`/same) and task range (min/max), convenience defaults moving to client-side presets (2026-10-12); "resolved and frozen at submit" replaced by the live merge of `job-config-merge` (brief updated). Dependencies: job-config-merge.
|
||||
- [ ] parjob-placement — `span` read from the effective resreq; span constraints in the fixed-N planner; range search bisects over the planner's allowed counts (block: multiples; ptile: multiples + max); LSF 10.1 findings recorded in its requirements.md notes; `span[stripe]` candidate. Dependencies: job-config-merge (for `span` from profiles; job-level `-R` span can land first).
|
||||
|
||||
## Existing Spec Updates (2026-09-28 `info job` discovery)
|
||||
|
|
@ -122,7 +122,8 @@ Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generi
|
|||
- [ ] project-guarantee — NEW 2026-10-09 (configuration layering, backlog). Project-owned guarantees (resource amount or host pool), borrowing of idle guaranteed capacity, reclaim through preemption. Brief: `.kiro/specs/project-guarantee/brief.md`. Dependencies: project-config, scheduler-preemption.
|
||||
- [ ] job-resize — NEW 2026-10-10 (from long-running services discovery; not on the services path). Grow/shrink a running parallel job's allocation (LSF resizable jobs, `bresize`) for elastic applications. Brief: `.kiro/specs/job-resize/brief.md`. Dependencies: parjob-placement, host-failure-job-recovery.
|
||||
- [ ] project-delegated-membership — NEW 2026-10-10 (from project-config requirements, user decision: future). Project managers edit their project's (and subtree's) members at runtime without an admin config push; needs a durable store that coexists with `xceadmin` config pushes (e.g. a separate delegated member list) and audit. Dependencies: project-config, rbac-persistence.
|
||||
- [ ] profile-usage-insight — NEW 2026-10-12 (from application-profile discussion, backlog; no brief yet). Per-profile report over recent jobs comparing the profile's defaults with actual use (requested vs. peak memory, CPU efficiency, OOM kills, pend time, runtime by variant) and suggesting better defaults, e.g. "rusage mem default 4000, p95 peak 1600 → suggest 2000" (Kubernetes VPA recommender precedent). **Advisory only** (user decision): the admin applies changes through `applications.yaml`; a profile never tunes itself. Optional user hint ("used 1.4G of 4G requested"). Dependencies: application-profile, run usage in job records.
|
||||
- [ ] xcecli-presets — NEW 2026-10-12 (from application-profile discussion, user decisions). Client-side presets replace convenience defaults in profiles: a preset is a set of job options (`--app`, `--queue`, `-R`, `-n`, env, limits) expanded by the client, a flag replacing the preset's value for that option; personal presets in `~/.config/xce/presets.yaml`; the apiserver only shares admin-deployed presets read-only (`GET /api/v1/presets[/{name}]`), and `preset pull` copies one locally so server-side removal never affects users (`preset diff`, `pull --force` user-initiated); user-to-user sharing by `preset export`/`import`; presets carry no privilege and are not a configuration level; optional preset-name job label; the portal does the same expansion. Brief: `.kiro/specs/xcecli-presets/brief.md`. Dependencies: application-profile.
|
||||
- [ ] profile-usage-insight — NEW 2026-10-12 (from application-profile discussion, backlog; no brief yet). Per-profile report over recent jobs comparing the profile's defaults with actual use (requested vs. peak memory, CPU efficiency, OOM kills, pend time, runtime by variant) and suggesting better values for site presets (or a profile essential), e.g. "preset openfoam rusage mem 4000, p95 peak 1600 → suggest 2000" (Kubernetes VPA recommender precedent). **Advisory only** (user decision): the admin applies changes to site presets or `applications.yaml`; nothing tunes itself. Optional user hint ("used 1.4G of 4G requested"). Dependencies: application-profile, run usage in job records.
|
||||
- [ ] lsf-config-migration — NEW 2026-10-09 (user idea, backlog; no brief yet). A tool (likely an `xceadmin` subcommand) that converts LSF configuration (`lsb.queues`, `lsb.applications`, `lsb.resources` Limit sections, `lsb.users`/`lsb.hosts` groups, `lsb.params` defaults, project settings) into XCE `configs/*.yaml`, with a report of what does not map: queue `RES_REQ` (XCE queues are policy only; move into a profile or drop), LSF's full per-section merge rules vs XCE's simplified ones, compound/alternative resreqs, unsupported parameters. Dependencies: job-config-merge, application-profile, project-config, parjob-placement (span), admin-config-tool. Promote via `/kiro-discovery` when those land.
|
||||
|
||||
- [ ] **xce-harness-component** — **PROMOTED to spec 2026-09-13** (`/kiro-discovery`; brief at `.kiro/specs/xce-harness-component/brief.md`). Elevate `xce_test` (the REPL-driven wire-protocol test client used by HA soak / multi-host / resource-value verification) out of `src/controller/testsuite/` into its own first-class `src/harness/` component — own `meson.build`, `CLAUDE.md`, and unit-test binary (carved out of `controller_unit_test`, which today recompiles ~14 of xce_test's own files a second time paired with white-box tests) — fully decoupled from `/xce-build controller` (today's implicit "controller always builds xce_test" coupling, `build-deployment.md:24`). Verified finding: none of xce_test's 15 source files include any controller-private header — only libxce public headers + generated capnp schema — so this is a pure relocation + build-graph split, not a dependency untangling problem, and no xce_test behavior changes. Touches `/xce-test-ha`, `/xce-deploy-ha`, `/xce-test-multihost`, `/xce-test-resources` (each must build `harness` explicitly once decoupled) and is adjacent to (not overlapping) the three existing behavior specs `xce-test-cli-ux`/`xce-test-e2e-primitives`/`xce-test-rc-file`, whose file-path references go stale after the move. Next: `/kiro-spec-init xce-harness-component`.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue