Rongsong Shen
332f86eaa1
The project tree is the only place concurrent use is capped: quotas are written on the project (quota, member_quota over the subtree) and compiled into the limit plugin, which becomes internal. Queues keep per-job ceilings only; host slot caps are supply. Project path is the identity and the only submit form; members are users and user groups, never sub-projects. Adds the limits-to-owners backlog item. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1023 lines
61 KiB
Markdown
1023 lines
61 KiB
Markdown
# Brief: application-profile
|
|
|
|
(Reshaped from `job-execution-mode` on 2026-09-29, by user direction. Execution
|
|
mode is an attribute of the job, determined by the **application** the user runs,
|
|
so it is carried by a named application profile rather than by a bare mode
|
|
option.)
|
|
|
|
**Update 2026-10-09 (`/kiro-discovery`, configuration layering; user
|
|
decisions):**
|
|
- Configuration is layered LSF-style: **queue** = scheduling policy only (no
|
|
resreq; queue `res_req` is deprecated), **application profile** =
|
|
application execution constraints *including the job's resource
|
|
requirement* (select / rusage / `span[...]` / same) and task limits
|
|
(min / default / max, LSF TASKLIMIT), **project** = resource ownership
|
|
(`project-config`).
|
|
- Every attribute has one owning level (`.kiro/steering/job-config-layering.md`).
|
|
How the levels interact is owned by **`job-config-merge`**; queue
|
|
ceilings and project quotas only check, never change, the job's values.
|
|
(The job-vs-profile rule of this update, *default*/*fixed* markers, is
|
|
replaced by the additive rule of 2026-10-12 below.)
|
|
- **Pre/post-exec moves here from the queue** (queue `pre_exec`/`post_exec`
|
|
deprecated). The profile's pre/post-exec is admin-defined and always runs
|
|
as the outer layer; the job's own pre/post-exec is the inner layer, allowed
|
|
or forbidden by the profile. Order: profile pre → job pre → job → job post
|
|
→ profile post (the worker's existing two-layer structure,
|
|
`xcecd_worker.c` `queue_pre_exec()`/`queue_post_exec()`).
|
|
- **The merge is *live*** (replaces the earlier "resolution at submit time",
|
|
removed from this brief 2026-10-12). The WAL
|
|
keeps the job's own values plus the profile name; the controller and
|
|
apiserver validate at submit, and the scheduler computes effective values
|
|
and recomputes them when a profile changes (a pending job that no longer
|
|
satisfies its profile pends with a reason; running jobs are unaffected).
|
|
Execution-side values (mode, options, launch type, env, pre/post-exec) are
|
|
still taken from the profile at dispatch time and frozen on the dispatch
|
|
record (decided 2026-10-12, "Dispatch record" below).
|
|
|
|
**Update 2026-10-10 (`/kiro-discovery`, long-running services):** a profile
|
|
may declare **kind `service`** with a service section: restart policy
|
|
(always / on-failure / never, backoff, give-up), liveness and readiness
|
|
probes (command / TCP / HTTP), named ports, replica range, spread rule and
|
|
rollout settings (how these follow the additive rule: "Essentials and the
|
|
additive rule" below). The section is defined with this spec and
|
|
consumed by `service-supervision` (worker probes and restarts),
|
|
`service-replica-set` (replicas, scaling, rolling update) and
|
|
`service-endpoint` (ports, endpoints).
|
|
|
|
**Update 2026-10-12 (discussion; user decisions):**
|
|
- **Split: execution modes move to `job-sandbox-execution`.** This spec is
|
|
the profile core: config, access and submit, job state (name, dispatch
|
|
record), the outer pre/post-exec layer. Its schema has `mode:` but
|
|
accepts only `direct`. The `sandbox` mode with its options, I/O
|
|
directories, worker capability advertisement and scheduler placement by
|
|
mode belong to `job-sandbox-execution`, since nothing uses a non-`direct`
|
|
mode before the sandbox exists. The worker's clear launch error for a
|
|
missing `direct` exec user stays here.
|
|
- **Order with `job-config-merge`:** requirements and design of this spec
|
|
first (it fixes the profile schema), then `job-config-merge`'s design;
|
|
implementation tasks interleave (schema/config, merge library, submit
|
|
validation and dispatch record).
|
|
- **Own configuration file.** Profiles live in a new
|
|
`src/configs/applications.yaml` with its own JSON schema, mapped into
|
|
`ClusterConfig` and `Conf4Sched` like queues and limits. See "Profile
|
|
configuration" below.
|
|
- **Cluster default** is set by `default_application` in `xce.yaml`. A name
|
|
that is not defined is a configuration error everywhere (`xceadmin`, live
|
|
upgrade, controller start), never a silent fallback. See "Default profile"
|
|
below.
|
|
- **This spec owns the profile's attribute schema** (which attributes exist
|
|
and what each holds). `job-config-merge` consumes that schema and owns the
|
|
merge and live re-evaluation.
|
|
- **A profile holds only essentials; convenience lives in client-side
|
|
presets** (user decision, 2026-10-12). "The profile holds what the job must
|
|
have; the preset holds what the user would like." A profile value is
|
|
something the application needs or the admin enforces, and it binds every
|
|
job of that profile. The *default*/*fixed* markers of 2026-10-09 are
|
|
dropped: there are no markers, and no per-term markers or ranges. See
|
|
"Essentials and the additive rule" below.
|
|
- **Additive rule:** the job may add to the profile's values, never change
|
|
or remove them. A conflict is rejected at submit with an error naming the
|
|
profile value, never resolved silently.
|
|
- **Exception: default reservations.** The profile's structured
|
|
`rusage` entries hold per-resource defaults the job may override (see
|
|
"Essentials and the additive rule"), so REST/portal users without
|
|
presets still reserve sensibly.
|
|
- **Other convenience defaults** (tuning such as `span[ptile=...]`, task
|
|
count, extra env, different `rusage` values) go into **presets**, handled
|
|
fully by the client (`xcecli-presets`). The server only shares
|
|
admin-deployed presets read-only; presets never reach the controller,
|
|
WAL or scheduler and carry no privilege.
|
|
- **Rule of thumb for admins:** something every job of the application must
|
|
have (required host properties, licenses, launch type, mode,
|
|
pre/post-exec, limits) goes into the profile; an optional variant or
|
|
tuning shortcut goes into a site preset, typically one with the profile's
|
|
name.
|
|
- **Recommendations are advisory** (user decision): usage-based
|
|
suggestions (requested vs. used, OOM kills, runtime by variant) are
|
|
reported to the admin, who changes site presets (or a profile, for an
|
|
essential); nothing tunes itself. Backlog: `profile-usage-insight`.
|
|
- Job details show the effective resreq with each term's origin (profile
|
|
or job).
|
|
- Deferred: a submit-time hook run for the application (LSF `esub`
|
|
style) that inspects or adjusts each submission; roadmap backlog
|
|
`app-submit-hook` (it cannot run in the controller, would add latency
|
|
and a failure mode to every apiserver submit, and bypasses the merge
|
|
rules; `params`, annotations, the job starter and pre-exec cover most
|
|
needs).
|
|
- Deferred: a privileged `app:override` for one-off exceptions (until
|
|
`job-modify` exists, an admin kills and resubmits, or uses a variant
|
|
profile; recurring variants are presets), and profile inheritance
|
|
(`base:`), no longer needed for variants.
|
|
- **Later sections are added by their consumers**: `service:`
|
|
(`service-supervision`), I/O claims (resource-model PR-6), relationship
|
|
and compound templates (PR-14/15), `container:` (R6). This spec only makes
|
|
the schema open to new top-level sections.
|
|
|
|
## Problem
|
|
XCE users run many kinds of applications: trusted in-house codes, untrusted
|
|
scripts, vendor binaries, MPI solvers. Each needs a different way of running:
|
|
- directly as the submitter, or jailed in a sandbox under a separate execution
|
|
identity (and later in a container or vm);
|
|
- a particular launch type (serial, `openmpi`, `mpich`);
|
|
- particular environment and pre/post-exec steps;
|
|
- resource requirements it cannot run without, and limits.
|
|
|
|
Today every one of these must be set per job by hand, and several (execution
|
|
mode, sandbox options) cannot be set at all. Many users log in through the
|
|
portal or REST API (JWT; `config_file`/PAM, later LDAP or OAuth2) and have **no
|
|
Unix account on the compute nodes**. They can only run jobs if their
|
|
applications run sandboxed. Admins have no way to say "this application runs
|
|
like this, and these users may run it".
|
|
|
|
This spec introduces **application profiles** and is the submission/policy half
|
|
of the job-sandbox initiative (roadmap "Job Sandbox Execution"). There is
|
|
precedent in LSF application profiles (`lsb.applications`, `bsub -app`), which
|
|
fits XCE's LSF-style job model.
|
|
|
|
## Current State
|
|
- **Identity.** A job's identity is only the JWT `username`
|
|
(`controllers/auth.go:26-37`). `SubmitUser` comes from the token
|
|
(`controllers/jobs.go:1379`, `:1459`), and `UserGroup` is taken from the client
|
|
request unchecked (`:1465`).
|
|
- **Execution user.** It is always the submitter: `resolveExecUser`
|
|
(`backend/convert_pure.go:162`, bug-1780).
|
|
- **Per-job settings that exist today** (`src/include/xce/types.h`):
|
|
- `job_startinfo_t` (:240-251): `execuser`, `exechome`, `execcwd`, `cmdline`,
|
|
`exdata`, `launch_type`, and pre/post-exec commands.
|
|
- `exdata` is a JSON object carrying the environment (`sub_env_vars`) and
|
|
the worker cgroup limits parsed by `parse_limits` (MemoryMax, CPUQuota,
|
|
TasksMax, RuntimeMax, ...). Since bug-1936 the apiserver sends it
|
|
(`backend/convert_pure.go:388`, capped at 16 KB by `maxExdataLen`); the
|
|
controller stores it opaquely in `start_info` (WAL submit fact,
|
|
snapshot) and the worker parses it, ignoring unknown top-level keys.
|
|
Remote tasks get the same exdata (`worker/src/controller_connector.c:764`).
|
|
- The resource requirement is `resreq` on the submit data.
|
|
- Remote tasks use `xce_taskhost_t` (:696-705).
|
|
- **Missing today:**
|
|
- any profile concept, any execution-mode field, any input/output directory
|
|
field;
|
|
- any way for workers to advertise execution capabilities to the scheduler.
|
|
- **Controller.** It validates only that the user is not empty
|
|
(`controller/job.c:2447`). The submit data, including `start_info`, is
|
|
persisted in the WAL submit fact and in snapshots.
|
|
- **Ownership.** RBAC ownership, scheduler per-user limits and the user table all
|
|
key on the **submit user**, not the exec user (`controllers/jobs.go:1010`,
|
|
`:2542`, `output.go:95`; `schedobj.c:121`; `plugins/limit.c`). So a separate
|
|
execution identity breaks none of them.
|
|
- **Access control.** RBAC is `config/rbac.yaml` plus `RequirePermission`
|
|
(`controllers/rbac_check.go`).
|
|
|
|
## Desired Outcome
|
|
- **Admins define named application profiles** in the cluster configuration. The
|
|
first version contains the following (user decision; the format must be
|
|
extensible for "maybe others"):
|
|
1. **Execution mode.** The `mode:` field, accepting only `direct` in this
|
|
spec; `job-sandbox-execution` adds `sandbox` with its options (allowed
|
|
input/output prefixes, isolation extras), later `container`/`vm`.
|
|
2. **Launch type.** Serial, `openmpi` or `mpich`.
|
|
3. **Environment and hooks.** Environment variables the application
|
|
needs, and the worker-run hooks `pre/post_job_exec` (first host) and
|
|
`pre/post_host_exec` (every host), see "Hooks and starters".
|
|
4. **Resource requirement and limits.** A structured resreq: the binding
|
|
terms the application cannot run without (`select`, and `span` /
|
|
`same` / `order` when set) and default reservations (`rusage`
|
|
entries), its task range (min / max), and per-job execution limits as
|
|
maximums (memory, runtime, CPU, tasks), mapped onto the existing worker
|
|
cgroup limits.
|
|
5. **Job starter and task starter** (decided 2026-10-12, user): two
|
|
wrappers, both frozen on the dispatch record.
|
|
- `job_starter` wraps the **full job**: the job command, started once
|
|
on the first host (for an MPI job, the launcher such as `mpirun`),
|
|
e.g. `module load openfoam &&`.
|
|
- `task_starter` wraps **every task startup** that XCE's workers
|
|
perform, on every host including remote tasks (`xce_taskhost_t`),
|
|
e.g. `apptainer exec img` or a binding/affinity wrapper.
|
|
- A serial (single-task) job's command is both the job and its task:
|
|
it is wrapped by both, `job_starter` outermost (design confirms).
|
|
- For MPI jobs XCE inserts the task starter into the `mpirun` line it
|
|
builds, so every rank starts through it (see "Hooks and starters").
|
|
6. **Application parameters (`params`)** (decided 2026-10-12): settings
|
|
XCE carries but never interprets, handled by the application, e.g.
|
|
|
|
```yaml
|
|
params:
|
|
- { name: solver, default: pimpleFoam, allowed: [pimpleFoam, simpleFoam] }
|
|
- { name: decompose, default: scotch }
|
|
- { name: case_dir, pattern: "^/shared/cases/.*" } # no default
|
|
```
|
|
- Each is delivered to the job as `XCE_APP_PARAM_<NAME>`, with the
|
|
profile env (separate from the job env, frozen on the dispatch
|
|
record), so the job, the job starter and the outer pre/post-exec can
|
|
read it.
|
|
- The job overrides with `--app-param name=value` (also in presets);
|
|
the job's value wins, like `rusage` defaults. Only declared names are
|
|
accepted (a typo is rejected at submit). XCE checks only the shape
|
|
(`allowed` list or `pattern`), never the meaning. A parameter with no
|
|
default and no job value is not set.
|
|
- The job's values are stored in its exdata (`app_params`), already
|
|
persisted; no new job field.
|
|
- The portal can build an application's submit form from the declared
|
|
parameters.
|
|
7. **Annotations** (decided 2026-10-12): free-form key/value facts about
|
|
the application (owner, version, docs URL, licence server name), like
|
|
Kubernetes annotations. Shown to tools (profile listing, portal, job
|
|
details) **and** exported to the job as `XCE_APP_ANN_<KEY>`, so an
|
|
admin writes a value once for both. The job cannot change them.
|
|
- Name conversion: keys may be Kubernetes-style
|
|
(`cfd.org/solver-family`); the env name is uppercased with every
|
|
non-alphanumeric character replaced by `_`
|
|
(`XCE_APP_ANN_CFD_ORG_SOLVER_FAMILY`). `xceadmin` rejects two
|
|
annotation keys (or two params) that convert to the same name.
|
|
- Limits on count and size, so a profile cannot bloat every job's env.
|
|
- Visible to every user who can view profiles: no secrets (the secret-name
|
|
warning applies). All are exported in v1; an opt-out only if
|
|
needed.
|
|
- `env` vs annotations vs params: `env` is a variable the application
|
|
expects under its own name; annotations are facts for tools and the
|
|
job; params are the user's choices. `XCE_APP_*` names are set by XCE
|
|
only; the reserved `XCE_` prefix keeps jobs and profiles from setting
|
|
them directly, so a job cannot fake one.
|
|
All of these are essentials under the additive rule, except the `rusage`
|
|
entries and `params`, which are defaults the job may override.
|
|
- **A profile attribute ships with its job feature** (user, 2026-10-12): a
|
|
profile attribute is added by the spec that builds the job-level feature
|
|
it configures, never ahead of it, so no profile setting is dead config.
|
|
None of the other candidates exists as a job feature today, so they are
|
|
deferred:
|
|
- requeue exit values and success exit values: `job-requeue` (backlog);
|
|
- checkpoint method/period/directory, and a warning signal before the run
|
|
limit: their own future specs;
|
|
- output path template: the PR-6/PR-7 I/O track (output options are
|
|
rejected at submit today);
|
|
- runtime estimate: not a profile attribute at all (it describes one run:
|
|
a job option or preset; the profile's runtime limit is the maximum).
|
|
- Found on the way: the REST submit accepts several of these options and
|
|
silently drops them (bug-1970).
|
|
- **Users pick a profile at submit** (REST, `xcecli jobs submit --app <name>`,
|
|
harness, libxce). A **cluster default profile** applies when none is given.
|
|
- **Jobs add, never override** (except `rusage` defaults). A job may add to
|
|
its profile's values (more `select` terms, other resources, its own env
|
|
variables, lower limits) and override a profile `rusage` default, but not
|
|
change or remove anything else. Anything else is rejected with a clear
|
|
error.
|
|
- **Access control is deferred** (user decision, 2026-10-12): v1 has no
|
|
per-profile permission. Any user who may submit (`queue:submit`) may use
|
|
any profile; listing and showing profiles reuses the existing
|
|
`cluster:view` permission (profiles are cluster configuration). The
|
|
design recorded below moves to the backlog spec `app-access-control`,
|
|
kept here so it is not lost:
|
|
- permissions `app:use` (submit with a profile) and `app:view` (list and
|
|
inspect profiles), a new `app` scope type whose values are profile
|
|
names or `all`, following the named-instance pattern of `queue:submit`
|
|
(`config/rbac.yaml`, `GetRequireResourcePermission`,
|
|
`controllers/jobs.go:1402-1409`);
|
|
- checks on job create (alongside, and independent of, `queue:submit`),
|
|
on job restart (re-check for the job's profile) and on listing;
|
|
- `rbac.yaml` defaults (`basic_user` → `app:use` on the default profile,
|
|
`app:view` on `all`); the current default profile always usable with
|
|
`queue:submit` (implicit `app:use`), so plain submits never break when
|
|
the default is repointed; aliases checked against the target profile;
|
|
- role API accepts the new scope type; `cluster_admin` keeps the single
|
|
existing bypass; API-made grants depend on `rbac-persistence`.
|
|
- **Queues and profiles are orthogonal** (user decision, 2026-09-29).
|
|
- **Queues** own scheduling policy: which jobs are considered and in what
|
|
order, priority, scheduling limits, and host partitioning.
|
|
- **Application profiles** own application-related configuration: how the
|
|
application is executed and what it needs to run.
|
|
- A queue does not list or restrict profiles, and a profile does not select or
|
|
constrain queues.
|
|
- A profile carries no scheduling-policy attributes (no priority, queue choice,
|
|
fairshare or preemption).
|
|
- Any combination of an allowed queue and a profile is valid. Queues keep
|
|
their access rules; per-profile access control comes later
|
|
(`app-access-control`).
|
|
- Resource values in a profile describe what the application needs (its
|
|
essential resource requirement and per-job execution limits). They are
|
|
not scheduling policy. Queue per-job ceilings and project quotas check
|
|
the merged result on top (no queue aggregate caps, 2026-10-12).
|
|
- **Live merge** (`job-config-merge`). The job records its profile name and
|
|
its own values, not the effective values. The scheduler computes the
|
|
effective values and recomputes them when the profile changes: a pending job
|
|
that no longer satisfies its profile pends with a reason, and running jobs
|
|
are unaffected. Job details show the profile name and the effective mode.
|
|
- **The apiserver never checks node accounts** (user decision, 2026-09-29).
|
|
- Submit-time validation covers profile existence, the additive rule and
|
|
paths only (no per-profile permission in v1).
|
|
- The execution account is the worker's concern. If a `direct` job's execution
|
|
user doesn't exist on the node, the worker fails the job at launch with a
|
|
clear, user-visible reason, instead of today's opaque systemd failure.
|
|
- Account-less users are served by profiles whose mode is `sandbox`.
|
|
- **Placement by mode** (moved to `job-sandbox-execution`): a job is placed
|
|
only on workers whose advertised capabilities support its mode (sandbox
|
|
needs systemd 242 or later and `nss-systemd`).
|
|
- **Ownership.** The submitter remains the owner for RBAC, limits and accounting
|
|
in every mode.
|
|
|
|
## Approach
|
|
- **Profile definitions** live in their own cluster configuration file (see
|
|
"Profile configuration" below), with xceadmin validation and live-upgrade
|
|
replication.
|
|
- **The controller validates the profile at submit**: the named profile
|
|
exists, and the job only adds to the profile's values (additive rule).
|
|
This makes it authoritative for every submit path (REST, libxce, harness).
|
|
It stores the profile name and the job's own values; the effective values
|
|
come from the live merge (`job-config-merge`).
|
|
- **The apiserver checks that the named profile exists** (user direction,
|
|
2026-09-29).
|
|
- At submit, before anything goes downstream (`preCheckJobSubmitRequest`), the
|
|
apiserver rejects a job that names a profile which does not exist, with a
|
|
clear client error naming the unknown profile.
|
|
- The apiserver already receives the cluster configuration from the controller
|
|
(`backend/convert.go:265`), so it knows the current set of profiles and keeps
|
|
it current across live config upgrades.
|
|
- The controller still re-validates at submit, because it is
|
|
authoritative for non-REST submit paths and for a profile removed between the
|
|
two checks. If the profile has gone by then, the job is rejected, never run
|
|
with a fallback profile.
|
|
- **The apiserver additionally checks the additive rule early**
|
|
(`preCheckJobSubmitRequest`); the middleware chain is unchanged. The
|
|
profile listing route uses `RequirePermission("cluster:view")`.
|
|
- **The job's own values stay in its existing fields** (`launch_type`,
|
|
pre/post exec, exdata env and limits, resreq). New fields (all additive
|
|
Cap'n Proto; WAL submit/dispatch facts and snapshot carry them with field
|
|
completeness):
|
|
- on the submit data: the profile name;
|
|
- on `start_info` (what the worker gets): the job starter, the task
|
|
starter (also on each `xce_taskhost_t` entry for remote tasks) and the
|
|
profile's env, kept **separate** from the job's env (see "Environment of
|
|
the outer layer"); the queue-named `qPreExec` / `qPostExec` carry the
|
|
profile's outer pre/post-exec (renamed in design);
|
|
- the dispatch record payload ("Dispatch record").
|
|
- **exdata is read by the merge** (owned by `job-config-merge`): the job's
|
|
env and limits live inside the exdata JSON (`sub_env_vars`, `limits`), so
|
|
the shared `src/lib` merge parses it to check the additive env rule, the
|
|
reserved `XCE_` prefix and "limits: the job may lower", and at dispatch
|
|
the controller writes the effective limits back into the exdata sent to
|
|
the worker, preserving every other key (including `meta`, used by
|
|
presets). Execution-side values (mode and options,
|
|
launch type, env, profile pre/post-exec) are read from the profile at
|
|
dispatch, as queue pre/post-exec is today (`get_queue_exec()`,
|
|
`controller/queue.c:199`).
|
|
- **Workers advertise their supported execution modes** and the scheduler
|
|
filters hosts by mode: `job-sandbox-execution` (the design picks the
|
|
mechanism, e.g. a host attribute or boolean resource).
|
|
|
|
### Profile configuration
|
|
- **One file per entity type**, as for hosts, queues, resources, limits,
|
|
users and scaler:
|
|
- `src/configs/applications.yaml`, top-level key `applications:` (named
|
|
after LSF `lsb.applications`; the user flag is `--app`);
|
|
- `src/configs/schema/applications.schema.json`, picked up by xceadmin's
|
|
file-to-schema naming rule (`xcecommon/validation.py:68`);
|
|
- `xcecommon/validation.py`: duplicate names rejected; cross-file
|
|
references checked (`rusage` entry names against `resources.yaml`,
|
|
`per:` values); aliases (no clashes, no chains); `XCE_*` env rejected;
|
|
a warning for env names that look like secrets;
|
|
- `src/schema/controller.capnp`: `applications: List(AppProfileConf)` in
|
|
`ClusterConfig` and in `Conf4Sched` (the scheduler runs the live merge).
|
|
`Conf4Worker` does not need it: the worker gets execution values through
|
|
`start_info` at dispatch.
|
|
- The apiserver already receives `ClusterConfig`
|
|
(`backend/convert.go:265`), so it gets the profiles with no new channel.
|
|
- **Shared config, per `config-design.md`:** controller, scheduler and
|
|
apiserver all need the same view (independence rule), and profiles change
|
|
at the rate of queues, as occasional admin edits (frequency rule).
|
|
- **The file is optional.** Without it there is one built-in profile
|
|
`default` (mode `direct`, no attributes), which is today's behaviour. A
|
|
profile named `default` in the file **replaces** the built-in one, so
|
|
admins can customise the default without setting `default_application`.
|
|
- **No secrets in profiles.** Profile env values and pre/post-exec paths
|
|
are visible to every user who can view profiles (`cluster:view`) and in
|
|
job details;
|
|
`xceadmin` warns on env names that look like secrets (`*TOKEN*`,
|
|
`*SECRET*`, `*PASSWORD*`, `*_KEY`), as for presets.
|
|
- **No override markers.** Every profile value binds the job (additive
|
|
rule), except the `rusage` entries' defaults, which the job may override.
|
|
Whether the job's own pre/post-exec is allowed is a profile setting
|
|
(`job_pre_post_exec: allow | forbid`).
|
|
- **Open to new sections.** Later specs add top-level sections to a profile
|
|
(`service:`, I/O claims, relationship templates, `container:`); the schema
|
|
must allow that without reshaping existing sections.
|
|
- Sketch (format decided in design):
|
|
|
|
```yaml
|
|
applications:
|
|
- name: openfoam
|
|
description: "OpenFOAM, MPI"
|
|
mode: direct
|
|
launch_type: openmpi
|
|
resreq:
|
|
select: "ib" # binding
|
|
rusage:
|
|
- { name: license_foam, per: job, default: 1 } # default reservation
|
|
- { name: mem, default: 4000 } # per task by default
|
|
tasks: { min: 32, max: 512 }
|
|
limits: { runtime: 24h, memory: 128G } # maximums
|
|
env: { FOAM_INST_DIR: /opt/openfoam }
|
|
job_starter: "module load openfoam &&"
|
|
task_starter: "/opt/xce/app/openfoam/bind.sh"
|
|
pre_job_exec: /opt/xce/app/openfoam/licence-out.sh
|
|
post_job_exec: /opt/xce/app/openfoam/licence-in.sh
|
|
pre_host_exec: /opt/xce/app/openfoam/scratch-setup.sh
|
|
post_host_exec: /opt/xce/app/openfoam/scratch-clean.sh
|
|
job_pre_post_exec: allow
|
|
|
|
# mode: sandbox and its options come with job-sandbox-execution;
|
|
# this spec accepts only mode: direct
|
|
- name: untrusted-python
|
|
mode: sandbox
|
|
sandbox: { inputs: ["/shared/in/"], outputs: ["/shared/out/"] }
|
|
job_pre_post_exec: forbid
|
|
```
|
|
|
|
Default reservations live in the profile's `rusage` entries (the job may
|
|
override them); other tuning such as `span[ptile=32]` and a task count of
|
|
64 is not in the profile: the admin publishes it as a site preset named
|
|
`openfoam`.
|
|
|
|
### Essentials and the additive rule
|
|
How each profile attribute combines with the job's own value (owned by
|
|
`job-config-merge`, which implements it):
|
|
|
|
| Profile attribute | Holds | The job may |
|
|
|---|---|---|
|
|
| resreq `select` | required terms (`ib`, GPU model, OS) | add terms (ANDed) |
|
|
| resreq `rusage` | entries with optional `per:` and `default` | add resources; override a profile default (job's term wins; unit must match, see below) |
|
|
| resreq `span` / `same` / `order` | only if essential (e.g. a code that runs on one host only) | set them if the profile does not; otherwise rejected |
|
|
| task range | `min` / `max` | choose within it; no `-n` means `min` |
|
|
| env | variables the application needs | add others; not redefine these |
|
|
| launch type, mode, sandbox options, job starter, task starter | fixed values | nothing |
|
|
| `job_pre_post_exec` | `allow` / `forbid` | use its own pre/post-exec only if `allow` |
|
|
| limits | maximums | ask for less (checked like a ceiling) |
|
|
| pre/post-exec | outer layer, always runs | add its own inner layer if the profile allows |
|
|
|
|
**Structured profile resreq; `rusage` defaults, the job's term wins** (user
|
|
decisions 2026-10-12). A profile's resreq is not one string but separate
|
|
parts, so each part's rule is visible from its shape:
|
|
|
|
```yaml
|
|
resreq:
|
|
select: "ib && os == rhel9" # binding: ANDed with the job's select
|
|
rusage: # defaults: the job may override per resource
|
|
- { name: license_foam, per: job, default: 1 }
|
|
- { name: mem, default: 4000 } # per: optional, defaults to task
|
|
- { name: gpu } # unit only: nothing reserved by default
|
|
span: "ptile=32" # binding if set
|
|
same: [switch]
|
|
order: "r15s"
|
|
```
|
|
|
|
| Part | Merge with the job's `-R` (an ordinary resreq string, parsed into terms) |
|
|
|---|---|
|
|
| `select` | ANDed with the job's |
|
|
| `rusage` entry with `default` | reserved at that value unless the job names the resource; then the job's term wins **whole** (value and `duration`/`decay`), in either direction |
|
|
| `rusage` entry without `default` | nothing reserved unless the job asks; it only fixes the unit |
|
|
| `per:` (optional, default `task`) | the unit for that resource: a job term without a per-type gets it; a job term with a **different** per-type is rejected at submit ("resource mem is per task in profile openfoam"), never silently reinterpreted (16000/job read as per task multiplies by the task count) |
|
|
| a resource only the job names | the job's term as written |
|
|
| `span` / `same` / `order` | the profile's if set (binding), otherwise the job's |
|
|
|
|
- `per:` defaults to `task`, today's behaviour: `PER_DEFAULT` takes the
|
|
caller's per-type (`lib/resreq.c:842`) and the controller passes
|
|
`PER_TASK` (`controller/job.c:3909`); `resources.yaml` has no per-type
|
|
field. A per-resource default there may come later.
|
|
- **A 0-valued term is removed by the merge** (decided 2026-10-12, user):
|
|
after combining profile and job, every `rusage` term whose value is 0 is
|
|
dropped, so the scheduler sees exactly "no term". `name=0` is how a job
|
|
drops a profile default. Needed because the scheduler does not treat 0 as
|
|
absent today: a 0 term still becomes a wanted entry
|
|
(`calc_wanted_host_resvals()`, `lib/resreq.c`), and `unsatisfy_()`
|
|
(`scheduler/src/resource.c:164`, `total < inuse + uncomit + wanted`)
|
|
rejects a host where that resource is already over-committed
|
|
(`inuse > total`, possible because memory is charged
|
|
`max(reserved, actual)`), and the term shows in decision-trace rows and
|
|
pend reasons; with no term the resource is not checked at all (fast path
|
|
at `resource.c:351`). Applies to every 0 term (profile, job, preset, or a
|
|
job without a profile). Job details still show "license: 0 (overrides the
|
|
profile default)".
|
|
- `xceadmin` validates each `rusage` entry against `resources.yaml` (name
|
|
exists) instead of parsing a string; later per-resource fields
|
|
(`duration`, a `min:` floor for counted resources such as licences, if
|
|
under-reservation proves a problem) slot into the entry without new
|
|
grammar.
|
|
- Consequence: `rusage` is the one profile part that holds **defaults**; a
|
|
profile cannot enforce a minimum reservation. Hard protection of shared
|
|
resources comes from project quotas (`limits.yaml` is retired, 2026-10-12).
|
|
- Side effects to handle in requirements: measured resources (memory) are
|
|
charged `max(reserved, actual)` once usage is reported
|
|
(`resource-accounting.md`), so lowering them mostly self-corrects, but
|
|
counted resources (licences) do not; xcecli and `--dry-run` warn when a
|
|
job lowers a profile default, and when a preset restates one (it would
|
|
pin the job against later profile changes); job details label each
|
|
`rusage` term as profile default or job override; queue ceilings and
|
|
project quotas check the merged value and the error names the job's
|
|
override; a profile `rusage` change changes the content hash, so
|
|
in-flight dispatches re-pend and pending jobs re-merge even if they
|
|
override that resource (accepted).
|
|
- A job with no `-R` gets the profile's binding terms and its `rusage`
|
|
defaults; the scheduler's existing empty-resreq default (`type == any`)
|
|
still applies to what neither sets.
|
|
- **Profile kind is checked** (decided 2026-10-12): a batch job naming a
|
|
`kind: service` profile is rejected ("profile X is a service profile");
|
|
the services specs reject the reverse.
|
|
- **A `select` no host matches** (decided 2026-10-12): a warning, not a
|
|
rejection, at submit and from `xceadmin` against `hosts.yaml`, since
|
|
host attributes can change (e.g. IB nodes added later).
|
|
- **The `service:` section is configured in the profile** (user,
|
|
2026-10-12; defined with this spec, consumed by `service-supervision`,
|
|
`service-replica-set`, `service-endpoint`) and follows the same rule:
|
|
- restart policy (always / on-failure / never, backoff, give-up),
|
|
liveness and readiness probes, and named ports are the application's
|
|
definition: profile values, the service cannot change them;
|
|
- replicas work like the task range: the profile may set `min` / `max`,
|
|
and the service chooses its count within them (from its own spec or a
|
|
preset);
|
|
- spread rule and rollout settings (max unavailable / surge): if the
|
|
profile sets them they bind; otherwise the service sets them.
|
|
- Other later sections (I/O claims, relationship templates, `container:`)
|
|
state in their own specs how they follow the additive rule.
|
|
- **Compound and alternative resreqs** (decided 2026-10-12, user,
|
|
simplified). Parts are unnamed today: only the profile's `select` applies,
|
|
ANDed into every part and every `||` alternative. If the profile has any
|
|
`rusage` entry with a `default`, or sets `span` / `same` / `order`, a
|
|
compound or alternative job resreq is **rejected** with an error naming
|
|
the part, until PR-15 named parts let a profile target a part by role.
|
|
`rusage` entries without a default only fix a unit and do not block.
|
|
Admin guidance (not enforceable): no part-specific property (e.g. a GPU
|
|
model only one role needs) in a profile's `select`.
|
|
|
|
### Hooks and starters
|
|
Decided 2026-10-12 (user). Run by XCE's worker, never by the job:
|
|
|
|
| Item | Runs | Typical use |
|
|
|---|---|---|
|
|
| `pre_job_exec` / `post_job_exec` | once, on the first host, before the job / after all tasks end | licence checkout and check-in, staging |
|
|
| `pre_host_exec` / `post_host_exec` | once per host, before its first task / after its last | scratch, local cache, cleanup (LSF `HOST_PRE_EXEC` / `HOST_POST_EXEC`) |
|
|
| `job_starter` | wraps the job command once (the `mpirun` line for MPI) | `module load ... &&` |
|
|
| `task_starter` | wraps every task | `apptainer exec img`, binding wrappers |
|
|
|
|
- **No per-task pre/post hooks.** With N tasks per host they would run N
|
|
times concurrently: setup needs locking, and "the last task cleans up"
|
|
needs reference counting that a SIGKILL breaks. For MPI jobs XCE does
|
|
not start the ranks (it builds one `mpirun` line and sets `ntasks = 1`,
|
|
`worker/src/controller_connector.c:993-1010`), so it could not run hooks
|
|
around them, and a wrapper there runs in the job's env. A task starter
|
|
can do per-task work itself (it is a wrapper).
|
|
- **Task starter reach (correction):** for serial and non-MPI parallel jobs
|
|
the worker wraps each process it starts (`nranks` per host, remote hosts
|
|
through `xce_taskhost_t`); for MPI jobs XCE inserts it into the `mpirun`
|
|
command line it builds (`create_mpi_cmdline()`: `mpirun <opts>
|
|
<task_starter> <app>`), so the MPI runtime starts every rank through it.
|
|
Only an application that calls its own launcher inside the job script
|
|
bypasses it. A serial job's command is wrapped by both starters,
|
|
`job_starter` outermost (design confirms).
|
|
- **Order on a multi-host job:**
|
|
```
|
|
first host: pre_job_exec → job's own pre-exec → job_starter(job command) … job's own post-exec → post_job_exec
|
|
every host: pre_host_exec → task_starter(task) … (all its tasks end) → post_host_exec
|
|
```
|
|
`pre_job_exec` finishes before any host runs `pre_host_exec`;
|
|
`post_job_exec` runs after every host's `post_host_exec`. The job's own
|
|
pre/post-exec stays job-level (inner layer, allowed by
|
|
`job_pre_post_exec`).
|
|
- **Clean environment** for all four hooks: system values, the profile env,
|
|
`XCE_APP_*`; never the job's env ("Environment of the outer layer").
|
|
- **Per-host helpers:** XCE exports `XCE_JOB_HOST_DIR` (a per-job, per-host
|
|
directory the worker creates before `pre_host_exec` and removes after
|
|
`post_host_exec`) and `XCE_HOST_NTASKS` / `XCE_HOST_TASK_INDEX`, so
|
|
starters and the application know their host-local position without
|
|
MPI-specific variables. A task starter may use a lock in
|
|
`XCE_JOB_HOST_DIR` for best-effort once-per-host work (e.g. a cache
|
|
warm-up), never for cleanup that must happen.
|
|
- **Failures** (decided, proposal 4): `pre_job_exec` fails → nothing else
|
|
starts, `post_job_exec` still runs, the job goes to EXIT with a reason
|
|
naming the hook; `pre_host_exec` fails on a host → its tasks do not
|
|
start, its `post_host_exec` still runs, the job fails. Requeue on hook
|
|
failure belongs to `job-requeue`. (Today `queue_pre_exec()` just exits:
|
|
`TODO: Handle pre-exec failure`.)
|
|
- **The post hooks must survive every way a job ends.** Checked
|
|
2026-10-12: a graceful kill is SIGINT → SIGTERM → SIGKILL with a grace
|
|
period (`worker/src/job_signal.c:52-55`); a forced kill is SIGKILL to the
|
|
job's processes; the run limit is systemd `RuntimeMaxUSec` on the unit
|
|
(`worker/src/xcecd_cgroups.c:753-766`), which stops the unit with
|
|
`KillMode=control-group`, `KillSignal=SIGTERM` (`:195-230`), the worker
|
|
included. The worker's `signal_term()` (`worker/src/xcecd_worker.c:293`)
|
|
forwards SIGTERM to the job and keeps running, so post-exec runs today,
|
|
but only until systemd's `TimeoutStopUSec` (unset by XCE, system default,
|
|
typically 90 s) SIGKILLs everything left in the unit. Requirements: the
|
|
unit gets an explicit stop timeout covering the post hooks (with a
|
|
configurable maximum hook duration), and design checks how the worker
|
|
survives an OOM kill inside its own unit.
|
|
- **Migration:** design checks on which hosts today's queue
|
|
`pre_exec`/`post_exec` runs (in `xcecd_worker`) and maps it to the
|
|
matching hook, so behaviour does not change.
|
|
|
|
### Environment of the outer layer and reserved names
|
|
Decided 2026-10-12 (user), replacing the idea of an env denylist:
|
|
- **A. The profile's pre/post-exec runs in a clean environment**: system
|
|
values (`XCE_*`, `HOME`, a fixed `PATH`) plus the profile's own env, never
|
|
the job's. The job's env applies after it, to the job's pre-exec, the job
|
|
and the job's post-exec. An admin script that needs job facts reads
|
|
`XCE_*`. Today the job env is applied first (`apply_env_vars()`,
|
|
`worker/src/xcecd_worker.c:835`, then `queue_pre_exec()` :843), so a job's
|
|
`PATH` or `BASH_ENV` can subvert the admin's script. Worker change in this
|
|
spec: the outer layer gets an explicitly built env (`uv_proc_pre_exec`
|
|
takes none today). It also fixes today's queue pre/post-exec.
|
|
- **The profile env travels separately from the job env** (decided
|
|
2026-10-12, review). Today all env is one `exdata.sub_env_vars` object,
|
|
also sent to remote tasks; if the profile env were merged into it, the
|
|
worker could not build the clean environment of A. The profile env goes
|
|
in its own `start_info` field (or an `exdata` key, decided in design), to
|
|
the head and every remote task; the job's env stays in `sub_env_vars`.
|
|
The job process gets both (profile first, the job's additions on top,
|
|
never redefining a profile variable).
|
|
- **B. The `XCE_` prefix is reserved**: neither a job nor a profile may set
|
|
`XCE_*` variables (rejected at submit by apiserver and controller, and by
|
|
`xceadmin` for profiles).
|
|
- Why no denylist: the whole job unit runs as the execution user
|
|
(`User=` on the transient unit, `worker/src/xcecd_cgroups.c:299-312`), in
|
|
sandbox mode inside the sandbox, so `LD_PRELOAD` and similar are not an
|
|
escalation; the real risks were the admin's outer layer and XCE's own
|
|
variables, which A and B close at the source. Sandbox-specific env
|
|
exposure stays with `job-sandbox-execution`.
|
|
|
|
### Presets (client side, `xcecli-presets`)
|
|
Not part of this spec, recorded here because they hold the convenience
|
|
defaults a profile does not (everything but default reservations):
|
|
- A preset is a set of job options (`--app`, `--queue`, `-R`, `-n`, env,
|
|
limits). The client expands it; a command-line flag replaces the preset's
|
|
value for that option as a whole. The submit carries only the expanded
|
|
values.
|
|
- Personal presets live one file per preset under `~/.config/xce/presets/`.
|
|
- The apiserver **only shares**: admin-deployed preset files served
|
|
read-only (`GET /xce/v1/presets`, `GET /xce/v1/presets/{name}` or
|
|
`{ns}/{name}`) from `<TopDir>/config/presets/`. Pulling copies a preset
|
|
into its own local file, so a server preset being removed never affects
|
|
users. `preset diff` / `pull --force` are user-initiated.
|
|
- User-to-user sharing by `preset export` / `import` (file, mail, git).
|
|
- Presets have no access control of their own (user, 2026-10-12): the
|
|
preset routes need no RBAC permission, and a submit made from a preset is
|
|
checked like any other submit. Presets carry no privilege.
|
|
|
|
### Dispatch record
|
|
Decided 2026-10-12 (user): what a job was scheduled and started with is
|
|
**frozen at dispatch**. A pending job follows the current profile; a
|
|
dispatched job uses only its dispatch record until it is requeued or
|
|
restarted, and the scheduler no longer re-merges it.
|
|
- The record holds both sides:
|
|
- scheduling: the effective resreq (`select`, `rusage`, `span`, `same`,
|
|
`order`) and task count, so a running job's reservations (e.g.
|
|
`rusage[license=1]`) do not change when the profile changes, and are
|
|
rebuilt identically after a scheduler failover or restart;
|
|
- execution: mode, mode options, launch type, job starter, task
|
|
starter, the
|
|
profile's env and the job's env (separately), outer pre/post-exec, the
|
|
`job_pre_post_exec` setting, effective limits;
|
|
- the resolved profile name (after aliases) and the content hash the
|
|
record was built from.
|
|
- At each dispatch the controller puts the execution values into the job's
|
|
`start_info` for the worker, and the whole record into an additive
|
|
payload on the WAL dispatch fact
|
|
(`persist_submit_job_dispatch_fact`, `controller/persist.c:8167`) and the
|
|
snapshot. Both apply seams decode it; `/xce-audit-invariants` checks the
|
|
round trip.
|
|
- Today the queue's pre/post-exec copied at dispatch (`controller/job.c`
|
|
1825-1897) is kept in memory only, so after a failover the new primary
|
|
does not know what a running job was given. The frozen record closes that
|
|
gap as pre/post-exec moves to the profile.
|
|
- A running or finished job's details show the frozen values, and say
|
|
"profile changed since dispatch" when the current profile differs.
|
|
- **Requeue or restart re-reads the profile** at the next dispatch, the same
|
|
as the live merge does for pending jobs.
|
|
- Running jobs therefore need nothing from the live profile: post-exec at
|
|
the end, later remote tasks or a resize, and a service replica's probes
|
|
and in-place restarts all use the record.
|
|
- Why: for sandboxed jobs the mode and mounts a job actually ran with is a
|
|
security/audit fact, and `job-sandbox-execution`'s mandatory cleanup
|
|
depends on it; `persistence.md` does not allow memory-only state without a
|
|
documented scope decision.
|
|
|
|
### Profile removed while jobs use it
|
|
Decided 2026-10-12 (user): **a job whose profile no longer exists pends**,
|
|
with the reason "application profile X is not defined".
|
|
- This applies to every job that reads the profile: pending jobs, requeued
|
|
or restarted jobs, and new service replicas (replacements, scale-up),
|
|
which are new jobs. Running jobs are unaffected (dispatch record); a
|
|
service whose profile is gone keeps its running replicas but shows as
|
|
degraded when one cannot be replaced.
|
|
- Putting the profile back releases the jobs. A rename is remove plus add.
|
|
- The removal is never refused (`xceadmin config build` works offline and
|
|
cannot know live jobs) and jobs never exit because of it.
|
|
- The controller's live-upgrade reply lists removed profiles still named by
|
|
pending jobs or services, with counts; `xceadmin config upgrade` prints
|
|
it. Needs a new scheduler pend reason (profile missing from
|
|
`Conf4Sched`).
|
|
- **Between the scheduler's decision and the controller's dispatch**
|
|
(decided 2026-10-12, user). The scheduler merges with its copy of the
|
|
profile (`Conf4Sched`); the controller reads the profile again when it
|
|
builds the dispatch record. If the profile is gone or has changed in
|
|
between, the **dispatch fails and the job goes back to PEND through the
|
|
requeue path** (re-reading the profile, as any requeue does): gone → it
|
|
pends with "application profile X is not defined"; changed → the
|
|
scheduler re-merges with the new profile. Never EXIT.
|
|
- New dispatch failure site (e.g. `XCE_DISPATCH_FAIL_SITE_APP_PROFILE`),
|
|
classified retryable, unlike today's terminal
|
|
`XCE_DISPATCH_FAIL_SITE_QUEUE_EXEC` (`controller/job.c`). Nothing is
|
|
committed on this path (`j->job.allocs` is committed only after a
|
|
successful `xce_startjob_req()`). Requirements decide that it does not
|
|
count as a run or against requeue limits.
|
|
- A scheduler still on the old config may re-dispatch and be declined
|
|
again until the controller's config push reaches it; the existing
|
|
dispatch retry backoff bounds that.
|
|
- After the controller has built the record, profile changes no longer
|
|
matter: the worker starts from `start_info`.
|
|
- **Profile content hash** (decided 2026-10-12, user): an internal version
|
|
of each profile.
|
|
- Computed, never stored as state: one `src/lib` function hashes the
|
|
canonical Cap'n Proto encoding of the profile, excluding
|
|
`description`. Controller, scheduler and apiserver get the same value
|
|
from the same profile, also after failover or restart.
|
|
- The scheduler's dispatch decision carries the hash it merged with
|
|
(additive field); the controller compares it with the current
|
|
profile's hash at dispatch. A scheduler that sends no hash (rolling
|
|
upgrade) skips the check.
|
|
- The dispatch record stores it; "profile changed since dispatch" in job
|
|
details is a hash comparison; job details show it short
|
|
(`openfoam@3f2a9c`). Admins never set it.
|
|
- Base for profile history and audit (`config-history-audit`, backlog).
|
|
- **Moving pending jobs to another profile is not in this spec**
|
|
(decided 2026-10-12, user). XCE has no job modification today
|
|
(`JOB_MODIFY` is only an enum, `include/xce/types.h:620`; the apiserver's
|
|
`"modify"` action is commented out, `controllers/jobs.go:2446`). A general
|
|
modify path (apiserver action, controller opcode, WAL fact, scheduler
|
|
re-merge) is its own backlog spec, `job-modify` (LSF `bmod`), with `--app`
|
|
among its first attributes. Until then: restore the profile, use an
|
|
alias, or kill and resubmit.
|
|
|
|
### Profile aliases
|
|
Decided 2026-10-12 (user), so a rename never strands jobs:
|
|
- A profile may list `aliases: [old-name, ...]`. An alias resolves to the
|
|
profile everywhere: merge, dispatch record, content hash (the target's),
|
|
and, once `app-access-control` lands, its permission check (against the
|
|
target profile's name).
|
|
- Jobs keep the name they recorded; the scheduler and controller resolve it
|
|
through the alias. New submits that name an alias are accepted and record
|
|
the **target's** name, so the alias can be removed once old jobs drain.
|
|
- Job details show the resolution, e.g. `openfoam (recorded as
|
|
openfoam-old)`.
|
|
- `xceadmin` rejects an alias that clashes with a profile name or another
|
|
alias, and alias chains (an alias naming an alias). `default_application`
|
|
must name a real profile, not an alias.
|
|
- Removing an alias while jobs still name it is the same as removing a
|
|
profile: they pend, and the live-upgrade warning lists them.
|
|
- Rename workflow: rename the profile and add the old name as an alias;
|
|
drop the alias when no pending job uses it (the upgrade warning shows
|
|
when that is).
|
|
|
|
### Default profile
|
|
- **Set by `default_application: <name>` in `xce.yaml`**, a cluster
|
|
parameter (as LSF keeps `DEFAULT_APPLICATION` in `lsb.params`), not a flag
|
|
on one profile.
|
|
- **Recorded at submit.** A job submitted without `--app` gets the current
|
|
default's *name* written on its record, so changing the default later does
|
|
not move pending jobs to another profile.
|
|
- **An empty profile name means "no profile"**, never "the current
|
|
default": jobs submitted before profiles existed (old WAL records and
|
|
snapshots) keep today's behaviour and are never silently given
|
|
`default_application`. Only new submits record the default's name.
|
|
- **Restart carries the profile name.** `POST /:jobids/restart` rebuilds the
|
|
new job field by field from the old job's record (bug-1968 shows what is
|
|
lost otherwise), so it must copy the profile name (or alias); it then
|
|
re-reads the profile like any new dispatch (and, once
|
|
`app-access-control` lands, re-checks `app:use`).
|
|
- **Array jobs** carry one profile name on the array's submit data; every
|
|
element uses it.
|
|
- **A name that is not defined is a configuration error, never a silent
|
|
fallback** (user decision, 2026-10-12):
|
|
|
|
| `default_application` | Behaviour |
|
|
|---|---|
|
|
| absent | built-in `default` profile (mode `direct`, no attributes), today's behaviour |
|
|
| names a defined profile | that profile is used for jobs submitted without `--app` |
|
|
| names an undefined profile | the configuration is rejected |
|
|
|
|
- `xceadmin config build` fails with an error naming the missing profile.
|
|
This is an error, not one of the warnings `validation.py` gives for
|
|
unknown host or user references: a dangling default would break every
|
|
submit without `--app`.
|
|
- The controller re-checks on live upgrade and at start, because config can
|
|
reach it without xceadmin. A live upgrade with a dangling default,
|
|
including one that removes or renames the current default profile, is
|
|
refused and the current config stays in force. At start the controller
|
|
refuses to run and logs the reason.
|
|
- Profiles and the default are in the same configuration, checked
|
|
atomically, so the default cannot become dangling at run time.
|
|
- **Queue deprecations surface here.** `xceadmin config build` warns when
|
|
`queues.yaml` still sets `res_req` or `pre_exec`/`post_exec` (the shipped
|
|
`night` queue does) and suggests moving them into a profile. The
|
|
deprecation itself is `job-config-merge`'s; the message needs
|
|
`applications.yaml` to exist.
|
|
|
|
## Scope
|
|
- **In**:
|
|
- The profile schema (essentials, additive rule, aliases) and
|
|
`applications.yaml`, and the default profile (`default_application`).
|
|
- Submit-time validation of the profile and the additive rule.
|
|
- Submit surfaces: REST, `xcecli --app`, harness, libxce.
|
|
- The profile name and the job's own values persisted on the job.
|
|
- The `mode:` schema field (only `direct`), the structured resreq schema
|
|
(`select`, `rusage` entries with `per:` / `default`, `span`, `same`,
|
|
`order`), the job and task starters, `params` (`--app-param`) and
|
|
annotations,
|
|
exported as `XCE_APP_PARAM_*` / `XCE_APP_ANN_*`.
|
|
- New `start_info` fields (job starter, task starter, profile env
|
|
separate from the job
|
|
env) and exdata handling with `job-config-merge`.
|
|
- Profile content hash (shared `src/lib` function) and its field on the
|
|
scheduler's dispatch decision.
|
|
- Restart and array submits carrying the profile name.
|
|
- A worker launch error when a `direct` exec user is missing.
|
|
- Dispatch record (scheduling and execution values, content hash) and the
|
|
retryable dispatch failure for a missing or changed profile.
|
|
- Profile pre/post-exec as the outer layer (moved from the queue), with
|
|
a clean environment; the reserved `XCE_` prefix.
|
|
- Aliases, live-upgrade warning, pend reason for a missing profile.
|
|
- Profile name and mode shown in job details.
|
|
- Listing available profiles (REST and xcecli).
|
|
- Profile `--dry-run` support: the effective values a job would get
|
|
(shared with `job-config-merge`).
|
|
- Tests.
|
|
- **Open for requirements / design**:
|
|
- Hooks and starters (decided, "Hooks and starters"); design confirms
|
|
the serial-job wrapping order, which hosts today's queue pre/post-exec
|
|
runs on, the unit stop timeout and maximum hook duration, and how the
|
|
worker survives an OOM kill.
|
|
- Confirm a dispatch declined for a missing or changed profile does not
|
|
count as a run or against requeue limits.
|
|
- ~~How resource defaults combine with a job's own resreq~~ — settled
|
|
2026-10-09, owned by `job-config-merge`.
|
|
- Settled 2026-10-12: scheduling and execution values frozen at
|
|
dispatch, re-read on requeue ("Dispatch record"); a removed profile
|
|
makes its jobs pend ("Profile removed while jobs use it"); clean env for
|
|
the outer layer and reserved `XCE_` instead of an env denylist; moving
|
|
pending jobs deferred to `job-modify`, profile aliases in v1; override granularity (no markers; additive rule),
|
|
limits a job may lower (profile limits are maximums), personal presets
|
|
(client side), `app:override` and inheritance (deferred); execution modes
|
|
split into `job-sandbox-execution`; v1 attributes are the four plus the
|
|
job starter (plus a task starter for every task startup, decided
|
|
2026-10-12), others ship with their job features; profile history and
|
|
change audit deferred to the roadmap (`config-history-audit`): the
|
|
content hash and the dispatch record already show what each run used;
|
|
no job labels needed here (presets record themselves in `exdata.meta`;
|
|
general labels are the `job-labels` backlog item); compound and
|
|
alternative resreqs (only the profile `select` applies); structured
|
|
resreq with `per:` default `task`; separate profile env; empty profile
|
|
name = no profile; a file profile named `default` replaces the built-in;
|
|
0-valued `rusage` terms removed by the merge; `params` and annotations
|
|
(exported to env) in v1; submit hook deferred (`app-submit-hook`).
|
|
- **Out**:
|
|
- The `sandbox` mode and its options, I/O directories, capability
|
|
advertisement, placement by mode and the worker sandbox mechanics
|
|
(`job-sandbox-execution`).
|
|
- Multi-node sandbox (`job-sandbox-multinode`).
|
|
- Container or vm execution itself (only room for them in the schema).
|
|
- Auth backends and JWT claims.
|
|
- Per-profile access control (`app:use` / `app:view`): deferred to
|
|
`app-access-control` (user decision, 2026-10-12).
|
|
|
|
## Boundary Candidates
|
|
- The profile model and config (schema, validation, live upgrade).
|
|
- Submit-time validation of the profile and the additive rule (controller).
|
|
- Early validation (apiserver).
|
|
- Submit and listing surfaces (REST, xcecli, harness, libxce).
|
|
- Persisting the profile name and the job's own values (wire, WAL submit
|
|
fact, snapshot).
|
|
|
|
## Out of Boundary
|
|
- Worker execution backends and sandbox mechanics.
|
|
- Output file location and retrieval inside the sandbox.
|
|
- Scheduler policy beyond capability filtering and the profile's essential
|
|
resreq.
|
|
- Presets (`xcecli-presets`): client side; the read-only preset listing on
|
|
the apiserver belongs to that spec.
|
|
|
|
## Upstream / Downstream
|
|
- **Upstream**:
|
|
- The existing submit path.
|
|
- The WAL submit fact (`ha-wal-replication`, the `raft-apply-decoupling` apply
|
|
seam).
|
|
- The config pipeline (`clusterconfig-schema-split`, `config-upgrade-snapshot`,
|
|
`admin-config-tool`).
|
|
- RBAC (`rbac_check.go`), only for the reused `cluster:view` on the
|
|
listing route.
|
|
- The scheduler-to-controller dispatch decision (gains the profile
|
|
content hash; `scheduler-dispatch-commit`).
|
|
- The restart path (`RestartJobByID`, bug-1968).
|
|
- **Downstream**:
|
|
- `job-sandbox-execution` adds the `sandbox` mode, its options and I/O
|
|
directories, capability advertisement and placement, on top of this
|
|
spec's `mode:` field and dispatch record.
|
|
- `job-sandbox-multinode`.
|
|
- Future `container`/`vm` modes, which add profile attributes.
|
|
- `job-status-detail-display`, which can show the profile and mode.
|
|
- `job-modify` (moves pending jobs between profiles), `job-requeue`
|
|
(requeue/success exit values as profile attributes),
|
|
`config-history-audit` (profile history), `xcecli-presets` (reads
|
|
profiles for warnings; `exdata.meta`).
|
|
|
|
## Existing Spec Touchpoints
|
|
- **Extends**: none directly.
|
|
- **Depends on**: `job-config-merge` (merge rules and live re-evaluation).
|
|
- **Feeds**: `parjob-placement` (the profile's `span[...]`).
|
|
- **Adjacent**:
|
|
- `job-status-detail-display`
|
|
- `controller-job-cleanup`
|
|
- `resreq-maxtasks-wiring` (task range)
|
|
- `xcecli-presets` (convenience defaults, client side)
|
|
- `profile-usage-insight` (advisory recommendations)
|
|
- `xce-node-manager`
|
|
|
|
## Constraints
|
|
- Middleware chain unchanged: WhitelistURL → WhitelistIP → RateLimit → JWT → CORS
|
|
→ RBAC.
|
|
- All wire changes are additive Cap'n Proto. Old WAL records and snapshots
|
|
decode as "no profile, mode `direct`", which is today's behaviour; an
|
|
empty profile name is never resolved to `default_application`.
|
|
- Save/restore field completeness (`persistence.md`): the profile name, the
|
|
job's own values, the new `start_info` fields and the dispatch record
|
|
round trip through the WAL submit and dispatch facts and the snapshot
|
|
(`/xce-audit-invariants controller`). The scheduler's own state for
|
|
dispatched jobs (frozen, no longer re-merged) and its `Conf4Sched`
|
|
profiles round trip through the scheduler snapshot and replay recordings
|
|
(`/xce-audit-invariants scheduler`).
|
|
- Input validation (`security.md`): the job's additions and paths are validated
|
|
server-side. No path traversal, and mount prefixes are restricted by the
|
|
profile.
|
|
- Size: still large after the split (config, access and submit, job state,
|
|
outer layer); one coherent feature, testable end to end with `direct`.
|
|
- HA/persistence spec: use the full 3-phase workflow.
|
|
|
|
## Future Discussion
|
|
Everything left to discuss, collected 2026-10-12 at the end of the brief
|
|
discussion. Nothing here blocks requirements.
|
|
|
|
### For this spec's design phase
|
|
1. **Serial job wrapping:** both starters wrap a single-task job's command;
|
|
confirm `job_starter` is outermost.
|
|
2. **Today's queue pre/post-exec:** on which hosts it runs (in
|
|
`xcecd_worker`), so it maps to `*_job_exec` or `*_host_exec` without a
|
|
behaviour change.
|
|
3. **Kill paths:** the unit's explicit stop timeout and a configurable
|
|
maximum hook duration (today post-exec is bounded only by systemd's
|
|
unset `TimeoutStopUSec`); how the worker survives an OOM kill inside its
|
|
own unit.
|
|
4. **Declined dispatch:** confirm a dispatch declined for a missing or
|
|
changed profile counts neither as a run nor against requeue limits.
|
|
5. **Field placement:** the profile env (a `start_info` field or an exdata
|
|
key), the task starter on each `xce_taskhost_t`, renaming the queue-named
|
|
`qPreExec` / `qPostExec`, and the dispatch-record payload layout.
|
|
6. **Task starter reach:** which launch types start tasks through the
|
|
worker and which through the MPI runtime; inserting the starter into
|
|
`create_mpi_cmdline()` for `openmpi` and `mpich`.
|
|
7. **Content hash:** canonical encoding, algorithm and length; which fields
|
|
besides `description` are excluded.
|
|
8. **params and annotations:** pattern syntax, count and size limits, env
|
|
name conversion edge cases; an opt-out for exporting annotations only if
|
|
someone needs it.
|
|
9. **Re-merge cost:** with the live merge, a profile change re-merges every
|
|
pending job of that profile in the scheduler; check the cost for large
|
|
pending queues and whether re-merging is batched or lazy.
|
|
10. **Messages:** the live-upgrade warning format, the new pend reason
|
|
("application profile X is not defined"), the `xceadmin` secret-name
|
|
patterns, and the warning when no host matches a profile's `select`.
|
|
|
|
### Deferred features (roadmap backlog)
|
|
| Item | What | Revisit when |
|
|
|---|---|---|
|
|
| `app-access-control` | per-profile `app:use` / `app:view` (design kept above) | profiles must be restricted to some users |
|
|
| `job-modify` | modify pending jobs (`--app`, `-R`, `-n`, queue; bulk `--app-from`) | renames or profile moves become common |
|
|
| `job-requeue` | requeue/success exit values (profile attributes), requeue on hook failure | an application needs automatic requeue |
|
|
| checkpoint, warning-before-kill | their own specs, each adding its profile attribute | an application needs them |
|
|
| output path template | PR-6/PR-7 I/O track | output options are supported |
|
|
| `job-labels` | server-side labels, filtering, summaries; reuses REST `tags` (bug-1970) | team-wide comparison or services need labels |
|
|
| `config-history-audit` | profile history, change audit, lookup by content hash | admins need to see past profiles |
|
|
| `app-submit-hook` | `esub`-style submit hook; client or apiserver | params, annotations, starters and hooks are not enough |
|
|
| `profile-usage-insight` | advisory suggestions for site presets and profile defaults | usage data is available per profile |
|
|
| `xcecli-presets` | client-side presets (brief exists, no `spec.json`) | after this spec's requirements |
|
|
| `app:override` | privileged one-off override of profile values | `job-modify` exists and admins still need it |
|
|
| profile inheritance (`base:`) | variants of a profile | presets prove insufficient for variants |
|
|
| counted-resource protection | a `min:` floor on a profile `rusage` entry, or a "counted" marker in `resources.yaml` | licence under-reservation is a real problem |
|
|
| per-resource default per-type | a per-type field in `resources.yaml` | a resource is almost always per host or per job |
|
|
| per-part profile terms | profile `rusage` / `span` / `same` / `order` for compound jobs | PR-15 named parts land |
|
|
| per-host hooks for the job | the job's own per-host pre/post-exec | users ask for it |
|
|
| writable server-side presets | publish presets from the portal | the portal needs it, after `rbac-persistence` |
|
|
| portal | preset storage, submit forms from `params` | portal work starts |
|
|
|
|
### Housekeeping
|
|
- Regenerate `job-config-merge`'s `requirements.md` after this spec's
|
|
design (it predates the structured resreq, the `rusage` rule, `params`,
|
|
exdata handling, frozen dispatched jobs and the `min`/`max` task range).
|
|
- Mirror bug-1970 to Gitea (`/gitea-sync`).
|
|
- `spec.json` for `xcecli-presets` and the backlog items when they are
|
|
promoted (gitea-sync skips briefs without one).
|
|
- Next step: `/kiro-spec-init application-profile`, then
|
|
`/kiro-spec-requirements application-profile`.
|