myxce/.kiro/specs/application-profile/brief.md

# Brief: application-profile

(Reshaped from `job-execution-mode` on 2026-09-29, by user direction. Execution
mode is an attribute of the job, determined by the **application** the user runs,
so it is carried by a named application profile rather than by a bare mode
option.)

**Update 2026-10-09 (`/kiro-discovery`, configuration layering; user
decisions):**
- Configuration is layered LSF-style: **queue** = scheduling policy only (no
  resreq; queue `res_req` is deprecated), **application profile** =
  application execution constraints *including the job's resource
  requirement* (select / rusage / `span[...]` / same) and task limits
  (min / default / max, LSF TASKLIMIT), **project** = resource ownership
  (`project-config`).
- Every attribute has one owning level (`.kiro/steering/job-config-layering.md`).
  How the levels interact is owned by **`job-config-merge`**; queue
  ceilings and project quotas only check, never change, the job's values.
  (The job-vs-profile rule of this update, *default*/*fixed* markers, is
  replaced by the additive rule of 2026-10-12 below.)
- **Pre/post-exec moves here from the queue** (queue `pre_exec`/`post_exec`
  deprecated). The profile's pre/post-exec is admin-defined and always runs
  as the outer layer; the job's own pre/post-exec is the inner layer, allowed
  or forbidden by the profile. Order: profile pre → job pre → job → job post
  → profile post (the worker's existing two-layer structure,
  `xcecd_worker.c` `queue_pre_exec()`/`queue_post_exec()`).
- **The merge is *live*** (replaces the earlier "resolution at submit time",
  removed from this brief 2026-10-12). The WAL
  keeps the job's own values plus the profile name; the controller and
  apiserver validate at submit, and the scheduler computes effective values
  and recomputes them when a profile changes (a pending job that no longer
  satisfies its profile pends with a reason; running jobs are unaffected).
  Execution-side values (mode, options, launch type, env, pre/post-exec) are
  still taken from the profile at dispatch time and frozen on the dispatch
  record (decided 2026-10-12, "Dispatch record" below).

**Update 2026-10-10 (`/kiro-discovery`, long-running services):** a profile
may declare **kind `service`** with a service section: restart policy
(always / on-failure / never, backoff, give-up), liveness and readiness
probes (command / TCP / HTTP), named ports, replica range, spread rule and
rollout settings (how these follow the additive rule: "Essentials and the
additive rule" below). The section is defined with this spec and
consumed by `service-supervision` (worker probes and restarts),
`service-replica-set` (replicas, scaling, rolling update) and
`service-endpoint` (ports, endpoints).

**Update 2026-10-12 (discussion; user decisions):**
- **Split: execution modes move to `job-sandbox-execution`.** This spec is
  the profile core: config, access and submit, job state (name, dispatch
  record), the outer pre/post-exec layer. Its schema has `mode:` but
  accepts only `direct`. The `sandbox` mode with its options, I/O
  directories, worker capability advertisement and scheduler placement by
  mode belong to `job-sandbox-execution`, since nothing uses a non-`direct`
  mode before the sandbox exists. The worker's clear launch error for a
  missing `direct` exec user stays here.
- **Order with `job-config-merge`:** requirements and design of this spec
  first (it fixes the profile schema), then `job-config-merge`'s design;
  implementation tasks interleave (schema/config, merge library, submit
  validation and dispatch record).
- **Own configuration file.** Profiles live in a new
  `src/configs/applications.yaml` with its own JSON schema, mapped into
  `ClusterConfig` and `Conf4Sched` like queues and limits. See "Profile
  configuration" below.
- **Cluster default** is set by `default_application` in `xce.yaml`. A name
  that is not defined is a configuration error everywhere (`xceadmin`, live
  upgrade, controller start), never a silent fallback. See "Default profile"
  below.
- **This spec owns the profile's attribute schema** (which attributes exist
  and what each holds). `job-config-merge` consumes that schema and owns the
  merge and live re-evaluation.
- **A profile holds only essentials; convenience lives in client-side
  presets** (user decision, 2026-10-12). "The profile holds what the job must
  have; the preset holds what the user would like." A profile value is
  something the application needs or the admin enforces, and it binds every
  job of that profile. The *default*/*fixed* markers of 2026-10-09 are
  dropped: there are no markers, and no per-term markers or ranges. See
  "Essentials and the additive rule" below.
  - **Additive rule:** the job may add to the profile's values, never change
    or remove them. A conflict is rejected at submit with an error naming the
    profile value, never resolved silently.
  - **Convenience defaults** (tuning such as `rusage[mem=...]`,
    `span[ptile=...]`, task count, extra env) go into **presets**, handled
    fully by the client (`xcecli-presets`). The server only shares
    admin-deployed presets read-only; presets never reach the controller,
    WAL or scheduler and carry no privilege.
  - **Rule of thumb for admins:** something every job of the application must
    have (required host properties, licenses, launch type, mode,
    pre/post-exec, limits) goes into the profile; an optional variant or
    tuning shortcut goes into a site preset, typically one with the profile's
    name.
  - **Recommendations are advisory** (user decision): usage-based
    suggestions (requested vs. used, OOM kills, runtime by variant) are
    reported to the admin, who changes site presets (or a profile, for an
    essential); nothing tunes itself. Backlog: `profile-usage-insight`.
  - Job details show the effective resreq with each term's origin (profile
    or job).
  - Deferred: a privileged `app:override` for one-off exceptions (an admin
    can modify the pending job instead; recurring variants are presets), and
    profile inheritance (`base:`), no longer needed for variants.
- **Later sections are added by their consumers**: `service:`
  (`service-supervision`), I/O claims (resource-model PR-6), relationship
  and compound templates (PR-14/15), `container:` (R6). This spec only makes
  the schema open to new top-level sections.

## Problem
XCE users run many kinds of applications: trusted in-house codes, untrusted
scripts, vendor binaries, MPI solvers. Each needs a different way of running:
- directly as the submitter, or jailed in a sandbox under a separate execution
  identity (and later in a container or vm);
- a particular launch type (serial, `openmpi`, `mpich`);
- particular environment and pre/post-exec steps;
- resource requirements it cannot run without, and limits.

Today every one of these must be set per job by hand, and several (execution
mode, sandbox options) cannot be set at all. Many users log in through the
portal or REST API (JWT; `config_file`/PAM, later LDAP or OAuth2) and have **no
Unix account on the compute nodes**. They can only run jobs if their
applications run sandboxed. Admins have no way to say "this application runs
like this, and these users may run it".

This spec introduces **application profiles** and is the submission/policy half
of the job-sandbox initiative (roadmap "Job Sandbox Execution"). There is
precedent in LSF application profiles (`lsb.applications`, `bsub -app`), which
fits XCE's LSF-style job model.

## Current State
- **Identity.** A job's identity is only the JWT `username`
  (`controllers/auth.go:26-37`). `SubmitUser` comes from the token
  (`controllers/jobs.go:1379`, `:1459`), and `UserGroup` is taken from the client
  request unchecked (`:1465`).
- **Execution user.** It is always the submitter: `resolveExecUser`
  (`backend/convert_pure.go:162`, bug-1780).
- **Per-job settings that exist today** (`src/include/xce/types.h`):
  - `job_startinfo_t` (:240-251): `execuser`, `exechome`, `execcwd`, `cmdline`,
    `exdata`, `launch_type`, and pre/post-exec commands.
  - `exdata` carries the environment (`sub_env_vars`) and the worker cgroup
    limits parsed by `parse_limits` (MemoryMax, CPUQuota, TasksMax, RuntimeMax,
    ...). The apiserver never sends exdata (`backend/convert.go:273-315`).
  - The resource requirement is `resreq` on the submit data.
  - Remote tasks use `xce_taskhost_t` (:696-705).
- **Missing today:**
  - any profile concept, any execution-mode field, any input/output directory
    field;
  - any way for workers to advertise execution capabilities to the scheduler.
- **Controller.** It validates only that the user is not empty
  (`controller/job.c:2447`). The submit data, including `start_info`, is
  persisted in the WAL submit fact and in snapshots.
- **Ownership.** RBAC ownership, scheduler per-user limits and the user table all
  key on the **submit user**, not the exec user (`controllers/jobs.go:1010`,
  `:2542`, `output.go:95`; `schedobj.c:121`; `plugins/limit.c`). So a separate
  execution identity breaks none of them.
- **Access control.** RBAC is `config/rbac.yaml` plus `RequirePermission`
  (`controllers/rbac_check.go`).

## Desired Outcome
- **Admins define named application profiles** in the cluster configuration. The
  first version contains the following (user decision; the format must be
  extensible for "maybe others"):
  1. **Execution mode.** The `mode:` field, accepting only `direct` in this
     spec; `job-sandbox-execution` adds `sandbox` with its options (allowed
     input/output prefixes, isolation extras), later `container`/`vm`.
  2. **Launch type.** Serial, `openmpi` or `mpich`.
  3. **Environment and pre/post-exec.** Environment variables the application
     needs, and the outer pre/post-exec commands.
  4. **Essential resource requirement and limits.** The resreq terms the
     application cannot run without (select / rusage / `span[...]` / same /
     order), its task range (min / max), and per-job execution limits as
     maximums (memory, runtime, CPU, tasks), mapped onto the existing worker
     cgroup limits.
  All of these are essentials under the additive rule; none is a
  convenience default.
- **Users pick a profile at submit** (REST, `xcecli jobs submit --app <name>`,
  harness, libxce). A **cluster default profile** applies when none is given.
- **Jobs add, never override.** A job may add to its profile's values (more
  `select` terms, other resources, its own env variables, its own I/O
  directories within the allowed prefixes, lower limits) but not change or
  remove any. Anything else is rejected with a clear error.
- **Access by RBAC permission.** No new access mechanism; this follows the
  existing named-instance pattern (user decision, 2026-09-29: fold profile RBAC
  into this spec). Today, `queue:submit` scoped to `type: queue`,
  `values: [normal, priority]` in `config/rbac.yaml` is enforced by
  `GetRequireResourcePermission` in `controllers/jobs.go:1402-1409`. Profiles add:
  - **Permissions:** `app:use` (submit a job with a profile) and `app:view` (list
    and inspect profiles).
  - **Scope:** a new `app` scope type whose values are profile names, or `all`.
  - **Checks:**
    - Job create checks `app:use` alongside, and independent of, `queue:submit`.
    - Job restart (`POST /:jobids/restart`, which today checks only
      `queue:submit`) re-checks `app:use` for the job's profile.
    - Listing profiles checks `app:view`.
  - **Defaults:** `rbac.yaml` gets defaults, for example `basic_user` →
    `app:use` on the default profile, and `app:view` on `all`.
  - **Role API:** role creation and update accept the new scope type.
  - **Bypass:** `cluster_admin` keeps the single existing bypass; no new bypass
    path.
  - **Persistence:** API-made grants are subject to the `rbac-persistence` gap
    until that spec lands.
- **Queues and profiles are orthogonal** (user decision, 2026-09-29).
  - **Queues** own scheduling policy: which jobs are considered and in what
    order, priority, scheduling limits, and host partitioning.
  - **Application profiles** own application-related configuration: how the
    application is executed and what it needs to run.
  - A queue does not list or restrict profiles, and a profile does not select or
    constrain queues.
  - A profile carries no scheduling-policy attributes (no priority, queue choice,
    fairshare or preemption).
  - Any combination of an allowed queue and an allowed profile is valid. Each is
    governed by its own access rules; for profiles, that is RBAC permission.
  - Resource values in a profile describe what the application needs (its
    essential resource requirement and per-job execution limits). They are
    not scheduling policy. Queue-level scheduling limits keep applying on top,
    unchanged.
- **Live merge** (`job-config-merge`). The job records its profile name and
  its own values, not the effective values. The scheduler computes the
  effective values and recomputes them when the profile changes: a pending job
  that no longer satisfies its profile pends with a reason, and running jobs
  are unaffected. Job details show the profile name and the effective mode.
- **The apiserver never checks node accounts** (user decision, 2026-09-29).
  - Submit-time validation covers profile existence, RBAC permission, the
    additive rule and paths only.
  - The execution account is the worker's concern. If a `direct` job's execution
    user doesn't exist on the node, the worker fails the job at launch with a
    clear, user-visible reason, instead of today's opaque systemd failure.
  - Account-less users are served by profiles whose mode is `sandbox`.
- **Placement by mode** (moved to `job-sandbox-execution`): a job is placed
  only on workers whose advertised capabilities support its mode (sandbox
  needs systemd 242 or later and `nss-systemd`).
- **Ownership.** The submitter remains the owner for RBAC, limits and accounting
  in every mode.

## Approach
- **Profile definitions** live in their own cluster configuration file (see
  "Profile configuration" below), with xceadmin validation and live-upgrade
  replication.
- **The controller validates the profile at submit**: the named profile
  exists, and the job only adds to the profile's values (additive rule).
  This makes it authoritative for every submit path (REST, libxce, harness).
  It stores the profile name and the job's own values; the effective values
  come from the live merge (`job-config-merge`).
- **The apiserver checks that the named profile exists** (user direction,
  2026-09-29).
  - At submit, before anything goes downstream (`preCheckJobSubmitRequest`), the
    apiserver rejects a job that names a profile which does not exist, with a
    clear client error naming the unknown profile.
  - The apiserver already receives the cluster configuration from the controller
    (`backend/convert.go:265`), so it knows the current set of profiles and keeps
    it current across live config upgrades.
  - The controller still re-validates at submit, because it is
    authoritative for non-REST submit paths and for a profile removed between the
    two checks. If the profile has gone by then, the job is rejected, never run
    with a fallback profile.
- **The apiserver additionally enforces the RBAC permission** for the profile
  (middleware chain unchanged; a `RequirePermission`-style check) and checks
  the additive rule early (`preCheckJobSubmitRequest`).
- **The job's own values stay in its existing fields** (`launch_type`,
  pre/post exec, exdata env and limits, resreq). New job fields are added only
  for the profile name and the job's own I/O directories. All wire changes
  are additive Cap'n Proto fields, and the WAL submit fact and snapshot carry
  them with field completeness. Execution-side values (mode and options,
  launch type, env, profile pre/post-exec) are read from the profile at
  dispatch, as queue pre/post-exec is today (`get_queue_exec()`,
  `controller/queue.c:199`).
- **Workers advertise their supported execution modes** and the scheduler
  filters hosts by mode: `job-sandbox-execution` (the design picks the
  mechanism, e.g. a host attribute or boolean resource).

### Profile configuration
- **One file per entity type**, as for hosts, queues, resources, limits,
  users and scaler:
  - `src/configs/applications.yaml`, top-level key `applications:` (named
    after LSF `lsb.applications`; the user flag is `--app`);
  - `src/configs/schema/applications.schema.json`, picked up by xceadmin's
    file-to-schema naming rule (`xcecommon/validation.py:68`);
  - `xcecommon/validation.py`: duplicate names rejected; cross-file
    references checked (resreq resource names against `resources.yaml`);
  - `src/schema/controller.capnp`: `applications: List(AppProfileConf)` in
    `ClusterConfig` and in `Conf4Sched` (the scheduler runs the live merge).
    `Conf4Worker` does not need it: the worker gets execution values through
    `start_info` at dispatch.
  - The apiserver already receives `ClusterConfig`
    (`backend/convert.go:265`), so it gets the profiles with no new channel.
- **Shared config, per `config-design.md`:** controller, scheduler and
  apiserver all need the same view (independence rule), and profiles change
  at the rate of queues, as occasional admin edits (frequency rule).
- **The file is optional.** Without it there is one built-in profile
  `default` (mode `direct`, no attributes), which is today's behaviour.
- **No override markers.** Every value in a profile is an essential and
  binds the job (additive rule). Whether the job's own pre/post-exec is
  allowed is a profile setting.
- **Open to new sections.** Later specs add top-level sections to a profile
  (`service:`, I/O claims, relationship templates, `container:`); the schema
  must allow that without reshaping existing sections.
- Sketch (format decided in design):

  ```yaml
  applications:
    - name: openfoam
      description: "OpenFOAM, MPI"
      mode: direct
      launch_type: openmpi
      resreq:   "select[ib] rusage[license_foam=1]"   # essentials only
      tasks:    { min: 32, max: 512 }
      limits:   { runtime: 24h, memory: 128G }       # maximums
      env:      { FOAM_INST_DIR: /opt/openfoam }
      pre_exec:  /opt/xce/app/openfoam/pre.sh
      post_exec: /opt/xce/app/openfoam/post.sh
      job_pre_post_exec: allow

    - name: untrusted-python
      mode: sandbox
      sandbox: { inputs: ["/shared/in/"], outputs: ["/shared/out/"] }
      job_pre_post_exec: forbid
  ```

  Tuning such as `rusage[mem=4000] span[ptile=32]` and a task count of 64
  is not in the profile; the admin publishes it as a site preset named
  `openfoam`.

### Essentials and the additive rule
How each profile attribute combines with the job's own value (owned by
`job-config-merge`, which implements it):

| Profile attribute | Holds | The job may |
|---|---|---|
| resreq `select` | required terms (`ib`, GPU model, OS) | add terms (ANDed) |
| resreq `rusage` | required consumptions (`license=1`) | add other resources; not change these |
| resreq `span` / `same` / `order` | only if essential (e.g. a code that runs on one host only) | set them if the profile does not; otherwise rejected |
| task range | `min` / `max` | choose within it; no `-n` means `min` |
| env | variables the application needs | add others; not redefine these |
| launch type, mode, sandbox options | fixed values | nothing |
| limits | maximums | ask for less (checked like a ceiling) |
| pre/post-exec | outer layer, always runs | add its own inner layer if the profile allows |

- A job with no `-R` gets the profile's essentials only; the scheduler's
  existing empty-resreq default (`type == any`) still applies to what
  neither sets.
- **The `service:` section is configured in the profile** (user,
  2026-10-12; defined with this spec, consumed by `service-supervision`,
  `service-replica-set`, `service-endpoint`) and follows the same rule:
  - restart policy (always / on-failure / never, backoff, give-up),
    liveness and readiness probes, and named ports are the application's
    definition: profile values, the service cannot change them;
  - replicas work like the task range: the profile may set `min` / `max`,
    and the service chooses its count within them (from its own spec or a
    preset);
  - spread rule and rollout settings (max unavailable / surge): if the
    profile sets them they bind; otherwise the service sets them.
- Other later sections (I/O claims, relationship templates, `container:`)
  state in their own specs how they follow the additive rule.
- Compound or alternative resreqs: a compound job resreq under a profile
  whose resreq sets terms is rejected until PR-15 named parts define
  per-part merging (leaning; requirements decide).

### Environment of the outer layer and reserved names
Decided 2026-10-12 (user), replacing the idea of an env denylist:
- **A. The profile's pre/post-exec runs in a clean environment**: system
  values (`XCE_*`, `HOME`, a fixed `PATH`) plus the profile's own env, never
  the job's. The job's env applies after it, to the job's pre-exec, the job
  and the job's post-exec. An admin script that needs job facts reads
  `XCE_*`. Today the job env is applied first (`apply_env_vars()`,
  `worker/src/xcecd_worker.c:835`, then `queue_pre_exec()` :843), so a job's
  `PATH` or `BASH_ENV` can subvert the admin's script. Worker change in this
  spec: the outer layer gets an explicitly built env (`uv_proc_pre_exec`
  takes none today). It also fixes today's queue pre/post-exec.
- **B. The `XCE_` prefix is reserved**: neither a job nor a profile may set
  `XCE_*` variables (rejected at submit by apiserver and controller, and by
  `xceadmin` for profiles).
- Why no denylist: the whole job unit runs as the execution user
  (`User=` on the transient unit, `worker/src/xcecd_cgroups.c:299-312`), in
  sandbox mode inside the sandbox, so `LD_PRELOAD` and similar are not an
  escalation; the real risks were the admin's outer layer and XCE's own
  variables, which A and B close at the source. Sandbox-specific env
  exposure stays with `job-sandbox-execution`.

### Presets (client side, `xcecli-presets`)
Not part of this spec, recorded here because they replace profile defaults:
- A preset is a set of job options (`--app`, `--queue`, `-R`, `-n`, env,
  limits). The client expands it; a command-line flag replaces the preset's
  value for that option as a whole. The submit carries only the expanded
  values.
- Personal presets live one file per preset under `~/.config/xce/presets/`.
- The apiserver **only shares**: admin-deployed preset files served
  read-only (`GET /xce/v1/presets`, `GET /xce/v1/presets/{name}`) from `<TopDir>/config/presets/`. Pulling
  copies a preset into the local file, so a server preset being removed
  never affects users. `preset diff` / `pull --force` are user-initiated.
- User-to-user sharing by `preset export` / `import` (file, mail, git).
- Presets carry no privilege: `app:use`, `queue:submit`, the profile's
  essentials, queue ceilings and project quotas still apply at submit.

### Dispatch record
Decided 2026-10-12 (user): what a job was scheduled and started with is
**frozen at dispatch**. A pending job follows the current profile; a
dispatched job uses only its dispatch record until it is requeued or
restarted, and the scheduler no longer re-merges it.
- The record holds both sides:
  - scheduling: the effective resreq (`select`, `rusage`, `span`, `same`,
    `order`) and task count, so a running job's reservations (e.g.
    `rusage[license=1]`) do not change when the profile changes, and are
    rebuilt identically after a scheduler failover or restart;
  - execution: mode, mode options, launch type, env, outer pre/post-exec,
    limits;
  - the profile content hash the record was built from.
- At each dispatch the controller puts the execution values into the job's
  `start_info` for the worker, and the whole record into an additive
  payload on the WAL dispatch fact
  (`persist_submit_job_dispatch_fact`, `controller/persist.c:8167`) and the
  snapshot. Both apply seams decode it; `/xce-audit-invariants` checks the
  round trip.
- Today the queue's pre/post-exec copied at dispatch (`controller/job.c`
  1825-1897) is kept in memory only, so after a failover the new primary
  does not know what a running job was given. The frozen record closes that
  gap as pre/post-exec moves to the profile.
- A running or finished job's details show the frozen values, and say
  "profile changed since dispatch" when the current profile differs.
- **Requeue or restart re-reads the profile** at the next dispatch, the same
  as the live merge does for pending jobs.
- Running jobs therefore need nothing from the live profile: post-exec at
  the end, later remote tasks or a resize, and a service replica's probes
  and in-place restarts all use the record.

### Profile removed while jobs use it
Decided 2026-10-12 (user): **a job whose profile no longer exists pends**,
with the reason "application profile X is not defined".
- This applies to every job that reads the profile: pending jobs, requeued
  or restarted jobs, and new service replicas (replacements, scale-up),
  which are new jobs. Running jobs are unaffected (dispatch record); a
  service whose profile is gone keeps its running replicas but shows as
  degraded when one cannot be replaced.
- Putting the profile back releases the jobs. A rename is remove plus add.
- The removal is never refused (`xceadmin config build` works offline and
  cannot know live jobs) and jobs never exit because of it.
- The controller's live-upgrade reply lists removed profiles still named by
  pending jobs or services, with counts; `xceadmin config upgrade` prints
  it. Needs a new scheduler pend reason (profile missing from
  `Conf4Sched`).
- **Between the scheduler's decision and the controller's dispatch**
  (decided 2026-10-12, user). The scheduler merges with its copy of the
  profile (`Conf4Sched`); the controller reads the profile again when it
  builds the dispatch record. If the profile is gone or has changed in
  between, the **dispatch fails and the job goes back to PEND through the
  requeue path** (re-reading the profile, as any requeue does): gone → it
  pends with "application profile X is not defined"; changed → the
  scheduler re-merges with the new profile. Never EXIT.
  - New dispatch failure site (e.g. `XCE_DISPATCH_FAIL_SITE_APP_PROFILE`),
    classified retryable, unlike today's terminal
    `XCE_DISPATCH_FAIL_SITE_QUEUE_EXEC` (`controller/job.c`). Nothing is
    committed on this path (`j->job.allocs` is committed only after a
    successful `xce_startjob_req()`). Requirements decide that it does not
    count as a run or against requeue limits.
  - A scheduler still on the old config may re-dispatch and be declined
    again until the controller's config push reaches it; the existing
    dispatch retry backoff bounds that.
  - After the controller has built the record, profile changes no longer
    matter: the worker starts from `start_info`.
- **Profile content hash** (decided 2026-10-12, user): an internal version
  of each profile.
  - Computed, never stored as state: one `src/lib` function hashes the
    canonical Cap'n Proto encoding of the profile, excluding
    `description`. Controller, scheduler and apiserver get the same value
    from the same profile, also after failover or restart.
  - The scheduler's dispatch decision carries the hash it merged with
    (additive field); the controller compares it with the current
    profile's hash at dispatch. A scheduler that sends no hash (rolling
    upgrade) skips the check.
  - The dispatch record stores it; "profile changed since dispatch" in job
    details is a hash comparison; job details show it short
    (`openfoam@3f2a9c`). Admins never set it.
  - Base for the open "profile versioning and audit" item.
- **Moving pending jobs to another profile is not in this spec**
  (decided 2026-10-12, user). XCE has no job modification today
  (`JOB_MODIFY` is only an enum, `include/xce/types.h:620`; the apiserver's
  `"modify"` action is commented out, `controllers/jobs.go:2446`). A general
  modify path (apiserver action, controller opcode, WAL fact, scheduler
  re-merge) is its own backlog spec, `job-modify` (LSF `bmod`), with `--app`
  among its first attributes. Until then: restore the profile, use an
  alias, or kill and resubmit.

### Profile aliases
Decided 2026-10-12 (user), so a rename never strands jobs:
- A profile may list `aliases: [old-name, ...]`. An alias resolves to the
  profile everywhere: merge, dispatch record, content hash (the target's),
  RBAC (`app:use` is checked against the target profile's name).
- Jobs keep the name they recorded; the scheduler and controller resolve it
  through the alias. New submits that name an alias are accepted and record
  the **target's** name, so the alias can be removed once old jobs drain.
- Job details show the resolution, e.g. `openfoam (recorded as
  openfoam-old)`.
- `xceadmin` rejects an alias that clashes with a profile name or another
  alias, and alias chains (an alias naming an alias). `default_application`
  must name a real profile, not an alias.
- Removing an alias while jobs still name it is the same as removing a
  profile: they pend, and the live-upgrade warning lists them.
- Rename workflow: rename the profile and add the old name as an alias;
  drop the alias when no pending job uses it (the upgrade warning shows
  when that is).
- Why: for sandboxed jobs the mode and mounts a job actually ran with is a
  security/audit fact, and `job-sandbox-execution`'s mandatory cleanup
  depends on it; `persistence.md` does not allow memory-only state without a
  documented scope decision.

### Default profile
- **Set by `default_application: <name>` in `xce.yaml`**, a cluster
  parameter (as LSF keeps `DEFAULT_APPLICATION` in `lsb.params`), not a flag
  on one profile.
- **Recorded at submit.** A job submitted without `--app` gets the current
  default's *name* written on its record, so changing the default later does
  not move pending jobs to another profile.
- **A name that is not defined is a configuration error, never a silent
  fallback** (user decision, 2026-10-12):

  | `default_application` | Behaviour |
  |---|---|
  | absent | built-in `default` profile (mode `direct`, no attributes), today's behaviour |
  | names a defined profile | that profile is used for jobs submitted without `--app` |
  | names an undefined profile | the configuration is rejected |

  - `xceadmin config build` fails with an error naming the missing profile.
    This is an error, not one of the warnings `validation.py` gives for
    unknown host or user references: a dangling default would break every
    submit without `--app`.
  - The controller re-checks on live upgrade and at start, because config can
    reach it without xceadmin. A live upgrade with a dangling default,
    including one that removes or renames the current default profile, is
    refused and the current config stays in force. At start the controller
    refuses to run and logs the reason.
  - Profiles and the default are in the same configuration, checked
    atomically, so the default cannot become dangling at run time.
- **Queue deprecations surface here.** `xceadmin config build` warns when
  `queues.yaml` still sets `res_req` or `pre_exec`/`post_exec` (the shipped
  `night` queue does) and suggests moving them into a profile. The
  deprecation itself is `job-config-merge`'s; the message needs
  `applications.yaml` to exist.

## Scope
- **In**:
  - The profile schema (essentials, additive rule, aliases) and
    `applications.yaml`, and the default profile (`default_application`).
  - Submit-time validation of the profile and the additive rule.
  - RBAC permission for profile use.
  - Submit surfaces: REST, `xcecli --app`, harness, libxce.
  - The profile name and the job's own values persisted on the job.
  - The `mode:` schema field (only `direct`).
  - A worker launch error when a `direct` exec user is missing.
  - Dispatch record (scheduling and execution values, content hash) and the
    retryable dispatch failure for a missing or changed profile.
  - Profile pre/post-exec as the outer layer (moved from the queue), with
    a clean environment; the reserved `XCE_` prefix.
  - Aliases, live-upgrade warning, pend reason for a missing profile.
  - Profile name and mode shown in job details.
  - Listing available profiles (REST and xcecli).
  - Profile `--dry-run` support: the effective values a job would get
    (shared with `job-config-merge`).
  - Tests.
- **Open for requirements**:
  - Which extra attributes go into v1 beyond the four ("maybe others"). Examples:
    requeue exit values, runtime estimate, job starter or wrapper, output path
    template, checkpoint settings.
  - ~~How resource defaults combine with a job's own resreq~~ — settled
    2026-10-09, owned by `job-config-merge`.
  - Profile versioning and audit.
  - Raised 2026-10-12, not yet decided:
    - Job labels (`--label key=value`, e.g. the preset name) and a summary
      grouped by label, to compare tuning variants: here,
      `job-status-detail-display`, or `xcecli-presets`?
  - Settled 2026-10-12: scheduling and execution values frozen at
    dispatch, re-read on requeue ("Dispatch record"); a removed profile
    makes its jobs pend ("Profile removed while jobs use it"); clean env for
    the outer layer and reserved `XCE_` instead of an env denylist; moving
    pending jobs deferred to `job-modify`, profile aliases in v1; override granularity (no markers; additive rule),
    limits a job may lower (profile limits are maximums), personal presets
    (client side), `app:override` and inheritance (deferred); execution modes
    split into `job-sandbox-execution`.
- **Out**:
  - The `sandbox` mode and its options, I/O directories, capability
    advertisement, placement by mode and the worker sandbox mechanics
    (`job-sandbox-execution`).
  - Multi-node sandbox (`job-sandbox-multinode`).
  - Container or vm execution itself (only room for them in the schema).
  - Auth backends and JWT claims.

## Boundary Candidates
- The profile model and config (schema, validation, live upgrade).
- Submit-time validation of the profile and the additive rule (controller).
- Access control (RBAC permission) and early validation (apiserver).
- Submit and listing surfaces (REST, xcecli, harness, libxce).
- Persisting the profile name and the job's own values (wire, WAL submit
  fact, snapshot).

## Out of Boundary
- Worker execution backends and sandbox mechanics.
- Output file location and retrieval inside the sandbox.
- Scheduler policy beyond capability filtering and the profile's essential
  resreq.
- Presets (`xcecli-presets`): client side; the read-only preset listing on
  the apiserver belongs to that spec.

## Upstream / Downstream
- **Upstream**:
  - The existing submit path.
  - The WAL submit fact (`ha-wal-replication`, the `raft-apply-decoupling` apply
    seam).
  - The config pipeline (`clusterconfig-schema-split`, `config-upgrade-snapshot`,
    `admin-config-tool`).
  - RBAC (`rbac_check.go`).
- **Downstream**:
  - `job-sandbox-execution` adds the `sandbox` mode, its options and I/O
    directories, capability advertisement and placement, on top of this
    spec's `mode:` field and dispatch record.
  - `job-sandbox-multinode`.
  - Future `container`/`vm` modes, which add profile attributes.
  - `job-status-detail-display`, which can show the profile and mode.

## Existing Spec Touchpoints
- **Extends**: none directly.
- **Depends on**: `job-config-merge` (merge rules and live re-evaluation).
- **Feeds**: `parjob-placement` (the profile's `span[...]`).
- **Adjacent**:
  - `job-status-detail-display`
  - `controller-job-cleanup`
  - `resreq-maxtasks-wiring` (task range)
  - `xcecli-presets` (convenience defaults, client side)
  - `profile-usage-insight` (advisory recommendations)
  - `xce-node-manager`

## Constraints
- Middleware chain unchanged: WhitelistURL → WhitelistIP → RateLimit → JWT → CORS
  → RBAC.
- All wire changes are additive Cap'n Proto. Old WAL records and snapshots
  decode as "no profile, mode `direct`", which is today's behaviour.
- Save/restore field completeness (`persistence.md`): the profile name and
  the job's own values (including I/O directories) round trip through the
  submit facts and snapshots. Run
  `/xce-audit-invariants`.
- Input validation (`security.md`): the job's additions and paths are validated
  server-side. No path traversal, and mount prefixes are restricted by the
  profile.
- Size: still large after the split (config, access and submit, job state,
  outer layer); one coherent feature, testable end to end with `direct`.
- HA/persistence spec: use the full 3-phase workflow.