myxce/.kiro/specs/application-profile/brief.md

Brief: application-profile

(Reshaped from job-execution-mode on 2026-09-29, by user direction. Execution mode is an attribute of the job, determined by the application the user runs, so it is carried by a named application profile rather than by a bare mode option.)

Update 2026-10-09 (/kiro-discovery, configuration layering; user decisions):

• Configuration is layered LSF-style: queue = scheduling policy only (no resreq; queue res_req is deprecated), application profile = application execution constraints including the job's resource requirement (select / rusage / span[...] / same) and task limits (min / default / max, LSF TASKLIMIT), project = resource ownership (project-config).
• Every attribute has one owning level (.kiro/steering/job-config-layering.md). How the levels interact is owned by job-config-merge; queue ceilings and project quotas only check, never change, the job's values. (The job-vs-profile rule of this update, default/fixed markers, is replaced by the additive rule of 2026-10-12 below.)
• Pre/post-exec moves here from the queue (queue pre_exec/post_exec deprecated). The profile's pre/post-exec is admin-defined and always runs as the outer layer; the job's own pre/post-exec is the inner layer, allowed or forbidden by the profile. Order: profile pre → job pre → job → job post → profile post (the worker's existing two-layer structure, xcecd_worker.c queue_pre_exec()/queue_post_exec()).
• The merge is live (replaces the earlier "resolution at submit time", removed from this brief 2026-10-12). The WAL keeps the job's own values plus the profile name; the controller and apiserver validate at submit, and the scheduler computes effective values and recomputes them when a profile changes (a pending job that no longer satisfies its profile pends with a reason; running jobs are unaffected). Execution-side values (mode, options, launch type, env, pre/post-exec) are still taken from the profile at dispatch time and frozen on the dispatch record (decided 2026-10-12, "Dispatch record" below).

Update 2026-10-10 (/kiro-discovery, long-running services): a profile may declare kind service with a service section: restart policy (always / on-failure / never, backoff, give-up), liveness and readiness probes (command / TCP / HTTP), named ports, replica range, spread rule and rollout settings (how these follow the additive rule: "Essentials and the additive rule" below). The section is defined with this spec and consumed by service-supervision (worker probes and restarts), service-replica-set (replicas, scaling, rolling update) and service-endpoint (ports, endpoints).

Update 2026-10-12 (discussion; user decisions):

• Split: execution modes move to job-sandbox-execution. This spec is the profile core: config, access and submit, job state (name, dispatch record), the outer pre/post-exec layer. Its schema has mode: but accepts only direct. The sandbox mode with its options, I/O directories, worker capability advertisement and scheduler placement by mode belong to job-sandbox-execution, since nothing uses a non-direct mode before the sandbox exists. The worker's clear launch error for a missing direct exec user stays here.
• Order with job-config-merge: requirements and design of this spec first (it fixes the profile schema), then job-config-merge's design; implementation tasks interleave (schema/config, merge library, submit validation and dispatch record).
• Own configuration file. Profiles live in a new src/configs/applications.yaml with its own JSON schema, mapped into ClusterConfig and Conf4Sched like queues and limits. See "Profile configuration" below.
• Cluster default is set by default_application in xce.yaml. A name that is not defined is a configuration error everywhere (xceadmin, live upgrade, controller start), never a silent fallback. See "Default profile" below.
• This spec owns the profile's attribute schema (which attributes exist and what each holds). job-config-merge consumes that schema and owns the merge and live re-evaluation.
• A profile holds only essentials; convenience lives in client-side presets (user decision, 2026-10-12). "The profile holds what the job must have; the preset holds what the user would like." A profile value is something the application needs or the admin enforces, and it binds every job of that profile. The default/fixed markers of 2026-10-09 are dropped: there are no markers, and no per-term markers or ranges. See "Essentials and the additive rule" below.
  • Additive rule: the job may add to the profile's values, never change or remove them. A conflict is rejected at submit with an error naming the profile value, never resolved silently.
  • Exception: default reservations. The profile's structured rusage entries hold per-resource defaults the job may override (see "Essentials and the additive rule"), so REST/portal users without presets still reserve sensibly.
  • Other convenience defaults (tuning such as span[ptile=...], task count, extra env, different rusage values) go into presets, handled fully by the client (xcecli-presets). The server only shares admin-deployed presets read-only; presets never reach the controller, WAL or scheduler and carry no privilege.
  • Rule of thumb for admins: something every job of the application must have (required host properties, licenses, launch type, mode, pre/post-exec, limits) goes into the profile; an optional variant or tuning shortcut goes into a site preset, typically one with the profile's name.
  • Recommendations are advisory (user decision): usage-based suggestions (requested vs. used, OOM kills, runtime by variant) are reported to the admin, who changes site presets (or a profile, for an essential); nothing tunes itself. Backlog: profile-usage-insight.
  • Job details show the effective resreq with each term's origin (profile or job).
  • Deferred: a submit-time hook run for the application (LSF esub style) that inspects or adjusts each submission; roadmap backlog app-submit-hook (it cannot run in the controller, would add latency and a failure mode to every apiserver submit, and bypasses the merge rules; params, annotations, the job starter and pre-exec cover most needs).
  • Deferred: a privileged app:override for one-off exceptions (until job-modify exists, an admin kills and resubmits, or uses a variant profile; recurring variants are presets), and profile inheritance (base:), no longer needed for variants.
• Later sections are added by their consumers: service: (service-supervision), I/O claims (resource-model PR-6), relationship and compound templates (PR-14/15), container: (R6). This spec only makes the schema open to new top-level sections.

Problem

XCE users run many kinds of applications: trusted in-house codes, untrusted scripts, vendor binaries, MPI solvers. Each needs a different way of running:

• directly as the submitter, or jailed in a sandbox under a separate execution identity (and later in a container or vm);
• a particular launch type (serial, openmpi, mpich);
• particular environment and pre/post-exec steps;
• resource requirements it cannot run without, and limits.

Today every one of these must be set per job by hand, and several (execution mode, sandbox options) cannot be set at all. Many users log in through the portal or REST API (JWT; config_file/PAM, later LDAP or OAuth2) and have no Unix account on the compute nodes. They can only run jobs if their applications run sandboxed. Admins have no way to say "this application runs like this, and these users may run it".

This spec introduces application profiles and is the submission/policy half of the job-sandbox initiative (roadmap "Job Sandbox Execution"). There is precedent in LSF application profiles (lsb.applications, bsub -app), which fits XCE's LSF-style job model.

Current State

• Identity. A job's identity is only the JWT username (controllers/auth.go:26-37). SubmitUser comes from the token (controllers/jobs.go:1379, :1459), and UserGroup is taken from the client request unchecked (:1465).
• Execution user. It is always the submitter: resolveExecUser (backend/convert_pure.go:162, bug-1780).
• Per-job settings that exist today (src/include/xce/types.h):
  • job_startinfo_t (:240-251): execuser, exechome, execcwd, cmdline, exdata, launch_type, and pre/post-exec commands.
  • exdata is a JSON object carrying the environment (sub_env_vars) and the worker cgroup limits parsed by parse_limits (MemoryMax, CPUQuota, TasksMax, RuntimeMax, ...). Since bug-1936 the apiserver sends it (backend/convert_pure.go:388, capped at 16 KB by maxExdataLen); the controller stores it opaquely in start_info (WAL submit fact, snapshot) and the worker parses it, ignoring unknown top-level keys. Remote tasks get the same exdata (worker/src/controller_connector.c:764).
  • The resource requirement is resreq on the submit data.
  • Remote tasks use xce_taskhost_t (:696-705).
• Missing today:
  • any profile concept, any execution-mode field, any input/output directory field;
  • any way for workers to advertise execution capabilities to the scheduler.
• Controller. It validates only that the user is not empty (controller/job.c:2447). The submit data, including start_info, is persisted in the WAL submit fact and in snapshots.
• Ownership. RBAC ownership, scheduler per-user limits and the user table all key on the submit user, not the exec user (controllers/jobs.go:1010, :2542, output.go:95; schedobj.c:121; plugins/limit.c). So a separate execution identity breaks none of them.
• Access control. RBAC is config/rbac.yaml plus RequirePermission (controllers/rbac_check.go).

Desired Outcome

• Admins define named application profiles in the cluster configuration. The first version contains the following (user decision; the format must be extensible for "maybe others"):

  1. Execution mode. The mode: field, accepting only direct in this spec; job-sandbox-execution adds sandbox with its options (allowed input/output prefixes, isolation extras), later container/vm.

  2. Launch type. Serial, openmpi or mpich.

  3. Environment and hooks. Environment variables the application needs, and the worker-run hooks pre/post_job_exec (first host) and pre/post_host_exec (every host), see "Hooks and starters".

  4. Resource requirement and limits. A structured resreq: the binding terms the application cannot run without (select, and span / same / order when set) and default reservations (rusage entries), its task range (min / max), and per-job execution limits as maximums (memory, runtime, CPU, tasks), mapped onto the existing worker cgroup limits.

  5. Job starter and task starter (decided 2026-10-12, user): two wrappers, both frozen on the dispatch record.

     • job_starter wraps the full job: the job command, started once on the first host (for an MPI job, the launcher such as mpirun), e.g. module load openfoam &&.
     • task_starter wraps every task startup that XCE's workers perform, on every host including remote tasks (xce_taskhost_t), e.g. apptainer exec img or a binding/affinity wrapper.
     • A serial (single-task) job's command is both the job and its task: it is wrapped by both, job_starter outermost (design confirms).
     • For MPI jobs XCE inserts the task starter into the mpirun line it builds, so every rank starts through it (see "Hooks and starters").

  6. Application parameters (params) (decided 2026-10-12): settings XCE carries but never interprets, handled by the application, e.g.

     params:
       - { name: solver, default: pimpleFoam, allowed: [pimpleFoam, simpleFoam] }
       - { name: decompose, default: scotch }
       - { name: case_dir, pattern: "^/shared/cases/.*" }   # no default
     

     • Each is delivered to the job as XCE_APP_PARAM_<NAME>, with the profile env (separate from the job env, frozen on the dispatch record), so the job, the job starter and the outer pre/post-exec can read it.
     • The job overrides with --app-param name=value (also in presets); the job's value wins, like rusage defaults. Only declared names are accepted (a typo is rejected at submit). XCE checks only the shape (allowed list or pattern), never the meaning. A parameter with no default and no job value is not set.
     • The job's values are stored in its exdata (app_params), already persisted; no new job field.
     • The portal can build an application's submit form from the declared parameters.

  7. Annotations (decided 2026-10-12): free-form key/value facts about the application (owner, version, docs URL, licence server name), like Kubernetes annotations. Shown to tools (profile listing, portal, job details) and exported to the job as XCE_APP_ANN_<KEY>, so an admin writes a value once for both. The job cannot change them.

     • Name conversion: keys may be Kubernetes-style (cfd.org/solver-family); the env name is uppercased with every non-alphanumeric character replaced by _ (XCE_APP_ANN_CFD_ORG_SOLVER_FAMILY). xceadmin rejects two annotation keys (or two params) that convert to the same name.
     • Limits on count and size, so a profile cannot bloat every job's env.
     • Visible to every app:view holder: no secrets (the secret-name warning applies). All are exported in v1; an opt-out only if needed.
  • env vs annotations vs params: env is a variable the application expects under its own name; annotations are facts for tools and the job; params are the user's choices. XCE_APP_* names are set by XCE only; the reserved XCE_ prefix keeps jobs and profiles from setting them directly, so a job cannot fake one. All of these are essentials under the additive rule, except the rusage entries and params, which are defaults the job may override.
• A profile attribute ships with its job feature (user, 2026-10-12): a profile attribute is added by the spec that builds the job-level feature it configures, never ahead of it, so no profile setting is dead config. None of the other candidates exists as a job feature today, so they are deferred:
  • requeue exit values and success exit values: job-requeue (backlog);
  • checkpoint method/period/directory, and a warning signal before the run limit: their own future specs;
  • output path template: the PR-6/PR-7 I/O track (output options are rejected at submit today);
  • runtime estimate: not a profile attribute at all (it describes one run: a job option or preset; the profile's runtime limit is the maximum).
  • Found on the way: the REST submit accepts several of these options and silently drops them (bug-1970).
• Users pick a profile at submit (REST, xcecli jobs submit --app <name>, harness, libxce). A cluster default profile applies when none is given.
• Jobs add, never override (except rusage defaults). A job may add to its profile's values (more select terms, other resources, its own env variables, lower limits) and override a profile rusage default, but not change or remove anything else. Anything else is rejected with a clear error.
• Access by RBAC permission. No new access mechanism; this follows the existing named-instance pattern (user decision, 2026-09-29: fold profile RBAC into this spec). Today, queue:submit scoped to type: queue, values: [normal, priority] in config/rbac.yaml is enforced by GetRequireResourcePermission in controllers/jobs.go:1402-1409. Profiles add:
  • Permissions: app:use (submit a job with a profile) and app:view (list and inspect profiles).
  • Scope: a new app scope type whose values are profile names, or all.
  • Checks:
    • Job create checks app:use alongside, and independent of, queue:submit.
    • Job restart (POST /:jobids/restart, which today checks only queue:submit) re-checks app:use for the job's profile.
    • Listing profiles checks app:view.
  • Defaults: rbac.yaml gets defaults, for example basic_user → app:use on the default profile, and app:view on all.
  • The current default profile is always usable (decided 2026-10-12): anyone with queue:submit has an implicit app:use on the profile default_application names (or the built-in default), so plain submits never break when an admin repoints the default.
  • Role API: role creation and update accept the new scope type.
  • Bypass: cluster_admin keeps the single existing bypass; no new bypass path.
  • Persistence: API-made grants are subject to the rbac-persistence gap until that spec lands.
• Queues and profiles are orthogonal (user decision, 2026-09-29).
  • Queues own scheduling policy: which jobs are considered and in what order, priority, scheduling limits, and host partitioning.
  • Application profiles own application-related configuration: how the application is executed and what it needs to run.
  • A queue does not list or restrict profiles, and a profile does not select or constrain queues.
  • A profile carries no scheduling-policy attributes (no priority, queue choice, fairshare or preemption).
  • Any combination of an allowed queue and an allowed profile is valid. Each is governed by its own access rules; for profiles, that is RBAC permission.
  • Resource values in a profile describe what the application needs (its essential resource requirement and per-job execution limits). They are not scheduling policy. Queue-level scheduling limits keep applying on top, unchanged.
• Live merge (job-config-merge). The job records its profile name and its own values, not the effective values. The scheduler computes the effective values and recomputes them when the profile changes: a pending job that no longer satisfies its profile pends with a reason, and running jobs are unaffected. Job details show the profile name and the effective mode.
• The apiserver never checks node accounts (user decision, 2026-09-29).
  • Submit-time validation covers profile existence, RBAC permission, the additive rule and paths only.
  • The execution account is the worker's concern. If a direct job's execution user doesn't exist on the node, the worker fails the job at launch with a clear, user-visible reason, instead of today's opaque systemd failure.
  • Account-less users are served by profiles whose mode is sandbox.
• Placement by mode (moved to job-sandbox-execution): a job is placed only on workers whose advertised capabilities support its mode (sandbox needs systemd 242 or later and nss-systemd).
• Ownership. The submitter remains the owner for RBAC, limits and accounting in every mode.

Approach

• Profile definitions live in their own cluster configuration file (see "Profile configuration" below), with xceadmin validation and live-upgrade replication.
• The controller validates the profile at submit: the named profile exists, and the job only adds to the profile's values (additive rule). This makes it authoritative for every submit path (REST, libxce, harness). It stores the profile name and the job's own values; the effective values come from the live merge (job-config-merge).
• The apiserver checks that the named profile exists (user direction, 2026-09-29).
  • At submit, before anything goes downstream (preCheckJobSubmitRequest), the apiserver rejects a job that names a profile which does not exist, with a clear client error naming the unknown profile.
  • The apiserver already receives the cluster configuration from the controller (backend/convert.go:265), so it knows the current set of profiles and keeps it current across live config upgrades.
  • The controller still re-validates at submit, because it is authoritative for non-REST submit paths and for a profile removed between the two checks. If the profile has gone by then, the job is rejected, never run with a fallback profile.
• The apiserver additionally enforces the RBAC permission for the profile (middleware chain unchanged; a RequirePermission-style check) and checks the additive rule early (preCheckJobSubmitRequest).
• The job's own values stay in its existing fields (launch_type, pre/post exec, exdata env and limits, resreq). New fields (all additive Cap'n Proto; WAL submit/dispatch facts and snapshot carry them with field completeness):
  • on the submit data: the profile name;
  • on start_info (what the worker gets): the job starter, the task starter (also on each xce_taskhost_t entry for remote tasks) and the profile's env, kept separate from the job's env (see "Environment of the outer layer"); the queue-named qPreExec / qPostExec carry the profile's outer pre/post-exec (renamed in design);
  • the dispatch record payload ("Dispatch record").
• exdata is read by the merge (owned by job-config-merge): the job's env and limits live inside the exdata JSON (sub_env_vars, limits), so the shared src/lib merge parses it to check the additive env rule, the reserved XCE_ prefix and "limits: the job may lower", and at dispatch the controller writes the effective limits back into the exdata sent to the worker, preserving every other key (including meta, used by presets). Execution-side values (mode and options, launch type, env, profile pre/post-exec) are read from the profile at dispatch, as queue pre/post-exec is today (get_queue_exec(), controller/queue.c:199).
• Workers advertise their supported execution modes and the scheduler filters hosts by mode: job-sandbox-execution (the design picks the mechanism, e.g. a host attribute or boolean resource).

Profile configuration

• One file per entity type, as for hosts, queues, resources, limits, users and scaler:

  • src/configs/applications.yaml, top-level key applications: (named after LSF lsb.applications; the user flag is --app);
  • src/configs/schema/applications.schema.json, picked up by xceadmin's file-to-schema naming rule (xcecommon/validation.py:68);
  • xcecommon/validation.py: duplicate names rejected; cross-file references checked (rusage entry names against resources.yaml, per: values); aliases (no clashes, no chains); XCE_* env rejected; a warning for env names that look like secrets;
  • src/schema/controller.capnp: applications: List(AppProfileConf) in ClusterConfig and in Conf4Sched (the scheduler runs the live merge). Conf4Worker does not need it: the worker gets execution values through start_info at dispatch.
  • The apiserver already receives ClusterConfig (backend/convert.go:265), so it gets the profiles with no new channel.

• Shared config, per config-design.md: controller, scheduler and apiserver all need the same view (independence rule), and profiles change at the rate of queues, as occasional admin edits (frequency rule).

• The file is optional. Without it there is one built-in profile default (mode direct, no attributes), which is today's behaviour. A profile named default in the file replaces the built-in one, so admins can customise the default without setting default_application.

• No secrets in profiles. Profile env values and pre/post-exec paths are visible to every app:view holder (profile listing, job details); xceadmin warns on env names that look like secrets (*TOKEN*, *SECRET*, *PASSWORD*, *_KEY), as for presets.

• No override markers. Every profile value binds the job (additive rule), except the rusage entries' defaults, which the job may override. Whether the job's own pre/post-exec is allowed is a profile setting (job_pre_post_exec: allow | forbid).

• Open to new sections. Later specs add top-level sections to a profile (service:, I/O claims, relationship templates, container:); the schema must allow that without reshaping existing sections.

• Sketch (format decided in design):

  applications:
    - name: openfoam
      description: "OpenFOAM, MPI"
      mode: direct
      launch_type: openmpi
      resreq:
        select: "ib"                                  # binding
        rusage:
          - { name: license_foam, per: job, default: 1 }  # default reservation
          - { name: mem, default: 4000 }                  # per task by default
      tasks:    { min: 32, max: 512 }
      limits:   { runtime: 24h, memory: 128G }       # maximums
      env:      { FOAM_INST_DIR: /opt/openfoam }
      job_starter:  "module load openfoam &&"
      task_starter: "/opt/xce/app/openfoam/bind.sh"
      pre_job_exec:   /opt/xce/app/openfoam/licence-out.sh
      post_job_exec:  /opt/xce/app/openfoam/licence-in.sh
      pre_host_exec:  /opt/xce/app/openfoam/scratch-setup.sh
      post_host_exec: /opt/xce/app/openfoam/scratch-clean.sh
      job_pre_post_exec: allow
  
    # mode: sandbox and its options come with job-sandbox-execution;
    # this spec accepts only mode: direct
    - name: untrusted-python
      mode: sandbox
      sandbox: { inputs: ["/shared/in/"], outputs: ["/shared/out/"] }
      job_pre_post_exec: forbid
  

  Default reservations live in the profile's rusage entries (the job may override them); other tuning such as span[ptile=32] and a task count of 64 is not in the profile: the admin publishes it as a site preset named openfoam.

Essentials and the additive rule

How each profile attribute combines with the job's own value (owned by job-config-merge, which implements it):

Profile attribute	Holds	The job may
resreq select	required terms (ib, GPU model, OS)	add terms (ANDed)
resreq rusage	entries with optional per: and default	add resources; override a profile default (job's term wins; unit must match, see below)
resreq span / same / order	only if essential (e.g. a code that runs on one host only)	set them if the profile does not; otherwise rejected
task range	min / max	choose within it; no -n means min
env	variables the application needs	add others; not redefine these
launch type, mode, sandbox options, job starter, task starter	fixed values	nothing
job_pre_post_exec	allow / forbid	use its own pre/post-exec only if allow
limits	maximums	ask for less (checked like a ceiling)
pre/post-exec	outer layer, always runs	add its own inner layer if the profile allows

Structured profile resreq; rusage defaults, the job's term wins (user decisions 2026-10-12). A profile's resreq is not one string but separate parts, so each part's rule is visible from its shape:

resreq:
  select: "ib && os == rhel9"        # binding: ANDed with the job's select
  rusage:                            # defaults: the job may override per resource
    - { name: license_foam, per: job, default: 1 }
    - { name: mem, default: 4000 }   # per: optional, defaults to task
    - { name: gpu }                  # unit only: nothing reserved by default
  span:  "ptile=32"                  # binding if set
  same:  [switch]
  order: "r15s"

Part	Merge with the job's -R (an ordinary resreq string, parsed into terms)
select	ANDed with the job's
rusage entry with default	reserved at that value unless the job names the resource; then the job's term wins whole (value and duration/decay), in either direction
rusage entry without default	nothing reserved unless the job asks; it only fixes the unit
per: (optional, default task)	the unit for that resource: a job term without a per-type gets it; a job term with a different per-type is rejected at submit ("resource mem is per task in profile openfoam"), never silently reinterpreted (16000/job read as per task multiplies by the task count)
a resource only the job names	the job's term as written
span / same / order	the profile's if set (binding), otherwise the job's

• per: defaults to task, today's behaviour: PER_DEFAULT takes the caller's per-type (lib/resreq.c:842) and the controller passes PER_TASK (controller/job.c:3909); resources.yaml has no per-type field. A per-resource default there may come later.
• A 0-valued term is removed by the merge (decided 2026-10-12, user): after combining profile and job, every rusage term whose value is 0 is dropped, so the scheduler sees exactly "no term". name=0 is how a job drops a profile default. Needed because the scheduler does not treat 0 as absent today: a 0 term still becomes a wanted entry (calc_wanted_host_resvals(), lib/resreq.c), and unsatisfy_() (scheduler/src/resource.c:164, total < inuse + uncomit + wanted) rejects a host where that resource is already over-committed (inuse > total, possible because memory is charged max(reserved, actual)), and the term shows in decision-trace rows and pend reasons; with no term the resource is not checked at all (fast path at resource.c:351). Applies to every 0 term (profile, job, preset, or a job without a profile). Job details still show "license: 0 (overrides the profile default)".
• xceadmin validates each rusage entry against resources.yaml (name exists) instead of parsing a string; later per-resource fields (duration, a min: floor for counted resources such as licences, if under-reservation proves a problem) slot into the entry without new grammar.
• Consequence: rusage is the one profile part that holds defaults; a profile cannot enforce a minimum reservation. Hard protection of shared resources comes from limits.yaml and project quotas.
• Side effects to handle in requirements: measured resources (memory) are charged max(reserved, actual) once usage is reported (resource-accounting.md), so lowering them mostly self-corrects, but counted resources (licences) do not; xcecli and --dry-run warn when a job lowers a profile default, and when a preset restates one (it would pin the job against later profile changes); job details label each rusage term as profile default or job override; queue ceilings and project quotas check the merged value and the error names the job's override; a profile rusage change changes the content hash, so in-flight dispatches re-pend and pending jobs re-merge even if they override that resource (accepted).
• A job with no -R gets the profile's binding terms and its rusage defaults; the scheduler's existing empty-resreq default (type == any) still applies to what neither sets.
• Profile kind is checked (decided 2026-10-12): a batch job naming a kind: service profile is rejected ("profile X is a service profile"); the services specs reject the reverse.
• A select no host matches (decided 2026-10-12): a warning, not a rejection, at submit and from xceadmin against hosts.yaml, since host attributes can change (e.g. IB nodes added later).
• The service: section is configured in the profile (user, 2026-10-12; defined with this spec, consumed by service-supervision, service-replica-set, service-endpoint) and follows the same rule:
  • restart policy (always / on-failure / never, backoff, give-up), liveness and readiness probes, and named ports are the application's definition: profile values, the service cannot change them;
  • replicas work like the task range: the profile may set min / max, and the service chooses its count within them (from its own spec or a preset);
  • spread rule and rollout settings (max unavailable / surge): if the profile sets them they bind; otherwise the service sets them.
• Other later sections (I/O claims, relationship templates, container:) state in their own specs how they follow the additive rule.
• Compound and alternative resreqs (decided 2026-10-12, user, simplified). Parts are unnamed today: only the profile's select applies, ANDed into every part and every || alternative. If the profile has any rusage entry with a default, or sets span / same / order, a compound or alternative job resreq is rejected with an error naming the part, until PR-15 named parts let a profile target a part by role. rusage entries without a default only fix a unit and do not block. Admin guidance (not enforceable): no part-specific property (e.g. a GPU model only one role needs) in a profile's select.

Hooks and starters

Decided 2026-10-12 (user). Run by XCE's worker, never by the job:

Item	Runs	Typical use
pre_job_exec / post_job_exec	once, on the first host, before the job / after all tasks end	licence checkout and check-in, staging
pre_host_exec / post_host_exec	once per host, before its first task / after its last	scratch, local cache, cleanup (LSF HOST_PRE_EXEC / HOST_POST_EXEC)
job_starter	wraps the job command once (the mpirun line for MPI)	module load ... &&
task_starter	wraps every task	apptainer exec img, binding wrappers

• No per-task pre/post hooks. With N tasks per host they would run N times concurrently: setup needs locking, and "the last task cleans up" needs reference counting that a SIGKILL breaks. For MPI jobs XCE does not start the ranks (it builds one mpirun line and sets ntasks = 1, worker/src/controller_connector.c:993-1010), so it could not run hooks around them, and a wrapper there runs in the job's env. A task starter can do per-task work itself (it is a wrapper).
• Task starter reach (correction): for serial and non-MPI parallel jobs the worker wraps each process it starts (nranks per host, remote hosts through xce_taskhost_t); for MPI jobs XCE inserts it into the mpirun command line it builds (create_mpi_cmdline(): mpirun <opts> <task_starter> <app>), so the MPI runtime starts every rank through it. Only an application that calls its own launcher inside the job script bypasses it. A serial job's command is wrapped by both starters, job_starter outermost (design confirms).
• Order on a multi-host job:

  first host:  pre_job_exec → job's own pre-exec → job_starter(job command) … job's own post-exec → post_job_exec
  every host:  pre_host_exec → task_starter(task) … (all its tasks end) → post_host_exec
  

  pre_job_exec finishes before any host runs pre_host_exec; post_job_exec runs after every host's post_host_exec. The job's own pre/post-exec stays job-level (inner layer, allowed by job_pre_post_exec).
• Clean environment for all four hooks: system values, the profile env, XCE_APP_*; never the job's env ("Environment of the outer layer").
• Per-host helpers: XCE exports XCE_JOB_HOST_DIR (a per-job, per-host directory the worker creates before pre_host_exec and removes after post_host_exec) and XCE_HOST_NTASKS / XCE_HOST_TASK_INDEX, so starters and the application know their host-local position without MPI-specific variables. A task starter may use a lock in XCE_JOB_HOST_DIR for best-effort once-per-host work (e.g. a cache warm-up), never for cleanup that must happen.
• Failures (decided, proposal 4): pre_job_exec fails → nothing else starts, post_job_exec still runs, the job goes to EXIT with a reason naming the hook; pre_host_exec fails on a host → its tasks do not start, its post_host_exec still runs, the job fails. Requeue on hook failure belongs to job-requeue. (Today queue_pre_exec() just exits: TODO: Handle pre-exec failure.)
• The post hooks must survive every way a job ends. Checked 2026-10-12: a graceful kill is SIGINT → SIGTERM → SIGKILL with a grace period (worker/src/job_signal.c:52-55); a forced kill is SIGKILL to the job's processes; the run limit is systemd RuntimeMaxUSec on the unit (worker/src/xcecd_cgroups.c:753-766), which stops the unit with KillMode=control-group, KillSignal=SIGTERM (:195-230), the worker included. The worker's signal_term() (worker/src/xcecd_worker.c:293) forwards SIGTERM to the job and keeps running, so post-exec runs today, but only until systemd's TimeoutStopUSec (unset by XCE, system default, typically 90 s) SIGKILLs everything left in the unit. Requirements: the unit gets an explicit stop timeout covering the post hooks (with a configurable maximum hook duration), and design checks how the worker survives an OOM kill inside its own unit.
• Migration: design checks on which hosts today's queue pre_exec/post_exec runs (in xcecd_worker) and maps it to the matching hook, so behaviour does not change.

Environment of the outer layer and reserved names

Decided 2026-10-12 (user), replacing the idea of an env denylist:

• A. The profile's pre/post-exec runs in a clean environment: system values (XCE_*, HOME, a fixed PATH) plus the profile's own env, never the job's. The job's env applies after it, to the job's pre-exec, the job and the job's post-exec. An admin script that needs job facts reads XCE_*. Today the job env is applied first (apply_env_vars(), worker/src/xcecd_worker.c:835, then queue_pre_exec() :843), so a job's PATH or BASH_ENV can subvert the admin's script. Worker change in this spec: the outer layer gets an explicitly built env (uv_proc_pre_exec takes none today). It also fixes today's queue pre/post-exec.
• The profile env travels separately from the job env (decided 2026-10-12, review). Today all env is one exdata.sub_env_vars object, also sent to remote tasks; if the profile env were merged into it, the worker could not build the clean environment of A. The profile env goes in its own start_info field (or an exdata key, decided in design), to the head and every remote task; the job's env stays in sub_env_vars. The job process gets both (profile first, the job's additions on top, never redefining a profile variable).
• B. The XCE_ prefix is reserved: neither a job nor a profile may set XCE_* variables (rejected at submit by apiserver and controller, and by xceadmin for profiles).
• Why no denylist: the whole job unit runs as the execution user (User= on the transient unit, worker/src/xcecd_cgroups.c:299-312), in sandbox mode inside the sandbox, so LD_PRELOAD and similar are not an escalation; the real risks were the admin's outer layer and XCE's own variables, which A and B close at the source. Sandbox-specific env exposure stays with job-sandbox-execution.

Presets (client side, xcecli-presets)

Not part of this spec, recorded here because they hold the convenience defaults a profile does not (everything but default reservations):

• A preset is a set of job options (--app, --queue, -R, -n, env, limits). The client expands it; a command-line flag replaces the preset's value for that option as a whole. The submit carries only the expanded values.
• Personal presets live one file per preset under ~/.config/xce/presets/.
• The apiserver only shares: admin-deployed preset files served read-only (GET /xce/v1/presets, GET /xce/v1/presets/{name} or {ns}/{name}) from <TopDir>/config/presets/. Pulling copies a preset into its own local file, so a server preset being removed never affects users. preset diff / pull --force are user-initiated.
• User-to-user sharing by preset export / import (file, mail, git).
• Presets carry no privilege: app:use, queue:submit, the profile's essentials, queue ceilings and project quotas still apply at submit.

Dispatch record

Decided 2026-10-12 (user): what a job was scheduled and started with is frozen at dispatch. A pending job follows the current profile; a dispatched job uses only its dispatch record until it is requeued or restarted, and the scheduler no longer re-merges it.

• The record holds both sides:
  • scheduling: the effective resreq (select, rusage, span, same, order) and task count, so a running job's reservations (e.g. rusage[license=1]) do not change when the profile changes, and are rebuilt identically after a scheduler failover or restart;
  • execution: mode, mode options, launch type, job starter, task starter, the profile's env and the job's env (separately), outer pre/post-exec, the job_pre_post_exec setting, effective limits;
  • the resolved profile name (after aliases) and the content hash the record was built from.
• At each dispatch the controller puts the execution values into the job's start_info for the worker, and the whole record into an additive payload on the WAL dispatch fact (persist_submit_job_dispatch_fact, controller/persist.c:8167) and the snapshot. Both apply seams decode it; /xce-audit-invariants checks the round trip.
• Today the queue's pre/post-exec copied at dispatch (controller/job.c 1825-1897) is kept in memory only, so after a failover the new primary does not know what a running job was given. The frozen record closes that gap as pre/post-exec moves to the profile.
• A running or finished job's details show the frozen values, and say "profile changed since dispatch" when the current profile differs.
• Requeue or restart re-reads the profile at the next dispatch, the same as the live merge does for pending jobs.
• Running jobs therefore need nothing from the live profile: post-exec at the end, later remote tasks or a resize, and a service replica's probes and in-place restarts all use the record.
• Why: for sandboxed jobs the mode and mounts a job actually ran with is a security/audit fact, and job-sandbox-execution's mandatory cleanup depends on it; persistence.md does not allow memory-only state without a documented scope decision.

Profile removed while jobs use it

Decided 2026-10-12 (user): a job whose profile no longer exists pends, with the reason "application profile X is not defined".

• This applies to every job that reads the profile: pending jobs, requeued or restarted jobs, and new service replicas (replacements, scale-up), which are new jobs. Running jobs are unaffected (dispatch record); a service whose profile is gone keeps its running replicas but shows as degraded when one cannot be replaced.
• Putting the profile back releases the jobs. A rename is remove plus add.
• The removal is never refused (xceadmin config build works offline and cannot know live jobs) and jobs never exit because of it.
• The controller's live-upgrade reply lists removed profiles still named by pending jobs or services, with counts; xceadmin config upgrade prints it. Needs a new scheduler pend reason (profile missing from Conf4Sched).
• Between the scheduler's decision and the controller's dispatch (decided 2026-10-12, user). The scheduler merges with its copy of the profile (Conf4Sched); the controller reads the profile again when it builds the dispatch record. If the profile is gone or has changed in between, the dispatch fails and the job goes back to PEND through the requeue path (re-reading the profile, as any requeue does): gone → it pends with "application profile X is not defined"; changed → the scheduler re-merges with the new profile. Never EXIT.
  • New dispatch failure site (e.g. XCE_DISPATCH_FAIL_SITE_APP_PROFILE), classified retryable, unlike today's terminal XCE_DISPATCH_FAIL_SITE_QUEUE_EXEC (controller/job.c). Nothing is committed on this path (j->job.allocs is committed only after a successful xce_startjob_req()). Requirements decide that it does not count as a run or against requeue limits.
  • A scheduler still on the old config may re-dispatch and be declined again until the controller's config push reaches it; the existing dispatch retry backoff bounds that.
  • After the controller has built the record, profile changes no longer matter: the worker starts from start_info.
• Profile content hash (decided 2026-10-12, user): an internal version of each profile.
  • Computed, never stored as state: one src/lib function hashes the canonical Cap'n Proto encoding of the profile, excluding description. Controller, scheduler and apiserver get the same value from the same profile, also after failover or restart.
  • The scheduler's dispatch decision carries the hash it merged with (additive field); the controller compares it with the current profile's hash at dispatch. A scheduler that sends no hash (rolling upgrade) skips the check.
  • The dispatch record stores it; "profile changed since dispatch" in job details is a hash comparison; job details show it short (openfoam@3f2a9c). Admins never set it.
  • Base for profile history and audit (config-history-audit, backlog).
• Moving pending jobs to another profile is not in this spec (decided 2026-10-12, user). XCE has no job modification today (JOB_MODIFY is only an enum, include/xce/types.h:620; the apiserver's "modify" action is commented out, controllers/jobs.go:2446). A general modify path (apiserver action, controller opcode, WAL fact, scheduler re-merge) is its own backlog spec, job-modify (LSF bmod), with --app among its first attributes. Until then: restore the profile, use an alias, or kill and resubmit.

Profile aliases

Decided 2026-10-12 (user), so a rename never strands jobs:

• A profile may list aliases: [old-name, ...]. An alias resolves to the profile everywhere: merge, dispatch record, content hash (the target's), RBAC (app:use is checked against the target profile's name).
• Jobs keep the name they recorded; the scheduler and controller resolve it through the alias. New submits that name an alias are accepted and record the target's name, so the alias can be removed once old jobs drain.
• Job details show the resolution, e.g. openfoam (recorded as openfoam-old).
• xceadmin rejects an alias that clashes with a profile name or another alias, and alias chains (an alias naming an alias). default_application must name a real profile, not an alias.
• Removing an alias while jobs still name it is the same as removing a profile: they pend, and the live-upgrade warning lists them.
• Rename workflow: rename the profile and add the old name as an alias; drop the alias when no pending job uses it (the upgrade warning shows when that is).

Default profile

• Set by default_application: <name> in xce.yaml, a cluster parameter (as LSF keeps DEFAULT_APPLICATION in lsb.params), not a flag on one profile.

• Recorded at submit. A job submitted without --app gets the current default's name written on its record, so changing the default later does not move pending jobs to another profile.

• An empty profile name means "no profile", never "the current default": jobs submitted before profiles existed (old WAL records and snapshots) keep today's behaviour and are never silently given default_application. Only new submits record the default's name.

• Restart carries the profile name. POST /:jobids/restart rebuilds the new job field by field from the old job's record (bug-1968 shows what is lost otherwise), so it must copy the profile name (or alias); it then re-checks app:use and re-reads the profile like any new dispatch.

• Array jobs carry one profile name on the array's submit data; every element uses it.

• A name that is not defined is a configuration error, never a silent fallback (user decision, 2026-10-12):

  default_application	Behaviour
  absent	built-in default profile (mode direct, no attributes), today's behaviour
  names a defined profile	that profile is used for jobs submitted without --app
  names an undefined profile	the configuration is rejected

  • xceadmin config build fails with an error naming the missing profile. This is an error, not one of the warnings validation.py gives for unknown host or user references: a dangling default would break every submit without --app.
  • The controller re-checks on live upgrade and at start, because config can reach it without xceadmin. A live upgrade with a dangling default, including one that removes or renames the current default profile, is refused and the current config stays in force. At start the controller refuses to run and logs the reason.
  • Profiles and the default are in the same configuration, checked atomically, so the default cannot become dangling at run time.

• Queue deprecations surface here. xceadmin config build warns when queues.yaml still sets res_req or pre_exec/post_exec (the shipped night queue does) and suggests moving them into a profile. The deprecation itself is job-config-merge's; the message needs applications.yaml to exist.

Scope

• In:
  • The profile schema (essentials, additive rule, aliases) and applications.yaml, and the default profile (default_application).
  • Submit-time validation of the profile and the additive rule.
  • RBAC permission for profile use.
  • Submit surfaces: REST, xcecli --app, harness, libxce.
  • The profile name and the job's own values persisted on the job.
  • The mode: schema field (only direct), the structured resreq schema (select, rusage entries with per: / default, span, same, order), the job and task starters, params (--app-param) and annotations, exported as XCE_APP_PARAM_* / XCE_APP_ANN_*.
  • New start_info fields (job starter, task starter, profile env separate from the job env) and exdata handling with job-config-merge.
  • Profile content hash (shared src/lib function) and its field on the scheduler's dispatch decision.
  • Restart and array submits carrying the profile name.
  • A worker launch error when a direct exec user is missing.
  • Dispatch record (scheduling and execution values, content hash) and the retryable dispatch failure for a missing or changed profile.
  • Profile pre/post-exec as the outer layer (moved from the queue), with a clean environment; the reserved XCE_ prefix.
  • Aliases, live-upgrade warning, pend reason for a missing profile.
  • Profile name and mode shown in job details.
  • Listing available profiles (REST and xcecli).
  • Profile --dry-run support: the effective values a job would get (shared with job-config-merge).
  • Tests.
• Open for requirements / design:
  • Hooks and starters (decided, "Hooks and starters"); design confirms the serial-job wrapping order, which hosts today's queue pre/post-exec runs on, the unit stop timeout and maximum hook duration, and how the worker survives an OOM kill.
  • Confirm a dispatch declined for a missing or changed profile does not count as a run or against requeue limits.
  • How resource defaults combine with a job's own resreq — settled 2026-10-09, owned by job-config-merge.
  • Settled 2026-10-12: scheduling and execution values frozen at dispatch, re-read on requeue ("Dispatch record"); a removed profile makes its jobs pend ("Profile removed while jobs use it"); clean env for the outer layer and reserved XCE_ instead of an env denylist; moving pending jobs deferred to job-modify, profile aliases in v1; override granularity (no markers; additive rule), limits a job may lower (profile limits are maximums), personal presets (client side), app:override and inheritance (deferred); execution modes split into job-sandbox-execution; v1 attributes are the four plus the job starter (plus a task starter for every task startup, decided 2026-10-12), others ship with their job features; profile history and change audit deferred to the roadmap (config-history-audit): the content hash and the dispatch record already show what each run used; no job labels needed here (presets record themselves in exdata.meta; general labels are the job-labels backlog item); compound and alternative resreqs (only the profile select applies); structured resreq with per: default task; separate profile env; empty profile name = no profile; a file profile named default replaces the built-in; 0-valued rusage terms removed by the merge; params and annotations (exported to env) in v1; submit hook deferred (app-submit-hook).
• Out:
  • The sandbox mode and its options, I/O directories, capability advertisement, placement by mode and the worker sandbox mechanics (job-sandbox-execution).
  • Multi-node sandbox (job-sandbox-multinode).
  • Container or vm execution itself (only room for them in the schema).
  • Auth backends and JWT claims.

Boundary Candidates

• The profile model and config (schema, validation, live upgrade).
• Submit-time validation of the profile and the additive rule (controller).
• Access control (RBAC permission) and early validation (apiserver).
• Submit and listing surfaces (REST, xcecli, harness, libxce).
• Persisting the profile name and the job's own values (wire, WAL submit fact, snapshot).

Out of Boundary

• Worker execution backends and sandbox mechanics.
• Output file location and retrieval inside the sandbox.
• Scheduler policy beyond capability filtering and the profile's essential resreq.
• Presets (xcecli-presets): client side; the read-only preset listing on the apiserver belongs to that spec.

Upstream / Downstream

• Upstream:
  • The existing submit path.
  • The WAL submit fact (ha-wal-replication, the raft-apply-decoupling apply seam).
  • The config pipeline (clusterconfig-schema-split, config-upgrade-snapshot, admin-config-tool).
  • RBAC (rbac_check.go).
  • The scheduler-to-controller dispatch decision (gains the profile content hash; scheduler-dispatch-commit).
  • The restart path (RestartJobByID, bug-1968).
• Downstream:
  • job-sandbox-execution adds the sandbox mode, its options and I/O directories, capability advertisement and placement, on top of this spec's mode: field and dispatch record.
  • job-sandbox-multinode.
  • Future container/vm modes, which add profile attributes.
  • job-status-detail-display, which can show the profile and mode.
  • job-modify (moves pending jobs between profiles), job-requeue (requeue/success exit values as profile attributes), config-history-audit (profile history), xcecli-presets (reads profiles for warnings; exdata.meta).

Existing Spec Touchpoints

• Extends: none directly.
• Depends on: job-config-merge (merge rules and live re-evaluation).
• Feeds: parjob-placement (the profile's span[...]).
• Adjacent:
  • job-status-detail-display
  • controller-job-cleanup
  • resreq-maxtasks-wiring (task range)
  • xcecli-presets (convenience defaults, client side)
  • profile-usage-insight (advisory recommendations)
  • xce-node-manager

Constraints

• Middleware chain unchanged: WhitelistURL → WhitelistIP → RateLimit → JWT → CORS → RBAC.
• All wire changes are additive Cap'n Proto. Old WAL records and snapshots decode as "no profile, mode direct", which is today's behaviour; an empty profile name is never resolved to default_application.
• Save/restore field completeness (persistence.md): the profile name, the job's own values, the new start_info fields and the dispatch record round trip through the WAL submit and dispatch facts and the snapshot (/xce-audit-invariants controller). The scheduler's own state for dispatched jobs (frozen, no longer re-merged) and its Conf4Sched profiles round trip through the scheduler snapshot and replay recordings (/xce-audit-invariants scheduler).
• Input validation (security.md): the job's additions and paths are validated server-side. No path traversal, and mount prefixes are restricted by the profile.
• Size: still large after the split (config, access and submit, job state, outer layer); one coherent feature, testable end to end with direct.
• HA/persistence spec: use the full 3-phase workflow.