myxce/.kiro/specs/application-profile/brief.md

Brief: application-profile

(Reshaped from job-execution-mode on 2026-09-29, by user direction. Execution mode is an attribute of the job, determined by the application the user runs, so it is carried by a named application profile rather than by a bare mode option.)

Update 2026-10-09 (/kiro-discovery, configuration layering; user decisions):

• Configuration is layered LSF-style: queue = scheduling policy only (no resreq; queue res_req is deprecated), application profile = application execution constraints including the job's resource requirement (select / rusage / span[...] / same) and task limits (min / default / max, LSF TASKLIMIT), project = resource ownership (project-config).
• Every attribute has one owning level (.kiro/steering/job-config-layering.md). How the levels interact is owned by job-config-merge; queue ceilings and project quotas only check, never change, the job's values. (The job-vs-profile rule of this update, default/fixed markers, is replaced by the additive rule of 2026-10-12 below.)
• Pre/post-exec moves here from the queue (queue pre_exec/post_exec deprecated). The profile's pre/post-exec is admin-defined and always runs as the outer layer; the job's own pre/post-exec is the inner layer, allowed or forbidden by the profile. Order: profile pre → job pre → job → job post → profile post (the worker's existing two-layer structure, xcecd_worker.c queue_pre_exec()/queue_post_exec()).
• The merge is live (replaces the earlier "resolution at submit time", removed from this brief 2026-10-12). The WAL keeps the job's own values plus the profile name; the controller and apiserver validate at submit, and the scheduler computes effective values and recomputes them when a profile changes (a pending job that no longer satisfies its profile pends with a reason; running jobs are unaffected). Execution-side values (mode, options, launch type, env, pre/post-exec) are still taken from the profile at dispatch time and frozen on the dispatch record (decided 2026-10-12, "Dispatch record" below).

Update 2026-10-10 (/kiro-discovery, long-running services): a profile may declare kind service with a service section: restart policy (always / on-failure / never, backoff, give-up), liveness and readiness probes (command / TCP / HTTP), named ports, replica range, spread rule and rollout settings (how these follow the additive rule: "Essentials and the additive rule" below). The section is defined with this spec and consumed by service-supervision (worker probes and restarts), service-replica-set (replicas, scaling, rolling update) and service-endpoint (ports, endpoints).

Update 2026-10-12 (discussion; user decisions):

• Own configuration file. Profiles live in a new src/configs/applications.yaml with its own JSON schema, mapped into ClusterConfig and Conf4Sched like queues and limits. See "Profile configuration" below.
• Cluster default is set by default_application in xce.yaml. A name that is not defined is a configuration error everywhere (xceadmin, live upgrade, controller start), never a silent fallback. See "Default profile" below.
• This spec owns the profile's attribute schema (which attributes exist and what each holds). job-config-merge consumes that schema and owns the merge and live re-evaluation.
• A profile holds only essentials; convenience lives in client-side presets (user decision, 2026-10-12). "The profile holds what the job must have; the preset holds what the user would like." A profile value is something the application needs or the admin enforces, and it binds every job of that profile. The default/fixed markers of 2026-10-09 are dropped: there are no markers, and no per-term markers or ranges. See "Essentials and the additive rule" below.
  • Additive rule: the job may add to the profile's values, never change or remove them. A conflict is rejected at submit with an error naming the profile value, never resolved silently.
  • Convenience defaults (tuning such as rusage[mem=...], span[ptile=...], task count, extra env) go into presets, handled fully by the client (xcecli-presets). The server only shares admin-deployed presets read-only; presets never reach the controller, WAL or scheduler and carry no privilege.
  • Rule of thumb for admins: something every job of the application must have (required host properties, licenses, launch type, mode, pre/post-exec, limits) goes into the profile; an optional variant or tuning shortcut goes into a site preset, typically one with the profile's name.
  • Recommendations are advisory (user decision): usage-based suggestions (requested vs. used, OOM kills, runtime by variant) are reported to the admin, who changes site presets (or a profile, for an essential); nothing tunes itself. Backlog: profile-usage-insight.
  • Job details show the effective resreq with each term's origin (profile or job).
  • Deferred: a privileged app:override for one-off exceptions (an admin can modify the pending job instead; recurring variants are presets), and profile inheritance (base:), no longer needed for variants.
• Later sections are added by their consumers: service: (service-supervision), I/O claims (resource-model PR-6), relationship and compound templates (PR-14/15), container: (R6). This spec only makes the schema open to new top-level sections.

Problem

XCE users run many kinds of applications: trusted in-house codes, untrusted scripts, vendor binaries, MPI solvers. Each needs a different way of running:

• directly as the submitter, or jailed in a sandbox under a separate execution identity (and later in a container or vm);
• a particular launch type (serial, openmpi, mpich);
• particular environment and pre/post-exec steps;
• resource requirements it cannot run without, and limits.

Today every one of these must be set per job by hand, and several (execution mode, sandbox options) cannot be set at all. Many users log in through the portal or REST API (JWT; config_file/PAM, later LDAP or OAuth2) and have no Unix account on the compute nodes. They can only run jobs if their applications run sandboxed. Admins have no way to say "this application runs like this, and these users may run it".

This spec introduces application profiles and is the submission/policy half of the job-sandbox initiative (roadmap "Job Sandbox Execution"). There is precedent in LSF application profiles (lsb.applications, bsub -app), which fits XCE's LSF-style job model.

Current State

• Identity. A job's identity is only the JWT username (controllers/auth.go:26-37). SubmitUser comes from the token (controllers/jobs.go:1379, :1459), and UserGroup is taken from the client request unchecked (:1465).
• Execution user. It is always the submitter: resolveExecUser (backend/convert_pure.go:162, bug-1780).
• Per-job settings that exist today (src/include/xce/types.h):
  • job_startinfo_t (:240-251): execuser, exechome, execcwd, cmdline, exdata, launch_type, and pre/post-exec commands.
  • exdata carries the environment (sub_env_vars) and the worker cgroup limits parsed by parse_limits (MemoryMax, CPUQuota, TasksMax, RuntimeMax, ...). The apiserver never sends exdata (backend/convert.go:273-315).
  • The resource requirement is resreq on the submit data.
  • Remote tasks use xce_taskhost_t (:696-705).
• Missing today:
  • any profile concept, any execution-mode field, any input/output directory field;
  • any way for workers to advertise execution capabilities to the scheduler.
• Controller. It validates only that the user is not empty (controller/job.c:2447). The submit data, including start_info, is persisted in the WAL submit fact and in snapshots.
• Ownership. RBAC ownership, scheduler per-user limits and the user table all key on the submit user, not the exec user (controllers/jobs.go:1010, :2542, output.go:95; schedobj.c:121; plugins/limit.c). So a separate execution identity breaks none of them.
• Access control. RBAC is config/rbac.yaml plus RequirePermission (controllers/rbac_check.go).

Desired Outcome

• Admins define named application profiles in the cluster configuration. The first version contains the following (user decision; the format must be extensible for "maybe others"):
  1. Execution mode and its options. direct or sandbox now, with room for container/vm. Sandbox options include the allowed input/output prefixes and isolation extras.
  2. Launch type. Serial, openmpi or mpich.
  3. Environment and pre/post-exec. Environment variables the application needs, and the outer pre/post-exec commands.
  4. Essential resource requirement and limits. The resreq terms the application cannot run without (select / rusage / span[...] / same / order), its task range (min / max), and per-job execution limits as maximums (memory, runtime, CPU, tasks), mapped onto the existing worker cgroup limits. All of these are essentials under the additive rule; none is a convenience default.
• Users pick a profile at submit (REST, xcecli jobs submit --app <name>, harness, libxce). A cluster default profile applies when none is given.
• Jobs add, never override. A job may add to its profile's values (more select terms, other resources, its own env variables, its own I/O directories within the allowed prefixes, lower limits) but not change or remove any. Anything else is rejected with a clear error.
• Access by RBAC permission. No new access mechanism; this follows the existing named-instance pattern (user decision, 2026-09-29: fold profile RBAC into this spec). Today, queue:submit scoped to type: queue, values: [normal, priority] in config/rbac.yaml is enforced by GetRequireResourcePermission in controllers/jobs.go:1402-1409. Profiles add:
  • Permissions: app:use (submit a job with a profile) and app:view (list and inspect profiles).
  • Scope: a new app scope type whose values are profile names, or all.
  • Checks:
    • Job create checks app:use alongside, and independent of, queue:submit.
    • Job restart (POST /:jobids/restart, which today checks only queue:submit) re-checks app:use for the job's profile.
    • Listing profiles checks app:view.
  • Defaults: rbac.yaml gets defaults, for example basic_user → app:use on the default profile, and app:view on all.
  • Role API: role creation and update accept the new scope type.
  • Bypass: cluster_admin keeps the single existing bypass; no new bypass path.
  • Persistence: API-made grants are subject to the rbac-persistence gap until that spec lands.
• Queues and profiles are orthogonal (user decision, 2026-09-29).
  • Queues own scheduling policy: which jobs are considered and in what order, priority, scheduling limits, and host partitioning.
  • Application profiles own application-related configuration: how the application is executed and what it needs to run.
  • A queue does not list or restrict profiles, and a profile does not select or constrain queues.
  • A profile carries no scheduling-policy attributes (no priority, queue choice, fairshare or preemption).
  • Any combination of an allowed queue and an allowed profile is valid. Each is governed by its own access rules; for profiles, that is RBAC permission.
  • Resource values in a profile describe what the application needs (its essential resource requirement and per-job execution limits). They are not scheduling policy. Queue-level scheduling limits keep applying on top, unchanged.
• Live merge (job-config-merge). The job records its profile name and its own values, not the effective values. The scheduler computes the effective values and recomputes them when the profile changes: a pending job that no longer satisfies its profile pends with a reason, and running jobs are unaffected. Job details show the profile name and the effective mode.
• The apiserver never checks node accounts (user decision, 2026-09-29).
  • Submit-time validation covers profile existence, RBAC permission, the additive rule and paths only.
  • The execution account is the worker's concern. If a direct job's execution user doesn't exist on the node, the worker fails the job at launch with a clear, user-visible reason, instead of today's opaque systemd failure.
  • Account-less users are served by profiles whose mode is sandbox.
• Placement. A job is placed only on workers whose advertised capabilities support its effective mode (sandbox needs systemd 242 or later and nss-systemd).
• Ownership. The submitter remains the owner for RBAC, limits and accounting in every mode.

Approach

• Profile definitions live in their own cluster configuration file (see "Profile configuration" below), with xceadmin validation and live-upgrade replication.
• The controller validates the profile at submit: the named profile exists, and the job only adds to the profile's values (additive rule). This makes it authoritative for every submit path (REST, libxce, harness). It stores the profile name and the job's own values; the effective values come from the live merge (job-config-merge).
• The apiserver checks that the named profile exists (user direction, 2026-09-29).
  • At submit, before anything goes downstream (preCheckJobSubmitRequest), the apiserver rejects a job that names a profile which does not exist, with a clear client error naming the unknown profile.
  • The apiserver already receives the cluster configuration from the controller (backend/convert.go:265), so it knows the current set of profiles and keeps it current across live config upgrades.
  • The controller still re-validates at submit, because it is authoritative for non-REST submit paths and for a profile removed between the two checks. If the profile has gone by then, the job is rejected, never run with a fallback profile.
• The apiserver additionally enforces the RBAC permission for the profile (middleware chain unchanged; a RequirePermission-style check) and checks the additive rule early (preCheckJobSubmitRequest).
• The job's own values stay in its existing fields (launch_type, pre/post exec, exdata env and limits, resreq). New job fields are added only for the profile name and the job's own I/O directories. All wire changes are additive Cap'n Proto fields, and the WAL submit fact and snapshot carry them with field completeness. Execution-side values (mode and options, launch type, env, profile pre/post-exec) are read from the profile at dispatch, as queue pre/post-exec is today (get_queue_exec(), controller/queue.c:199).
• Workers advertise their supported execution modes. The scheduler filters hosts by the job's effective mode. The design picks the mechanism, for example a host attribute or boolean resource.

Profile configuration

• One file per entity type, as for hosts, queues, resources, limits, users and scaler:

  • src/configs/applications.yaml, top-level key applications: (named after LSF lsb.applications; the user flag is --app);
  • src/configs/schema/applications.schema.json, picked up by xceadmin's file-to-schema naming rule (xcecommon/validation.py:68);
  • xcecommon/validation.py: duplicate names rejected; cross-file references checked (resreq resource names against resources.yaml);
  • src/schema/controller.capnp: applications: List(AppProfileConf) in ClusterConfig and in Conf4Sched (the scheduler runs the live merge). Conf4Worker does not need it: the worker gets execution values through start_info at dispatch.
  • The apiserver already receives ClusterConfig (backend/convert.go:265), so it gets the profiles with no new channel.

• Shared config, per config-design.md: controller, scheduler and apiserver all need the same view (independence rule), and profiles change at the rate of queues, as occasional admin edits (frequency rule).

• The file is optional. Without it there is one built-in profile default (mode direct, no attributes), which is today's behaviour.

• No override markers. Every value in a profile is an essential and binds the job (additive rule). Whether the job's own pre/post-exec is allowed is a profile setting.

• Open to new sections. Later specs add top-level sections to a profile (service:, I/O claims, relationship templates, container:); the schema must allow that without reshaping existing sections.

• Sketch (format decided in design):

  applications:
    - name: openfoam
      description: "OpenFOAM, MPI"
      mode: direct
      launch_type: openmpi
      resreq:   "select[ib] rusage[license_foam=1]"   # essentials only
      tasks:    { min: 32, max: 512 }
      limits:   { runtime: 24h, memory: 128G }       # maximums
      env:      { FOAM_INST_DIR: /opt/openfoam }
      pre_exec:  /opt/xce/app/openfoam/pre.sh
      post_exec: /opt/xce/app/openfoam/post.sh
      job_pre_post_exec: allow
  
    - name: untrusted-python
      mode: sandbox
      sandbox: { inputs: ["/shared/in/"], outputs: ["/shared/out/"] }
      job_pre_post_exec: forbid
  

  Tuning such as rusage[mem=4000] span[ptile=32] and a task count of 64 is not in the profile; the admin publishes it as a site preset named openfoam.

Essentials and the additive rule

How each profile attribute combines with the job's own value (owned by job-config-merge, which implements it):

Profile attribute	Holds	The job may
resreq select	required terms (ib, GPU model, OS)	add terms (ANDed)
resreq rusage	required consumptions (license=1)	add other resources; not change these
resreq span / same / order	only if essential (e.g. a code that runs on one host only)	set them if the profile does not; otherwise rejected
task range	min / max	choose within it; no -n means min
env	variables the application needs	add others; not redefine these
launch type, mode, sandbox options	fixed values	nothing
limits	maximums	ask for less (checked like a ceiling)
pre/post-exec	outer layer, always runs	add its own inner layer if the profile allows

• A job with no -R gets the profile's essentials only; the scheduler's existing empty-resreq default (type == any) still applies to what neither sets.
• The service: section is configured in the profile (user, 2026-10-12; defined with this spec, consumed by service-supervision, service-replica-set, service-endpoint) and follows the same rule:
  • restart policy (always / on-failure / never, backoff, give-up), liveness and readiness probes, and named ports are the application's definition: profile values, the service cannot change them;
  • replicas work like the task range: the profile may set min / max, and the service chooses its count within them (from its own spec or a preset);
  • spread rule and rollout settings (max unavailable / surge): if the profile sets them they bind; otherwise the service sets them.
• Other later sections (I/O claims, relationship templates, container:) state in their own specs how they follow the additive rule.
• Compound or alternative resreqs: a compound job resreq under a profile whose resreq sets terms is rejected until PR-15 named parts define per-part merging (leaning; requirements decide).

Presets (client side, xcecli-presets)

Not part of this spec, recorded here because they replace profile defaults:

• A preset is a set of job options (--app, --queue, -R, -n, env, limits). The client expands it; a command-line flag replaces the preset's value for that option as a whole. The submit carries only the expanded values.
• Personal presets live in ~/.config/xce/presets.yaml.
• The apiserver only shares: admin-deployed preset files served read-only (GET /api/v1/presets, GET /api/v1/presets/{name}). Pulling copies a preset into the local file, so a server preset being removed never affects users. preset diff / pull --force are user-initiated.
• User-to-user sharing by preset export / import (file, mail, git).
• Presets carry no privilege: app:use, queue:submit, the profile's essentials, queue ceilings and project quotas still apply at submit.

Dispatch record

Decided 2026-10-12 (user): what a job was scheduled and started with is frozen at dispatch. A pending job follows the current profile; a dispatched job uses only its dispatch record until it is requeued or restarted, and the scheduler no longer re-merges it.

• The record holds both sides:
  • scheduling: the effective resreq (select, rusage, span, same, order) and task count, so a running job's reservations (e.g. rusage[license=1]) do not change when the profile changes, and are rebuilt identically after a scheduler failover or restart;
  • execution: mode, mode options, launch type, env, outer pre/post-exec, limits.
• At each dispatch the controller puts the execution values into the job's start_info for the worker, and the whole record into an additive payload on the WAL dispatch fact (persist_submit_job_dispatch_fact, controller/persist.c:8167) and the snapshot. Both apply seams decode it; /xce-audit-invariants checks the round trip.
• Today the queue's pre/post-exec copied at dispatch (controller/job.c 1825-1897) is kept in memory only, so after a failover the new primary does not know what a running job was given. The frozen record closes that gap as pre/post-exec moves to the profile.
• A running or finished job's details show the frozen values, and say "profile changed since dispatch" when the current profile differs.
• Requeue or restart re-reads the profile at the next dispatch, the same as the live merge does for pending jobs.
• Running jobs therefore need nothing from the live profile: post-exec at the end, later remote tasks or a resize, and a service replica's probes and in-place restarts all use the record.

Profile removed while jobs use it

Decided 2026-10-12 (user): a job whose profile no longer exists pends, with the reason "application profile X is not defined".

• This applies to every job that reads the profile: pending jobs, requeued or restarted jobs, and new service replicas (replacements, scale-up), which are new jobs. Running jobs are unaffected (dispatch record); a service whose profile is gone keeps its running replicas but shows as degraded when one cannot be replaced.
• Putting the profile back releases the jobs. A rename is remove plus add.
• The removal is never refused (xceadmin config build works offline and cannot know live jobs) and jobs never exit because of it.
• The controller's live-upgrade reply lists removed profiles still named by pending jobs or services, with counts; xceadmin config upgrade prints it. Needs a new scheduler pend reason (profile missing from Conf4Sched).
• Leaning, for requirements: jobmod --app <new> moves a pending job to another profile (the owner, with app:use on the new one; admins in bulk, e.g. --app-from <old>); the scheduler re-evaluates.
• Why: for sandboxed jobs the mode and mounts a job actually ran with is a security/audit fact, and job-sandbox-execution's mandatory cleanup depends on it; persistence.md does not allow memory-only state without a documented scope decision.

Default profile

• Set by default_application: <name> in xce.yaml, a cluster parameter (as LSF keeps DEFAULT_APPLICATION in lsb.params), not a flag on one profile.

• Recorded at submit. A job submitted without --app gets the current default's name written on its record, so changing the default later does not move pending jobs to another profile.

• A name that is not defined is a configuration error, never a silent fallback (user decision, 2026-10-12):

  default_application	Behaviour
  absent	built-in default profile (mode direct, no attributes), today's behaviour
  names a defined profile	that profile is used for jobs submitted without --app
  names an undefined profile	the configuration is rejected

  • xceadmin config build fails with an error naming the missing profile. This is an error, not one of the warnings validation.py gives for unknown host or user references: a dangling default would break every submit without --app.
  • The controller re-checks on live upgrade and at start, because config can reach it without xceadmin. A live upgrade with a dangling default, including one that removes or renames the current default profile, is refused and the current config stays in force. At start the controller refuses to run and logs the reason.
  • Profiles and the default are in the same configuration, checked atomically, so the default cannot become dangling at run time.

• Queue deprecations surface here. xceadmin config build warns when queues.yaml still sets res_req or pre_exec/post_exec (the shipped night queue does) and suggests moving them into a profile. The deprecation itself is job-config-merge's; the message needs applications.yaml to exist.

Scope

• In:
  • The profile schema (essentials, additive rule) and applications.yaml, and the default profile (default_application).
  • Submit-time validation of the profile and the additive rule.
  • RBAC permission for profile use.
  • Submit surfaces: REST, xcecli --app, harness, libxce.
  • The profile name and the job's own values persisted on the job.
  • Execution-mode and I/O-directory fields end to end.
  • Capability advertisement and placement.
  • A worker launch error when a direct exec user is missing.
  • Profile name and mode shown in job details.
  • Listing available profiles (REST and xcecli).
  • Profile --dry-run support: the effective values a job would get (shared with job-config-merge).
  • Tests.
• Open for requirements:
  • Which extra attributes go into v1 beyond the four ("maybe others"). Examples: requeue exit values, runtime estimate, job starter or wrapper, output path template, checkpoint settings.
  • How resource defaults combine with a job's own resreq — settled 2026-10-09, owned by job-config-merge.
  • Profile versioning and audit.
  • jobmod --app for a pending job: who may, and the bulk form (leaning: the owner with app:use on the new profile; admins in bulk).
  • Raised 2026-10-12, not yet decided:
    • A denylist of env variables a sandbox profile forbids jobs to add (e.g. LD_PRELOAD)?
    • Job labels (--label key=value, e.g. the preset name) and a summary grouped by label, to compare tuning variants: here, job-status-detail-display, or xcecli-presets?
  • Settled 2026-10-12: scheduling and execution values frozen at dispatch, re-read on requeue ("Dispatch record"); a removed profile makes its jobs pend ("Profile removed while jobs use it"); override granularity (no markers; additive rule), limits a job may lower (profile limits are maximums), personal presets (client side), app:override and inheritance (deferred).
    • Splitting the spec: execution mode, I/O directories, capability advertisement and placement as a separate spec (or folded into job-sandbox-execution), leaving this spec the profile core.
• Out:
  • The worker sandbox mechanics (job-sandbox-execution).
  • Multi-node sandbox (job-sandbox-multinode).
  • Container or vm execution itself (only room for them in the schema).
  • Auth backends and JWT claims.

Boundary Candidates

• The profile model and config (schema, validation, live upgrade).
• Submit-time validation of the profile and the additive rule (controller).
• Access control (RBAC permission) and early validation (apiserver).
• Submit and listing surfaces (REST, xcecli, harness, libxce).
• Persisting the profile name and the job's own values (wire, WAL submit fact, snapshot).
• Capability advertisement and scheduler placement.

Out of Boundary

• Worker execution backends and sandbox mechanics.
• Output file location and retrieval inside the sandbox.
• Scheduler policy beyond capability filtering and the profile's essential resreq.
• Presets (xcecli-presets): client side; the read-only preset listing on the apiserver belongs to that spec.

Upstream / Downstream

• Upstream:
  • The existing submit path.
  • The WAL submit fact (ha-wal-replication, the raft-apply-decoupling apply seam).
  • The config pipeline (clusterconfig-schema-split, config-upgrade-snapshot, admin-config-tool).
  • RBAC (rbac_check.go).
  • Resource/attribute reporting from worker to scheduler.
• Downstream:
  • job-sandbox-execution consumes the effective mode, options and directories.
  • job-sandbox-multinode.
  • Future container/vm modes, which add profile attributes.
  • job-status-detail-display, which can show the profile and mode.

Existing Spec Touchpoints

• Extends: none directly.
• Depends on: job-config-merge (merge rules and live re-evaluation).
• Feeds: parjob-placement (the profile's span[...]).
• Adjacent:
  • job-status-detail-display
  • controller-job-cleanup
  • resreq-maxtasks-wiring (task range)
  • xcecli-presets (convenience defaults, client side)
  • profile-usage-insight (advisory recommendations)
  • xce-node-manager

Constraints

• Middleware chain unchanged: WhitelistURL → WhitelistIP → RateLimit → JWT → CORS → RBAC.
• All wire changes are additive Cap'n Proto. Old WAL records and snapshots decode as "no profile, mode direct", which is today's behaviour.
• Save/restore field completeness (persistence.md): the profile name and the job's own values (including I/O directories) round trip through the submit facts and snapshots. Run /xce-audit-invariants.
• Input validation (security.md): the job's additions and paths are validated server-side. No path traversal, and mount prefixes are restricted by the profile.
• Size: this is likely 20 or more tasks. If requirements confirm that, split as listed under "Open for requirements".
• HA/persistence spec: use the full 3-phase workflow.