myxce/.kiro/specs/application-profile/brief.md

Brief: application-profile

(Reshaped from job-execution-mode on 2026-09-29, by user direction. Execution mode is an attribute of the job, determined by the application the user runs, so it is carried by a named application profile rather than by a bare mode option.)

Update 2026-10-09 (/kiro-discovery, configuration layering; user decisions):

• Configuration is layered LSF-style: queue = scheduling policy only (no resreq; queue res_req is deprecated), application profile = application execution constraints including the job's resource requirement (select / rusage / span[...] / same) and task limits (min / default / max, LSF TASKLIMIT), project = resource ownership (project-config).
• How the levels combine is owned by job-config-merge: job overrides profile section by section, select is ANDed, queue hard and task limits are ceilings that reject. This answers the open question "how resource defaults combine with a job's own resreq" below.
• Replaces "Resolution at submit time" below: the merge is live. The WAL keeps the job's own values plus the profile name; the controller and apiserver validate at submit, and the scheduler computes effective values and recomputes them when a profile changes (a pending job that no longer satisfies its profile pends with a reason; running jobs are unaffected). Execution-side values (mode, options, launch type, env, pre/post-exec) are still taken from the profile at dispatch time; requirements decide whether dispatch freezes them on the job record.

Problem

XCE users run many kinds of applications: trusted in-house codes, untrusted scripts, vendor binaries, MPI solvers. Each needs a different way of running:

• directly as the submitter, or jailed in a sandbox under a separate execution identity (and later in a container or vm);
• a particular launch type (serial, openmpi, mpich);
• particular environment and pre/post-exec steps;
• sensible resource defaults and limits.

Today every one of these must be set per job by hand, and several (execution mode, sandbox options) cannot be set at all. Many users log in through the portal or REST API (JWT; config_file/PAM, later LDAP or OAuth2) and have no Unix account on the compute nodes. They can only run jobs if their applications run sandboxed. Admins have no way to say "this application runs like this, and these users may run it".

This spec introduces application profiles and is the submission/policy half of the job-sandbox initiative (roadmap "Job Sandbox Execution"). There is precedent in LSF application profiles (lsb.applications, bsub -app), which fits XCE's LSF-style job model.

Current State

• Identity. A job's identity is only the JWT username (controllers/auth.go:26-37). SubmitUser comes from the token (controllers/jobs.go:1379, :1459), and UserGroup is taken from the client request unchecked (:1465).
• Execution user. It is always the submitter: resolveExecUser (backend/convert_pure.go:162, bug-1780).
• Per-job settings that exist today (src/include/xce/types.h):
  • job_startinfo_t (:240-251): execuser, exechome, execcwd, cmdline, exdata, launch_type, and pre/post-exec commands.
  • exdata carries the environment (sub_env_vars) and the worker cgroup limits parsed by parse_limits (MemoryMax, CPUQuota, TasksMax, RuntimeMax, ...). The apiserver never sends exdata (backend/convert.go:273-315).
  • The resource requirement is resreq on the submit data.
  • Remote tasks use xce_taskhost_t (:696-705).
• Missing today:
  • any profile concept, any execution-mode field, any input/output directory field;
  • any way for workers to advertise execution capabilities to the scheduler.
• Controller. It validates only that the user is not empty (controller/job.c:2447). The submit data, including start_info, is persisted in the WAL submit fact and in snapshots.
• Ownership. RBAC ownership, scheduler per-user limits and the user table all key on the submit user, not the exec user (controllers/jobs.go:1010, :2542, output.go:95; schedobj.c:121; plugins/limit.c). So a separate execution identity breaks none of them.
• Access control. RBAC is config/rbac.yaml plus RequirePermission (controllers/rbac_check.go).

Desired Outcome

• Admins define named application profiles in the cluster configuration. The first version contains the following (user decision; the format must be extensible for "maybe others"):
  1. Execution mode and its options. direct or sandbox now, with room for container/vm. Sandbox options include allowed and default input/output mounts and isolation extras.
  2. Launch type. Serial, openmpi or mpich.
  3. Environment and pre/post-exec. Default environment variables and pre/post-exec commands.
  4. Resource defaults and limits. A default resource requirement, plus per-job execution limits (memory, runtime, CPU, tasks), mapped onto the existing worker cgroup limits.
• Users pick a profile at submit (REST, xcecli jobs submit --app <name>, harness, libxce). A cluster default profile applies when none is given.
• Limited overrides. A job may override only the attributes its profile marks as overridable (for example its own input/output directories within the allowed prefixes, or env additions). Anything else is rejected with a clear error.
• Access by RBAC permission. No new access mechanism; this follows the existing named-instance pattern (user decision, 2026-09-29: fold profile RBAC into this spec). Today, queue:submit scoped to type: queue, values: [normal, priority] in config/rbac.yaml is enforced by GetRequireResourcePermission in controllers/jobs.go:1402-1409. Profiles add:
  • Permissions: app:use (submit a job with a profile) and app:view (list and inspect profiles).
  • Scope: a new app scope type whose values are profile names, or all.
  • Checks:
    • Job create checks app:use alongside, and independent of, queue:submit.
    • Job restart (POST /:jobids/restart, which today checks only queue:submit) re-checks app:use for the job's profile.
    • Listing profiles checks app:view.
  • Defaults: rbac.yaml gets defaults, for example basic_user → app:use on the default profile, and app:view on all.
  • Role API: role creation and update accept the new scope type.
  • Bypass: cluster_admin keeps the single existing bypass; no new bypass path.
  • Persistence: API-made grants are subject to the rbac-persistence gap until that spec lands.
• Queues and profiles are orthogonal (user decision, 2026-09-29).
  • Queues own scheduling policy: which jobs are considered and in what order, priority, scheduling limits, and host partitioning.
  • Application profiles own application-related configuration: how the application is executed and what it needs to run.
  • A queue does not list or restrict profiles, and a profile does not select or constrain queues.
  • A profile carries no scheduling-policy attributes (no priority, queue choice, fairshare or preemption).
  • Any combination of an allowed queue and an allowed profile is valid. Each is governed by its own access rules; for profiles, that is RBAC permission.
  • Resource defaults in a profile describe what the application needs (its default resource requirement and per-job execution limits). They are not scheduling policy. Queue-level scheduling limits keep applying on top, unchanged.
• Resolution at submit time. The effective values are stored on the job (mode, options, launch type, env, limits, resreq). Editing a profile later does not change queued or running jobs. The job records its profile name, and its details show the effective mode.
• The apiserver never checks node accounts (user decision, 2026-09-29).
  • Submit-time validation covers profile existence, RBAC permission, overrides and paths only.
  • The execution account is the worker's concern. If a direct job's execution user doesn't exist on the node, the worker fails the job at launch with a clear, user-visible reason, instead of today's opaque systemd failure.
  • Account-less users are served by profiles whose mode is sandbox.
• Placement. A job is placed only on workers whose advertised capabilities support its effective mode (sandbox needs systemd 242 or later and nss-systemd).
• Ownership. The submitter remains the owner for RBAC, limits and accounting in every mode.

Approach

• Profile definitions live in the cluster configuration: schema, YAML, JSON schema, xceadmin mapping, and live-upgrade replication, following the config-design.md placement rules.
• The controller resolves the profile at submit and merges it with the job's permitted overrides. This makes it authoritative for every submit path (REST, libxce, harness).
• The apiserver checks that the named profile exists (user direction, 2026-09-29).
  • At submit, before anything goes downstream (preCheckJobSubmitRequest), the apiserver rejects a job that names a profile which does not exist, with a clear client error naming the unknown profile.
  • The apiserver already receives the cluster configuration from the controller (backend/convert.go:265), so it knows the current set of profiles and keeps it current across live config upgrades.
  • The controller still re-validates when it resolves the profile, because it is authoritative for non-REST submit paths and for a profile removed between the two checks. If the profile has gone by then, the job is rejected, never run with a fallback profile.
• The apiserver additionally enforces the RBAC permission for the profile (middleware chain unchanged; a RequirePermission-style check) and validates overrides early (preCheckJobSubmitRequest).
• Effective values go into the job's existing fields where they exist (launch_type, pre/post exec, exdata env and limits, resreq). New fields are added only for the profile name, execution mode, mode options and I/O directories. All wire changes are additive Cap'n Proto fields, and the WAL submit fact and snapshot carry them with field completeness.
• Workers advertise their supported execution modes. The scheduler filters hosts by the job's effective mode. The design picks the mechanism, for example a host attribute or boolean resource.

Scope

• In:
  • The profile schema and config, and a default profile.
  • Resolution and override rules.
  • RBAC permission for profile use.
  • Submit surfaces: REST, xcecli --app, harness, libxce.
  • Effective values persisted on the job.
  • Execution-mode and I/O-directory fields end to end.
  • Capability advertisement and placement.
  • A worker launch error when a direct exec user is missing.
  • Profile name and mode shown in job details.
  • Listing available profiles (REST and xcecli).
  • Tests.
• Open for requirements:
  • The exact per-attribute override rules.
  • Which extra attributes go into v1 beyond the four ("maybe others"). Examples: requeue exit values, runtime estimate, job starter or wrapper, output path template, checkpoint settings.
  • How resource defaults combine with a job's own resreq — settled 2026-10-09, owned by job-config-merge.
  • Profile versioning and audit.
  • Whether the profile can be changed for a pending job.
• Out:
  • The worker sandbox mechanics (job-sandbox-execution).
  • Multi-node sandbox (job-sandbox-multinode).
  • Container or vm execution itself (only room for them in the schema).
  • Auth backends and JWT claims.

Boundary Candidates

• The profile model and config (schema, validation, live upgrade).
• Resolution and override engine (controller).
• Access control (RBAC permission) and early validation (apiserver).
• Submit and listing surfaces (REST, xcecli, harness, libxce).
• Persisting effective values (wire, WAL submit fact, snapshot).
• Capability advertisement and scheduler placement.

Out of Boundary

• Worker execution backends and sandbox mechanics.
• Output file location and retrieval inside the sandbox.
• Scheduler policy beyond capability filtering and resreq defaults.

Upstream / Downstream

• Upstream:
  • The existing submit path.
  • The WAL submit fact (ha-wal-replication, the raft-apply-decoupling apply seam).
  • The config pipeline (clusterconfig-schema-split, config-upgrade-snapshot, admin-config-tool).
  • RBAC (rbac_check.go).
  • Resource/attribute reporting from worker to scheduler.
• Downstream:
  • job-sandbox-execution consumes the effective mode, options and directories.
  • job-sandbox-multinode.
  • Future container/vm modes, which add profile attributes.
  • job-status-detail-display, which can show the profile and mode.

Existing Spec Touchpoints

• Extends: none directly.
• Depends on: job-config-merge (merge rules and live re-evaluation).
• Feeds: parjob-placement (the profile's span[...]).
• Adjacent:
  • job-status-detail-display
  • controller-job-cleanup
  • resreq-maxtasks-wiring (resreq defaults)
  • xce-node-manager

Constraints

• Middleware chain unchanged: WhitelistURL → WhitelistIP → RateLimit → JWT → CORS → RBAC.
• All wire changes are additive Cap'n Proto. Old WAL records and snapshots decode as "no profile, mode direct", which is today's behaviour.
• Save/restore field completeness (persistence.md): the effective profile values round trip through the submit facts and snapshots. Run /xce-audit-invariants.
• Input validation (security.md): overrides and paths are validated server-side. No path traversal, and mount prefixes are restricted by the profile.
• Size: this is likely 20 or more tasks. If requirements confirm that, consider splitting "capability advertisement and placement" into its own spec.
• HA/persistence spec: use the full 3-phase workflow.