myxce/.kiro/specs/application-profile/brief.md

# Brief: application-profile

(Reshaped from `job-execution-mode` on 2026-09-29, by user direction. Execution
mode is an attribute of the job, determined by the **application** the user runs,
so it is carried by a named application profile rather than by a bare mode
option.)

**Update 2026-10-09 (`/kiro-discovery`, configuration layering; user
decisions):**
- Configuration is layered LSF-style: **queue** = scheduling policy only (no
  resreq; queue `res_req` is deprecated), **application profile** =
  application execution constraints *including the job's resource
  requirement* (select / rusage / `span[...]` / same) and task limits
  (min / default / max, LSF TASKLIMIT), **project** = resource ownership
  (`project-config`).
- Every attribute has one owning level (`.kiro/steering/job-config-layering.md`).
  How the levels interact is owned by **`job-config-merge`**: each profile
  attribute is *default* (the job may replace it) or *fixed* (the job may
  not; for `select` it may only add ANDed terms); queue ceilings and project
  quotas only check, never change, the job's values. This answers the open
  question "how resource defaults combine with a job's own resreq" below.
- **Pre/post-exec moves here from the queue** (queue `pre_exec`/`post_exec`
  deprecated). The profile's pre/post-exec is admin-defined and always runs
  as the outer layer; the job's own pre/post-exec is the inner layer, allowed
  or forbidden by the profile. Order: profile pre → job pre → job → job post
  → profile post (the worker's existing two-layer structure,
  `xcecd_worker.c` `queue_pre_exec()`/`queue_post_exec()`).
- **Replaces "Resolution at submit time"** below: the merge is *live*. The WAL
  keeps the job's own values plus the profile name; the controller and
  apiserver validate at submit, and the scheduler computes effective values
  and recomputes them when a profile changes (a pending job that no longer
  satisfies its profile pends with a reason; running jobs are unaffected).
  Execution-side values (mode, options, launch type, env, pre/post-exec) are
  still taken from the profile at dispatch time; requirements decide whether
  dispatch freezes them on the job record.

**Update 2026-10-10 (`/kiro-discovery`, long-running services):** a profile
may declare **kind `service`** with a service section: restart policy
(always / on-failure / never, backoff, give-up), liveness and readiness
probes (command / TCP / HTTP), named ports, default replica count, spread
rule and rollout settings. The section is defined with this spec and
consumed by `service-supervision` (worker probes and restarts),
`service-replica-set` (replicas, scaling, rolling update) and
`service-endpoint` (ports, endpoints).

## Problem
XCE users run many kinds of applications: trusted in-house codes, untrusted
scripts, vendor binaries, MPI solvers. Each needs a different way of running:
- directly as the submitter, or jailed in a sandbox under a separate execution
  identity (and later in a container or vm);
- a particular launch type (serial, `openmpi`, `mpich`);
- particular environment and pre/post-exec steps;
- sensible resource defaults and limits.

Today every one of these must be set per job by hand, and several (execution
mode, sandbox options) cannot be set at all. Many users log in through the
portal or REST API (JWT; `config_file`/PAM, later LDAP or OAuth2) and have **no
Unix account on the compute nodes**. They can only run jobs if their
applications run sandboxed. Admins have no way to say "this application runs
like this, and these users may run it".

This spec introduces **application profiles** and is the submission/policy half
of the job-sandbox initiative (roadmap "Job Sandbox Execution"). There is
precedent in LSF application profiles (`lsb.applications`, `bsub -app`), which
fits XCE's LSF-style job model.

## Current State
- **Identity.** A job's identity is only the JWT `username`
  (`controllers/auth.go:26-37`). `SubmitUser` comes from the token
  (`controllers/jobs.go:1379`, `:1459`), and `UserGroup` is taken from the client
  request unchecked (`:1465`).
- **Execution user.** It is always the submitter: `resolveExecUser`
  (`backend/convert_pure.go:162`, bug-1780).
- **Per-job settings that exist today** (`src/include/xce/types.h`):
  - `job_startinfo_t` (:240-251): `execuser`, `exechome`, `execcwd`, `cmdline`,
    `exdata`, `launch_type`, and pre/post-exec commands.
  - `exdata` carries the environment (`sub_env_vars`) and the worker cgroup
    limits parsed by `parse_limits` (MemoryMax, CPUQuota, TasksMax, RuntimeMax,
    ...). The apiserver never sends exdata (`backend/convert.go:273-315`).
  - The resource requirement is `resreq` on the submit data.
  - Remote tasks use `xce_taskhost_t` (:696-705).
- **Missing today:**
  - any profile concept, any execution-mode field, any input/output directory
    field;
  - any way for workers to advertise execution capabilities to the scheduler.
- **Controller.** It validates only that the user is not empty
  (`controller/job.c:2447`). The submit data, including `start_info`, is
  persisted in the WAL submit fact and in snapshots.
- **Ownership.** RBAC ownership, scheduler per-user limits and the user table all
  key on the **submit user**, not the exec user (`controllers/jobs.go:1010`,
  `:2542`, `output.go:95`; `schedobj.c:121`; `plugins/limit.c`). So a separate
  execution identity breaks none of them.
- **Access control.** RBAC is `config/rbac.yaml` plus `RequirePermission`
  (`controllers/rbac_check.go`).

## Desired Outcome
- **Admins define named application profiles** in the cluster configuration. The
  first version contains the following (user decision; the format must be
  extensible for "maybe others"):
  1. **Execution mode and its options.** `direct` or `sandbox` now, with room for
     `container`/`vm`. Sandbox options include allowed and default input/output
     mounts and isolation extras.
  2. **Launch type.** Serial, `openmpi` or `mpich`.
  3. **Environment and pre/post-exec.** Default environment variables and
     pre/post-exec commands.
  4. **Resource defaults and limits.** A default resource requirement, plus
     per-job execution limits (memory, runtime, CPU, tasks), mapped onto the
     existing worker cgroup limits.
- **Users pick a profile at submit** (REST, `xcecli jobs submit --app <name>`,
  harness, libxce). A **cluster default profile** applies when none is given.
- **Limited overrides.** A job may override only the attributes its profile
  marks as overridable (for example its own input/output directories within the
  allowed prefixes, or env additions). Anything else is rejected with a clear
  error.
- **Access by RBAC permission.** No new access mechanism; this follows the
  existing named-instance pattern (user decision, 2026-09-29: fold profile RBAC
  into this spec). Today, `queue:submit` scoped to `type: queue`,
  `values: [normal, priority]` in `config/rbac.yaml` is enforced by
  `GetRequireResourcePermission` in `controllers/jobs.go:1402-1409`. Profiles add:
  - **Permissions:** `app:use` (submit a job with a profile) and `app:view` (list
    and inspect profiles).
  - **Scope:** a new `app` scope type whose values are profile names, or `all`.
  - **Checks:**
    - Job create checks `app:use` alongside, and independent of, `queue:submit`.
    - Job restart (`POST /:jobids/restart`, which today checks only
      `queue:submit`) re-checks `app:use` for the job's profile.
    - Listing profiles checks `app:view`.
  - **Defaults:** `rbac.yaml` gets defaults, for example `basic_user` →
    `app:use` on the default profile, and `app:view` on `all`.
  - **Role API:** role creation and update accept the new scope type.
  - **Bypass:** `cluster_admin` keeps the single existing bypass; no new bypass
    path.
  - **Persistence:** API-made grants are subject to the `rbac-persistence` gap
    until that spec lands.
- **Queues and profiles are orthogonal** (user decision, 2026-09-29).
  - **Queues** own scheduling policy: which jobs are considered and in what
    order, priority, scheduling limits, and host partitioning.
  - **Application profiles** own application-related configuration: how the
    application is executed and what it needs to run.
  - A queue does not list or restrict profiles, and a profile does not select or
    constrain queues.
  - A profile carries no scheduling-policy attributes (no priority, queue choice,
    fairshare or preemption).
  - Any combination of an allowed queue and an allowed profile is valid. Each is
    governed by its own access rules; for profiles, that is RBAC permission.
  - Resource defaults in a profile describe what the application needs (its
    default resource requirement and per-job execution limits). They are not
    scheduling policy. Queue-level scheduling limits keep applying on top,
    unchanged.
- **Resolution at submit time.** The effective values are stored on the job
  (mode, options, launch type, env, limits, resreq). Editing a profile later does
  not change queued or running jobs. The job records its profile name, and its
  details show the effective mode.
- **The apiserver never checks node accounts** (user decision, 2026-09-29).
  - Submit-time validation covers profile existence, RBAC permission, overrides
    and paths only.
  - The execution account is the worker's concern. If a `direct` job's execution
    user doesn't exist on the node, the worker fails the job at launch with a
    clear, user-visible reason, instead of today's opaque systemd failure.
  - Account-less users are served by profiles whose mode is `sandbox`.
- **Placement.** A job is placed only on workers whose advertised capabilities
  support its effective mode (sandbox needs systemd 242 or later and
  `nss-systemd`).
- **Ownership.** The submitter remains the owner for RBAC, limits and accounting
  in every mode.

## Approach
- **Profile definitions** live in the cluster configuration: schema, YAML, JSON
  schema, xceadmin mapping, and live-upgrade replication, following the
  `config-design.md` placement rules.
- **The controller resolves the profile at submit** and merges it with the job's
  permitted overrides. This makes it authoritative for every submit path (REST,
  libxce, harness).
- **The apiserver checks that the named profile exists** (user direction,
  2026-09-29).
  - At submit, before anything goes downstream (`preCheckJobSubmitRequest`), the
    apiserver rejects a job that names a profile which does not exist, with a
    clear client error naming the unknown profile.
  - The apiserver already receives the cluster configuration from the controller
    (`backend/convert.go:265`), so it knows the current set of profiles and keeps
    it current across live config upgrades.
  - The controller still re-validates when it resolves the profile, because it is
    authoritative for non-REST submit paths and for a profile removed between the
    two checks. If the profile has gone by then, the job is rejected, never run
    with a fallback profile.
- **The apiserver additionally enforces the RBAC permission** for the profile
  (middleware chain unchanged; a `RequirePermission`-style check) and validates
  overrides early (`preCheckJobSubmitRequest`).
- **Effective values go into the job's existing fields where they exist**
  (`launch_type`, pre/post exec, exdata env and limits, resreq). New fields are
  added only for the profile name, execution mode, mode options and I/O
  directories. All wire changes are additive Cap'n Proto fields, and the WAL
  submit fact and snapshot carry them with field completeness.
- **Workers advertise their supported execution modes.** The scheduler filters
  hosts by the job's effective mode. The design picks the mechanism, for example
  a host attribute or boolean resource.

## Scope
- **In**:
  - The profile schema and config, and a default profile.
  - Resolution and override rules.
  - RBAC permission for profile use.
  - Submit surfaces: REST, `xcecli --app`, harness, libxce.
  - Effective values persisted on the job.
  - Execution-mode and I/O-directory fields end to end.
  - Capability advertisement and placement.
  - A worker launch error when a `direct` exec user is missing.
  - Profile name and mode shown in job details.
  - Listing available profiles (REST and xcecli).
  - Tests.
- **Open for requirements**:
  - The exact per-attribute override rules.
  - Which extra attributes go into v1 beyond the four ("maybe others"). Examples:
    requeue exit values, runtime estimate, job starter or wrapper, output path
    template, checkpoint settings.
  - ~~How resource defaults combine with a job's own resreq~~ — settled
    2026-10-09, owned by `job-config-merge`.
  - Profile versioning and audit.
  - Whether the profile can be changed for a pending job.
- **Out**:
  - The worker sandbox mechanics (`job-sandbox-execution`).
  - Multi-node sandbox (`job-sandbox-multinode`).
  - Container or vm execution itself (only room for them in the schema).
  - Auth backends and JWT claims.

## Boundary Candidates
- The profile model and config (schema, validation, live upgrade).
- Resolution and override engine (controller).
- Access control (RBAC permission) and early validation (apiserver).
- Submit and listing surfaces (REST, xcecli, harness, libxce).
- Persisting effective values (wire, WAL submit fact, snapshot).
- Capability advertisement and scheduler placement.

## Out of Boundary
- Worker execution backends and sandbox mechanics.
- Output file location and retrieval inside the sandbox.
- Scheduler policy beyond capability filtering and resreq defaults.

## Upstream / Downstream
- **Upstream**:
  - The existing submit path.
  - The WAL submit fact (`ha-wal-replication`, the `raft-apply-decoupling` apply
    seam).
  - The config pipeline (`clusterconfig-schema-split`, `config-upgrade-snapshot`,
    `admin-config-tool`).
  - RBAC (`rbac_check.go`).
  - Resource/attribute reporting from worker to scheduler.
- **Downstream**:
  - `job-sandbox-execution` consumes the effective mode, options and directories.
  - `job-sandbox-multinode`.
  - Future `container`/`vm` modes, which add profile attributes.
  - `job-status-detail-display`, which can show the profile and mode.

## Existing Spec Touchpoints
- **Extends**: none directly.
- **Depends on**: `job-config-merge` (merge rules and live re-evaluation).
- **Feeds**: `parjob-placement` (the profile's `span[...]`).
- **Adjacent**:
  - `job-status-detail-display`
  - `controller-job-cleanup`
  - `resreq-maxtasks-wiring` (resreq defaults)
  - `xce-node-manager`

## Constraints
- Middleware chain unchanged: WhitelistURL → WhitelistIP → RateLimit → JWT → CORS
  → RBAC.
- All wire changes are additive Cap'n Proto. Old WAL records and snapshots
  decode as "no profile, mode `direct`", which is today's behaviour.
- Save/restore field completeness (`persistence.md`): the effective profile
  values round trip through the submit facts and snapshots. Run
  `/xce-audit-invariants`.
- Input validation (`security.md`): overrides and paths are validated
  server-side. No path traversal, and mount prefixes are restricted by the
  profile.
- Size: this is likely 20 or more tasks. If requirements confirm that, consider
  splitting "capability advertisement and placement" into its own spec.
- HA/persistence spec: use the full 3-phase workflow.