docs(spec): split execution modes from application-profile into job-sandbox-execution

- application-profile keeps the profile core: config, access and submit,
  dispatch record, outer pre/post-exec layer; mode: accepts only direct
- sandbox mode + options, I/O directories, capability advertisement and
  placement by mode move to job-sandbox-execution
- order: application-profile requirements/design before job-config-merge
  design; implementation interleaves

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.kiro/specs/application-profile/brief.md

@@ -45,6 +45,18 @@ consumed by `service-supervision` (worker probes and restarts),
 `service-endpoint` (ports, endpoints).
 
 **Update 2026-10-12 (discussion; user decisions):**
+- **Split: execution modes move to `job-sandbox-execution`.** This spec is
+  the profile core: config, access and submit, job state (name, dispatch
+  record), the outer pre/post-exec layer. Its schema has `mode:` but
+  accepts only `direct`. The `sandbox` mode with its options, I/O
+  directories, worker capability advertisement and scheduler placement by
+  mode belong to `job-sandbox-execution`, since nothing uses a non-`direct`
+  mode before the sandbox exists. The worker's clear launch error for a
+  missing `direct` exec user stays here.
+- **Order with `job-config-merge`:** requirements and design of this spec
+  first (it fixes the profile schema), then `job-config-merge`'s design;
+  implementation tasks interleave (schema/config, merge library, submit
+  validation and dispatch record).
 - **Own configuration file.** Profiles live in a new
   `src/configs/applications.yaml` with its own JSON schema, mapped into
   `ClusterConfig` and `Conf4Sched` like queues and limits. See "Profile
@@ -144,9 +156,9 @@ fits XCE's LSF-style job model.
 - **Admins define named application profiles** in the cluster configuration. The
   first version contains the following (user decision; the format must be
   extensible for "maybe others"):
-  1. **Execution mode and its options.** `direct` or `sandbox` now, with room for
-     `container`/`vm`. Sandbox options include the allowed input/output
-     prefixes and isolation extras.
+  1. **Execution mode.** The `mode:` field, accepting only `direct` in this
+     spec; `job-sandbox-execution` adds `sandbox` with its options (allowed
+     input/output prefixes, isolation extras), later `container`/`vm`.
   2. **Launch type.** Serial, `openmpi` or `mpich`.
   3. **Environment and pre/post-exec.** Environment variables the application
      needs, and the outer pre/post-exec commands.
@@ -210,9 +222,9 @@ fits XCE's LSF-style job model.
     user doesn't exist on the node, the worker fails the job at launch with a
     clear, user-visible reason, instead of today's opaque systemd failure.
   - Account-less users are served by profiles whose mode is `sandbox`.
-- **Placement.** A job is placed only on workers whose advertised capabilities
-  support its effective mode (sandbox needs systemd 242 or later and
-  `nss-systemd`).
+- **Placement by mode** (moved to `job-sandbox-execution`): a job is placed
+  only on workers whose advertised capabilities support its mode (sandbox
+  needs systemd 242 or later and `nss-systemd`).
 - **Ownership.** The submitter remains the owner for RBAC, limits and accounting
   in every mode.
 
@@ -248,9 +260,9 @@ fits XCE's LSF-style job model.
   launch type, env, profile pre/post-exec) are read from the profile at
   dispatch, as queue pre/post-exec is today (`get_queue_exec()`,
   `controller/queue.c:199`).
-- **Workers advertise their supported execution modes.** The scheduler filters
-  hosts by the job's effective mode. The design picks the mechanism, for example
-  a host attribute or boolean resource.
+- **Workers advertise their supported execution modes** and the scheduler
+  filters hosts by mode: `job-sandbox-execution` (the design picks the
+  mechanism, e.g. a host attribute or boolean resource).
 
 ### Profile configuration
 - **One file per entity type**, as for hosts, queues, resources, limits,
@@ -527,9 +539,13 @@ Decided 2026-10-12 (user), so a rename never strands jobs:
   - RBAC permission for profile use.
   - Submit surfaces: REST, `xcecli --app`, harness, libxce.
   - The profile name and the job's own values persisted on the job.
-  - Execution-mode and I/O-directory fields end to end.
-  - Capability advertisement and placement.
+  - The `mode:` schema field (only `direct`).
   - A worker launch error when a `direct` exec user is missing.
+  - Dispatch record (scheduling and execution values, content hash) and the
+    retryable dispatch failure for a missing or changed profile.
+  - Profile pre/post-exec as the outer layer (moved from the queue), with
+    a clean environment; the reserved `XCE_` prefix.
+  - Aliases, live-upgrade warning, pend reason for a missing profile.
   - Profile name and mode shown in job details.
   - Listing available profiles (REST and xcecli).
   - Profile `--dry-run` support: the effective values a job would get
@@ -552,12 +568,12 @@ Decided 2026-10-12 (user), so a rename never strands jobs:
     the outer layer and reserved `XCE_` instead of an env denylist; moving
     pending jobs deferred to `job-modify`, profile aliases in v1; override granularity (no markers; additive rule),
     limits a job may lower (profile limits are maximums), personal presets
-    (client side), `app:override` and inheritance (deferred).
-    - Splitting the spec: execution mode, I/O directories, capability
-      advertisement and placement as a separate spec (or folded into
-      `job-sandbox-execution`), leaving this spec the profile core.
+    (client side), `app:override` and inheritance (deferred); execution modes
+    split into `job-sandbox-execution`.
 - **Out**:
-  - The worker sandbox mechanics (`job-sandbox-execution`).
+  - The `sandbox` mode and its options, I/O directories, capability
+    advertisement, placement by mode and the worker sandbox mechanics
+    (`job-sandbox-execution`).
   - Multi-node sandbox (`job-sandbox-multinode`).
   - Container or vm execution itself (only room for them in the schema).
   - Auth backends and JWT claims.
@@ -569,7 +585,6 @@ Decided 2026-10-12 (user), so a rename never strands jobs:
 - Submit and listing surfaces (REST, xcecli, harness, libxce).
 - Persisting the profile name and the job's own values (wire, WAL submit
   fact, snapshot).
-- Capability advertisement and scheduler placement.
 
 ## Out of Boundary
 - Worker execution backends and sandbox mechanics.
@@ -587,9 +602,10 @@ Decided 2026-10-12 (user), so a rename never strands jobs:
   - The config pipeline (`clusterconfig-schema-split`, `config-upgrade-snapshot`,
     `admin-config-tool`).
   - RBAC (`rbac_check.go`).
-  - Resource/attribute reporting from worker to scheduler.
 - **Downstream**:
-  - `job-sandbox-execution` consumes the effective mode, options and directories.
+  - `job-sandbox-execution` adds the `sandbox` mode, its options and I/O
+    directories, capability advertisement and placement, on top of this
+    spec's `mode:` field and dispatch record.
   - `job-sandbox-multinode`.
   - Future `container`/`vm` modes, which add profile attributes.
   - `job-status-detail-display`, which can show the profile and mode.
@@ -618,6 +634,6 @@ Decided 2026-10-12 (user), so a rename never strands jobs:
 - Input validation (`security.md`): the job's additions and paths are validated
   server-side. No path traversal, and mount prefixes are restricted by the
   profile.
-- Size: this is likely 20 or more tasks. If requirements confirm that, split
-  as listed under "Open for requirements".
+- Size: still large after the split (config, access and submit, job state,
+  outer layer); one coherent feature, testable end to end with `direct`.
 - HA/persistence spec: use the full 3-phase workflow.

.kiro/specs/job-sandbox-execution/brief.md

@@ -1,5 +1,16 @@
 # Brief: job-sandbox-execution
 
+**Update 2026-10-12 (application-profile split; user decision):** this spec
+now also owns the submission side of execution modes, moved out of
+`application-profile` (whose schema keeps `mode:` accepting only `direct`):
+the `sandbox` mode value and its options (allowed input/output prefixes,
+isolation extras), the job's I/O-directory fields end to end (wire, WAL
+submit fact and snapshot, field completeness), worker capability
+advertisement, and scheduler placement by mode. The mode and options reach
+the worker through `application-profile`'s dispatch record. Sandbox-specific
+env exposure is here; the outer pre/post-exec clean environment and the
+reserved `XCE_` prefix are `application-profile`'s.
+
 ## Problem
 A job whose execution mode is `sandbox` (from `application-profile`) must run on a
 worker node:
@@ -103,7 +114,8 @@ Code exploration, 2026-09-29.
 - **Capability**:
   - The worker detects its sandbox capability (systemd 242 or later, `systemd`
     in `nsswitch.conf` passwd/group, and D-Bus property support) and advertises
-    it to `application-profile`'s placement.
+    it; the scheduler places sandbox jobs only on capable workers (this
+    spec).
   - It refuses, with a clear error, a sandbox job it cannot honour.
 - **Direct mode is unchanged.**
 
@@ -150,11 +162,14 @@ Viability check (2026-09-29, systemd NEWS, systemd.exec(5), v239 sources):
   - Input/output mounts.
   - `StandardOutput=file:` output and retrieval.
   - Helper authentication by unit or cgroup.
-  - Capability detection and advertisement.
+  - Capability detection and advertisement, and scheduler placement by mode.
+  - The `sandbox` mode value and options in the profile schema, and the
+    job's I/O-directory fields end to end (wire, WAL, snapshot).
   - Single-node MPI in one unit.
   - Tests: unit tests plus a live test on a systemd 242+ node.
 - **Out**:
-  - The submission side: mode field, policy, placement (`application-profile`).
+  - The profile core (`application-profile`): config, `mode:` field,
+    access, dispatch record.
   - Multi-node sandbox launch (`job-sandbox-multinode`).
   - Non-systemd backends and container/vm modes (the interface only, not the
     implementations).
@@ -169,15 +184,15 @@ Viability check (2026-09-29, systemd NEWS, systemd.exec(5), v239 sources):
 - Helper authentication.
 
 ## Out of Boundary
-- Job mode selection and policy.
+- Profile selection and access (`application-profile`).
 - Cross-node uid consistency and remote launch.
 - Worker cgroup resource limits, which are unchanged and simply apply to sandbox
   units too.
 
 ## Upstream / Downstream
 - **Upstream**:
-  - `application-profile`: mode and directory fields on `job_startinfo_t`, and
-    capability-based placement.
+  - `application-profile`: the profile core, `mode:` field and the dispatch
+    record that carries mode and options to the worker.
   - The remote-task root fix.
 - **Downstream**:
   - `job-sandbox-multinode`.

.kiro/steering/roadmap.md

@@ -63,8 +63,8 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
 - [x] ha-commit-durability — closes bug-661 (acknowledgment did not imply durability; committed jobs lost across failover). Makes acknowledged ⇒ durable on a majority: the primary counts its own copy toward the commit majority only at its durable LSN (`majority_candidate_locked` caps self at `persist_durable_lsn`, not the appended tip); `fsync_worker` repurposed timer→demand-driven group-commit flusher (one `fdatasync` over pending appends, advance `durable_lsn`, wake commit-eval) — absorbs the former "WAL group-commit batching" backlog item (see below); recovery integrity verdict (clean vs corruption-truncated WAL) + durable-backed freshness floor (candidate advertises bare `persist_durable_floor`, voter floor = `max(durable_floor, learned_committed)`) + pre-promotion eligibility fence. Establishes the **controllers-never-self-terminate** invariant: a recovery-truncated/behind member that cannot catch up is a LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — never `exit`/`abort`), self-heals on peer return (see `ha-async-io.md` process-lifecycle invariant + `src/controller/CLAUDE.md`). Dependencies: ha-commit-engine (consumes commit-position semantics), ha-wal-replication (standby ACK/catch-up), ha-commit-barrier-rollout (reply gate, unchanged). (specs done 2026-06-18; **IMPLEMENTED 2026-06-18**: 17/17 tasks (1.1–5.3), validate-impl GO, controller suite 0 failures under ASAN; deterministic ack-before-fdatasync E2E RED→GREEN + rolling-SIGKILL soak zero acked-job loss. Verified by tests + Protocol Verification Checklist, NOT by raft.tla (which assumes a durable log). Fixed bug-760 (adjacent `test_fact_hook_append` raced the async flusher — test made durable-deterministic via `persist_flush`). raft.tla durability-model extension deferred unless explicitly opted in.)
 - [x] scheduler-allocator-runner — NEW 2026-09-28 (`/kiro-discovery`, Path C). The framework owns `job_alloc_fn` through a new `xce_sched_allocator_iface_t` (accepts guard + plan iterator) and a single `sched_allocator_run()`; both `my_alloc()`s are deleted (seqjob/parjob), and `alloc_iter.h` `init` is retyped to `xce_sched_core_t*`. Behaviour must stay identical (bug-1882/1767/1768). Closes `issues/new_requirements.md` item 4. Brief: `.kiro/specs/scheduler-allocator-runner/brief.md`. Dependencies: scheduler-allocation, scheduler-core (both done). **IMPLEMENTED 2026-09-30** (11/11 tasks, e2266582..28cff179; validate-impl GO; scheduler ASAN suite 96/96): runner in `src/scheduler/src/alloc_runner.c`; decided along the way: the runner clears `alloc_req.host` before every candidate so every short host gets pend reasons (both allocators), and fixes bug-1893 (shared reasons were never recorded). Follow-ups: publish pend reasons only when changed; sim-mode parjob allocation parity fixture; bug-1892; bug-1897.
 - [ ] scheduler-alter-pend-reasons — NEW 2026-09-28 (`/kiro-discovery`, Path E). An ALTER-resreq job keeps a pend reason for EVERY alternative instead of only the last (`scheduler.c:424` FIXME). Wire shape (user decision): additive `alternatives @3: List(JobPendReason)` on `JobPendReason`, per-alternative attribution; carried end-to-end through controller job record, WAL preason fact, recordings and `job_preason_eq`. Also fixes the confirmed `obj->reason` overwrite leak (`scheduler.c:388`). Brief: `.kiro/specs/scheduler-alter-pend-reasons/brief.md`. Dependencies: libxce-alloc-preason-equality (HARD: extends its `job_preason_eq`); sequence after scheduler-allocator-runner (its "no pend-reason content change" rule).
-- [ ] application-profile — NEW 2026-09-29 (`/kiro-discovery`, Job Sandbox Execution initiative, Path D; reshaped from `job-execution-mode` by user direction: execution mode is a job attribute determined by the APPLICATION). Admin-defined named application profiles (LSF `-app` precedent) in cluster config: execution mode + options (`direct`|`sandbox`, room for `container`/`vm`), launch type, env + pre/post exec, resource defaults/limits (extensible). Users pick a profile at submit (`--app`), cluster default profile otherwise; queues (scheduling policy) and profiles (application config) are orthogonal — neither restricts the other; a profile holds only essentials and the job may add to them but never change them (additive rule, 2026-10-12; convenience defaults are client-side presets, `xcecli-presets`); access by RBAC permission (new `app:use`/`app:view` with `app` domain type, folded into this spec); apiserver checks profile existence; controller validates at submit and persists the profile name + the job's own values (live merge via job-config-merge; wire additive, WAL/snapshot field-completeness); own config file `applications.yaml`, cluster default `default_application` in `xce.yaml` (undefined name = config error, never a fallback; updated 2026-10-12). apiserver never checks node accounts; worker fails a `direct` job with a clear reason if the exec user is missing. Worker capability advertisement + scheduler placement. Brief: `.kiro/specs/application-profile/brief.md`. Dependencies: none.
-- [ ] job-sandbox-execution — NEW 2026-09-29 (Job Sandbox Execution). Worker: pluggable execution-backend interface (`direct` refactored + `systemd-sandbox`; non-systemd backends later). Sandbox = existing transient unit + `DynamicUser` with fresh per-job name (never the submitter's), ProtectSystem=strict/ProtectHome/PrivateTmp/NoNewPrivileges, Bind(ReadOnly)Paths for job I/O dirs or worker-created temp workspace (handed to the uid via SO_PEERCRED), `StandardOutput=file:` output, helper auth by unit/cgroup, MANDATORY post-job cleanup/ownership normalization (dynamic uid reuse), systemd ≥ 242 + nss-systemd capability probe; single-node incl. single-node MPI. Viability-checked 2026-09-29. Brief: `.kiro/specs/job-sandbox-execution/brief.md`. Dependencies: application-profile; remote-task-root fix (Direct-Implementation).
+- [ ] application-profile — NEW 2026-09-29 (`/kiro-discovery`, Job Sandbox Execution initiative, Path D; reshaped from `job-execution-mode` by user direction: execution mode is a job attribute determined by the APPLICATION). Admin-defined named application profiles (LSF `-app` precedent) in cluster config: execution mode + options (`direct`|`sandbox`, room for `container`/`vm`), launch type, env + pre/post exec, resource defaults/limits (extensible). Users pick a profile at submit (`--app`), cluster default profile otherwise; queues (scheduling policy) and profiles (application config) are orthogonal — neither restricts the other; a profile holds only essentials and the job may add to them but never change them (additive rule, 2026-10-12; convenience defaults are client-side presets, `xcecli-presets`); access by RBAC permission (new `app:use`/`app:view` with `app` domain type, folded into this spec); apiserver checks profile existence; controller validates at submit and persists the profile name + the job's own values (live merge via job-config-merge; wire additive, WAL/snapshot field-completeness); own config file `applications.yaml`, cluster default `default_application` in `xce.yaml` (undefined name = config error, never a fallback; updated 2026-10-12). apiserver never checks node accounts; worker fails a `direct` job with a clear reason if the exec user is missing. **Split 2026-10-12 (user):** execution modes (`sandbox` value + options, I/O dirs, worker capability advertisement, placement by mode) moved to job-sandbox-execution; this spec keeps `mode: direct`, the dispatch record (resreq + execution values + profile content hash; retryable dispatch failure when the profile is gone/changed), aliases, outer pre/post-exec with clean env, reserved `XCE_`. Order: this spec's requirements+design before job-config-merge's design. Brief: `.kiro/specs/application-profile/brief.md`. Dependencies: none.
+- [ ] job-sandbox-execution — NEW 2026-09-29 (Job Sandbox Execution). Worker: pluggable execution-backend interface (`direct` refactored + `systemd-sandbox`; non-systemd backends later). Sandbox = existing transient unit + `DynamicUser` with fresh per-job name (never the submitter's), ProtectSystem=strict/ProtectHome/PrivateTmp/NoNewPrivileges, Bind(ReadOnly)Paths for job I/O dirs or worker-created temp workspace (handed to the uid via SO_PEERCRED), `StandardOutput=file:` output, helper auth by unit/cgroup, MANDATORY post-job cleanup/ownership normalization (dynamic uid reuse), systemd ≥ 242 + nss-systemd capability probe; single-node incl. single-node MPI. **2026-10-12:** also owns the submission side of execution modes moved from application-profile (`sandbox` mode + options, job I/O-directory fields end to end, capability advertisement, scheduler placement by mode). Viability-checked 2026-09-29. Brief: `.kiro/specs/job-sandbox-execution/brief.md`. Dependencies: application-profile; remote-task-root fix (Direct-Implementation).
 - [ ] job-sandbox-multinode — NEW 2026-09-29 (Job Sandbox Execution). Multi-node sandbox jobs: cluster-consistent ephemeral uid (expected: controller-allocated, WAL-replicated uid lease from a reserved range), worker-driven remote/MPI launch (no ssh; PMIx/prted via worker), shared-FS ownership normalization, one unit per job per node. Brief: `.kiro/specs/job-sandbox-multinode/brief.md`. Dependencies: job-sandbox-execution, application-profile.
 - [ ] rbac-persistence — NEW 2026-09-29 (`/kiro-discovery`, raised during application-profile discovery). Closes the deferred `security.md` gap: runtime RBAC mutations (roles, permissions, user-role assignments, Logon auto-assignment) are in-memory only and silently revert to `rbac.yaml` on apiserver restart — including revocations. Durable-or-rejected mutations, audit trail, defaults-vs-runtime reconciliation, multi-apiserver consistency; storage approach (rbac.yaml write-through / controller-replicated entity / embedded store / read-only) decided in design. Brief: `.kiro/specs/rbac-persistence/brief.md`. Dependencies: none (independent of the sandbox initiative; application-profile's API-made `app:use` grants rely on it for durability).
 - [ ] job-time-window — NEW 2026-09-29 (`/kiro-discovery`, Path C). Job begin time (scheduler gate in `sched_job_is_ready()` via `xce_core_now`, pend reason BEGIN_TIME_NOT_REACHED) and end time (WORKER enforces running jobs: helper SIGTERM→grace→SIGKILL, exit reason END_TIME_REACHED persisted, unit RuntimeMaxUSec backstop; PEND jobs whose end time can't be met stay PEND with END_TIME_UNMEETABLE until user kills/modifies — no controller sweep, primary not involved). Adds runtime estimate (`-We/--estimated_run_time`, scheduling hint only, never enforced; estimate > run limit rejected); unmeetable rule `now + (estimate ‖ run_limit ‖ 0) > end`. Times/estimate modifiable only while PEND (new `xce jobs modify` + harness `jobmod`). Both xcecli and harness `jobsub` gain all options (`-b`/`-t`/`-W`/estimate). Also fixes the silent drop of xcecli `-b`/`-t`/`-W` and REST `run_time_estimation` (apiserver `modelJobToSubmit` never sets them nor `exdata`). Brief: `.kiro/specs/job-time-window/brief.md`. Dependencies: none (new reason codes need text in job-status-detail-display).