docs(spec): application-profile hooks and starters; kill-path requirement; proposals 2-5

- worker-run hooks: pre/post_job_exec (once, first host) and
  pre/post_host_exec (once per host); no per-task hooks
- task starter reaches MPI ranks through the mpirun line XCE builds
  (correction); XCE_JOB_HOST_DIR, XCE_HOST_NTASKS, XCE_HOST_TASK_INDEX
- hook failure: job EXIT with a reason naming the hook; post hooks still run
- post hooks must survive every way a job ends: run limit is systemd
  RuntimeMaxUSec stopping the unit (worker included) and post-exec is only
  bounded by the unset TimeoutStopUSec; requirement for an explicit stop
  timeout and OOM survival (not logged as a bug: post-exec does run today)
- default profile always usable with queue:submit; service profile
  rejected for batch jobs; warning when no host matches the profile select

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.kiro/specs/application-profile/brief.md

@@ -175,8 +175,9 @@ fits XCE's LSF-style job model.
      spec; `job-sandbox-execution` adds `sandbox` with its options (allowed
      input/output prefixes, isolation extras), later `container`/`vm`.
   2. **Launch type.** Serial, `openmpi` or `mpich`.
-  3. **Environment and pre/post-exec.** Environment variables the application
+  3. **Environment and hooks.** Environment variables the application
-     needs, and the outer pre/post-exec commands.
+     needs, and the worker-run hooks `pre/post_job_exec` (first host) and
+     `pre/post_host_exec` (every host), see "Hooks and starters".
   4. **Resource requirement and limits.** A structured resreq: the binding
      terms the application cannot run without (`select`, and `span` /
      `same` / `order` when set) and default reservations (`rusage`
@@ -193,10 +194,8 @@ fits XCE's LSF-style job model.
        e.g. `apptainer exec img` or a binding/affinity wrapper.
      - A serial (single-task) job's command is both the job and its task:
        it is wrapped by both, `job_starter` outermost (design confirms).
-     - Tasks an MPI runtime starts by itself (its own daemons, not through
+     - For MPI jobs XCE inserts the task starter into the `mpirun` line it
-       XCE's workers) are out of XCE's reach; for those the job starter or
+       builds, so every rank starts through it (see "Hooks and starters").
-       the launch type's arguments apply. Which launch types start tasks
-       through XCE is settled in design.
   6. **Application parameters (`params`)** (decided 2026-10-12): settings
      XCE carries but never interprets, handled by the application, e.g.
 
@@ -276,6 +275,10 @@ fits XCE's LSF-style job model.
     - Listing profiles checks `app:view`.
   - **Defaults:** `rbac.yaml` gets defaults, for example `basic_user` →
     `app:use` on the default profile, and `app:view` on `all`.
+  - **The current default profile is always usable** (decided 2026-10-12):
+    anyone with `queue:submit` has an implicit `app:use` on the profile
+    `default_application` names (or the built-in `default`), so plain
+    submits never break when an admin repoints the default.
   - **Role API:** role creation and update accept the new scope type.
   - **Bypass:** `cluster_admin` keeps the single existing bypass; no new bypass
     path.
@@ -416,8 +419,10 @@ fits XCE's LSF-style job model.
       env:      { FOAM_INST_DIR: /opt/openfoam }
       job_starter:  "module load openfoam &&"
       task_starter: "/opt/xce/app/openfoam/bind.sh"
-      pre_exec:  /opt/xce/app/openfoam/pre.sh
+      pre_job_exec:   /opt/xce/app/openfoam/licence-out.sh
-      post_exec: /opt/xce/app/openfoam/post.sh
+      post_job_exec:  /opt/xce/app/openfoam/licence-in.sh
+      pre_host_exec:  /opt/xce/app/openfoam/scratch-setup.sh
+      post_host_exec: /opt/xce/app/openfoam/scratch-clean.sh
       job_pre_post_exec: allow
 
     # mode: sandbox and its options come with job-sandbox-execution;
@@ -514,6 +519,12 @@ resreq:
 - A job with no `-R` gets the profile's binding terms and its `rusage`
   defaults; the scheduler's existing empty-resreq default (`type == any`)
   still applies to what neither sets.
+- **Profile kind is checked** (decided 2026-10-12): a batch job naming a
+  `kind: service` profile is rejected ("profile X is a service profile");
+  the services specs reject the reverse.
+- **A `select` no host matches** (decided 2026-10-12): a warning, not a
+  rejection, at submit and from `xceadmin` against `hosts.yaml`, since
+  host attributes can change (e.g. IB nodes added later).
 - **The `service:` section is configured in the profile** (user,
   2026-10-12; defined with this spec, consumed by `service-supervision`,
   `service-replica-set`, `service-endpoint`) and follows the same rule:
@@ -537,6 +548,72 @@ resreq:
   Admin guidance (not enforceable): no part-specific property (e.g. a GPU
   model only one role needs) in a profile's `select`.
 
+### Hooks and starters
+Decided 2026-10-12 (user). Run by XCE's worker, never by the job:
+
+| Item | Runs | Typical use |
+|---|---|---|
+| `pre_job_exec` / `post_job_exec` | once, on the first host, before the job / after all tasks end | licence checkout and check-in, staging |
+| `pre_host_exec` / `post_host_exec` | once per host, before its first task / after its last | scratch, local cache, cleanup (LSF `HOST_PRE_EXEC` / `HOST_POST_EXEC`) |
+| `job_starter` | wraps the job command once (the `mpirun` line for MPI) | `module load ... &&` |
+| `task_starter` | wraps every task | `apptainer exec img`, binding wrappers |
+
+- **No per-task pre/post hooks.** With N tasks per host they would run N
+  times concurrently: setup needs locking, and "the last task cleans up"
+  needs reference counting that a SIGKILL breaks. For MPI jobs XCE does
+  not start the ranks (it builds one `mpirun` line and sets `ntasks = 1`,
+  `worker/src/controller_connector.c:993-1010`), so it could not run hooks
+  around them, and a wrapper there runs in the job's env. A task starter
+  can do per-task work itself (it is a wrapper).
+- **Task starter reach (correction):** for serial and non-MPI parallel jobs
+  the worker wraps each process it starts (`nranks` per host, remote hosts
+  through `xce_taskhost_t`); for MPI jobs XCE inserts it into the `mpirun`
+  command line it builds (`create_mpi_cmdline()`: `mpirun <opts>
+  <task_starter> <app>`), so the MPI runtime starts every rank through it.
+  Only an application that calls its own launcher inside the job script
+  bypasses it. A serial job's command is wrapped by both starters,
+  `job_starter` outermost (design confirms).
+- **Order on a multi-host job:**
+  ```
+  first host:  pre_job_exec → job's own pre-exec → job_starter(job command) … job's own post-exec → post_job_exec
+  every host:  pre_host_exec → task_starter(task) … (all its tasks end) → post_host_exec
+  ```
+  `pre_job_exec` finishes before any host runs `pre_host_exec`;
+  `post_job_exec` runs after every host's `post_host_exec`. The job's own
+  pre/post-exec stays job-level (inner layer, allowed by
+  `job_pre_post_exec`).
+- **Clean environment** for all four hooks: system values, the profile env,
+  `XCE_APP_*`; never the job's env ("Environment of the outer layer").
+- **Per-host helpers:** XCE exports `XCE_JOB_HOST_DIR` (a per-job, per-host
+  directory the worker creates before `pre_host_exec` and removes after
+  `post_host_exec`) and `XCE_HOST_NTASKS` / `XCE_HOST_TASK_INDEX`, so
+  starters and the application know their host-local position without
+  MPI-specific variables. A task starter may use a lock in
+  `XCE_JOB_HOST_DIR` for best-effort once-per-host work (e.g. a cache
+  warm-up), never for cleanup that must happen.
+- **Failures** (decided, proposal 4): `pre_job_exec` fails → nothing else
+  starts, `post_job_exec` still runs, the job goes to EXIT with a reason
+  naming the hook; `pre_host_exec` fails on a host → its tasks do not
+  start, its `post_host_exec` still runs, the job fails. Requeue on hook
+  failure belongs to `job-requeue`. (Today `queue_pre_exec()` just exits:
+  `TODO: Handle pre-exec failure`.)
+- **The post hooks must survive every way a job ends.** Checked
+  2026-10-12: a graceful kill is SIGINT → SIGTERM → SIGKILL with a grace
+  period (`worker/src/job_signal.c:52-55`); a forced kill is SIGKILL to the
+  job's processes; the run limit is systemd `RuntimeMaxUSec` on the unit
+  (`worker/src/xcecd_cgroups.c:753-766`), which stops the unit with
+  `KillMode=control-group`, `KillSignal=SIGTERM` (`:195-230`), the worker
+  included. The worker's `signal_term()` (`worker/src/xcecd_worker.c:293`)
+  forwards SIGTERM to the job and keeps running, so post-exec runs today,
+  but only until systemd's `TimeoutStopUSec` (unset by XCE, system default,
+  typically 90 s) SIGKILLs everything left in the unit. Requirements: the
+  unit gets an explicit stop timeout covering the post hooks (with a
+  configurable maximum hook duration), and design checks how the worker
+  survives an OOM kill inside its own unit.
+- **Migration:** design checks on which hosts today's queue
+  `pre_exec`/`post_exec` runs (in `xcecd_worker`) and maps it to the
+  matching hook, so behaviour does not change.
+
 ### Environment of the outer layer and reserved names
 Decided 2026-10-12 (user), replacing the idea of an env denylist:
 - **A. The profile's pre/post-exec runs in a clean environment**: system
@@ -771,10 +848,10 @@ Decided 2026-10-12 (user), so a rename never strands jobs:
     (shared with `job-config-merge`).
   - Tests.
 - **Open for requirements / design**:
-  - Multi-host jobs: does the profile's outer pre/post-exec run once (first
+  - Hooks and starters (decided, "Hooks and starters"); design confirms
-    host) or on every host (today's queue pre-exec runs in `xcecd_worker`
+    the serial-job wrapping order, which hosts today's queue pre/post-exec
-    on each host it starts)? Remote tasks get the profile env and the job
+    runs on, the unit stop timeout and maximum hook duration, and how the
-    env separately. (Job vs. task starter: decided, item 5 above.)
+    worker survives an OOM kill.
   - Confirm a dispatch declined for a missing or changed profile does not
     count as a run or against requeue limits.
   - ~~How resource defaults combine with a job's own resreq~~ — settled

.kiro/steering/job-config-layering.md

@@ -41,7 +41,7 @@ One line per part; details below.
 | env | the job adds; cannot redefine the profile's |
 | `params` | declared by the profile; the job's value wins (`--app-param`); undeclared names rejected; exported as `XCE_APP_PARAM_*` |
 | annotations | the profile's; exported as `XCE_APP_ANN_*`; the job cannot change them |
-| mode, launch type, job starter (wraps the full job), task starter (wraps every task startup), outer pre/post-exec | the profile's |
+| mode, launch type, job starter (wraps the full job), task starter (wraps every task), hooks `pre/post_job_exec` (first host) and `pre/post_host_exec` (every host), run by the worker | the profile's |
 | compound / alternative job resreq | only the profile's `select` applies; a profile `rusage` default or `span`/`same`/`order` → rejected (until PR-15) |
 | queue ceilings, project quotas | check the merged result; never change it |
 
@@ -148,12 +148,17 @@ preset.
 - Moves from the queue to the application profile. Queue `pre_exec` /
   `post_exec` are deprecated (`xceadmin config build` warns, then rejects).
 - Two layers remain, each with one owner:
-  1. **Profile pre/post-exec**: admin-defined, always runs (fixed); the job
+  1. **Profile hooks**: admin-defined, always run (fixed), run by the worker
-     cannot replace or skip it.
+     in a clean environment; the job cannot replace or skip them. Since
+     2026-10-12 they are `pre/post_job_exec` (once, first host) and
+     `pre/post_host_exec` (once per host); details and kill-path
+     requirements in the `application-profile` brief, "Hooks and starters".
   2. **Job pre/post-exec**: the user's own (`start_info` today); allowed or
      forbidden by the profile (e.g. a sandbox profile may forbid it).
-- Order: profile pre → job pre → job → job post → profile post. A failed
+- Order: profile `pre_job_exec` → job pre → job → job post → profile
-  profile pre-exec skips the rest; profile post-exec still runs for cleanup.
+  `post_job_exec` on the first host; `pre_host_exec` / `post_host_exec`
+  around each host's tasks. A failed profile pre hook skips the rest; the
+  matching post hook still runs for cleanup.
 - The worker already runs exactly two layers (`queue_pre_exec()` /
   `queue_post_exec()`, `worker/src/xcecd_worker.c`; stages
   `JOB_EXEC_STAGE_Q_PRE_EXEC` / `Q_POST_EXEC`), so the worker keeps its