From 863ba98ac893db98effda4b8a5ceaa73b37ff6ac Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 13 Oct 2026 03:28:10 +0000 Subject: [PATCH] docs(spec): application-profile hooks and starters; kill-path requirement; proposals 2-5 - worker-run hooks: pre/post_job_exec (once, first host) and pre/post_host_exec (once per host); no per-task hooks - task starter reaches MPI ranks through the mpirun line XCE builds (correction); XCE_JOB_HOST_DIR, XCE_HOST_NTASKS, XCE_HOST_TASK_INDEX - hook failure: job EXIT with a reason naming the hook; post hooks still run - post hooks must survive every way a job ends: run limit is systemd RuntimeMaxUSec stopping the unit (worker included) and post-exec is only bounded by the unset TimeoutStopUSec; requirement for an explicit stop timeout and OOM survival (not logged as a bug: post-exec does run today) - default profile always usable with queue:submit; service profile rejected for batch jobs; warning when no host matches the profile select Co-Authored-By: Claude Opus 5.5 (1M context) --- .kiro/specs/application-profile/brief.md | 101 ++++++++++++++++++++--- .kiro/steering/job-config-layering.md | 15 ++-- 2 files changed, 99 insertions(+), 17 deletions(-) diff --git a/.kiro/specs/application-profile/brief.md b/.kiro/specs/application-profile/brief.md index 2c6b497f..18d6d161 100644 --- a/.kiro/specs/application-profile/brief.md +++ b/.kiro/specs/application-profile/brief.md @@ -175,8 +175,9 @@ fits XCE's LSF-style job model. spec; `job-sandbox-execution` adds `sandbox` with its options (allowed input/output prefixes, isolation extras), later `container`/`vm`. 2. **Launch type.** Serial, `openmpi` or `mpich`. - 3. **Environment and pre/post-exec.** Environment variables the application - needs, and the outer pre/post-exec commands. + 3. **Environment and hooks.** Environment variables the application + needs, and the worker-run hooks `pre/post_job_exec` (first host) and + `pre/post_host_exec` (every host), see "Hooks and starters". 4. **Resource requirement and limits.** A structured resreq: the binding terms the application cannot run without (`select`, and `span` / `same` / `order` when set) and default reservations (`rusage` @@ -193,10 +194,8 @@ fits XCE's LSF-style job model. e.g. `apptainer exec img` or a binding/affinity wrapper. - A serial (single-task) job's command is both the job and its task: it is wrapped by both, `job_starter` outermost (design confirms). - - Tasks an MPI runtime starts by itself (its own daemons, not through - XCE's workers) are out of XCE's reach; for those the job starter or - the launch type's arguments apply. Which launch types start tasks - through XCE is settled in design. + - For MPI jobs XCE inserts the task starter into the `mpirun` line it + builds, so every rank starts through it (see "Hooks and starters"). 6. **Application parameters (`params`)** (decided 2026-10-12): settings XCE carries but never interprets, handled by the application, e.g. @@ -276,6 +275,10 @@ fits XCE's LSF-style job model. - Listing profiles checks `app:view`. - **Defaults:** `rbac.yaml` gets defaults, for example `basic_user` → `app:use` on the default profile, and `app:view` on `all`. + - **The current default profile is always usable** (decided 2026-10-12): + anyone with `queue:submit` has an implicit `app:use` on the profile + `default_application` names (or the built-in `default`), so plain + submits never break when an admin repoints the default. - **Role API:** role creation and update accept the new scope type. - **Bypass:** `cluster_admin` keeps the single existing bypass; no new bypass path. @@ -416,8 +419,10 @@ fits XCE's LSF-style job model. env: { FOAM_INST_DIR: /opt/openfoam } job_starter: "module load openfoam &&" task_starter: "/opt/xce/app/openfoam/bind.sh" - pre_exec: /opt/xce/app/openfoam/pre.sh - post_exec: /opt/xce/app/openfoam/post.sh + pre_job_exec: /opt/xce/app/openfoam/licence-out.sh + post_job_exec: /opt/xce/app/openfoam/licence-in.sh + pre_host_exec: /opt/xce/app/openfoam/scratch-setup.sh + post_host_exec: /opt/xce/app/openfoam/scratch-clean.sh job_pre_post_exec: allow # mode: sandbox and its options come with job-sandbox-execution; @@ -514,6 +519,12 @@ resreq: - A job with no `-R` gets the profile's binding terms and its `rusage` defaults; the scheduler's existing empty-resreq default (`type == any`) still applies to what neither sets. +- **Profile kind is checked** (decided 2026-10-12): a batch job naming a + `kind: service` profile is rejected ("profile X is a service profile"); + the services specs reject the reverse. +- **A `select` no host matches** (decided 2026-10-12): a warning, not a + rejection, at submit and from `xceadmin` against `hosts.yaml`, since + host attributes can change (e.g. IB nodes added later). - **The `service:` section is configured in the profile** (user, 2026-10-12; defined with this spec, consumed by `service-supervision`, `service-replica-set`, `service-endpoint`) and follows the same rule: @@ -537,6 +548,72 @@ resreq: Admin guidance (not enforceable): no part-specific property (e.g. a GPU model only one role needs) in a profile's `select`. +### Hooks and starters +Decided 2026-10-12 (user). Run by XCE's worker, never by the job: + +| Item | Runs | Typical use | +|---|---|---| +| `pre_job_exec` / `post_job_exec` | once, on the first host, before the job / after all tasks end | licence checkout and check-in, staging | +| `pre_host_exec` / `post_host_exec` | once per host, before its first task / after its last | scratch, local cache, cleanup (LSF `HOST_PRE_EXEC` / `HOST_POST_EXEC`) | +| `job_starter` | wraps the job command once (the `mpirun` line for MPI) | `module load ... &&` | +| `task_starter` | wraps every task | `apptainer exec img`, binding wrappers | + +- **No per-task pre/post hooks.** With N tasks per host they would run N + times concurrently: setup needs locking, and "the last task cleans up" + needs reference counting that a SIGKILL breaks. For MPI jobs XCE does + not start the ranks (it builds one `mpirun` line and sets `ntasks = 1`, + `worker/src/controller_connector.c:993-1010`), so it could not run hooks + around them, and a wrapper there runs in the job's env. A task starter + can do per-task work itself (it is a wrapper). +- **Task starter reach (correction):** for serial and non-MPI parallel jobs + the worker wraps each process it starts (`nranks` per host, remote hosts + through `xce_taskhost_t`); for MPI jobs XCE inserts it into the `mpirun` + command line it builds (`create_mpi_cmdline()`: `mpirun + `), so the MPI runtime starts every rank through it. + Only an application that calls its own launcher inside the job script + bypasses it. A serial job's command is wrapped by both starters, + `job_starter` outermost (design confirms). +- **Order on a multi-host job:** + ``` + first host: pre_job_exec → job's own pre-exec → job_starter(job command) … job's own post-exec → post_job_exec + every host: pre_host_exec → task_starter(task) … (all its tasks end) → post_host_exec + ``` + `pre_job_exec` finishes before any host runs `pre_host_exec`; + `post_job_exec` runs after every host's `post_host_exec`. The job's own + pre/post-exec stays job-level (inner layer, allowed by + `job_pre_post_exec`). +- **Clean environment** for all four hooks: system values, the profile env, + `XCE_APP_*`; never the job's env ("Environment of the outer layer"). +- **Per-host helpers:** XCE exports `XCE_JOB_HOST_DIR` (a per-job, per-host + directory the worker creates before `pre_host_exec` and removes after + `post_host_exec`) and `XCE_HOST_NTASKS` / `XCE_HOST_TASK_INDEX`, so + starters and the application know their host-local position without + MPI-specific variables. A task starter may use a lock in + `XCE_JOB_HOST_DIR` for best-effort once-per-host work (e.g. a cache + warm-up), never for cleanup that must happen. +- **Failures** (decided, proposal 4): `pre_job_exec` fails → nothing else + starts, `post_job_exec` still runs, the job goes to EXIT with a reason + naming the hook; `pre_host_exec` fails on a host → its tasks do not + start, its `post_host_exec` still runs, the job fails. Requeue on hook + failure belongs to `job-requeue`. (Today `queue_pre_exec()` just exits: + `TODO: Handle pre-exec failure`.) +- **The post hooks must survive every way a job ends.** Checked + 2026-10-12: a graceful kill is SIGINT → SIGTERM → SIGKILL with a grace + period (`worker/src/job_signal.c:52-55`); a forced kill is SIGKILL to the + job's processes; the run limit is systemd `RuntimeMaxUSec` on the unit + (`worker/src/xcecd_cgroups.c:753-766`), which stops the unit with + `KillMode=control-group`, `KillSignal=SIGTERM` (`:195-230`), the worker + included. The worker's `signal_term()` (`worker/src/xcecd_worker.c:293`) + forwards SIGTERM to the job and keeps running, so post-exec runs today, + but only until systemd's `TimeoutStopUSec` (unset by XCE, system default, + typically 90 s) SIGKILLs everything left in the unit. Requirements: the + unit gets an explicit stop timeout covering the post hooks (with a + configurable maximum hook duration), and design checks how the worker + survives an OOM kill inside its own unit. +- **Migration:** design checks on which hosts today's queue + `pre_exec`/`post_exec` runs (in `xcecd_worker`) and maps it to the + matching hook, so behaviour does not change. + ### Environment of the outer layer and reserved names Decided 2026-10-12 (user), replacing the idea of an env denylist: - **A. The profile's pre/post-exec runs in a clean environment**: system @@ -771,10 +848,10 @@ Decided 2026-10-12 (user), so a rename never strands jobs: (shared with `job-config-merge`). - Tests. - **Open for requirements / design**: - - Multi-host jobs: does the profile's outer pre/post-exec run once (first - host) or on every host (today's queue pre-exec runs in `xcecd_worker` - on each host it starts)? Remote tasks get the profile env and the job - env separately. (Job vs. task starter: decided, item 5 above.) + - Hooks and starters (decided, "Hooks and starters"); design confirms + the serial-job wrapping order, which hosts today's queue pre/post-exec + runs on, the unit stop timeout and maximum hook duration, and how the + worker survives an OOM kill. - Confirm a dispatch declined for a missing or changed profile does not count as a run or against requeue limits. - ~~How resource defaults combine with a job's own resreq~~ — settled diff --git a/.kiro/steering/job-config-layering.md b/.kiro/steering/job-config-layering.md index 6cbb157c..0c3bbeb9 100644 --- a/.kiro/steering/job-config-layering.md +++ b/.kiro/steering/job-config-layering.md @@ -41,7 +41,7 @@ One line per part; details below. | env | the job adds; cannot redefine the profile's | | `params` | declared by the profile; the job's value wins (`--app-param`); undeclared names rejected; exported as `XCE_APP_PARAM_*` | | annotations | the profile's; exported as `XCE_APP_ANN_*`; the job cannot change them | -| mode, launch type, job starter (wraps the full job), task starter (wraps every task startup), outer pre/post-exec | the profile's | +| mode, launch type, job starter (wraps the full job), task starter (wraps every task), hooks `pre/post_job_exec` (first host) and `pre/post_host_exec` (every host), run by the worker | the profile's | | compound / alternative job resreq | only the profile's `select` applies; a profile `rusage` default or `span`/`same`/`order` → rejected (until PR-15) | | queue ceilings, project quotas | check the merged result; never change it | @@ -148,12 +148,17 @@ preset. - Moves from the queue to the application profile. Queue `pre_exec` / `post_exec` are deprecated (`xceadmin config build` warns, then rejects). - Two layers remain, each with one owner: - 1. **Profile pre/post-exec**: admin-defined, always runs (fixed); the job - cannot replace or skip it. + 1. **Profile hooks**: admin-defined, always run (fixed), run by the worker + in a clean environment; the job cannot replace or skip them. Since + 2026-10-12 they are `pre/post_job_exec` (once, first host) and + `pre/post_host_exec` (once per host); details and kill-path + requirements in the `application-profile` brief, "Hooks and starters". 2. **Job pre/post-exec**: the user's own (`start_info` today); allowed or forbidden by the profile (e.g. a sandbox profile may forbid it). -- Order: profile pre → job pre → job → job post → profile post. A failed - profile pre-exec skips the rest; profile post-exec still runs for cleanup. +- Order: profile `pre_job_exec` → job pre → job → job post → profile + `post_job_exec` on the first host; `pre_host_exec` / `post_host_exec` + around each host's tasks. A failed profile pre hook skips the rest; the + matching post hook still runs for cleanup. - The worker already runs exactly two layers (`queue_pre_exec()` / `queue_post_exec()`, `worker/src/xcecd_worker.c`; stages `JOB_EXEC_STAGE_Q_PRE_EXEC` / `Q_POST_EXEC`), so the worker keeps its