From f2406caaa76cd1f5e6987e868390f16ae10c3164 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 13 Oct 2026 03:15:43 +0000 Subject: [PATCH] docs(spec): application-profile: job starter wraps the full job, task starter wraps every task startup Co-Authored-By: Claude Opus 5.5 (1M context) --- .kiro/specs/application-profile/brief.md | 42 ++++++++++++++++-------- .kiro/steering/job-config-layering.md | 2 +- 2 files changed, 30 insertions(+), 14 deletions(-) diff --git a/.kiro/specs/application-profile/brief.md b/.kiro/specs/application-profile/brief.md index 83714549..2c6b497f 100644 --- a/.kiro/specs/application-profile/brief.md +++ b/.kiro/specs/application-profile/brief.md @@ -183,10 +183,20 @@ fits XCE's LSF-style job model. entries), its task range (min / max), and per-job execution limits as maximums (memory, runtime, CPU, tasks), mapped onto the existing worker cgroup limits. - 5. **Job starter** (decided 2026-10-12): a wrapper the worker puts in - front of the job command (e.g. `module load openfoam &&`, - `apptainer exec img`), frozen on the dispatch record. Design decides - whether it wraps the MPI launcher or each task. + 5. **Job starter and task starter** (decided 2026-10-12, user): two + wrappers, both frozen on the dispatch record. + - `job_starter` wraps the **full job**: the job command, started once + on the first host (for an MPI job, the launcher such as `mpirun`), + e.g. `module load openfoam &&`. + - `task_starter` wraps **every task startup** that XCE's workers + perform, on every host including remote tasks (`xce_taskhost_t`), + e.g. `apptainer exec img` or a binding/affinity wrapper. + - A serial (single-task) job's command is both the job and its task: + it is wrapped by both, `job_starter` outermost (design confirms). + - Tasks an MPI runtime starts by itself (its own daemons, not through + XCE's workers) are out of XCE's reach; for those the job starter or + the launch type's arguments apply. Which launch types start tasks + through XCE is settled in design. 6. **Application parameters (`params`)** (decided 2026-10-12): settings XCE carries but never interprets, handled by the application, e.g. @@ -333,7 +343,8 @@ fits XCE's LSF-style job model. Cap'n Proto; WAL submit/dispatch facts and snapshot carry them with field completeness): - on the submit data: the profile name; - - on `start_info` (what the worker gets): the job starter and the + - on `start_info` (what the worker gets): the job starter, the task + starter (also on each `xce_taskhost_t` entry for remote tasks) and the profile's env, kept **separate** from the job's env (see "Environment of the outer layer"); the queue-named `qPreExec` / `qPostExec` carry the profile's outer pre/post-exec (renamed in design); @@ -403,7 +414,8 @@ fits XCE's LSF-style job model. tasks: { min: 32, max: 512 } limits: { runtime: 24h, memory: 128G } # maximums env: { FOAM_INST_DIR: /opt/openfoam } - job_starter: "module load openfoam &&" + job_starter: "module load openfoam &&" + task_starter: "/opt/xce/app/openfoam/bind.sh" pre_exec: /opt/xce/app/openfoam/pre.sh post_exec: /opt/xce/app/openfoam/post.sh job_pre_post_exec: allow @@ -432,7 +444,7 @@ How each profile attribute combines with the job's own value (owned by | resreq `span` / `same` / `order` | only if essential (e.g. a code that runs on one host only) | set them if the profile does not; otherwise rejected | | task range | `min` / `max` | choose within it; no `-n` means `min` | | env | variables the application needs | add others; not redefine these | -| launch type, mode, sandbox options, job starter | fixed values | nothing | +| launch type, mode, sandbox options, job starter, task starter | fixed values | nothing | | `job_pre_post_exec` | `allow` / `forbid` | use its own pre/post-exec only if `allow` | | limits | maximums | ask for less (checked like a ceiling) | | pre/post-exec | outer layer, always runs | add its own inner layer if the profile allows | @@ -581,7 +593,8 @@ restarted, and the scheduler no longer re-merges it. `order`) and task count, so a running job's reservations (e.g. `rusage[license=1]`) do not change when the profile changes, and are rebuilt identically after a scheduler failover or restart; - - execution: mode, mode options, launch type, job starter, the + - execution: mode, mode options, launch type, job starter, task + starter, the profile's env and the job's env (separately), outer pre/post-exec, the `job_pre_post_exec` setting, effective limits; - the resolved profile name (after aliases) and the content hash the @@ -737,9 +750,11 @@ Decided 2026-10-12 (user), so a rename never strands jobs: - The profile name and the job's own values persisted on the job. - The `mode:` schema field (only `direct`), the structured resreq schema (`select`, `rusage` entries with `per:` / `default`, `span`, `same`, - `order`), the job starter, `params` (`--app-param`) and annotations, + `order`), the job and task starters, `params` (`--app-param`) and + annotations, exported as `XCE_APP_PARAM_*` / `XCE_APP_ANN_*`. - - New `start_info` fields (job starter, profile env separate from the job + - New `start_info` fields (job starter, task starter, profile env + separate from the job env) and exdata handling with `job-config-merge`. - Profile content hash (shared `src/lib` function) and its field on the scheduler's dispatch decision. @@ -758,8 +773,8 @@ Decided 2026-10-12 (user), so a rename never strands jobs: - **Open for requirements / design**: - Multi-host jobs: does the profile's outer pre/post-exec run once (first host) or on every host (today's queue pre-exec runs in `xcecd_worker` - on each host it starts)? Does the job starter wrap the MPI launcher or - each task? Remote tasks get the profile env and the job env separately. + on each host it starts)? Remote tasks get the profile env and the job + env separately. (Job vs. task starter: decided, item 5 above.) - Confirm a dispatch declined for a missing or changed profile does not count as a run or against requeue limits. - ~~How resource defaults combine with a job's own resreq~~ — settled @@ -772,7 +787,8 @@ Decided 2026-10-12 (user), so a rename never strands jobs: limits a job may lower (profile limits are maximums), personal presets (client side), `app:override` and inheritance (deferred); execution modes split into `job-sandbox-execution`; v1 attributes are the four plus the - job starter, others ship with their job features; profile history and + job starter (plus a task starter for every task startup, decided + 2026-10-12), others ship with their job features; profile history and change audit deferred to the roadmap (`config-history-audit`): the content hash and the dispatch record already show what each run used; no job labels needed here (presets record themselves in `exdata.meta`; diff --git a/.kiro/steering/job-config-layering.md b/.kiro/steering/job-config-layering.md index dfbca94f..6cbb157c 100644 --- a/.kiro/steering/job-config-layering.md +++ b/.kiro/steering/job-config-layering.md @@ -41,7 +41,7 @@ One line per part; details below. | env | the job adds; cannot redefine the profile's | | `params` | declared by the profile; the job's value wins (`--app-param`); undeclared names rejected; exported as `XCE_APP_PARAM_*` | | annotations | the profile's; exported as `XCE_APP_ANN_*`; the job cannot change them | -| mode, launch type, job starter, outer pre/post-exec | the profile's | +| mode, launch type, job starter (wraps the full job), task starter (wraps every task startup), outer pre/post-exec | the profile's | | compound / alternative job resreq | only the profile's `select` applies; a profile `rusage` default or `span`/`same`/`order` → rejected (until PR-15) | | queue ceilings, project quotas | check the merged result; never change it |