diff --git a/.kiro/specs/application-profile/brief.md b/.kiro/specs/application-profile/brief.md index 1afbf727..e25a553a 100644 --- a/.kiro/specs/application-profile/brief.md +++ b/.kiro/specs/application-profile/brief.md @@ -98,9 +98,10 @@ consumed by `service-supervision` (worker probes and restarts), essential); nothing tunes itself. Backlog: `profile-usage-insight`. - Job details show the effective resreq with each term's origin (profile or job). - - Deferred: a privileged `app:override` for one-off exceptions (an admin - can modify the pending job instead; recurring variants are presets), and - profile inheritance (`base:`), no longer needed for variants. + - Deferred: a privileged `app:override` for one-off exceptions (until + `job-modify` exists, an admin kills and resubmits, or uses a variant + profile; recurring variants are presets), and profile inheritance + (`base:`), no longer needed for variants. - **Later sections are added by their consumers**: `service:` (`service-supervision`), I/O claims (resource-model PR-6), relationship and compound templates (PR-14/15), `container:` (R6). This spec only makes @@ -137,9 +138,13 @@ fits XCE's LSF-style job model. - **Per-job settings that exist today** (`src/include/xce/types.h`): - `job_startinfo_t` (:240-251): `execuser`, `exechome`, `execcwd`, `cmdline`, `exdata`, `launch_type`, and pre/post-exec commands. - - `exdata` carries the environment (`sub_env_vars`) and the worker cgroup - limits parsed by `parse_limits` (MemoryMax, CPUQuota, TasksMax, RuntimeMax, - ...). The apiserver never sends exdata (`backend/convert.go:273-315`). + - `exdata` is a JSON object carrying the environment (`sub_env_vars`) and + the worker cgroup limits parsed by `parse_limits` (MemoryMax, CPUQuota, + TasksMax, RuntimeMax, ...). Since bug-1936 the apiserver sends it + (`backend/convert_pure.go:388`, capped at 16 KB by `maxExdataLen`); the + controller stores it opaquely in `start_info` (WAL submit fact, + snapshot) and the worker parses it, ignoring unknown top-level keys. + Remote tasks get the same exdata (`worker/src/controller_connector.c:764`). - The resource requirement is `resreq` on the submit data. - Remote tasks use `xce_taskhost_t` (:696-705). - **Missing today:** @@ -166,17 +171,18 @@ fits XCE's LSF-style job model. 2. **Launch type.** Serial, `openmpi` or `mpich`. 3. **Environment and pre/post-exec.** Environment variables the application needs, and the outer pre/post-exec commands. - 4. **Essential resource requirement and limits.** The resreq terms the - application cannot run without (select / rusage / `span[...]` / same / - order), its task range (min / max), and per-job execution limits as + 4. **Resource requirement and limits.** A structured resreq: the binding + terms the application cannot run without (`select`, and `span` / + `same` / `order` when set) and default reservations (`rusage` + entries), its task range (min / max), and per-job execution limits as maximums (memory, runtime, CPU, tasks), mapped onto the existing worker cgroup limits. 5. **Job starter** (decided 2026-10-12): a wrapper the worker puts in front of the job command (e.g. `module load openfoam &&`, `apptainer exec img`), frozen on the dispatch record. Design decides whether it wraps the MPI launcher or each task. - All of these are essentials under the additive rule; none is a - convenience default. + All of these are essentials under the additive rule, except the `rusage` + entries, which are default reservations the job may override. - **A profile attribute ships with its job feature** (user, 2026-10-12): a profile attribute is added by the spec that builds the job-level feature it configures, never ahead of it, so no profile setting is dead config. @@ -193,10 +199,11 @@ fits XCE's LSF-style job model. silently drops them (bug-1970). - **Users pick a profile at submit** (REST, `xcecli jobs submit --app `, harness, libxce). A **cluster default profile** applies when none is given. -- **Jobs add, never override.** A job may add to its profile's values (more - `select` terms, other resources, its own env variables, its own I/O - directories within the allowed prefixes, lower limits) but not change or - remove any. Anything else is rejected with a clear error. +- **Jobs add, never override** (except `rusage` defaults). A job may add to + its profile's values (more `select` terms, other resources, its own env + variables, lower limits) and override a profile `rusage` default, but not + change or remove anything else. Anything else is rejected with a clear + error. - **Access by RBAC permission.** No new access mechanism; this follows the existing named-instance pattern (user decision, 2026-09-29: fold profile RBAC into this spec). Today, `queue:submit` scoped to `type: queue`, @@ -275,10 +282,22 @@ fits XCE's LSF-style job model. (middleware chain unchanged; a `RequirePermission`-style check) and checks the additive rule early (`preCheckJobSubmitRequest`). - **The job's own values stay in its existing fields** (`launch_type`, - pre/post exec, exdata env and limits, resreq). New job fields are added only - for the profile name and the job's own I/O directories. All wire changes - are additive Cap'n Proto fields, and the WAL submit fact and snapshot carry - them with field completeness. Execution-side values (mode and options, + pre/post exec, exdata env and limits, resreq). New fields (all additive + Cap'n Proto; WAL submit/dispatch facts and snapshot carry them with field + completeness): + - on the submit data: the profile name; + - on `start_info` (what the worker gets): the job starter and the + profile's env, kept **separate** from the job's env (see "Environment of + the outer layer"); the queue-named `qPreExec` / `qPostExec` carry the + profile's outer pre/post-exec (renamed in design); + - the dispatch record payload ("Dispatch record"). +- **exdata is read by the merge** (owned by `job-config-merge`): the job's + env and limits live inside the exdata JSON (`sub_env_vars`, `limits`), so + the shared `src/lib` merge parses it to check the additive env rule, the + reserved `XCE_` prefix and "limits: the job may lower", and at dispatch + the controller writes the effective limits back into the exdata sent to + the worker, preserving every other key (including `meta`, used by + presets). Execution-side values (mode and options, launch type, env, profile pre/post-exec) are read from the profile at dispatch, as queue pre/post-exec is today (`get_queue_exec()`, `controller/queue.c:199`). @@ -294,7 +313,9 @@ fits XCE's LSF-style job model. - `src/configs/schema/applications.schema.json`, picked up by xceadmin's file-to-schema naming rule (`xcecommon/validation.py:68`); - `xcecommon/validation.py`: duplicate names rejected; cross-file - references checked (resreq resource names against `resources.yaml`); + references checked (`rusage` entry names against `resources.yaml`, + `per:` values); aliases (no clashes, no chains); `XCE_*` env rejected; + a warning for env names that look like secrets; - `src/schema/controller.capnp`: `applications: List(AppProfileConf)` in `ClusterConfig` and in `Conf4Sched` (the scheduler runs the live merge). `Conf4Worker` does not need it: the worker gets execution values through @@ -305,10 +326,17 @@ fits XCE's LSF-style job model. apiserver all need the same view (independence rule), and profiles change at the rate of queues, as occasional admin edits (frequency rule). - **The file is optional.** Without it there is one built-in profile - `default` (mode `direct`, no attributes), which is today's behaviour. -- **No override markers.** Every value in a profile is an essential and - binds the job (additive rule). Whether the job's own pre/post-exec is - allowed is a profile setting. + `default` (mode `direct`, no attributes), which is today's behaviour. A + profile named `default` in the file **replaces** the built-in one, so + admins can customise the default without setting `default_application`. +- **No secrets in profiles.** Profile env values and pre/post-exec paths + are visible to every `app:view` holder (profile listing, job details); + `xceadmin` warns on env names that look like secrets (`*TOKEN*`, + `*SECRET*`, `*PASSWORD*`, `*_KEY`), as for presets. +- **No override markers.** Every profile value binds the job (additive + rule), except the `rusage` entries' defaults, which the job may override. + Whether the job's own pre/post-exec is allowed is a profile setting + (`job_pre_post_exec: allow | forbid`). - **Open to new sections.** Later specs add top-level sections to a profile (`service:`, I/O claims, relationship templates, `container:`); the schema must allow that without reshaping existing sections. @@ -328,10 +356,13 @@ fits XCE's LSF-style job model. tasks: { min: 32, max: 512 } limits: { runtime: 24h, memory: 128G } # maximums env: { FOAM_INST_DIR: /opt/openfoam } + job_starter: "module load openfoam &&" pre_exec: /opt/xce/app/openfoam/pre.sh post_exec: /opt/xce/app/openfoam/post.sh job_pre_post_exec: allow + # mode: sandbox and its options come with job-sandbox-execution; + # this spec accepts only mode: direct - name: untrusted-python mode: sandbox sandbox: { inputs: ["/shared/in/"], outputs: ["/shared/out/"] } @@ -354,7 +385,8 @@ How each profile attribute combines with the job's own value (owned by | resreq `span` / `same` / `order` | only if essential (e.g. a code that runs on one host only) | set them if the profile does not; otherwise rejected | | task range | `min` / `max` | choose within it; no `-n` means `min` | | env | variables the application needs | add others; not redefine these | -| launch type, mode, sandbox options | fixed values | nothing | +| launch type, mode, sandbox options, job starter | fixed values | nothing | +| `job_pre_post_exec` | `allow` / `forbid` | use its own pre/post-exec only if `allow` | | limits | maximums | ask for less (checked like a ceiling) | | pre/post-exec | outer layer, always runs | add its own inner layer if the profile allows | @@ -409,9 +441,9 @@ resreq: override; a profile `rusage` change changes the content hash, so in-flight dispatches re-pend and pending jobs re-merge even if they override that resource (accepted). -- A job with no `-R` gets the profile's essentials only; the scheduler's - existing empty-resreq default (`type == any`) still applies to what - neither sets. +- A job with no `-R` gets the profile's binding terms and its `rusage` + defaults; the scheduler's existing empty-resreq default (`type == any`) + still applies to what neither sets. - **The `service:` section is configured in the profile** (user, 2026-10-12; defined with this spec, consumed by `service-supervision`, `service-replica-set`, `service-endpoint`) and follows the same rule: @@ -446,6 +478,14 @@ Decided 2026-10-12 (user), replacing the idea of an env denylist: `PATH` or `BASH_ENV` can subvert the admin's script. Worker change in this spec: the outer layer gets an explicitly built env (`uv_proc_pre_exec` takes none today). It also fixes today's queue pre/post-exec. +- **The profile env travels separately from the job env** (decided + 2026-10-12, review). Today all env is one `exdata.sub_env_vars` object, + also sent to remote tasks; if the profile env were merged into it, the + worker could not build the clean environment of A. The profile env goes + in its own `start_info` field (or an `exdata` key, decided in design), to + the head and every remote task; the job's env stays in `sub_env_vars`. + The job process gets both (profile first, the job's additions on top, + never redefining a profile variable). - **B. The `XCE_` prefix is reserved**: neither a job nor a profile may set `XCE_*` variables (rejected at submit by apiserver and controller, and by `xceadmin` for profiles). @@ -457,16 +497,18 @@ Decided 2026-10-12 (user), replacing the idea of an env denylist: exposure stays with `job-sandbox-execution`. ### Presets (client side, `xcecli-presets`) -Not part of this spec, recorded here because they replace profile defaults: +Not part of this spec, recorded here because they hold the convenience +defaults a profile does not (everything but default reservations): - A preset is a set of job options (`--app`, `--queue`, `-R`, `-n`, env, limits). The client expands it; a command-line flag replaces the preset's value for that option as a whole. The submit carries only the expanded values. - Personal presets live one file per preset under `~/.config/xce/presets/`. - The apiserver **only shares**: admin-deployed preset files served - read-only (`GET /xce/v1/presets`, `GET /xce/v1/presets/{name}`) from `/config/presets/`. Pulling - copies a preset into the local file, so a server preset being removed - never affects users. `preset diff` / `pull --force` are user-initiated. + read-only (`GET /xce/v1/presets`, `GET /xce/v1/presets/{name}` or + `{ns}/{name}`) from `/config/presets/`. Pulling copies a preset + into its own local file, so a server preset being removed never affects + users. `preset diff` / `pull --force` are user-initiated. - User-to-user sharing by `preset export` / `import` (file, mail, git). - Presets carry no privilege: `app:use`, `queue:submit`, the profile's essentials, queue ceilings and project quotas still apply at submit. @@ -481,9 +523,11 @@ restarted, and the scheduler no longer re-merges it. `order`) and task count, so a running job's reservations (e.g. `rusage[license=1]`) do not change when the profile changes, and are rebuilt identically after a scheduler failover or restart; - - execution: mode, mode options, launch type, env, outer pre/post-exec, - limits; - - the profile content hash the record was built from. + - execution: mode, mode options, launch type, job starter, the + profile's env and the job's env (separately), outer pre/post-exec, the + `job_pre_post_exec` setting, effective limits; + - the resolved profile name (after aliases) and the content hash the + record was built from. - At each dispatch the controller puts the execution values into the job's `start_info` for the worker, and the whole record into an additive payload on the WAL dispatch fact @@ -501,6 +545,10 @@ restarted, and the scheduler no longer re-merges it. - Running jobs therefore need nothing from the live profile: post-exec at the end, later remote tasks or a resize, and a service replica's probes and in-place restarts all use the record. +- Why: for sandboxed jobs the mode and mounts a job actually ran with is a + security/audit fact, and `job-sandbox-execution`'s mandatory cleanup + depends on it; `persistence.md` does not allow memory-only state without a + documented scope decision. ### Profile removed while jobs use it Decided 2026-10-12 (user): **a job whose profile no longer exists pends**, @@ -549,7 +597,7 @@ with the reason "application profile X is not defined". - The dispatch record stores it; "profile changed since dispatch" in job details is a hash comparison; job details show it short (`openfoam@3f2a9c`). Admins never set it. - - Base for the open "profile versioning and audit" item. + - Base for profile history and audit (`config-history-audit`, backlog). - **Moving pending jobs to another profile is not in this spec** (decided 2026-10-12, user). XCE has no job modification today (`JOB_MODIFY` is only an enum, `include/xce/types.h:620`; the apiserver's @@ -577,10 +625,6 @@ Decided 2026-10-12 (user), so a rename never strands jobs: - Rename workflow: rename the profile and add the old name as an alias; drop the alias when no pending job uses it (the upgrade warning shows when that is). -- Why: for sandboxed jobs the mode and mounts a job actually ran with is a - security/audit fact, and `job-sandbox-execution`'s mandatory cleanup - depends on it; `persistence.md` does not allow memory-only state without a - documented scope decision. ### Default profile - **Set by `default_application: ` in `xce.yaml`**, a cluster @@ -589,6 +633,16 @@ Decided 2026-10-12 (user), so a rename never strands jobs: - **Recorded at submit.** A job submitted without `--app` gets the current default's *name* written on its record, so changing the default later does not move pending jobs to another profile. +- **An empty profile name means "no profile"**, never "the current + default": jobs submitted before profiles existed (old WAL records and + snapshots) keep today's behaviour and are never silently given + `default_application`. Only new submits record the default's name. +- **Restart carries the profile name.** `POST /:jobids/restart` rebuilds the + new job field by field from the old job's record (bug-1968 shows what is + lost otherwise), so it must copy the profile name (or alias); it then + re-checks `app:use` and re-reads the profile like any new dispatch. +- **Array jobs** carry one profile name on the array's submit data; every + element uses it. - **A name that is not defined is a configuration error, never a silent fallback** (user decision, 2026-10-12): @@ -623,7 +677,14 @@ Decided 2026-10-12 (user), so a rename never strands jobs: - RBAC permission for profile use. - Submit surfaces: REST, `xcecli --app`, harness, libxce. - The profile name and the job's own values persisted on the job. - - The `mode:` schema field (only `direct`). + - The `mode:` schema field (only `direct`), the structured resreq schema + (`select`, `rusage` entries with `per:` / `default`, `span`, `same`, + `order`) and the job starter. + - New `start_info` fields (job starter, profile env separate from the job + env) and exdata handling with `job-config-merge`. + - Profile content hash (shared `src/lib` function) and its field on the + scheduler's dispatch decision. + - Restart and array submits carrying the profile name. - A worker launch error when a `direct` exec user is missing. - Dispatch record (scheduling and execution values, content hash) and the retryable dispatch failure for a missing or changed profile. @@ -635,7 +696,14 @@ Decided 2026-10-12 (user), so a rename never strands jobs: - Profile `--dry-run` support: the effective values a job would get (shared with `job-config-merge`). - Tests. -- **Open for requirements**: +- **Open for requirements / design**: + - Multi-host jobs: does the profile's outer pre/post-exec run once (first + host) or on every host (today's queue pre-exec runs in `xcecd_worker` + on each host it starts)? Does the job starter wrap the MPI launcher or + each task? Remote tasks get the profile env and the job env separately. + - Confirm a `rusage` value of `0` means "reserve none" in the scheduler. + - Confirm a dispatch declined for a missing or changed profile does not + count as a run or against requeue limits. - ~~How resource defaults combine with a job's own resreq~~ — settled 2026-10-09, owned by `job-config-merge`. - Settled 2026-10-12: scheduling and execution values frozen at @@ -650,7 +718,10 @@ Decided 2026-10-12 (user), so a rename never strands jobs: change audit deferred to the roadmap (`config-history-audit`): the content hash and the dispatch record already show what each run used; no job labels needed here (presets record themselves in `exdata.meta`; - general labels are the `job-labels` backlog item). + general labels are the `job-labels` backlog item); compound and + alternative resreqs (only the profile `select` applies); structured + resreq with `per:` default `task`; separate profile env; empty profile + name = no profile; a file profile named `default` replaces the built-in. - **Out**: - The `sandbox` mode and its options, I/O directories, capability advertisement, placement by mode and the worker sandbox mechanics @@ -683,6 +754,9 @@ Decided 2026-10-12 (user), so a rename never strands jobs: - The config pipeline (`clusterconfig-schema-split`, `config-upgrade-snapshot`, `admin-config-tool`). - RBAC (`rbac_check.go`). + - The scheduler-to-controller dispatch decision (gains the profile + content hash; `scheduler-dispatch-commit`). + - The restart path (`RestartJobByID`, bug-1968). - **Downstream**: - `job-sandbox-execution` adds the `sandbox` mode, its options and I/O directories, capability advertisement and placement, on top of this @@ -690,6 +764,10 @@ Decided 2026-10-12 (user), so a rename never strands jobs: - `job-sandbox-multinode`. - Future `container`/`vm` modes, which add profile attributes. - `job-status-detail-display`, which can show the profile and mode. + - `job-modify` (moves pending jobs between profiles), `job-requeue` + (requeue/success exit values as profile attributes), + `config-history-audit` (profile history), `xcecli-presets` (reads + profiles for warnings; `exdata.meta`). ## Existing Spec Touchpoints - **Extends**: none directly. @@ -707,11 +785,15 @@ Decided 2026-10-12 (user), so a rename never strands jobs: - Middleware chain unchanged: WhitelistURL → WhitelistIP → RateLimit → JWT → CORS → RBAC. - All wire changes are additive Cap'n Proto. Old WAL records and snapshots - decode as "no profile, mode `direct`", which is today's behaviour. -- Save/restore field completeness (`persistence.md`): the profile name and - the job's own values (including I/O directories) round trip through the - submit facts and snapshots. Run - `/xce-audit-invariants`. + decode as "no profile, mode `direct`", which is today's behaviour; an + empty profile name is never resolved to `default_application`. +- Save/restore field completeness (`persistence.md`): the profile name, the + job's own values, the new `start_info` fields and the dispatch record + round trip through the WAL submit and dispatch facts and the snapshot + (`/xce-audit-invariants controller`). The scheduler's own state for + dispatched jobs (frozen, no longer re-merged) and its `Conf4Sched` + profiles round trip through the scheduler snapshot and replay recordings + (`/xce-audit-invariants scheduler`). - Input validation (`security.md`): the job's additions and paths are validated server-side. No path traversal, and mount prefixes are restricted by the profile. diff --git a/.kiro/specs/job-config-merge/brief.md b/.kiro/specs/job-config-merge/brief.md index a0b809d8..18a62687 100644 --- a/.kiro/specs/job-config-merge/brief.md +++ b/.kiro/specs/job-config-merge/brief.md @@ -107,7 +107,14 @@ values and the three names must round-trip). config change; queue hard/task limit wiring (YAML, schema, `mapping.py`, capnp, C); queue `res_req` and pre/post-exec deprecation; outer pre/post-exec sourced from the profile; effective values in job details and - the decision trace; tests. + the decision trace; **exdata handling** (added 2026-10-12, + application-profile review): the job's env and limits live inside the + exdata JSON (`sub_env_vars`, `limits`), which the controller treats as + opaque today, so the merge library parses it to check the additive env + rule, the reserved `XCE_` prefix and "limits: the job may lower", and at + dispatch writes the effective limits back into the exdata sent to the + worker, preserving every other key (including `meta`, used by + `xcecli-presets`); tests. - **Out**: the profile model itself (`application-profile`); project definitions (`project-config`); `span` placement semantics (`parjob-placement`); LSF compound/alternative merge stages; fairshare. diff --git a/.kiro/specs/xcecli-presets/brief.md b/.kiro/specs/xcecli-presets/brief.md index ca33b394..cab31f39 100644 --- a/.kiro/specs/xcecli-presets/brief.md +++ b/.kiro/specs/xcecli-presets/brief.md @@ -137,7 +137,8 @@ configuration level. queries return it as `extend_data` (`apiserver/backend/convert_pure.go` :388, :537); 16 KB cap (`maxExdataLen`, :395). - `meta` is client-only metadata: apiserver and controller pass it - through uninterpreted. The worker's `parse_exdata()` + through uninterpreted (the controller's dispatch-time rewrite of exdata + limits, `job-config-merge`, preserves it). The worker's `parse_exdata()` (`worker/src/xcecd_exdata.c`) ignores unknown top-level keys but logs a NOTICE for each; this spec makes `meta` a known, silently skipped key (one-line worker change). diff --git a/.kiro/steering/job-config-layering.md b/.kiro/steering/job-config-layering.md index 3f98dc78..c44df0b4 100644 --- a/.kiro/steering/job-config-layering.md +++ b/.kiro/steering/job-config-layering.md @@ -20,7 +20,7 @@ XCE instead gives **every attribute exactly one owning level**. | Level | Owns | Never owns | |---|---|---| | **Queue**: scheduling policy | priority and ordering, host partition, dispatch/run windows, preemption policy, policy ceilings (max run time, max tasks, max hosts, ...) | resource requirement, anything about how a job executes (pre/post-exec, env, launch, limits enforced on the job) | -| **Application profile**: what the application needs and how it runs (essentials plus default reservations, 2026-10-12) | resreq (select / rusage / span / same / order), task range (min / default / max), execution mode and options, launch type, env, **pre/post-exec**, execution limits (mem, runtime, CPU, tasks) | priority, queue choice, quotas | +| **Application profile**: what the application needs and how it runs (essentials plus default reservations, 2026-10-12) | resreq (select / rusage / span / same / order), task range (min / max), execution mode and options, launch type, env, **pre/post-exec**, execution limits (mem, runtime, CPU, tasks) | priority, queue choice, quotas | | **Project**: resource ownership | membership, quotas / limits, later guarantees, accounting | how a job runs or is ordered | | **Job** | its own values, only for attributes its profile allows | n/a |