diff --git a/.kiro/specs/application-profile/brief.md b/.kiro/specs/application-profile/brief.md index e25a553a..83714549 100644 --- a/.kiro/specs/application-profile/brief.md +++ b/.kiro/specs/application-profile/brief.md @@ -98,6 +98,12 @@ consumed by `service-supervision` (worker probes and restarts), essential); nothing tunes itself. Backlog: `profile-usage-insight`. - Job details show the effective resreq with each term's origin (profile or job). + - Deferred: a submit-time hook run for the application (LSF `esub` + style) that inspects or adjusts each submission; roadmap backlog + `app-submit-hook` (it cannot run in the controller, would add latency + and a failure mode to every apiserver submit, and bypasses the merge + rules; `params`, annotations, the job starter and pre-exec cover most + needs). - Deferred: a privileged `app:override` for one-off exceptions (until `job-modify` exists, an admin kills and resubmits, or uses a variant profile; recurring variants are presets), and profile inheritance @@ -181,8 +187,49 @@ fits XCE's LSF-style job model. front of the job command (e.g. `module load openfoam &&`, `apptainer exec img`), frozen on the dispatch record. Design decides whether it wraps the MPI launcher or each task. + 6. **Application parameters (`params`)** (decided 2026-10-12): settings + XCE carries but never interprets, handled by the application, e.g. + + ```yaml + params: + - { name: solver, default: pimpleFoam, allowed: [pimpleFoam, simpleFoam] } + - { name: decompose, default: scotch } + - { name: case_dir, pattern: "^/shared/cases/.*" } # no default + ``` + - Each is delivered to the job as `XCE_APP_PARAM_`, with the + profile env (separate from the job env, frozen on the dispatch + record), so the job, the job starter and the outer pre/post-exec can + read it. + - The job overrides with `--app-param name=value` (also in presets); + the job's value wins, like `rusage` defaults. Only declared names are + accepted (a typo is rejected at submit). XCE checks only the shape + (`allowed` list or `pattern`), never the meaning. A parameter with no + default and no job value is not set. + - The job's values are stored in its exdata (`app_params`), already + persisted; no new job field. + - The portal can build an application's submit form from the declared + parameters. + 7. **Annotations** (decided 2026-10-12): free-form key/value facts about + the application (owner, version, docs URL, licence server name), like + Kubernetes annotations. Shown to tools (profile listing, portal, job + details) **and** exported to the job as `XCE_APP_ANN_`, so an + admin writes a value once for both. The job cannot change them. + - Name conversion: keys may be Kubernetes-style + (`cfd.org/solver-family`); the env name is uppercased with every + non-alphanumeric character replaced by `_` + (`XCE_APP_ANN_CFD_ORG_SOLVER_FAMILY`). `xceadmin` rejects two + annotation keys (or two params) that convert to the same name. + - Limits on count and size, so a profile cannot bloat every job's env. + - Visible to every `app:view` holder: no secrets (the secret-name + warning applies). All are exported in v1; an opt-out only if + needed. + - `env` vs annotations vs params: `env` is a variable the application + expects under its own name; annotations are facts for tools and the + job; params are the user's choices. `XCE_APP_*` names are set by XCE + only; the reserved `XCE_` prefix keeps jobs and profiles from setting + them directly, so a job cannot fake one. All of these are essentials under the additive rule, except the `rusage` - entries, which are default reservations the job may override. + entries and `params`, which are defaults the job may override. - **A profile attribute ships with its job feature** (user, 2026-10-12): a profile attribute is added by the spec that builds the job-level feature it configures, never ahead of it, so no profile setting is dead config. @@ -419,9 +466,20 @@ resreq: caller's per-type (`lib/resreq.c:842`) and the controller passes `PER_TASK` (`controller/job.c:3909`); `resources.yaml` has no per-type field. A per-resource default there may come later. -- `name=0` in the job means "reserve none" (the only way to drop a profile - default); requirements must confirm the scheduler treats a 0 reservation - that way. +- **A 0-valued term is removed by the merge** (decided 2026-10-12, user): + after combining profile and job, every `rusage` term whose value is 0 is + dropped, so the scheduler sees exactly "no term". `name=0` is how a job + drops a profile default. Needed because the scheduler does not treat 0 as + absent today: a 0 term still becomes a wanted entry + (`calc_wanted_host_resvals()`, `lib/resreq.c`), and `unsatisfy_()` + (`scheduler/src/resource.c:164`, `total < inuse + uncomit + wanted`) + rejects a host where that resource is already over-committed + (`inuse > total`, possible because memory is charged + `max(reserved, actual)`), and the term shows in decision-trace rows and + pend reasons; with no term the resource is not checked at all (fast path + at `resource.c:351`). Applies to every 0 term (profile, job, preset, or a + job without a profile). Job details still show "license: 0 (overrides the + profile default)". - `xceadmin` validates each `rusage` entry against `resources.yaml` (name exists) instead of parsing a string; later per-resource fields (`duration`, a `min:` floor for counted resources such as licences, if @@ -679,7 +737,8 @@ Decided 2026-10-12 (user), so a rename never strands jobs: - The profile name and the job's own values persisted on the job. - The `mode:` schema field (only `direct`), the structured resreq schema (`select`, `rusage` entries with `per:` / `default`, `span`, `same`, - `order`) and the job starter. + `order`), the job starter, `params` (`--app-param`) and annotations, + exported as `XCE_APP_PARAM_*` / `XCE_APP_ANN_*`. - New `start_info` fields (job starter, profile env separate from the job env) and exdata handling with `job-config-merge`. - Profile content hash (shared `src/lib` function) and its field on the @@ -701,7 +760,6 @@ Decided 2026-10-12 (user), so a rename never strands jobs: host) or on every host (today's queue pre-exec runs in `xcecd_worker` on each host it starts)? Does the job starter wrap the MPI launcher or each task? Remote tasks get the profile env and the job env separately. - - Confirm a `rusage` value of `0` means "reserve none" in the scheduler. - Confirm a dispatch declined for a missing or changed profile does not count as a run or against requeue limits. - ~~How resource defaults combine with a job's own resreq~~ — settled @@ -721,7 +779,9 @@ Decided 2026-10-12 (user), so a rename never strands jobs: general labels are the `job-labels` backlog item); compound and alternative resreqs (only the profile `select` applies); structured resreq with `per:` default `task`; separate profile env; empty profile - name = no profile; a file profile named `default` replaces the built-in. + name = no profile; a file profile named `default` replaces the built-in; + 0-valued `rusage` terms removed by the merge; `params` and annotations + (exported to env) in v1; submit hook deferred (`app-submit-hook`). - **Out**: - The `sandbox` mode and its options, I/O directories, capability advertisement, placement by mode and the worker sandbox mechanics diff --git a/.kiro/specs/job-config-merge/brief.md b/.kiro/specs/job-config-merge/brief.md index 18a62687..490f19f1 100644 --- a/.kiro/specs/job-config-merge/brief.md +++ b/.kiro/specs/job-config-merge/brief.md @@ -55,7 +55,9 @@ apiserver and display would disagree about what a job actually asked for. profile value binds, except `rusage`: union of profile and job resources, the job's term wins whole for a resource in both, the profile entry's `per:` (optional, default `task`) fixes the unit (an explicitly - different per-type is rejected), `0` means none; profile resreq is + different per-type is rejected), 0-valued terms are dropped after the + merge so the scheduler sees no term; `params` declared by the profile, + the job's value wins, undeclared names rejected; profile resreq is structured (`select`, `rusage` entries, `span`, `same`, `order`), see the at-a-glance table in `job-config-layering.md`; otherwise the job may add (more `select` terms, `span`/`same`/`order` the profile does not set, env variables, lower limits, an inner pre/post-exec if allowed) diff --git a/.kiro/specs/job-config-merge/requirements.md b/.kiro/specs/job-config-merge/requirements.md index 72f78efc..f585874d 100644 --- a/.kiro/specs/job-config-merge/requirements.md +++ b/.kiro/specs/job-config-merge/requirements.md @@ -14,7 +14,7 @@ Discovery brief: `.kiro/specs/job-config-merge/brief.md`. Governing rules: `.kir **What should change:** - One merge function in `src/lib`, shared by controller, scheduler and (through cgo) apiserver. Its inputs are the job's own values, the application profile, the queue's policy and the project; its output is the job's effective values: resreq (select / rusage / span / same / order), task range, execution limits, pre/post-exec, env. - **One owner per attribute.** The queue owns scheduling policy, the application profile owns what the application needs and how it runs, and the project owns resource ownership. Only two interactions exist: - - **Addition** (2026-10-12, replacing *default*/*fixed* markers): a profile holds only essentials and every profile value binds, except `rusage`: the union of profile and job resources, the job's term winning whole for a resource in both, with the unit from the profile entry's optional `per:` (default `task`; an explicitly different per-type is rejected) and `0` meaning none. The profile resreq is structured (`select`, `rusage` entries with `per:`/`default`, `span`, `same`, `order`); rules at a glance: `job-config-layering.md`. Otherwise the job may add to them (more `select` terms, `span`/`same`/`order` the profile does not set, env variables, lower limits) but never change or remove them; a conflict is rejected at submit. Convenience defaults are client-side presets and never reach the merge. + - **Addition** (2026-10-12, replacing *default*/*fixed* markers): a profile holds only essentials and every profile value binds, except `rusage`: the union of profile and job resources, the job's term winning whole for a resource in both, with the unit from the profile entry's optional `per:` (default `task`; an explicitly different per-type is rejected) and 0-valued terms dropped after the merge (the scheduler then sees no term; today a 0 term still keeps a job off hosts where the resource is over-committed). Profile `params`: declared names only, the job's value wins. The profile resreq is structured (`select`, `rusage` entries with `per:`/`default`, `span`, `same`, `order`); rules at a glance: `job-config-layering.md`. Otherwise the job may add to them (more `select` terms, `span`/`same`/`order` the profile does not set, env variables, lower limits) but never change or remove them; a conflict is rejected at submit. Convenience defaults are client-side presets and never reach the merge. - **Ceiling:** queue ceilings and project quotas never change a value. A value that can never be satisfied is rejected at submit; a quota that is only full right now makes the job pend with a reason naming the queue or project. - **Live merge.** The WAL keeps only the job's own values plus its queue, profile and project names. - The controller (authoritative) and the apiserver (early) run the merge at submit to validate. diff --git a/.kiro/specs/xcecli-presets/brief.md b/.kiro/specs/xcecli-presets/brief.md index cab31f39..ba0f52b4 100644 --- a/.kiro/specs/xcecli-presets/brief.md +++ b/.kiro/specs/xcecli-presets/brief.md @@ -15,7 +15,7 @@ configuration level. ## Desired Outcome - **A preset is a set of job options** (`--app`, `--queue`, `-R`, `-n`, - env, limits, ...), named and stored by the client. `xcecli jobs submit + env, limits, `--app-param`, ...), named and stored by the client. `xcecli jobs submit --preset ` expands it; any command-line flag replaces the preset's value for that option as a whole (no resreq parsing). The submit carries only the expanded values. diff --git a/.kiro/steering/job-config-layering.md b/.kiro/steering/job-config-layering.md index c44df0b4..dfbca94f 100644 --- a/.kiro/steering/job-config-layering.md +++ b/.kiro/steering/job-config-layering.md @@ -34,11 +34,13 @@ One line per part; details below. | Part | Rule | |---|---| | `select` | profile and job ANDed | -| `rusage` | union; the job's term wins for a resource in both; the unit (`per:`, default `task`) is the profile's and must match | +| `rusage` | union; the job's term wins for a resource in both; the unit (`per:`, default `task`) is the profile's and must match; 0-valued terms are dropped | | `span` / `same` / `order` | the profile's if set, otherwise the job's | | task range | the job chooses within the profile's `min` / `max` | | limits | the profile's are maximums; the job may lower | | env | the job adds; cannot redefine the profile's | +| `params` | declared by the profile; the job's value wins (`--app-param`); undeclared names rejected; exported as `XCE_APP_PARAM_*` | +| annotations | the profile's; exported as `XCE_APP_ANN_*`; the job cannot change them | | mode, launch type, job starter, outer pre/post-exec | the profile's | | compound / alternative job resreq | only the profile's `select` applies; a profile `rusage` default or `span`/`same`/`order` → rejected (until PR-15) | | queue ceilings, project quotas | check the merged result; never change it | @@ -94,9 +96,10 @@ One line per part; details below. caller's per-type (`lib/resreq.c:842`) and the controller passes `PER_TASK` (`controller/job.c:3909`); `resources.yaml` has no per-type field. A per-resource default there may come later. - - `name=0` in the job means "reserve none" (the only way to drop a profile - default); requirements must confirm the scheduler treats a 0 reservation - that way. + - A 0-valued term is removed by the merge (profile, job or preset), so + the scheduler sees exactly "no term"; `name=0` is how a job drops a + profile default. (Today a 0 term is still checked: it keeps the job + off hosts where the resource is over-committed.) - `xceadmin` validates each `rusage` entry against `resources.yaml` (name exists) instead of parsing a string; later per-resource fields (`duration`, a `min:` floor for counted resources such as licences, if diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index 70422f5a..288bea16 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -125,6 +125,7 @@ Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generi - [ ] job-modify — NEW 2026-10-12 (from application-profile discussion, user decision; backlog, no brief yet). Modify a pending job (LSF `bmod`): apiserver action (today `"modify"` is commented out, `controllers/jobs.go:2446`; `JOB_MODIFY` only an enum), controller opcode, WAL fact with snapshot field-completeness, scheduler re-merge, xcecli. First attributes: `--app` (owner with `app:use` on the new profile; admins in bulk, e.g. `--app-from `), `-R`, `-n`, queue; later begin/end time (`job-time-window`), dependencies. Dependencies: application-profile, job-config-merge. - [ ] config-history-audit — NEW 2026-10-12 (from application-profile discussion, user decision: needed, roadmap). Keep past cluster-config versions (including `applications.yaml` profiles) with who changed what and when: `xceadmin config history` / `diff`, look up a profile by its content hash (the application-profile hash recorded on dispatch records), audit trail of live upgrades. Per-job display of the values a run used is already covered by the application-profile dispatch record; this adds admin history and audit. Covers the whole cluster config, not only profiles (the controller already tracks `configVersion`/`configChecksum`). Dependencies: application-profile, admin-config-tool. - [ ] job-labels — NEW 2026-10-12 (from application-profile discussion; backlog, no brief yet). General Kubernetes-style job labels: `--label key=value` (repeatable, limited charset and size), an additive submit-data field through WAL and snapshot (field completeness), server-side filtering (`jobs list -l k=v`) and summaries grouped by label (runtime, peak memory, efficiency); metadata only, never used for placement. Reuses or replaces the REST `tags` field that is silently dropped today (bug-1970). Not needed for presets (those use `exdata.meta`); candidates: team-wide comparison, services labelling replicas. Dependencies: none. +- [ ] app-submit-hook — NEW 2026-10-12 (from application-profile discussion, user decision: backlog only). A profile-named program that inspects or adjusts each submission (LSF `esub` style). Cannot run in the controller (no-fork, non-blocking rules); in the apiserver it adds latency and a failure mode to every submit and bypasses the merge rules, so decide client (xcecli) vs apiserver first. Application-profile `params`, annotations, the job starter and pre-exec cover most needs meanwhile. Dependencies: application-profile. - [ ] job-requeue — NEW 2026-10-12 (from application-profile discussion, user decision; backlog, no brief yet). Automatic requeue on given exit values (LSF `REQUEUE_EXIT_VALUES`) and success exit values (`SUCCESS_EXIT_VALUES`), with a requeue loop limit; wires the commented-out `requeue` job action (`apiserver/controllers/jobs.go`) and the REST `requeue_error_values` / `rerunable` options that are silently dropped today (bug-1970). Adds the matching application-profile attributes (a profile attribute ships with its job feature). Later, same principle: checkpoint settings and warning-before-kill each get their own spec. Dependencies: application-profile. - [ ] xcecli-presets — NEW 2026-10-12 (from application-profile discussion, user decisions). Client-side presets replace convenience defaults in profiles: a preset is a set of job options (`--app`, `--queue`, `-R`, `-n`, env, limits) expanded by the client, a flag replacing the preset's value for that option; one file per preset (`~/.config/xce/presets//.yaml`, same layout in the apiserver's presets dir), env capture by name (`pass:`; `XCE_*` and secret-looking names excluded); the apiserver only shares admin-deployed presets read-only (`GET /xce/v1/presets[/{ns}/{name}]`, files in `/config/presets/` or `presets.dir`, re-read on change; not in ClusterConfig), and `preset pull` copies one locally so server-side removal never affects users; on a name collision pull refuses unless it is an update of the same server preset (`--as` / `--force` with `.bak`); submit uses local presets only (`preset diff`, `pull --force` user-initiated); user-to-user sharing by `preset export`/`import`; presets carry no privilege and are not a configuration level; no preset field on the server: the client records preset name + hash in the job's `exdata.meta` (passed through uninterpreted; worker skips `meta`), and `preset report` filters jobs on it client-side; the portal does the same expansion. Brief: `.kiro/specs/xcecli-presets/brief.md`. Dependencies: application-profile. - [ ] profile-usage-insight — NEW 2026-10-12 (from application-profile discussion, backlog; no brief yet). Per-profile report over recent jobs comparing the profile's defaults with actual use (requested vs. peak memory, CPU efficiency, OOM kills, pend time, runtime by variant) and suggesting better values for site presets (or a profile essential), e.g. "preset openfoam rusage mem 4000, p95 peak 1600 → suggest 2000" (Kubernetes VPA recommender precedent). **Advisory only** (user decision): the admin applies changes to site presets or `applications.yaml`; nothing tunes itself. Optional user hint ("used 1.4G of 4G requested"). Dependencies: application-profile, run usage in job records.