lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(spec): application-profile brief: own config file, default_application, live merge cleanup

Browse source 
- profiles in src/configs/applications.yaml (+ schema, ClusterConfig and
  Conf4Sched), optional file with a built-in 'default' profile
- default_application in xce.yaml; an undefined name is a config error at
  xceadmin build, live upgrade and controller start, never a fallback
- remove the superseded submit-time resolution text: the job keeps its
  profile name and own values, effective values come from job-config-merge
- this spec owns the profile attribute schema and override markers
- record the 2026-10-12 open questions with leanings

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-12 09:11:06 +00:00
parent 19c9f616fb
commit 84501d30da
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
2 changed files with 161 additions and 31 deletions
Download patch file Download diff file Expand all files Collapse all files

190
.kiro/specs/application-profile/brief.md
View file

@ -25,7 +25,8 @@ decisions):**
or forbidden by the profile. Order: profile pre → job pre → job → job post
→ profile post (the worker's existing two-layer structure,
`xcecd_worker.c` `queue_pre_exec()`/`queue_post_exec()`).
- **Replaces "Resolution at submit time"** below: the merge is *live*. The WAL
- **The merge is *live*** (replaces the earlier "resolution at submit time",
removed from this brief 2026-10-12). The WAL
keeps the job's own values plus the profile name; the controller and
apiserver validate at submit, and the scheduler computes effective values
and recomputes them when a profile changes (a pending job that no longer
@ -43,6 +44,23 @@ consumed by `service-supervision` (worker probes and restarts),
`service-replica-set` (replicas, scaling, rolling update) and
`service-endpoint` (ports, endpoints).
**Update 2026-10-12 (discussion; user decisions):**
- **Own configuration file.** Profiles live in a new
`src/configs/applications.yaml` with its own JSON schema, mapped into
`ClusterConfig` and `Conf4Sched` like queues and limits. See "Profile
configuration" below.
- **Cluster default** is set by `default_application` in `xce.yaml`. A name
that is not defined is a configuration error everywhere (`xceadmin`, live
upgrade, controller start), never a silent fallback. See "Default profile"
below.
- **This spec owns the profile's attribute schema**: which attributes exist
and how each is marked *default* or *fixed*. `job-config-merge` consumes
that schema and owns only the merge and live re-evaluation.
- **Later sections are added by their consumers**: `service:`
(`service-supervision`), I/O claims (resource-model PR-6), relationship
and compound templates (PR-14/15), `container:` (R6). This spec only makes
the schema open to new top-level sections.
## Problem
XCE users run many kinds of applications: trusted in-house codes, untrusted
scripts, vendor binaries, MPI solvers. Each needs a different way of running:
@ -103,9 +121,10 @@ fits XCE's LSF-style job model.
2. **Launch type.** Serial, `openmpi` or `mpich`.
3. **Environment and pre/post-exec.** Default environment variables and
pre/post-exec commands.
4. **Resource defaults and limits.** A default resource requirement, plus
per-job execution limits (memory, runtime, CPU, tasks), mapped onto the
existing worker cgroup limits.
4. **Resource requirement and limits.** The job's resource requirement
(select / rusage / `span[...]` / same / order), its task range (min /
default / max), and per-job execution limits (memory, runtime, CPU,
tasks), mapped onto the existing worker cgroup limits.
- **Users pick a profile at submit** (REST, `xcecli jobs submit --app <name>`,
harness, libxce). A **cluster default profile** applies when none is given.
- **Limited overrides.** A job may override only the attributes its profile
@ -147,10 +166,11 @@ fits XCE's LSF-style job model.
default resource requirement and per-job execution limits). They are not
scheduling policy. Queue-level scheduling limits keep applying on top,
unchanged.
- **Resolution at submit time.** The effective values are stored on the job
(mode, options, launch type, env, limits, resreq). Editing a profile later does
not change queued or running jobs. The job records its profile name, and its
details show the effective mode.
- **Live merge** (`job-config-merge`). The job records its profile name and
its own values, not the effective values. The scheduler computes the
effective values and recomputes them when the profile changes: a pending job
that no longer satisfies its profile pends with a reason, and running jobs
are unaffected. Job details show the profile name and the effective mode.
- **The apiserver never checks node accounts** (user decision, 2026-09-29).
- Submit-time validation covers profile existence, RBAC permission, overrides
and paths only.
@ -165,12 +185,14 @@ fits XCE's LSF-style job model.
in every mode.
## Approach
- **Profile definitions** live in the cluster configuration: schema, YAML, JSON
schema, xceadmin mapping, and live-upgrade replication, following the
`config-design.md` placement rules.
- **The controller resolves the profile at submit** and merges it with the job's
permitted overrides. This makes it authoritative for every submit path (REST,
libxce, harness).
- **Profile definitions** live in their own cluster configuration file (see
"Profile configuration" below), with xceadmin validation and live-upgrade
replication.
- **The controller validates the profile at submit**: the named profile
exists, and the job overrides only attributes the profile marks *default*.
This makes it authoritative for every submit path (REST, libxce, harness).
It stores the profile name and the job's own values; the effective values
come from the live merge (`job-config-merge`).
- **The apiserver checks that the named profile exists** (user direction,
2026-09-29).
- At submit, before anything goes downstream (`preCheckJobSubmitRequest`), the
@ -179,29 +201,118 @@ fits XCE's LSF-style job model.
- The apiserver already receives the cluster configuration from the controller
(`backend/convert.go:265`), so it knows the current set of profiles and keeps
it current across live config upgrades.
- The controller still re-validates when it resolves the profile, because it is
- The controller still re-validates at submit, because it is
authoritative for non-REST submit paths and for a profile removed between the
two checks. If the profile has gone by then, the job is rejected, never run
with a fallback profile.
- **The apiserver additionally enforces the RBAC permission** for the profile
(middleware chain unchanged; a `RequirePermission`-style check) and validates
overrides early (`preCheckJobSubmitRequest`).
- **Effective values go into the job's existing fields where they exist**
(`launch_type`, pre/post exec, exdata env and limits, resreq). New fields are
added only for the profile name, execution mode, mode options and I/O
directories. All wire changes are additive Cap'n Proto fields, and the WAL
submit fact and snapshot carry them with field completeness.
- **The job's own values stay in its existing fields** (`launch_type`,
pre/post exec, exdata env and limits, resreq). New job fields are added only
for the profile name and the job's own I/O directories. All wire changes
are additive Cap'n Proto fields, and the WAL submit fact and snapshot carry
them with field completeness. Execution-side values (mode and options,
launch type, env, profile pre/post-exec) are read from the profile at
dispatch, as queue pre/post-exec is today (`get_queue_exec()`,
`controller/queue.c:199`).
- **Workers advertise their supported execution modes.** The scheduler filters
hosts by the job's effective mode. The design picks the mechanism, for example
a host attribute or boolean resource.
### Profile configuration
- **One file per entity type**, as for hosts, queues, resources, limits,
users and scaler:
- `src/configs/applications.yaml`, top-level key `applications:` (named
after LSF `lsb.applications`; the user flag is `--app`);
- `src/configs/schema/applications.schema.json`, picked up by xceadmin's
file-to-schema naming rule (`xcecommon/validation.py:68`);
- `xcecommon/validation.py`: duplicate names rejected; cross-file
references checked (resreq resource names against `resources.yaml`);
- `src/schema/controller.capnp`: `applications: List(AppProfileConf)` in
`ClusterConfig` and in `Conf4Sched` (the scheduler runs the live merge).
`Conf4Worker` does not need it: the worker gets execution values through
`start_info` at dispatch.
- The apiserver already receives `ClusterConfig`
(`backend/convert.go:265`), so it gets the profiles with no new channel.
- **Shared config, per `config-design.md`:** controller, scheduler and
apiserver all need the same view (independence rule), and profiles change
at the rate of queues, as occasional admin edits (frequency rule).
- **The file is optional.** Without it there is one built-in profile
`default` (mode `direct`, no attributes), which is today's behaviour.
- **Every attribute carries its override marker** (`default` or `fixed`,
`job-config-layering.md`). Profile pre/post-exec is always fixed; whether
the job's own pre/post-exec is allowed is a profile setting.
- **Open to new sections.** Later specs add top-level sections to a profile
(`service:`, I/O claims, relationship templates, `container:`); the schema
must allow that without reshaping existing sections.
- Sketch (format decided in design):
```yaml
applications:
- name: openfoam
description: "OpenFOAM, MPI"
mode: direct
launch_type: openmpi
resreq: { value: "select[ib] span[ptile=32] rusage[mem=4000]", override: default }
tasks: { min: 32, default: 64, max: 512 }
limits:
runtime: { value: 24h, override: fixed }
memory: { value: 128G, override: fixed }
env:
vars: { FOAM_INST_DIR: /opt/openfoam }
job_env: allow
pre_exec: /opt/xce/app/openfoam/pre.sh
post_exec: /opt/xce/app/openfoam/post.sh
job_pre_post_exec: allow
- name: untrusted-python
mode: sandbox
sandbox: { inputs: ["/shared/in/"], outputs: ["/shared/out/"] }
job_pre_post_exec: forbid
```
### Default profile
- **Set by `default_application: <name>` in `xce.yaml`**, a cluster
parameter (as LSF keeps `DEFAULT_APPLICATION` in `lsb.params`), not a flag
on one profile.
- **Recorded at submit.** A job submitted without `--app` gets the current
default's *name* written on its record, so changing the default later does
not move pending jobs to another profile.
- **A name that is not defined is a configuration error, never a silent
fallback** (user decision, 2026-10-12):
| `default_application` | Behaviour |
|---|---|
| absent | built-in `default` profile (mode `direct`, no attributes), today's behaviour |
| names a defined profile | that profile is used for jobs submitted without `--app` |
| names an undefined profile | the configuration is rejected |
- `xceadmin config build` fails with an error naming the missing profile.
This is an error, not one of the warnings `validation.py` gives for
unknown host or user references: a dangling default would break every
submit without `--app`.
- The controller re-checks on live upgrade and at start, because config can
reach it without xceadmin. A live upgrade with a dangling default,
including one that removes or renames the current default profile, is
refused and the current config stays in force. At start the controller
refuses to run and logs the reason.
- Profiles and the default are in the same configuration, checked
atomically, so the default cannot become dangling at run time.
- **Queue deprecations surface here.** `xceadmin config build` warns when
`queues.yaml` still sets `res_req` or `pre_exec`/`post_exec` (the shipped
`night` queue does) and suggests moving them into a profile. The
deprecation itself is `job-config-merge`'s; the message needs
`applications.yaml` to exist.
## Scope
- **In**:
- The profile schema and config, and a default profile.
- Resolution and override rules.
- The profile schema (attributes and override markers) and
`applications.yaml`, and the default profile (`default_application`).
- Submit-time validation of the profile and the job's overrides.
- RBAC permission for profile use.
- Submit surfaces: REST, `xcecli --app`, harness, libxce.
- Effective values persisted on the job.
- The profile name and the job's own values persisted on the job.
- Execution-mode and I/O-directory fields end to end.
- Capability advertisement and placement.
- A worker launch error when a `direct` exec user is missing.
@ -216,7 +327,24 @@ fits XCE's LSF-style job model.
- ~~How resource defaults combine with a job's own resreq~~ — settled
2026-10-09, owned by `job-config-merge`.
- Profile versioning and audit.
- Whether the profile can be changed for a pending job.
- Whether the profile can be changed for a pending job (leaning yes:
re-check `app:use`, store the new name, the scheduler re-evaluates).
- Raised 2026-10-12, not yet decided:
- Whether execution-side values read at dispatch are frozen on that
run's record (leaning yes: display and post-dispatch failover use what
actually ran; a requeue reads the current profile again).
- Removing a non-default profile that pending jobs still name (leaning:
xceadmin warns, the jobs pend with "profile X not defined", running
jobs continue).
- Limits a job may lower but not raise: either a third "bounded" marker,
or the profile value is a fixed maximum and the job's own value a
request checked against it like a ceiling (keeps the two-interaction
rule of `job-config-layering.md`, which should then say so).
- Env merge: per-variable default/fixed, plus a denylist for sandbox
profiles (e.g. `LD_PRELOAD`)?
- Splitting the spec: execution mode, I/O directories, capability
advertisement and placement as a separate spec (or folded into
`job-sandbox-execution`), leaving this spec the profile core.
- **Out**:
- The worker sandbox mechanics (`job-sandbox-execution`).
- Multi-node sandbox (`job-sandbox-multinode`).
@ -225,10 +353,11 @@ fits XCE's LSF-style job model.
## Boundary Candidates
- The profile model and config (schema, validation, live upgrade).
- Resolution and override engine (controller).
- Submit-time validation of the profile and overrides (controller).
- Access control (RBAC permission) and early validation (apiserver).
- Submit and listing surfaces (REST, xcecli, harness, libxce).
- Persisting effective values (wire, WAL submit fact, snapshot).
- Persisting the profile name and the job's own values (wire, WAL submit
fact, snapshot).
- Capability advertisement and scheduler placement.
## Out of Boundary
@ -266,12 +395,13 @@ fits XCE's LSF-style job model.
→ RBAC.
- All wire changes are additive Cap'n Proto. Old WAL records and snapshots
decode as "no profile, mode `direct`", which is today's behaviour.
- Save/restore field completeness (`persistence.md`): the effective profile
values round trip through the submit facts and snapshots. Run
- Save/restore field completeness (`persistence.md`): the profile name and
the job's own values (including I/O directories) round trip through the
submit facts and snapshots. Run
`/xce-audit-invariants`.
- Input validation (`security.md`): overrides and paths are validated
server-side. No path traversal, and mount prefixes are restricted by the
profile.
- Size: this is likely 20 or more tasks. If requirements confirm that, consider
splitting "capability advertisement and placement" into its own spec.
- Size: this is likely 20 or more tasks. If requirements confirm that, split
as listed under "Open for requirements".
- HA/persistence spec: use the full 3-phase workflow.