From 84501d30dadd69a31bc586aadc96a4daacb3ea6b Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Mon, 12 Oct 2026 09:11:06 +0000 Subject: [PATCH] docs(spec): application-profile brief: own config file, default_application, live merge cleanup - profiles in src/configs/applications.yaml (+ schema, ClusterConfig and Conf4Sched), optional file with a built-in 'default' profile - default_application in xce.yaml; an undefined name is a config error at xceadmin build, live upgrade and controller start, never a fallback - remove the superseded submit-time resolution text: the job keeps its profile name and own values, effective values come from job-config-merge - this spec owns the profile attribute schema and override markers - record the 2026-10-12 open questions with leanings Co-Authored-By: Claude Opus 5.5 (1M context) --- .kiro/specs/application-profile/brief.md | 190 +++++++++++++++++++---- .kiro/steering/roadmap.md | 2 +- 2 files changed, 161 insertions(+), 31 deletions(-) diff --git a/.kiro/specs/application-profile/brief.md b/.kiro/specs/application-profile/brief.md index e1cf663b..91cb4874 100644 --- a/.kiro/specs/application-profile/brief.md +++ b/.kiro/specs/application-profile/brief.md @@ -25,7 +25,8 @@ decisions):** or forbidden by the profile. Order: profile pre → job pre → job → job post → profile post (the worker's existing two-layer structure, `xcecd_worker.c` `queue_pre_exec()`/`queue_post_exec()`). -- **Replaces "Resolution at submit time"** below: the merge is *live*. The WAL +- **The merge is *live*** (replaces the earlier "resolution at submit time", + removed from this brief 2026-10-12). The WAL keeps the job's own values plus the profile name; the controller and apiserver validate at submit, and the scheduler computes effective values and recomputes them when a profile changes (a pending job that no longer @@ -43,6 +44,23 @@ consumed by `service-supervision` (worker probes and restarts), `service-replica-set` (replicas, scaling, rolling update) and `service-endpoint` (ports, endpoints). +**Update 2026-10-12 (discussion; user decisions):** +- **Own configuration file.** Profiles live in a new + `src/configs/applications.yaml` with its own JSON schema, mapped into + `ClusterConfig` and `Conf4Sched` like queues and limits. See "Profile + configuration" below. +- **Cluster default** is set by `default_application` in `xce.yaml`. A name + that is not defined is a configuration error everywhere (`xceadmin`, live + upgrade, controller start), never a silent fallback. See "Default profile" + below. +- **This spec owns the profile's attribute schema**: which attributes exist + and how each is marked *default* or *fixed*. `job-config-merge` consumes + that schema and owns only the merge and live re-evaluation. +- **Later sections are added by their consumers**: `service:` + (`service-supervision`), I/O claims (resource-model PR-6), relationship + and compound templates (PR-14/15), `container:` (R6). This spec only makes + the schema open to new top-level sections. + ## Problem XCE users run many kinds of applications: trusted in-house codes, untrusted scripts, vendor binaries, MPI solvers. Each needs a different way of running: @@ -103,9 +121,10 @@ fits XCE's LSF-style job model. 2. **Launch type.** Serial, `openmpi` or `mpich`. 3. **Environment and pre/post-exec.** Default environment variables and pre/post-exec commands. - 4. **Resource defaults and limits.** A default resource requirement, plus - per-job execution limits (memory, runtime, CPU, tasks), mapped onto the - existing worker cgroup limits. + 4. **Resource requirement and limits.** The job's resource requirement + (select / rusage / `span[...]` / same / order), its task range (min / + default / max), and per-job execution limits (memory, runtime, CPU, + tasks), mapped onto the existing worker cgroup limits. - **Users pick a profile at submit** (REST, `xcecli jobs submit --app `, harness, libxce). A **cluster default profile** applies when none is given. - **Limited overrides.** A job may override only the attributes its profile @@ -147,10 +166,11 @@ fits XCE's LSF-style job model. default resource requirement and per-job execution limits). They are not scheduling policy. Queue-level scheduling limits keep applying on top, unchanged. -- **Resolution at submit time.** The effective values are stored on the job - (mode, options, launch type, env, limits, resreq). Editing a profile later does - not change queued or running jobs. The job records its profile name, and its - details show the effective mode. +- **Live merge** (`job-config-merge`). The job records its profile name and + its own values, not the effective values. The scheduler computes the + effective values and recomputes them when the profile changes: a pending job + that no longer satisfies its profile pends with a reason, and running jobs + are unaffected. Job details show the profile name and the effective mode. - **The apiserver never checks node accounts** (user decision, 2026-09-29). - Submit-time validation covers profile existence, RBAC permission, overrides and paths only. @@ -165,12 +185,14 @@ fits XCE's LSF-style job model. in every mode. ## Approach -- **Profile definitions** live in the cluster configuration: schema, YAML, JSON - schema, xceadmin mapping, and live-upgrade replication, following the - `config-design.md` placement rules. -- **The controller resolves the profile at submit** and merges it with the job's - permitted overrides. This makes it authoritative for every submit path (REST, - libxce, harness). +- **Profile definitions** live in their own cluster configuration file (see + "Profile configuration" below), with xceadmin validation and live-upgrade + replication. +- **The controller validates the profile at submit**: the named profile + exists, and the job overrides only attributes the profile marks *default*. + This makes it authoritative for every submit path (REST, libxce, harness). + It stores the profile name and the job's own values; the effective values + come from the live merge (`job-config-merge`). - **The apiserver checks that the named profile exists** (user direction, 2026-09-29). - At submit, before anything goes downstream (`preCheckJobSubmitRequest`), the @@ -179,29 +201,118 @@ fits XCE's LSF-style job model. - The apiserver already receives the cluster configuration from the controller (`backend/convert.go:265`), so it knows the current set of profiles and keeps it current across live config upgrades. - - The controller still re-validates when it resolves the profile, because it is + - The controller still re-validates at submit, because it is authoritative for non-REST submit paths and for a profile removed between the two checks. If the profile has gone by then, the job is rejected, never run with a fallback profile. - **The apiserver additionally enforces the RBAC permission** for the profile (middleware chain unchanged; a `RequirePermission`-style check) and validates overrides early (`preCheckJobSubmitRequest`). -- **Effective values go into the job's existing fields where they exist** - (`launch_type`, pre/post exec, exdata env and limits, resreq). New fields are - added only for the profile name, execution mode, mode options and I/O - directories. All wire changes are additive Cap'n Proto fields, and the WAL - submit fact and snapshot carry them with field completeness. +- **The job's own values stay in its existing fields** (`launch_type`, + pre/post exec, exdata env and limits, resreq). New job fields are added only + for the profile name and the job's own I/O directories. All wire changes + are additive Cap'n Proto fields, and the WAL submit fact and snapshot carry + them with field completeness. Execution-side values (mode and options, + launch type, env, profile pre/post-exec) are read from the profile at + dispatch, as queue pre/post-exec is today (`get_queue_exec()`, + `controller/queue.c:199`). - **Workers advertise their supported execution modes.** The scheduler filters hosts by the job's effective mode. The design picks the mechanism, for example a host attribute or boolean resource. +### Profile configuration +- **One file per entity type**, as for hosts, queues, resources, limits, + users and scaler: + - `src/configs/applications.yaml`, top-level key `applications:` (named + after LSF `lsb.applications`; the user flag is `--app`); + - `src/configs/schema/applications.schema.json`, picked up by xceadmin's + file-to-schema naming rule (`xcecommon/validation.py:68`); + - `xcecommon/validation.py`: duplicate names rejected; cross-file + references checked (resreq resource names against `resources.yaml`); + - `src/schema/controller.capnp`: `applications: List(AppProfileConf)` in + `ClusterConfig` and in `Conf4Sched` (the scheduler runs the live merge). + `Conf4Worker` does not need it: the worker gets execution values through + `start_info` at dispatch. + - The apiserver already receives `ClusterConfig` + (`backend/convert.go:265`), so it gets the profiles with no new channel. +- **Shared config, per `config-design.md`:** controller, scheduler and + apiserver all need the same view (independence rule), and profiles change + at the rate of queues, as occasional admin edits (frequency rule). +- **The file is optional.** Without it there is one built-in profile + `default` (mode `direct`, no attributes), which is today's behaviour. +- **Every attribute carries its override marker** (`default` or `fixed`, + `job-config-layering.md`). Profile pre/post-exec is always fixed; whether + the job's own pre/post-exec is allowed is a profile setting. +- **Open to new sections.** Later specs add top-level sections to a profile + (`service:`, I/O claims, relationship templates, `container:`); the schema + must allow that without reshaping existing sections. +- Sketch (format decided in design): + + ```yaml + applications: + - name: openfoam + description: "OpenFOAM, MPI" + mode: direct + launch_type: openmpi + resreq: { value: "select[ib] span[ptile=32] rusage[mem=4000]", override: default } + tasks: { min: 32, default: 64, max: 512 } + limits: + runtime: { value: 24h, override: fixed } + memory: { value: 128G, override: fixed } + env: + vars: { FOAM_INST_DIR: /opt/openfoam } + job_env: allow + pre_exec: /opt/xce/app/openfoam/pre.sh + post_exec: /opt/xce/app/openfoam/post.sh + job_pre_post_exec: allow + + - name: untrusted-python + mode: sandbox + sandbox: { inputs: ["/shared/in/"], outputs: ["/shared/out/"] } + job_pre_post_exec: forbid + ``` + +### Default profile +- **Set by `default_application: ` in `xce.yaml`**, a cluster + parameter (as LSF keeps `DEFAULT_APPLICATION` in `lsb.params`), not a flag + on one profile. +- **Recorded at submit.** A job submitted without `--app` gets the current + default's *name* written on its record, so changing the default later does + not move pending jobs to another profile. +- **A name that is not defined is a configuration error, never a silent + fallback** (user decision, 2026-10-12): + + | `default_application` | Behaviour | + |---|---| + | absent | built-in `default` profile (mode `direct`, no attributes), today's behaviour | + | names a defined profile | that profile is used for jobs submitted without `--app` | + | names an undefined profile | the configuration is rejected | + + - `xceadmin config build` fails with an error naming the missing profile. + This is an error, not one of the warnings `validation.py` gives for + unknown host or user references: a dangling default would break every + submit without `--app`. + - The controller re-checks on live upgrade and at start, because config can + reach it without xceadmin. A live upgrade with a dangling default, + including one that removes or renames the current default profile, is + refused and the current config stays in force. At start the controller + refuses to run and logs the reason. + - Profiles and the default are in the same configuration, checked + atomically, so the default cannot become dangling at run time. +- **Queue deprecations surface here.** `xceadmin config build` warns when + `queues.yaml` still sets `res_req` or `pre_exec`/`post_exec` (the shipped + `night` queue does) and suggests moving them into a profile. The + deprecation itself is `job-config-merge`'s; the message needs + `applications.yaml` to exist. + ## Scope - **In**: - - The profile schema and config, and a default profile. - - Resolution and override rules. + - The profile schema (attributes and override markers) and + `applications.yaml`, and the default profile (`default_application`). + - Submit-time validation of the profile and the job's overrides. - RBAC permission for profile use. - Submit surfaces: REST, `xcecli --app`, harness, libxce. - - Effective values persisted on the job. + - The profile name and the job's own values persisted on the job. - Execution-mode and I/O-directory fields end to end. - Capability advertisement and placement. - A worker launch error when a `direct` exec user is missing. @@ -216,7 +327,24 @@ fits XCE's LSF-style job model. - ~~How resource defaults combine with a job's own resreq~~ — settled 2026-10-09, owned by `job-config-merge`. - Profile versioning and audit. - - Whether the profile can be changed for a pending job. + - Whether the profile can be changed for a pending job (leaning yes: + re-check `app:use`, store the new name, the scheduler re-evaluates). + - Raised 2026-10-12, not yet decided: + - Whether execution-side values read at dispatch are frozen on that + run's record (leaning yes: display and post-dispatch failover use what + actually ran; a requeue reads the current profile again). + - Removing a non-default profile that pending jobs still name (leaning: + xceadmin warns, the jobs pend with "profile X not defined", running + jobs continue). + - Limits a job may lower but not raise: either a third "bounded" marker, + or the profile value is a fixed maximum and the job's own value a + request checked against it like a ceiling (keeps the two-interaction + rule of `job-config-layering.md`, which should then say so). + - Env merge: per-variable default/fixed, plus a denylist for sandbox + profiles (e.g. `LD_PRELOAD`)? + - Splitting the spec: execution mode, I/O directories, capability + advertisement and placement as a separate spec (or folded into + `job-sandbox-execution`), leaving this spec the profile core. - **Out**: - The worker sandbox mechanics (`job-sandbox-execution`). - Multi-node sandbox (`job-sandbox-multinode`). @@ -225,10 +353,11 @@ fits XCE's LSF-style job model. ## Boundary Candidates - The profile model and config (schema, validation, live upgrade). -- Resolution and override engine (controller). +- Submit-time validation of the profile and overrides (controller). - Access control (RBAC permission) and early validation (apiserver). - Submit and listing surfaces (REST, xcecli, harness, libxce). -- Persisting effective values (wire, WAL submit fact, snapshot). +- Persisting the profile name and the job's own values (wire, WAL submit + fact, snapshot). - Capability advertisement and scheduler placement. ## Out of Boundary @@ -266,12 +395,13 @@ fits XCE's LSF-style job model. → RBAC. - All wire changes are additive Cap'n Proto. Old WAL records and snapshots decode as "no profile, mode `direct`", which is today's behaviour. -- Save/restore field completeness (`persistence.md`): the effective profile - values round trip through the submit facts and snapshots. Run +- Save/restore field completeness (`persistence.md`): the profile name and + the job's own values (including I/O directories) round trip through the + submit facts and snapshots. Run `/xce-audit-invariants`. - Input validation (`security.md`): overrides and paths are validated server-side. No path traversal, and mount prefixes are restricted by the profile. -- Size: this is likely 20 or more tasks. If requirements confirm that, consider - splitting "capability advertisement and placement" into its own spec. +- Size: this is likely 20 or more tasks. If requirements confirm that, split + as listed under "Open for requirements". - HA/persistence spec: use the full 3-phase workflow. diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index c98344af..229854b5 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -63,7 +63,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han - [x] ha-commit-durability — closes bug-661 (acknowledgment did not imply durability; committed jobs lost across failover). Makes acknowledged ⇒ durable on a majority: the primary counts its own copy toward the commit majority only at its durable LSN (`majority_candidate_locked` caps self at `persist_durable_lsn`, not the appended tip); `fsync_worker` repurposed timer→demand-driven group-commit flusher (one `fdatasync` over pending appends, advance `durable_lsn`, wake commit-eval) — absorbs the former "WAL group-commit batching" backlog item (see below); recovery integrity verdict (clean vs corruption-truncated WAL) + durable-backed freshness floor (candidate advertises bare `persist_durable_floor`, voter floor = `max(durable_floor, learned_committed)`) + pre-promotion eligibility fence. Establishes the **controllers-never-self-terminate** invariant: a recovery-truncated/behind member that cannot catch up is a LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — never `exit`/`abort`), self-heals on peer return (see `ha-async-io.md` process-lifecycle invariant + `src/controller/CLAUDE.md`). Dependencies: ha-commit-engine (consumes commit-position semantics), ha-wal-replication (standby ACK/catch-up), ha-commit-barrier-rollout (reply gate, unchanged). (specs done 2026-06-18; **IMPLEMENTED 2026-06-18**: 17/17 tasks (1.1–5.3), validate-impl GO, controller suite 0 failures under ASAN; deterministic ack-before-fdatasync E2E RED→GREEN + rolling-SIGKILL soak zero acked-job loss. Verified by tests + Protocol Verification Checklist, NOT by raft.tla (which assumes a durable log). Fixed bug-760 (adjacent `test_fact_hook_append` raced the async flusher — test made durable-deterministic via `persist_flush`). raft.tla durability-model extension deferred unless explicitly opted in.) - [x] scheduler-allocator-runner — NEW 2026-09-28 (`/kiro-discovery`, Path C). The framework owns `job_alloc_fn` through a new `xce_sched_allocator_iface_t` (accepts guard + plan iterator) and a single `sched_allocator_run()`; both `my_alloc()`s are deleted (seqjob/parjob), and `alloc_iter.h` `init` is retyped to `xce_sched_core_t*`. Behaviour must stay identical (bug-1882/1767/1768). Closes `issues/new_requirements.md` item 4. Brief: `.kiro/specs/scheduler-allocator-runner/brief.md`. Dependencies: scheduler-allocation, scheduler-core (both done). **IMPLEMENTED 2026-09-30** (11/11 tasks, e2266582..28cff179; validate-impl GO; scheduler ASAN suite 96/96): runner in `src/scheduler/src/alloc_runner.c`; decided along the way: the runner clears `alloc_req.host` before every candidate so every short host gets pend reasons (both allocators), and fixes bug-1893 (shared reasons were never recorded). Follow-ups: publish pend reasons only when changed; sim-mode parjob allocation parity fixture; bug-1892; bug-1897. - [ ] scheduler-alter-pend-reasons — NEW 2026-09-28 (`/kiro-discovery`, Path E). An ALTER-resreq job keeps a pend reason for EVERY alternative instead of only the last (`scheduler.c:424` FIXME). Wire shape (user decision): additive `alternatives @3: List(JobPendReason)` on `JobPendReason`, per-alternative attribution; carried end-to-end through controller job record, WAL preason fact, recordings and `job_preason_eq`. Also fixes the confirmed `obj->reason` overwrite leak (`scheduler.c:388`). Brief: `.kiro/specs/scheduler-alter-pend-reasons/brief.md`. Dependencies: libxce-alloc-preason-equality (HARD: extends its `job_preason_eq`); sequence after scheduler-allocator-runner (its "no pend-reason content change" rule). -- [ ] application-profile — NEW 2026-09-29 (`/kiro-discovery`, Job Sandbox Execution initiative, Path D; reshaped from `job-execution-mode` by user direction: execution mode is a job attribute determined by the APPLICATION). Admin-defined named application profiles (LSF `-app` precedent) in cluster config: execution mode + options (`direct`|`sandbox`, room for `container`/`vm`), launch type, env + pre/post exec, resource defaults/limits (extensible). Users pick a profile at submit (`--app`), cluster default profile otherwise; queues (scheduling policy) and profiles (application config) are orthogonal — neither restricts the other; only profile-marked overridable attributes may be overridden; access by RBAC permission (new `app:use`/`app:view` with `app` domain type, folded into this spec); apiserver checks profile existence; controller resolves at submit and persists effective values on the job (wire additive, WAL/snapshot field-completeness). apiserver never checks node accounts; worker fails a `direct` job with a clear reason if the exec user is missing. Worker capability advertisement + scheduler placement. Brief: `.kiro/specs/application-profile/brief.md`. Dependencies: none. +- [ ] application-profile — NEW 2026-09-29 (`/kiro-discovery`, Job Sandbox Execution initiative, Path D; reshaped from `job-execution-mode` by user direction: execution mode is a job attribute determined by the APPLICATION). Admin-defined named application profiles (LSF `-app` precedent) in cluster config: execution mode + options (`direct`|`sandbox`, room for `container`/`vm`), launch type, env + pre/post exec, resource defaults/limits (extensible). Users pick a profile at submit (`--app`), cluster default profile otherwise; queues (scheduling policy) and profiles (application config) are orthogonal — neither restricts the other; only profile-marked overridable attributes may be overridden; access by RBAC permission (new `app:use`/`app:view` with `app` domain type, folded into this spec); apiserver checks profile existence; controller validates at submit and persists the profile name + the job's own values (live merge via job-config-merge; wire additive, WAL/snapshot field-completeness); own config file `applications.yaml`, cluster default `default_application` in `xce.yaml` (undefined name = config error, never a fallback; updated 2026-10-12). apiserver never checks node accounts; worker fails a `direct` job with a clear reason if the exec user is missing. Worker capability advertisement + scheduler placement. Brief: `.kiro/specs/application-profile/brief.md`. Dependencies: none. - [ ] job-sandbox-execution — NEW 2026-09-29 (Job Sandbox Execution). Worker: pluggable execution-backend interface (`direct` refactored + `systemd-sandbox`; non-systemd backends later). Sandbox = existing transient unit + `DynamicUser` with fresh per-job name (never the submitter's), ProtectSystem=strict/ProtectHome/PrivateTmp/NoNewPrivileges, Bind(ReadOnly)Paths for job I/O dirs or worker-created temp workspace (handed to the uid via SO_PEERCRED), `StandardOutput=file:` output, helper auth by unit/cgroup, MANDATORY post-job cleanup/ownership normalization (dynamic uid reuse), systemd ≥ 242 + nss-systemd capability probe; single-node incl. single-node MPI. Viability-checked 2026-09-29. Brief: `.kiro/specs/job-sandbox-execution/brief.md`. Dependencies: application-profile; remote-task-root fix (Direct-Implementation). - [ ] job-sandbox-multinode — NEW 2026-09-29 (Job Sandbox Execution). Multi-node sandbox jobs: cluster-consistent ephemeral uid (expected: controller-allocated, WAL-replicated uid lease from a reserved range), worker-driven remote/MPI launch (no ssh; PMIx/prted via worker), shared-FS ownership normalization, one unit per job per node. Brief: `.kiro/specs/job-sandbox-multinode/brief.md`. Dependencies: job-sandbox-execution, application-profile. - [ ] rbac-persistence — NEW 2026-09-29 (`/kiro-discovery`, raised during application-profile discovery). Closes the deferred `security.md` gap: runtime RBAC mutations (roles, permissions, user-role assignments, Logon auto-assignment) are in-memory only and silently revert to `rbac.yaml` on apiserver restart — including revocations. Durable-or-rejected mutations, audit trail, defaults-vs-runtime reconciliation, multi-apiserver consistency; storage approach (rbac.yaml write-through / controller-replicated entity / embedded store / read-only) decided in design. Brief: `.kiro/specs/rbac-persistence/brief.md`. Dependencies: none (independent of the sandbox initiative; application-profile's API-made `app:use` grants rely on it for durability).