docs(spec): defer per-profile access control to app-access-control

- v1: any queue:submit user may use any profile; listing reuses cluster:view
- app:use / app:view design kept in the brief and moved to the new backlog
  item app-access-control (depends on rbac-persistence for API grants)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-13 03:31:00 +00:00
parent 863ba98ac8
commit e48d36054d
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
3 changed files with 45 additions and 43 deletions

View file

@ -229,7 +229,7 @@ fits XCE's LSF-style job model.
(`XCE_APP_ANN_CFD_ORG_SOLVER_FAMILY`). `xceadmin` rejects two (`XCE_APP_ANN_CFD_ORG_SOLVER_FAMILY`). `xceadmin` rejects two
annotation keys (or two params) that convert to the same name. annotation keys (or two params) that convert to the same name.
- Limits on count and size, so a profile cannot bloat every job's env. - Limits on count and size, so a profile cannot bloat every job's env.
- Visible to every `app:view` holder: no secrets (the secret-name - Visible to every user who can view profiles: no secrets (the secret-name
warning applies). All are exported in v1; an opt-out only if warning applies). All are exported in v1; an opt-out only if
needed. needed.
- `env` vs annotations vs params: `env` is a variable the application - `env` vs annotations vs params: `env` is a variable the application
@ -260,30 +260,25 @@ fits XCE's LSF-style job model.
variables, lower limits) and override a profile `rusage` default, but not variables, lower limits) and override a profile `rusage` default, but not
change or remove anything else. Anything else is rejected with a clear change or remove anything else. Anything else is rejected with a clear
error. error.
- **Access by RBAC permission.** No new access mechanism; this follows the - **Access control is deferred** (user decision, 2026-10-12): v1 has no
existing named-instance pattern (user decision, 2026-09-29: fold profile RBAC per-profile permission. Any user who may submit (`queue:submit`) may use
into this spec). Today, `queue:submit` scoped to `type: queue`, any profile; listing and showing profiles reuses the existing
`values: [normal, priority]` in `config/rbac.yaml` is enforced by `cluster:view` permission (profiles are cluster configuration). The
`GetRequireResourcePermission` in `controllers/jobs.go:1402-1409`. Profiles add: design recorded below moves to the backlog spec `app-access-control`,
- **Permissions:** `app:use` (submit a job with a profile) and `app:view` (list kept here so it is not lost:
and inspect profiles). - permissions `app:use` (submit with a profile) and `app:view` (list and
- **Scope:** a new `app` scope type whose values are profile names, or `all`. inspect profiles), a new `app` scope type whose values are profile
- **Checks:** names or `all`, following the named-instance pattern of `queue:submit`
- Job create checks `app:use` alongside, and independent of, `queue:submit`. (`config/rbac.yaml`, `GetRequireResourcePermission`,
- Job restart (`POST /:jobids/restart`, which today checks only `controllers/jobs.go:1402-1409`);
`queue:submit`) re-checks `app:use` for the job's profile. - checks on job create (alongside, and independent of, `queue:submit`),
- Listing profiles checks `app:view`. on job restart (re-check for the job's profile) and on listing;
- **Defaults:** `rbac.yaml` gets defaults, for example `basic_user` → - `rbac.yaml` defaults (`basic_user` → `app:use` on the default profile,
`app:use` on the default profile, and `app:view` on `all`. `app:view` on `all`); the current default profile always usable with
- **The current default profile is always usable** (decided 2026-10-12): `queue:submit` (implicit `app:use`), so plain submits never break when
anyone with `queue:submit` has an implicit `app:use` on the profile the default is repointed; aliases checked against the target profile;
`default_application` names (or the built-in `default`), so plain - role API accepts the new scope type; `cluster_admin` keeps the single
submits never break when an admin repoints the default. existing bypass; API-made grants depend on `rbac-persistence`.
- **Role API:** role creation and update accept the new scope type.
- **Bypass:** `cluster_admin` keeps the single existing bypass; no new bypass
path.
- **Persistence:** API-made grants are subject to the `rbac-persistence` gap
until that spec lands.
- **Queues and profiles are orthogonal** (user decision, 2026-09-29). - **Queues and profiles are orthogonal** (user decision, 2026-09-29).
- **Queues** own scheduling policy: which jobs are considered and in what - **Queues** own scheduling policy: which jobs are considered and in what
order, priority, scheduling limits, and host partitioning. order, priority, scheduling limits, and host partitioning.
@ -293,8 +288,9 @@ fits XCE's LSF-style job model.
constrain queues. constrain queues.
- A profile carries no scheduling-policy attributes (no priority, queue choice, - A profile carries no scheduling-policy attributes (no priority, queue choice,
fairshare or preemption). fairshare or preemption).
- Any combination of an allowed queue and an allowed profile is valid. Each is - Any combination of an allowed queue and a profile is valid. Queues keep
governed by its own access rules; for profiles, that is RBAC permission. their access rules; per-profile access control comes later
(`app-access-control`).
- Resource values in a profile describe what the application needs (its - Resource values in a profile describe what the application needs (its
essential resource requirement and per-job execution limits). They are essential resource requirement and per-job execution limits). They are
not scheduling policy. Queue-level scheduling limits keep applying on top, not scheduling policy. Queue-level scheduling limits keep applying on top,
@ -305,8 +301,8 @@ fits XCE's LSF-style job model.
that no longer satisfies its profile pends with a reason, and running jobs that no longer satisfies its profile pends with a reason, and running jobs
are unaffected. Job details show the profile name and the effective mode. are unaffected. Job details show the profile name and the effective mode.
- **The apiserver never checks node accounts** (user decision, 2026-09-29). - **The apiserver never checks node accounts** (user decision, 2026-09-29).
- Submit-time validation covers profile existence, RBAC permission, the - Submit-time validation covers profile existence, the additive rule and
additive rule and paths only. paths only (no per-profile permission in v1).
- The execution account is the worker's concern. If a `direct` job's execution - The execution account is the worker's concern. If a `direct` job's execution
user doesn't exist on the node, the worker fails the job at launch with a user doesn't exist on the node, the worker fails the job at launch with a
clear, user-visible reason, instead of today's opaque systemd failure. clear, user-visible reason, instead of today's opaque systemd failure.
@ -338,9 +334,9 @@ fits XCE's LSF-style job model.
authoritative for non-REST submit paths and for a profile removed between the authoritative for non-REST submit paths and for a profile removed between the
two checks. If the profile has gone by then, the job is rejected, never run two checks. If the profile has gone by then, the job is rejected, never run
with a fallback profile. with a fallback profile.
- **The apiserver additionally enforces the RBAC permission** for the profile - **The apiserver additionally checks the additive rule early**
(middleware chain unchanged; a `RequirePermission`-style check) and checks (`preCheckJobSubmitRequest`); the middleware chain is unchanged. The
the additive rule early (`preCheckJobSubmitRequest`). profile listing route uses `RequirePermission("cluster:view")`.
- **The job's own values stay in its existing fields** (`launch_type`, - **The job's own values stay in its existing fields** (`launch_type`,
pre/post exec, exdata env and limits, resreq). New fields (all additive pre/post exec, exdata env and limits, resreq). New fields (all additive
Cap'n Proto; WAL submit/dispatch facts and snapshot carry them with field Cap'n Proto; WAL submit/dispatch facts and snapshot carry them with field
@ -391,7 +387,8 @@ fits XCE's LSF-style job model.
profile named `default` in the file **replaces** the built-in one, so profile named `default` in the file **replaces** the built-in one, so
admins can customise the default without setting `default_application`. admins can customise the default without setting `default_application`.
- **No secrets in profiles.** Profile env values and pre/post-exec paths - **No secrets in profiles.** Profile env values and pre/post-exec paths
are visible to every `app:view` holder (profile listing, job details); are visible to every user who can view profiles (`cluster:view`) and in
job details;
`xceadmin` warns on env names that look like secrets (`*TOKEN*`, `xceadmin` warns on env names that look like secrets (`*TOKEN*`,
`*SECRET*`, `*PASSWORD*`, `*_KEY`), as for presets. `*SECRET*`, `*PASSWORD*`, `*_KEY`), as for presets.
- **No override markers.** Every profile value binds the job (additive - **No override markers.** Every profile value binds the job (additive
@ -657,7 +654,7 @@ defaults a profile does not (everything but default reservations):
into its own local file, so a server preset being removed never affects into its own local file, so a server preset being removed never affects
users. `preset diff` / `pull --force` are user-initiated. users. `preset diff` / `pull --force` are user-initiated.
- User-to-user sharing by `preset export` / `import` (file, mail, git). - User-to-user sharing by `preset export` / `import` (file, mail, git).
- Presets carry no privilege: `app:use`, `queue:submit`, the profile's - Presets carry no privilege: `queue:submit` (later `app:use`), the profile's
essentials, queue ceilings and project quotas still apply at submit. essentials, queue ceilings and project quotas still apply at submit.
### Dispatch record ### Dispatch record
@ -759,7 +756,8 @@ with the reason "application profile X is not defined".
Decided 2026-10-12 (user), so a rename never strands jobs: Decided 2026-10-12 (user), so a rename never strands jobs:
- A profile may list `aliases: [old-name, ...]`. An alias resolves to the - A profile may list `aliases: [old-name, ...]`. An alias resolves to the
profile everywhere: merge, dispatch record, content hash (the target's), profile everywhere: merge, dispatch record, content hash (the target's),
RBAC (`app:use` is checked against the target profile's name). and, once `app-access-control` lands, its permission check (against the
target profile's name).
- Jobs keep the name they recorded; the scheduler and controller resolve it - Jobs keep the name they recorded; the scheduler and controller resolve it
through the alias. New submits that name an alias are accepted and record through the alias. New submits that name an alias are accepted and record
the **target's** name, so the alias can be removed once old jobs drain. the **target's** name, so the alias can be removed once old jobs drain.
@ -788,7 +786,8 @@ Decided 2026-10-12 (user), so a rename never strands jobs:
- **Restart carries the profile name.** `POST /:jobids/restart` rebuilds the - **Restart carries the profile name.** `POST /:jobids/restart` rebuilds the
new job field by field from the old job's record (bug-1968 shows what is new job field by field from the old job's record (bug-1968 shows what is
lost otherwise), so it must copy the profile name (or alias); it then lost otherwise), so it must copy the profile name (or alias); it then
re-checks `app:use` and re-reads the profile like any new dispatch. re-reads the profile like any new dispatch (and, once
`app-access-control` lands, re-checks `app:use`).
- **Array jobs** carry one profile name on the array's submit data; every - **Array jobs** carry one profile name on the array's submit data; every
element uses it. element uses it.
- **A name that is not defined is a configuration error, never a silent - **A name that is not defined is a configuration error, never a silent
@ -822,7 +821,6 @@ Decided 2026-10-12 (user), so a rename never strands jobs:
- The profile schema (essentials, additive rule, aliases) and - The profile schema (essentials, additive rule, aliases) and
`applications.yaml`, and the default profile (`default_application`). `applications.yaml`, and the default profile (`default_application`).
- Submit-time validation of the profile and the additive rule. - Submit-time validation of the profile and the additive rule.
- RBAC permission for profile use.
- Submit surfaces: REST, `xcecli --app`, harness, libxce. - Submit surfaces: REST, `xcecli --app`, harness, libxce.
- The profile name and the job's own values persisted on the job. - The profile name and the job's own values persisted on the job.
- The `mode:` schema field (only `direct`), the structured resreq schema - The `mode:` schema field (only `direct`), the structured resreq schema
@ -882,11 +880,13 @@ Decided 2026-10-12 (user), so a rename never strands jobs:
- Multi-node sandbox (`job-sandbox-multinode`). - Multi-node sandbox (`job-sandbox-multinode`).
- Container or vm execution itself (only room for them in the schema). - Container or vm execution itself (only room for them in the schema).
- Auth backends and JWT claims. - Auth backends and JWT claims.
- Per-profile access control (`app:use` / `app:view`): deferred to
`app-access-control` (user decision, 2026-10-12).
## Boundary Candidates ## Boundary Candidates
- The profile model and config (schema, validation, live upgrade). - The profile model and config (schema, validation, live upgrade).
- Submit-time validation of the profile and the additive rule (controller). - Submit-time validation of the profile and the additive rule (controller).
- Access control (RBAC permission) and early validation (apiserver). - Early validation (apiserver).
- Submit and listing surfaces (REST, xcecli, harness, libxce). - Submit and listing surfaces (REST, xcecli, harness, libxce).
- Persisting the profile name and the job's own values (wire, WAL submit - Persisting the profile name and the job's own values (wire, WAL submit
fact, snapshot). fact, snapshot).
@ -906,7 +906,8 @@ Decided 2026-10-12 (user), so a rename never strands jobs:
seam). seam).
- The config pipeline (`clusterconfig-schema-split`, `config-upgrade-snapshot`, - The config pipeline (`clusterconfig-schema-split`, `config-upgrade-snapshot`,
`admin-config-tool`). `admin-config-tool`).
- RBAC (`rbac_check.go`). - RBAC (`rbac_check.go`), only for the reused `cluster:view` on the
listing route.
- The scheduler-to-controller dispatch decision (gains the profile - The scheduler-to-controller dispatch decision (gains the profile
content hash; `scheduler-dispatch-commit`). content hash; `scheduler-dispatch-commit`).
- The restart path (`RestartJobByID`, bug-1968). - The restart path (`RestartJobByID`, bug-1968).

View file

@ -21,7 +21,7 @@ configuration level.
only the expanded values. only the expanded values.
- **Handled fully on the client.** Presets never reach the controller, WAL - **Handled fully on the client.** Presets never reach the controller, WAL
or scheduler, own no attribute, are not a configuration level and carry or scheduler, own no attribute, are not a configuration level and carry
no privilege: `app:use`, `queue:submit`, the profile's essentials, queue no privilege: `queue:submit` (later `app:use`), the profile's essentials, queue
ceilings and project quotas still apply at submit. ceilings and project quotas still apply at submit.
- **One file per preset** (user, 2026-10-12), the same layout on client - **One file per preset** (user, 2026-10-12), the same layout on client
and server: and server:
@ -113,7 +113,7 @@ configuration level.
in the job record; an explicit `set:` overrides. in the job record; an explicit `set:` overrides.
- A captured name the job's profile defines is dropped by the client with - A captured name the job's profile defines is dropped by the client with
a note; an explicit `set:` that conflicts is still rejected by the a note; an explicit `set:` that conflicts is still rejected by the
server (additive rule). The client reads the profile (`app:view`). server (additive rule). The client reads the profile (`cluster:view` in v1).
- `pass: all` is opt-in; no capture is the default (today's behaviour). - `pass: all` is opt-in; no capture is the default (today's behaviour).
CLI only: portal and REST have no shell env and use `set:` only. CLI only: portal and REST have no shell env and use `set:` only.
- Today's `xcecli jobs submit --env` (whole `os.Environ()`, - Today's `xcecli jobs submit --env` (whole `os.Environ()`,

View file

@ -63,10 +63,10 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
- [x] ha-commit-durability — closes bug-661 (acknowledgment did not imply durability; committed jobs lost across failover). Makes acknowledged ⇒ durable on a majority: the primary counts its own copy toward the commit majority only at its durable LSN (`majority_candidate_locked` caps self at `persist_durable_lsn`, not the appended tip); `fsync_worker` repurposed timer→demand-driven group-commit flusher (one `fdatasync` over pending appends, advance `durable_lsn`, wake commit-eval) — absorbs the former "WAL group-commit batching" backlog item (see below); recovery integrity verdict (clean vs corruption-truncated WAL) + durable-backed freshness floor (candidate advertises bare `persist_durable_floor`, voter floor = `max(durable_floor, learned_committed)`) + pre-promotion eligibility fence. Establishes the **controllers-never-self-terminate** invariant: a recovery-truncated/behind member that cannot catch up is a LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — never `exit`/`abort`), self-heals on peer return (see `ha-async-io.md` process-lifecycle invariant + `src/controller/CLAUDE.md`). Dependencies: ha-commit-engine (consumes commit-position semantics), ha-wal-replication (standby ACK/catch-up), ha-commit-barrier-rollout (reply gate, unchanged). (specs done 2026-06-18; **IMPLEMENTED 2026-06-18**: 17/17 tasks (1.1–5.3), validate-impl GO, controller suite 0 failures under ASAN; deterministic ack-before-fdatasync E2E RED→GREEN + rolling-SIGKILL soak zero acked-job loss. Verified by tests + Protocol Verification Checklist, NOT by raft.tla (which assumes a durable log). Fixed bug-760 (adjacent `test_fact_hook_append` raced the async flusher — test made durable-deterministic via `persist_flush`). raft.tla durability-model extension deferred unless explicitly opted in.) - [x] ha-commit-durability — closes bug-661 (acknowledgment did not imply durability; committed jobs lost across failover). Makes acknowledged ⇒ durable on a majority: the primary counts its own copy toward the commit majority only at its durable LSN (`majority_candidate_locked` caps self at `persist_durable_lsn`, not the appended tip); `fsync_worker` repurposed timer→demand-driven group-commit flusher (one `fdatasync` over pending appends, advance `durable_lsn`, wake commit-eval) — absorbs the former "WAL group-commit batching" backlog item (see below); recovery integrity verdict (clean vs corruption-truncated WAL) + durable-backed freshness floor (candidate advertises bare `persist_durable_floor`, voter floor = `max(durable_floor, learned_committed)`) + pre-promotion eligibility fence. Establishes the **controllers-never-self-terminate** invariant: a recovery-truncated/behind member that cannot catch up is a LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — never `exit`/`abort`), self-heals on peer return (see `ha-async-io.md` process-lifecycle invariant + `src/controller/CLAUDE.md`). Dependencies: ha-commit-engine (consumes commit-position semantics), ha-wal-replication (standby ACK/catch-up), ha-commit-barrier-rollout (reply gate, unchanged). (specs done 2026-06-18; **IMPLEMENTED 2026-06-18**: 17/17 tasks (1.1–5.3), validate-impl GO, controller suite 0 failures under ASAN; deterministic ack-before-fdatasync E2E RED→GREEN + rolling-SIGKILL soak zero acked-job loss. Verified by tests + Protocol Verification Checklist, NOT by raft.tla (which assumes a durable log). Fixed bug-760 (adjacent `test_fact_hook_append` raced the async flusher — test made durable-deterministic via `persist_flush`). raft.tla durability-model extension deferred unless explicitly opted in.)
- [x] scheduler-allocator-runner — NEW 2026-09-28 (`/kiro-discovery`, Path C). The framework owns `job_alloc_fn` through a new `xce_sched_allocator_iface_t` (accepts guard + plan iterator) and a single `sched_allocator_run()`; both `my_alloc()`s are deleted (seqjob/parjob), and `alloc_iter.h` `init` is retyped to `xce_sched_core_t*`. Behaviour must stay identical (bug-1882/1767/1768). Closes `issues/new_requirements.md` item 4. Brief: `.kiro/specs/scheduler-allocator-runner/brief.md`. Dependencies: scheduler-allocation, scheduler-core (both done). **IMPLEMENTED 2026-09-30** (11/11 tasks, e2266582..28cff179; validate-impl GO; scheduler ASAN suite 96/96): runner in `src/scheduler/src/alloc_runner.c`; decided along the way: the runner clears `alloc_req.host` before every candidate so every short host gets pend reasons (both allocators), and fixes bug-1893 (shared reasons were never recorded). Follow-ups: publish pend reasons only when changed; sim-mode parjob allocation parity fixture; bug-1892; bug-1897. - [x] scheduler-allocator-runner — NEW 2026-09-28 (`/kiro-discovery`, Path C). The framework owns `job_alloc_fn` through a new `xce_sched_allocator_iface_t` (accepts guard + plan iterator) and a single `sched_allocator_run()`; both `my_alloc()`s are deleted (seqjob/parjob), and `alloc_iter.h` `init` is retyped to `xce_sched_core_t*`. Behaviour must stay identical (bug-1882/1767/1768). Closes `issues/new_requirements.md` item 4. Brief: `.kiro/specs/scheduler-allocator-runner/brief.md`. Dependencies: scheduler-allocation, scheduler-core (both done). **IMPLEMENTED 2026-09-30** (11/11 tasks, e2266582..28cff179; validate-impl GO; scheduler ASAN suite 96/96): runner in `src/scheduler/src/alloc_runner.c`; decided along the way: the runner clears `alloc_req.host` before every candidate so every short host gets pend reasons (both allocators), and fixes bug-1893 (shared reasons were never recorded). Follow-ups: publish pend reasons only when changed; sim-mode parjob allocation parity fixture; bug-1892; bug-1897.
- [ ] scheduler-alter-pend-reasons — NEW 2026-09-28 (`/kiro-discovery`, Path E). An ALTER-resreq job keeps a pend reason for EVERY alternative instead of only the last (`scheduler.c:424` FIXME). Wire shape (user decision): additive `alternatives @3: List(JobPendReason)` on `JobPendReason`, per-alternative attribution; carried end-to-end through controller job record, WAL preason fact, recordings and `job_preason_eq`. Also fixes the confirmed `obj->reason` overwrite leak (`scheduler.c:388`). Brief: `.kiro/specs/scheduler-alter-pend-reasons/brief.md`. Dependencies: libxce-alloc-preason-equality (HARD: extends its `job_preason_eq`); sequence after scheduler-allocator-runner (its "no pend-reason content change" rule). - [ ] scheduler-alter-pend-reasons — NEW 2026-09-28 (`/kiro-discovery`, Path E). An ALTER-resreq job keeps a pend reason for EVERY alternative instead of only the last (`scheduler.c:424` FIXME). Wire shape (user decision): additive `alternatives @3: List(JobPendReason)` on `JobPendReason`, per-alternative attribution; carried end-to-end through controller job record, WAL preason fact, recordings and `job_preason_eq`. Also fixes the confirmed `obj->reason` overwrite leak (`scheduler.c:388`). Brief: `.kiro/specs/scheduler-alter-pend-reasons/brief.md`. Dependencies: libxce-alloc-preason-equality (HARD: extends its `job_preason_eq`); sequence after scheduler-allocator-runner (its "no pend-reason content change" rule).
- [ ] application-profile — NEW 2026-09-29 (`/kiro-discovery`, Job Sandbox Execution initiative, Path D; reshaped from `job-execution-mode` by user direction: execution mode is a job attribute determined by the APPLICATION). Admin-defined named application profiles (LSF `-app` precedent) in cluster config: execution mode + options (`direct`|`sandbox`, room for `container`/`vm`), launch type, env + pre/post exec, resource defaults/limits (extensible). Users pick a profile at submit (`--app`), cluster default profile otherwise; queues (scheduling policy) and profiles (application config) are orthogonal — neither restricts the other; a profile holds only essentials and the job may add to them but never change them (additive rule, 2026-10-12; convenience defaults are client-side presets, `xcecli-presets`); access by RBAC permission (new `app:use`/`app:view` with `app` domain type, folded into this spec); apiserver checks profile existence; controller validates at submit and persists the profile name + the job's own values (live merge via job-config-merge; wire additive, WAL/snapshot field-completeness); own config file `applications.yaml`, cluster default `default_application` in `xce.yaml` (undefined name = config error, never a fallback; updated 2026-10-12). apiserver never checks node accounts; worker fails a `direct` job with a clear reason if the exec user is missing. **Split 2026-10-12 (user):** execution modes (`sandbox` value + options, I/O dirs, worker capability advertisement, placement by mode) moved to job-sandbox-execution; this spec keeps `mode: direct`, the dispatch record (resreq + execution values + profile content hash; retryable dispatch failure when the profile is gone/changed), aliases, outer pre/post-exec with clean env, reserved `XCE_`. Order: this spec's requirements+design before job-config-merge's design. Brief: `.kiro/specs/application-profile/brief.md`. Dependencies: none. - [ ] application-profile — NEW 2026-09-29 (`/kiro-discovery`, Job Sandbox Execution initiative, Path D; reshaped from `job-execution-mode` by user direction: execution mode is a job attribute determined by the APPLICATION). Admin-defined named application profiles (LSF `-app` precedent) in cluster config: execution mode + options (`direct`|`sandbox`, room for `container`/`vm`), launch type, env + pre/post exec, resource defaults/limits (extensible). Users pick a profile at submit (`--app`), cluster default profile otherwise; queues (scheduling policy) and profiles (application config) are orthogonal — neither restricts the other; a profile holds only essentials and the job may add to them but never change them (additive rule, 2026-10-12; convenience defaults are client-side presets, `xcecli-presets`); access control deferred (user decision 2026-10-12: v1 lets any `queue:submit` user use any profile, listing reuses `cluster:view`; per-profile `app:use`/`app:view` moved to `app-access-control`); apiserver checks profile existence; controller validates at submit and persists the profile name + the job's own values (live merge via job-config-merge; wire additive, WAL/snapshot field-completeness); own config file `applications.yaml`, cluster default `default_application` in `xce.yaml` (undefined name = config error, never a fallback; updated 2026-10-12). apiserver never checks node accounts; worker fails a `direct` job with a clear reason if the exec user is missing. **Split 2026-10-12 (user):** execution modes (`sandbox` value + options, I/O dirs, worker capability advertisement, placement by mode) moved to job-sandbox-execution; this spec keeps `mode: direct`, the dispatch record (resreq + execution values + profile content hash; retryable dispatch failure when the profile is gone/changed), aliases, outer pre/post-exec with clean env, reserved `XCE_`. Order: this spec's requirements+design before job-config-merge's design. Brief: `.kiro/specs/application-profile/brief.md`. Dependencies: none.
- [ ] job-sandbox-execution — NEW 2026-09-29 (Job Sandbox Execution). Worker: pluggable execution-backend interface (`direct` refactored + `systemd-sandbox`; non-systemd backends later). Sandbox = existing transient unit + `DynamicUser` with fresh per-job name (never the submitter's), ProtectSystem=strict/ProtectHome/PrivateTmp/NoNewPrivileges, Bind(ReadOnly)Paths for job I/O dirs or worker-created temp workspace (handed to the uid via SO_PEERCRED), `StandardOutput=file:` output, helper auth by unit/cgroup, MANDATORY post-job cleanup/ownership normalization (dynamic uid reuse), systemd ≥ 242 + nss-systemd capability probe; single-node incl. single-node MPI. **2026-10-12:** also owns the submission side of execution modes moved from application-profile (`sandbox` mode + options, job I/O-directory fields end to end, capability advertisement, scheduler placement by mode). Viability-checked 2026-09-29. Brief: `.kiro/specs/job-sandbox-execution/brief.md`. Dependencies: application-profile; remote-task-root fix (Direct-Implementation). - [ ] job-sandbox-execution — NEW 2026-09-29 (Job Sandbox Execution). Worker: pluggable execution-backend interface (`direct` refactored + `systemd-sandbox`; non-systemd backends later). Sandbox = existing transient unit + `DynamicUser` with fresh per-job name (never the submitter's), ProtectSystem=strict/ProtectHome/PrivateTmp/NoNewPrivileges, Bind(ReadOnly)Paths for job I/O dirs or worker-created temp workspace (handed to the uid via SO_PEERCRED), `StandardOutput=file:` output, helper auth by unit/cgroup, MANDATORY post-job cleanup/ownership normalization (dynamic uid reuse), systemd ≥ 242 + nss-systemd capability probe; single-node incl. single-node MPI. **2026-10-12:** also owns the submission side of execution modes moved from application-profile (`sandbox` mode + options, job I/O-directory fields end to end, capability advertisement, scheduler placement by mode). Viability-checked 2026-09-29. Brief: `.kiro/specs/job-sandbox-execution/brief.md`. Dependencies: application-profile; remote-task-root fix (Direct-Implementation).
- [ ] job-sandbox-multinode — NEW 2026-09-29 (Job Sandbox Execution). Multi-node sandbox jobs: cluster-consistent ephemeral uid (expected: controller-allocated, WAL-replicated uid lease from a reserved range), worker-driven remote/MPI launch (no ssh; PMIx/prted via worker), shared-FS ownership normalization, one unit per job per node. Brief: `.kiro/specs/job-sandbox-multinode/brief.md`. Dependencies: job-sandbox-execution, application-profile. - [ ] job-sandbox-multinode — NEW 2026-09-29 (Job Sandbox Execution). Multi-node sandbox jobs: cluster-consistent ephemeral uid (expected: controller-allocated, WAL-replicated uid lease from a reserved range), worker-driven remote/MPI launch (no ssh; PMIx/prted via worker), shared-FS ownership normalization, one unit per job per node. Brief: `.kiro/specs/job-sandbox-multinode/brief.md`. Dependencies: job-sandbox-execution, application-profile.
- [ ] rbac-persistence — NEW 2026-09-29 (`/kiro-discovery`, raised during application-profile discovery). Closes the deferred `security.md` gap: runtime RBAC mutations (roles, permissions, user-role assignments, Logon auto-assignment) are in-memory only and silently revert to `rbac.yaml` on apiserver restart — including revocations. Durable-or-rejected mutations, audit trail, defaults-vs-runtime reconciliation, multi-apiserver consistency; storage approach (rbac.yaml write-through / controller-replicated entity / embedded store / read-only) decided in design. Brief: `.kiro/specs/rbac-persistence/brief.md`. Dependencies: none (independent of the sandbox initiative; application-profile's API-made `app:use` grants rely on it for durability). - [ ] rbac-persistence — NEW 2026-09-29 (`/kiro-discovery`, raised during application-profile discovery). Closes the deferred `security.md` gap: runtime RBAC mutations (roles, permissions, user-role assignments, Logon auto-assignment) are in-memory only and silently revert to `rbac.yaml` on apiserver restart — including revocations. Durable-or-rejected mutations, audit trail, defaults-vs-runtime reconciliation, multi-apiserver consistency; storage approach (rbac.yaml write-through / controller-replicated entity / embedded store / read-only) decided in design. Brief: `.kiro/specs/rbac-persistence/brief.md`. Dependencies: none (independent of the sandbox initiative; app-access-control's API-made `app:use` grants rely on it for durability).
- [ ] job-time-window — NEW 2026-09-29 (`/kiro-discovery`, Path C). Job begin time (scheduler gate in `sched_job_is_ready()` via `xce_core_now`, pend reason BEGIN_TIME_NOT_REACHED) and end time (WORKER enforces running jobs: helper SIGTERM→grace→SIGKILL, exit reason END_TIME_REACHED persisted, unit RuntimeMaxUSec backstop; PEND jobs whose end time can't be met stay PEND with END_TIME_UNMEETABLE until user kills/modifies — no controller sweep, primary not involved). Adds runtime estimate (`-We/--estimated_run_time`, scheduling hint only, never enforced; estimate > run limit rejected); unmeetable rule `now + (estimate ‖ run_limit ‖ 0) > end`. Times/estimate modifiable only while PEND (new `xce jobs modify` + harness `jobmod`). Both xcecli and harness `jobsub` gain all options (`-b`/`-t`/`-W`/estimate). Also fixes the silent drop of xcecli `-b`/`-t`/`-W` and REST `run_time_estimation` (apiserver `modelJobToSubmit` never sets them nor `exdata`). Brief: `.kiro/specs/job-time-window/brief.md`. Dependencies: none (new reason codes need text in job-status-detail-display). - [ ] job-time-window — NEW 2026-09-29 (`/kiro-discovery`, Path C). Job begin time (scheduler gate in `sched_job_is_ready()` via `xce_core_now`, pend reason BEGIN_TIME_NOT_REACHED) and end time (WORKER enforces running jobs: helper SIGTERM→grace→SIGKILL, exit reason END_TIME_REACHED persisted, unit RuntimeMaxUSec backstop; PEND jobs whose end time can't be met stay PEND with END_TIME_UNMEETABLE until user kills/modifies — no controller sweep, primary not involved). Adds runtime estimate (`-We/--estimated_run_time`, scheduling hint only, never enforced; estimate > run limit rejected); unmeetable rule `now + (estimate ‖ run_limit ‖ 0) > end`. Times/estimate modifiable only while PEND (new `xce jobs modify` + harness `jobmod`). Both xcecli and harness `jobsub` gain all options (`-b`/`-t`/`-W`/estimate). Also fixes the silent drop of xcecli `-b`/`-t`/`-W` and REST `run_time_estimation` (apiserver `modelJobToSubmit` never sets them nor `exdata`). Brief: `.kiro/specs/job-time-window/brief.md`. Dependencies: none (new reason codes need text in job-status-detail-display).
- [ ] job-dependency — NEW 2026-09-29 (`/kiro-discovery`, Path D; ref LSF 10.1 Admin Guide "Job dependency and job priority" pp. 459-465 — dependency only, priority deferred). `-w` expressions by job ID: done/ended/exit(id[,op code])/started/post_done/post_err (PDONE/PERR) + bare id, `&& || !` (). Shared libxce grammar (Go via cgo); apiserver early validation (syntax, parent exists, submitter has `job:view`); controller authoritative re-validation, stores expression (additive JobSubmitData field) + in-memory reverse index rebuilt on replay (implements controller-job-cleanup's `job_has_dependents()`); scheduler evaluates in `sched_job_is_ready()` (pend reasons DEP_NOT_SATISFIED / DEP_NEVER_SATISFIED; orphans stay PEND). Fixes scheduler exit-code field-completeness gap (xce_sched_job_t/SchedJob/snapshot lack exit code). xcecli `-w`, `modify -w/-wn`; harness `jobsub -w`/`jobmod`. Brief: `.kiro/specs/job-dependency/brief.md`. Dependencies: job-time-window (shared gate + modify path) — coordinate. - [ ] job-dependency — NEW 2026-09-29 (`/kiro-discovery`, Path D; ref LSF 10.1 Admin Guide "Job dependency and job priority" pp. 459-465 — dependency only, priority deferred). `-w` expressions by job ID: done/ended/exit(id[,op code])/started/post_done/post_err (PDONE/PERR) + bare id, `&& || !` (). Shared libxce grammar (Go via cgo); apiserver early validation (syntax, parent exists, submitter has `job:view`); controller authoritative re-validation, stores expression (additive JobSubmitData field) + in-memory reverse index rebuilt on replay (implements controller-job-cleanup's `job_has_dependents()`); scheduler evaluates in `sched_job_is_ready()` (pend reasons DEP_NOT_SATISFIED / DEP_NEVER_SATISFIED; orphans stay PEND). Fixes scheduler exit-code field-completeness gap (xce_sched_job_t/SchedJob/snapshot lack exit code). xcecli `-w`, `modify -w/-wn`; harness `jobsub -w`/`jobmod`. Brief: `.kiro/specs/job-dependency/brief.md`. Dependencies: job-time-window (shared gate + modify path) — coordinate.
- [ ] job-dependency-insight — NEW 2026-09-29. `bjdepinfo`-style query (parents/children/-p blocking/-r levels/-l verdicts, RBAC-filtered) via REST/xcecli/harness, dependency info in job details, optional orphan termination (per-job `-ti` + cluster grace period, default off; EXIT reason ORPHAN; failover-safe persisted grace start; minimal primary periodic work). Brief: `.kiro/specs/job-dependency-insight/brief.md`. Dependencies: job-dependency, job-time-window (exit-reason field). - [ ] job-dependency-insight — NEW 2026-09-29. `bjdepinfo`-style query (parents/children/-p blocking/-r levels/-l verdicts, RBAC-filtered) via REST/xcecli/harness, dependency info in job details, optional orphan termination (per-job `-ti` + cluster grace period, default off; EXIT reason ORPHAN; failover-safe persisted grace start; minimal primary periodic work). Brief: `.kiro/specs/job-dependency-insight/brief.md`. Dependencies: job-dependency, job-time-window (exit-reason field).
@ -122,6 +122,7 @@ Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generi
- [ ] project-guarantee — NEW 2026-10-09 (configuration layering, backlog). Project-owned guarantees (resource amount or host pool), borrowing of idle guaranteed capacity, reclaim through preemption. Brief: `.kiro/specs/project-guarantee/brief.md`. Dependencies: project-config, scheduler-preemption. - [ ] project-guarantee — NEW 2026-10-09 (configuration layering, backlog). Project-owned guarantees (resource amount or host pool), borrowing of idle guaranteed capacity, reclaim through preemption. Brief: `.kiro/specs/project-guarantee/brief.md`. Dependencies: project-config, scheduler-preemption.
- [ ] job-resize — NEW 2026-10-10 (from long-running services discovery; not on the services path). Grow/shrink a running parallel job's allocation (LSF resizable jobs, `bresize`) for elastic applications. Brief: `.kiro/specs/job-resize/brief.md`. Dependencies: parjob-placement, host-failure-job-recovery. - [ ] job-resize — NEW 2026-10-10 (from long-running services discovery; not on the services path). Grow/shrink a running parallel job's allocation (LSF resizable jobs, `bresize`) for elastic applications. Brief: `.kiro/specs/job-resize/brief.md`. Dependencies: parjob-placement, host-failure-job-recovery.
- [ ] project-delegated-membership — NEW 2026-10-10 (from project-config requirements, user decision: future). Project managers edit their project's (and subtree's) members at runtime without an admin config push; needs a durable store that coexists with `xceadmin` config pushes (e.g. a separate delegated member list) and audit. Dependencies: project-config, rbac-persistence. - [ ] project-delegated-membership — NEW 2026-10-10 (from project-config requirements, user decision: future). Project managers edit their project's (and subtree's) members at runtime without an admin config push; needs a durable store that coexists with `xceadmin` config pushes (e.g. a separate delegated member list) and audit. Dependencies: project-config, rbac-persistence.
- [ ] app-access-control — NEW 2026-10-12 (deferred from application-profile, user decision; backlog, design kept in the application-profile brief). Per-profile RBAC: `app:use` / `app:view` with an `app` scope type (profile names or `all`), checks on job create, restart and listing, `rbac.yaml` defaults, the current default profile always usable with `queue:submit`, aliases checked against the target profile, role API support, `cluster_admin` bypass only. Dependencies: application-profile; rbac-persistence for durable API-made grants.
- [ ] job-modify — NEW 2026-10-12 (from application-profile discussion, user decision; backlog, no brief yet). Modify a pending job (LSF `bmod`): apiserver action (today `"modify"` is commented out, `controllers/jobs.go:2446`; `JOB_MODIFY` only an enum), controller opcode, WAL fact with snapshot field-completeness, scheduler re-merge, xcecli. First attributes: `--app` (owner with `app:use` on the new profile; admins in bulk, e.g. `--app-from <old>`), `-R`, `-n`, queue; later begin/end time (`job-time-window`), dependencies. Dependencies: application-profile, job-config-merge. - [ ] job-modify — NEW 2026-10-12 (from application-profile discussion, user decision; backlog, no brief yet). Modify a pending job (LSF `bmod`): apiserver action (today `"modify"` is commented out, `controllers/jobs.go:2446`; `JOB_MODIFY` only an enum), controller opcode, WAL fact with snapshot field-completeness, scheduler re-merge, xcecli. First attributes: `--app` (owner with `app:use` on the new profile; admins in bulk, e.g. `--app-from <old>`), `-R`, `-n`, queue; later begin/end time (`job-time-window`), dependencies. Dependencies: application-profile, job-config-merge.
- [ ] config-history-audit — NEW 2026-10-12 (from application-profile discussion, user decision: needed, roadmap). Keep past cluster-config versions (including `applications.yaml` profiles) with who changed what and when: `xceadmin config history` / `diff`, look up a profile by its content hash (the application-profile hash recorded on dispatch records), audit trail of live upgrades. Per-job display of the values a run used is already covered by the application-profile dispatch record; this adds admin history and audit. Covers the whole cluster config, not only profiles (the controller already tracks `configVersion`/`configChecksum`). Dependencies: application-profile, admin-config-tool. - [ ] config-history-audit — NEW 2026-10-12 (from application-profile discussion, user decision: needed, roadmap). Keep past cluster-config versions (including `applications.yaml` profiles) with who changed what and when: `xceadmin config history` / `diff`, look up a profile by its content hash (the application-profile hash recorded on dispatch records), audit trail of live upgrades. Per-job display of the values a run used is already covered by the application-profile dispatch record; this adds admin history and audit. Covers the whole cluster config, not only profiles (the controller already tracks `configVersion`/`configChecksum`). Dependencies: application-profile, admin-config-tool.
- [ ] job-labels — NEW 2026-10-12 (from application-profile discussion; backlog, no brief yet). General Kubernetes-style job labels: `--label key=value` (repeatable, limited charset and size), an additive submit-data field through WAL and snapshot (field completeness), server-side filtering (`jobs list -l k=v`) and summaries grouped by label (runtime, peak memory, efficiency); metadata only, never used for placement. Reuses or replaces the REST `tags` field that is silently dropped today (bug-1970). Not needed for presets (those use `exdata.meta`); candidates: team-wide comparison, services labelling replicas. Dependencies: none. - [ ] job-labels — NEW 2026-10-12 (from application-profile discussion; backlog, no brief yet). General Kubernetes-style job labels: `--label key=value` (repeatable, limited charset and size), an additive submit-data field through WAL and snapshot (field completeness), server-side filtering (`jobs list -l k=v`) and summaries grouped by label (runtime, peak memory, efficiency); metadata only, never used for placement. Reuses or replaces the REST `tags` field that is silently dropped today (bug-1970). Not needed for presets (those use `exdata.meta`); candidates: team-wide comparison, services labelling replicas. Dependencies: none.