docs(spec): discovery briefs for job status detail, job cleanup, sandbox execution and RBAC persistence
New specs from /kiro-discovery (2026-09-28/29), added to roadmap.md: - job-status-detail-display: show pending reasons and reserved-vs-used resources in REST and xcecli. Requirements generated (7 requirements). - scheduler-alter-pend-reasons: keep a pending reason for every alternative resreq (per-alternative JobPendReason) and fix the obj->reason leak. - controller-job-cleanup: primary-only sweep that removes finished jobs past a configurable retention, via a replicated WAL removal fact. - application-profile: admin-defined application profiles (execution mode, launch type, env, resource defaults) chosen per job, with app:use/app:view RBAC. Queues and profiles stay orthogonal. - job-sandbox-execution / job-sandbox-multinode: run jobs in a systemd DynamicUser sandbox behind a pluggable execution backend. - rbac-persistence: make RBAC changes made through the API survive an apiserver restart (closes the security.md gap). roadmap.md also records the scheduler-sim-diagnostics info job update and a security item: remote-task units appear to run as root (found by code reading). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
b283f3b93c
commit
7fe9b22381
GPG key ID:
80533536B0EB651C
9 changed files with 1318 additions and 0 deletions
237
.kiro/specs/application-profile/brief.md
Normal file
237
.kiro/specs/application-profile/brief.md
Normal file
|
|
@ -0,0 +1,237 @@
|
|||
# Brief: application-profile
|
||||
|
||||
(Reshaped from `job-execution-mode` on 2026-09-29, by user direction. Execution
|
||||
mode is an attribute of the job, determined by the **application** the user runs,
|
||||
so it is carried by a named application profile rather than by a bare mode
|
||||
option.)
|
||||
|
||||
## Problem
|
||||
XCE users run many kinds of applications: trusted in-house codes, untrusted
|
||||
scripts, vendor binaries, MPI solvers. Each needs a different way of running:
|
||||
- directly as the submitter, or jailed in a sandbox under a separate execution
|
||||
identity (and later in a container or vm);
|
||||
- a particular launch type (serial, `openmpi`, `mpich`);
|
||||
- particular environment and pre/post-exec steps;
|
||||
- sensible resource defaults and limits.
|
||||
|
||||
Today every one of these must be set per job by hand, and several (execution
|
||||
mode, sandbox options) cannot be set at all. Many users log in through the
|
||||
portal or REST API (JWT; `config_file`/PAM, later LDAP or OAuth2) and have **no
|
||||
Unix account on the compute nodes**. They can only run jobs if their
|
||||
applications run sandboxed. Admins have no way to say "this application runs
|
||||
like this, and these users may run it".
|
||||
|
||||
This spec introduces **application profiles** and is the submission/policy half
|
||||
of the job-sandbox initiative (roadmap "Job Sandbox Execution"). There is
|
||||
precedent in LSF application profiles (`lsb.applications`, `bsub -app`), which
|
||||
fits XCE's LSF-style job model.
|
||||
|
||||
## Current State
|
||||
- **Identity.** A job's identity is only the JWT `username`
|
||||
(`controllers/auth.go:26-37`). `SubmitUser` comes from the token
|
||||
(`controllers/jobs.go:1379`, `:1459`), and `UserGroup` is taken from the client
|
||||
request unchecked (`:1465`).
|
||||
- **Execution user.** It is always the submitter: `resolveExecUser`
|
||||
(`backend/convert_pure.go:162`, bug-1780).
|
||||
- **Per-job settings that exist today** (`src/include/xce/types.h`):
|
||||
- `job_startinfo_t` (:240-251): `execuser`, `exechome`, `execcwd`, `cmdline`,
|
||||
`exdata`, `launch_type`, and pre/post-exec commands.
|
||||
- `exdata` carries the environment (`sub_env_vars`) and the worker cgroup
|
||||
limits parsed by `parse_limits` (MemoryMax, CPUQuota, TasksMax, RuntimeMax,
|
||||
...). The apiserver never sends exdata (`backend/convert.go:273-315`).
|
||||
- The resource requirement is `resreq` on the submit data.
|
||||
- Remote tasks use `xce_taskhost_t` (:696-705).
|
||||
- **Missing today:**
|
||||
- any profile concept, any execution-mode field, any input/output directory
|
||||
field;
|
||||
- any way for workers to advertise execution capabilities to the scheduler.
|
||||
- **Controller.** It validates only that the user is not empty
|
||||
(`controller/job.c:2447`). The submit data, including `start_info`, is
|
||||
persisted in the WAL submit fact and in snapshots.
|
||||
- **Ownership.** RBAC ownership, scheduler per-user limits and the user table all
|
||||
key on the **submit user**, not the exec user (`controllers/jobs.go:1010`,
|
||||
`:2542`, `output.go:95`; `schedobj.c:121`; `plugins/limit.c`). So a separate
|
||||
execution identity breaks none of them.
|
||||
- **Access control.** RBAC is `config/rbac.yaml` plus `RequirePermission`
|
||||
(`controllers/rbac_check.go`).
|
||||
|
||||
## Desired Outcome
|
||||
- **Admins define named application profiles** in the cluster configuration. The
|
||||
first version contains the following (user decision; the format must be
|
||||
extensible for "maybe others"):
|
||||
1. **Execution mode and its options.** `direct` or `sandbox` now, with room for
|
||||
`container`/`vm`. Sandbox options include allowed and default input/output
|
||||
mounts and isolation extras.
|
||||
2. **Launch type.** Serial, `openmpi` or `mpich`.
|
||||
3. **Environment and pre/post-exec.** Default environment variables and
|
||||
pre/post-exec commands.
|
||||
4. **Resource defaults and limits.** A default resource requirement, plus
|
||||
per-job execution limits (memory, runtime, CPU, tasks), mapped onto the
|
||||
existing worker cgroup limits.
|
||||
- **Users pick a profile at submit** (REST, `xcecli jobs submit --app <name>`,
|
||||
harness, libxce). A **cluster default profile** applies when none is given.
|
||||
- **Limited overrides.** A job may override only the attributes its profile
|
||||
marks as overridable (for example its own input/output directories within the
|
||||
allowed prefixes, or env additions). Anything else is rejected with a clear
|
||||
error.
|
||||
- **Access by RBAC permission.** No new access mechanism; this follows the
|
||||
existing named-instance pattern (user decision, 2026-09-29: fold profile RBAC
|
||||
into this spec). Today, `queue:submit` scoped to `type: queue`,
|
||||
`values: [normal, priority]` in `config/rbac.yaml` is enforced by
|
||||
`GetRequireResourcePermission` in `controllers/jobs.go:1402-1409`. Profiles add:
|
||||
- **Permissions:** `app:use` (submit a job with a profile) and `app:view` (list
|
||||
and inspect profiles).
|
||||
- **Scope:** a new `app` scope type whose values are profile names, or `all`.
|
||||
- **Checks:**
|
||||
- Job create checks `app:use` alongside, and independent of, `queue:submit`.
|
||||
- Job restart (`POST /:jobids/restart`, which today checks only
|
||||
`queue:submit`) re-checks `app:use` for the job's profile.
|
||||
- Listing profiles checks `app:view`.
|
||||
- **Defaults:** `rbac.yaml` gets defaults, for example `basic_user` →
|
||||
`app:use` on the default profile, and `app:view` on `all`.
|
||||
- **Role API:** role creation and update accept the new scope type.
|
||||
- **Bypass:** `cluster_admin` keeps the single existing bypass; no new bypass
|
||||
path.
|
||||
- **Persistence:** API-made grants are subject to the `rbac-persistence` gap
|
||||
until that spec lands.
|
||||
- **Queues and profiles are orthogonal** (user decision, 2026-09-29).
|
||||
- **Queues** own scheduling policy: which jobs are considered and in what
|
||||
order, priority, scheduling limits, and host partitioning.
|
||||
- **Application profiles** own application-related configuration: how the
|
||||
application is executed and what it needs to run.
|
||||
- A queue does not list or restrict profiles, and a profile does not select or
|
||||
constrain queues.
|
||||
- A profile carries no scheduling-policy attributes (no priority, queue choice,
|
||||
fairshare or preemption).
|
||||
- Any combination of an allowed queue and an allowed profile is valid. Each is
|
||||
governed by its own access rules; for profiles, that is RBAC permission.
|
||||
- Resource defaults in a profile describe what the application needs (its
|
||||
default resource requirement and per-job execution limits). They are not
|
||||
scheduling policy. Queue-level scheduling limits keep applying on top,
|
||||
unchanged.
|
||||
- **Resolution at submit time.** The effective values are stored on the job
|
||||
(mode, options, launch type, env, limits, resreq). Editing a profile later does
|
||||
not change queued or running jobs. The job records its profile name, and its
|
||||
details show the effective mode.
|
||||
- **The apiserver never checks node accounts** (user decision, 2026-09-29).
|
||||
- Submit-time validation covers profile existence, RBAC permission, overrides
|
||||
and paths only.
|
||||
- The execution account is the worker's concern. If a `direct` job's execution
|
||||
user doesn't exist on the node, the worker fails the job at launch with a
|
||||
clear, user-visible reason, instead of today's opaque systemd failure.
|
||||
- Account-less users are served by profiles whose mode is `sandbox`.
|
||||
- **Placement.** A job is placed only on workers whose advertised capabilities
|
||||
support its effective mode (sandbox needs systemd 242 or later and
|
||||
`nss-systemd`).
|
||||
- **Ownership.** The submitter remains the owner for RBAC, limits and accounting
|
||||
in every mode.
|
||||
|
||||
## Approach
|
||||
- **Profile definitions** live in the cluster configuration: schema, YAML, JSON
|
||||
schema, xceadmin mapping, and live-upgrade replication, following the
|
||||
`config-design.md` placement rules.
|
||||
- **The controller resolves the profile at submit** and merges it with the job's
|
||||
permitted overrides. This makes it authoritative for every submit path (REST,
|
||||
libxce, harness).
|
||||
- **The apiserver checks that the named profile exists** (user direction,
|
||||
2026-09-29).
|
||||
- At submit, before anything goes downstream (`preCheckJobSubmitRequest`), the
|
||||
apiserver rejects a job that names a profile which does not exist, with a
|
||||
clear client error naming the unknown profile.
|
||||
- The apiserver already receives the cluster configuration from the controller
|
||||
(`backend/convert.go:265`), so it knows the current set of profiles and keeps
|
||||
it current across live config upgrades.
|
||||
- The controller still re-validates when it resolves the profile, because it is
|
||||
authoritative for non-REST submit paths and for a profile removed between the
|
||||
two checks. If the profile has gone by then, the job is rejected, never run
|
||||
with a fallback profile.
|
||||
- **The apiserver additionally enforces the RBAC permission** for the profile
|
||||
(middleware chain unchanged; a `RequirePermission`-style check) and validates
|
||||
overrides early (`preCheckJobSubmitRequest`).
|
||||
- **Effective values go into the job's existing fields where they exist**
|
||||
(`launch_type`, pre/post exec, exdata env and limits, resreq). New fields are
|
||||
added only for the profile name, execution mode, mode options and I/O
|
||||
directories. All wire changes are additive Cap'n Proto fields, and the WAL
|
||||
submit fact and snapshot carry them with field completeness.
|
||||
- **Workers advertise their supported execution modes.** The scheduler filters
|
||||
hosts by the job's effective mode. The design picks the mechanism, for example
|
||||
a host attribute or boolean resource.
|
||||
|
||||
## Scope
|
||||
- **In**:
|
||||
- The profile schema and config, and a default profile.
|
||||
- Resolution and override rules.
|
||||
- RBAC permission for profile use.
|
||||
- Submit surfaces: REST, `xcecli --app`, harness, libxce.
|
||||
- Effective values persisted on the job.
|
||||
- Execution-mode and I/O-directory fields end to end.
|
||||
- Capability advertisement and placement.
|
||||
- A worker launch error when a `direct` exec user is missing.
|
||||
- Profile name and mode shown in job details.
|
||||
- Listing available profiles (REST and xcecli).
|
||||
- Tests.
|
||||
- **Open for requirements**:
|
||||
- The exact per-attribute override rules.
|
||||
- Which extra attributes go into v1 beyond the four ("maybe others"). Examples:
|
||||
requeue exit values, runtime estimate, job starter or wrapper, output path
|
||||
template, checkpoint settings.
|
||||
- How resource defaults combine with a job's own resreq (replace, or fill in
|
||||
what's missing).
|
||||
- Profile versioning and audit.
|
||||
- Whether the profile can be changed for a pending job.
|
||||
- **Out**:
|
||||
- The worker sandbox mechanics (`job-sandbox-execution`).
|
||||
- Multi-node sandbox (`job-sandbox-multinode`).
|
||||
- Container or vm execution itself (only room for them in the schema).
|
||||
- Auth backends and JWT claims.
|
||||
|
||||
## Boundary Candidates
|
||||
- The profile model and config (schema, validation, live upgrade).
|
||||
- Resolution and override engine (controller).
|
||||
- Access control (RBAC permission) and early validation (apiserver).
|
||||
- Submit and listing surfaces (REST, xcecli, harness, libxce).
|
||||
- Persisting effective values (wire, WAL submit fact, snapshot).
|
||||
- Capability advertisement and scheduler placement.
|
||||
|
||||
## Out of Boundary
|
||||
- Worker execution backends and sandbox mechanics.
|
||||
- Output file location and retrieval inside the sandbox.
|
||||
- Scheduler policy beyond capability filtering and resreq defaults.
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream**:
|
||||
- The existing submit path.
|
||||
- The WAL submit fact (`ha-wal-replication`, the `raft-apply-decoupling` apply
|
||||
seam).
|
||||
- The config pipeline (`clusterconfig-schema-split`, `config-upgrade-snapshot`,
|
||||
`admin-config-tool`).
|
||||
- RBAC (`rbac_check.go`).
|
||||
- Resource/attribute reporting from worker to scheduler.
|
||||
- **Downstream**:
|
||||
- `job-sandbox-execution` consumes the effective mode, options and directories.
|
||||
- `job-sandbox-multinode`.
|
||||
- Future `container`/`vm` modes, which add profile attributes.
|
||||
- `job-status-detail-display`, which can show the profile and mode.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends**: none directly.
|
||||
- **Adjacent**:
|
||||
- `job-status-detail-display`
|
||||
- `controller-job-cleanup`
|
||||
- `resreq-maxtasks-wiring` (resreq defaults)
|
||||
- `xce-node-manager`
|
||||
|
||||
## Constraints
|
||||
- Middleware chain unchanged: WhitelistURL → WhitelistIP → RateLimit → JWT → CORS
|
||||
→ RBAC.
|
||||
- All wire changes are additive Cap'n Proto. Old WAL records and snapshots
|
||||
decode as "no profile, mode `direct`", which is today's behaviour.
|
||||
- Save/restore field completeness (`persistence.md`): the effective profile
|
||||
values round trip through the submit facts and snapshots. Run
|
||||
`/xce-audit-invariants`.
|
||||
- Input validation (`security.md`): overrides and paths are validated
|
||||
server-side. No path traversal, and mount prefixes are restricted by the
|
||||
profile.
|
||||
- Size: this is likely 20 or more tasks. If requirements confirm that, consider
|
||||
splitting "capability advertisement and placement" into its own spec.
|
||||
- HA/persistence spec: use the full 3-phase workflow.
|
||||
211
.kiro/specs/controller-job-cleanup/brief.md
Normal file
211
.kiro/specs/controller-job-cleanup/brief.md
Normal file
|
|
@ -0,0 +1,211 @@
|
|||
# Brief: controller-job-cleanup
|
||||
|
||||
## Problem
|
||||
Operators of long-running XCE clusters have no way to get rid of finished jobs.
|
||||
- The controller keeps every job it has ever accepted in memory, forever. So do
|
||||
the scheduler (`core->jobs`) and the apiserver store.
|
||||
- Memory use, full dataset pushes (scheduler join, apiserver connect), and
|
||||
`xcecli jobs list` output all grow without bound.
|
||||
- A hidden, hard-coded one-hour rule drops finished jobs only from **snapshots**,
|
||||
not from memory. Nodes therefore disagree about which jobs exist:
|
||||
- a restarted node holds a different job set than one that never restarted;
|
||||
- a standby that installs a snapshot keeps jobs the snapshot omits.
|
||||
|
||||
## Current State
|
||||
Codebase exploration, 2026-09-29.
|
||||
|
||||
**Job table**
|
||||
- Jobs live in `c->jobmgr`, an `xkvmgr` table with integer keys
|
||||
(`controller/cluster.c:1668-1685`). It has **no delete operation**: `.del_key =
|
||||
NULL` (`src/lib/kvmgr.c:289`, bug-1443).
|
||||
- The underlying pieces exist: `kvtab_del` (`kv.c:288`) and `vclear` (the rollback
|
||||
path at `kvmgr.c:107-128`).
|
||||
- Jobs are reference-counted RTTI objects, and `job_obj_killer`
|
||||
(`cluster.c:1591`) does nothing.
|
||||
|
||||
**WAL**
|
||||
- There is no "job removed" entry type (`include/wal_types.h:132-310`).
|
||||
Numbers 13-19 are free.
|
||||
- A new type has to be handled in:
|
||||
- both apply paths, `apply_wal_entry` (`persist.c:4195`) and
|
||||
`apply_committed_entry` (`persist.c:4607`), with identical decoding
|
||||
(`persistence.md:68`);
|
||||
- `wal_classify()` (`persist.c:7395`, where an unclassified type asserts under
|
||||
XCE_TESTING);
|
||||
- the replay verdict. `persist_apply_verdict` (`persist.c:6660-6690`) treats a
|
||||
shrinking job table as DIVERGENT, and `apply_expectation_t` (`persist.h:162`)
|
||||
has to count removals.
|
||||
- Fact append pattern to copy: `persist_submit_job_fact_ex` (`persist.c:8105`) →
|
||||
`wal_hook_append_classified(..., WAL_MODE_RETAIN, ...)`.
|
||||
|
||||
**The existing snapshot-only retention rule**
|
||||
- `job_should_retain()` (`persist.c:8695-8714`) keeps a terminal (EXIT|DONE|PERR)
|
||||
job until `finish_time + finished_retain_s` has passed.
|
||||
- `finished_retain_s` defaults to `DEFAULT_FINISHED_RETAIN_S=3600`
|
||||
(`include/config.h:91`) and cannot be configured.
|
||||
- Its only caller is `collect_job_frames` (`persist.c:2513`), which runs at
|
||||
snapshot time, per node, using that node's `time(NULL)`.
|
||||
- The same rule also expires TRANSIENT jobs after `transient_job_timeout_s`.
|
||||
- `load_snapshot_entities` (`persist.c:3827`) only adds jobs and never clears the
|
||||
table.
|
||||
|
||||
**Terminal states**
|
||||
- `JOB_IS_FINISH` = EXIT || DONE (`include/xce/types.h:425-429`).
|
||||
- Two definitions of "terminal" disagree: `job.c:45` says DONE|EXIT, and
|
||||
`persist.c:8690` says EXIT|DONE|PERR.
|
||||
- `finish_time` is set live (`job.c:2756`), carried in the WAL_JOB_FINISH payload,
|
||||
and restored on apply and resync.
|
||||
|
||||
**Periodic, primary-only work**
|
||||
- Pattern: `ha_start_primary_heartbeat` (`ha.c:10142`) uses `delay_task_ms(...,
|
||||
-1, fn, c)`, and the callback does nothing unless `role == CTRL_PRIMARY`.
|
||||
- Offload pattern: `checkpoint_task_fn` (`persist.c:3545-3592`) runs on the
|
||||
delaymgr thread and calls `slowlane_submit(...)` (`slowlane.c:185`).
|
||||
- delaymgr is trigger-only (`ha-async-io.md`).
|
||||
|
||||
**Cluster config**
|
||||
- Schema `ClusterConfig` (`schema/controller.capnp:850-869`); the next free ordinal
|
||||
is `@15`.
|
||||
- C struct: `cluster_config_t` (`include/clusterconfig.h`).
|
||||
- YAML and validation: `configs/xce.yaml` and `configs/schema/xce.schema.json`
|
||||
(`additionalProperties:false`).
|
||||
- xceadmin: `_XCE_SCALAR_FIELDS` (`xceadmin/src/xcecommon/mapping.py:176`).
|
||||
- Defaults: `myconf.c:65-76`.
|
||||
- Live config upgrade swaps `myconf->cluster_config` (`config_repl.c:747`), so
|
||||
read the value on every sweep.
|
||||
- Model to follow: `worker_resync_timeout_ms`.
|
||||
|
||||
**Downstream**
|
||||
- Scheduler: no job-removed hook detail exists (`include/xce/cluster_def.h:43-67`),
|
||||
and `core->jobs` never deletes.
|
||||
- Apiserver: `OnJobDeleted` (`backend/notification.go:11`, `handler.go:83`) exists
|
||||
but nothing calls it. Reconnect never prunes the store.
|
||||
- libxce: querying a missing jobid returns 0 jobs with status 0
|
||||
(`opget.c:159-176`, `xceapi.c:787`). Control operations return `-ENOENT`.
|
||||
- Worker resync ignores unknown jobids (`resync.c:638-647`), which is safe.
|
||||
- Array jobs: there is no parent object. Each element is an independent job with
|
||||
a jobid in `[base, base+array_size)`.
|
||||
- There is no accounting or history file, so a removed job leaves no record.
|
||||
|
||||
**Job dependencies**
|
||||
- None exist (no field in `job_submit_data_t`). The user asked for a stub.
|
||||
|
||||
## Desired Outcome
|
||||
- Finished jobs (DONE or EXIT) are removed permanently once they have been
|
||||
finished longer than a configured retention period and no other job depends on
|
||||
them.
|
||||
- Every controller node removes the same jobs at the same point in the log.
|
||||
Removal survives replay, restart and failover, and snapshots match memory.
|
||||
- The scheduler and apiserver drop removed jobs too, so a removed job disappears
|
||||
from `xcecli` and from REST.
|
||||
- Both periods can be configured in the cluster configuration, and cleanup can be
|
||||
turned off.
|
||||
|
||||
## Approach
|
||||
**Chosen: approach A, a replicated removal record** (user decision, 2026-09-29).
|
||||
|
||||
1. **Trigger (primary only).** A delaymgr timer fires every
|
||||
`job_cleanup_period`. The callback returns unless this node is PRIMARY, and
|
||||
otherwise hands the sweep to the slow lane. It never blocks.
|
||||
2. **Sweep (slow lane).** Walk `jobmgr` under a read lock and pick jobs that meet
|
||||
all of these:
|
||||
- state is DONE or EXIT, not TRANSIENT;
|
||||
- `finish_time + job_retention < now`;
|
||||
- the dependency stub (`job_has_dependents(job)`, which always returns false
|
||||
for now) lets it go.
|
||||
|
||||
Append the chosen jobids as new WAL job-removal facts (RETAIN), batched and
|
||||
bounded per sweep.
|
||||
3. **Apply (every node, both apply paths).** Remove the job from `jobmgr` through
|
||||
a new `xkvmgr` delete operation that drops the table's reference; in-flight
|
||||
holders keep the job alive until they release it. Update the replay-verdict
|
||||
expectations so that removals are counted.
|
||||
4. **Propagate.** Tell the scheduler to drop the job from `core->jobs` (a new
|
||||
hook detail or event), and tell the apiserver (wire up the existing
|
||||
`OnJobDeleted`).
|
||||
5. **Retire the snapshot-only filter** for finished jobs, so a snapshot contains
|
||||
exactly the jobs in memory. The requirements decide what happens to the
|
||||
TRANSIENT-timeout part of `job_should_retain` (keep it separate or fold it in).
|
||||
|
||||
Rejected alternatives:
|
||||
- **B, each node cleans up locally without replication.** Clock skew makes nodes
|
||||
disagree, and the replay verdict flags a shrinking table as DIVERGENT.
|
||||
- **C, extend the snapshot filter.** Still decided per node, and makes the
|
||||
existing drift permanent.
|
||||
|
||||
## Scope
|
||||
- **In**:
|
||||
- Two cluster configuration parameters, both configurable (names are
|
||||
provisional):
|
||||
- `job_retention`: how long a finished job is kept. Default 3600 s, matching
|
||||
today's snapshot rule. 0 turns cleanup off, and jobs are kept forever.
|
||||
- `job_cleanup_period`: the sweep interval. Default 300 s.
|
||||
- Both need schema, YAML, JSON-schema validation, xceadmin mapping, defaults
|
||||
and live-upgrade pickup.
|
||||
- The primary-only periodic sweep.
|
||||
- The dependency-check stub.
|
||||
- The new WAL removal fact, handled on both apply paths and in classification,
|
||||
with replay-verdict accounting.
|
||||
- The `xkvmgr` delete operation, safe with reference counts.
|
||||
- Removal on every node.
|
||||
- Scheduler and apiserver propagation.
|
||||
- Retiring the snapshot-only filter for finished jobs, so snapshot and memory
|
||||
agree.
|
||||
- Tests: unit, persistence round-trip, and live HA.
|
||||
- **Out**:
|
||||
- A real job-dependency feature. Only the stub is in scope.
|
||||
- Accounting or job-history archiving before removal.
|
||||
- Manual per-job purge commands.
|
||||
- Cleaning up PEND, RUN or other non-terminal jobs.
|
||||
- The lost-and-found drain (`roadmap.md`).
|
||||
|
||||
## Boundary Candidates
|
||||
- libxce `xkvmgr` delete primitive (reference-count lifetime).
|
||||
- Controller cleanup policy: config, sweep, dependency stub.
|
||||
- Persistence: removal fact, both apply paths, classification, replay verdict,
|
||||
snapshot alignment.
|
||||
- Propagation: scheduler removal event, apiserver `OnJobDeleted`.
|
||||
|
||||
## Out of Boundary
|
||||
- The job-dependency model and its submit option.
|
||||
- Job history or accounting storage.
|
||||
- How jobs are displayed (`job-status-detail-display`).
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream**:
|
||||
- The WAL and replication machinery (`ha-wal-replication`,
|
||||
`raft-apply-decoupling`'s `apply_committed_entry` seam).
|
||||
- `ha-entity-snapshots` (where `job_should_retain` lives).
|
||||
- The config pipeline (`clusterconfig-schema-split`, `config-upgrade-snapshot`,
|
||||
`admin-config-tool`).
|
||||
- **Downstream**:
|
||||
- A future job-dependency feature, which replaces the stub.
|
||||
- A future accounting or history feature, which archives before removal.
|
||||
- `job-status-detail-display` (a removed job simply stops appearing).
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends**: `ha-entity-snapshots`. Its finished-job retention filter is retired
|
||||
or replaced; note this in that spec's history.
|
||||
- **Adjacent**:
|
||||
- `worker-rejoin-resync` (unknown jobids on resync)
|
||||
- `job-status-detail-display`
|
||||
- `scheduler-core` (its job table)
|
||||
|
||||
## Constraints
|
||||
- HA/persistence spec: use the full 3-phase workflow, not `/kiro-spec-quick`.
|
||||
- Only the primary decides a removal, and every removal goes through the
|
||||
replicated WAL. No node removes a job on its own.
|
||||
- The delaymgr callback is trigger-only. The sweep runs on the slow lane.
|
||||
- Save/restore field-completeness (`persistence.md`):
|
||||
- the removal fact decodes identically on both apply paths;
|
||||
- snapshot and memory agree after removal;
|
||||
- run `/xce-audit-invariants controller` (dimension 7).
|
||||
- Wire and schema changes are additive Cap'n Proto only. Old WAL and snapshots
|
||||
must still load. The requirements must also cover what happens when a node
|
||||
without this feature meets the new fact (`WAL_ENTRY_TYPE_MAX` skip-with-WARN).
|
||||
- Removal must not free a job that another thread still references.
|
||||
- Sweeps must be bounded, so a large backlog does not create one giant WAL entry
|
||||
or stall the slow lane.
|
||||
- Known edge case: a very late resubmit retry of a removed job's jobid is
|
||||
accepted as a new job, because dedup checks `jobmgr` presence. Record it as
|
||||
accepted or mitigated.
|
||||
202
.kiro/specs/job-sandbox-execution/brief.md
Normal file
202
.kiro/specs/job-sandbox-execution/brief.md
Normal file
|
|
@ -0,0 +1,202 @@
|
|||
# Brief: job-sandbox-execution
|
||||
|
||||
## Problem
|
||||
A job whose execution mode is `sandbox` (from `application-profile`) must run on a
|
||||
worker node:
|
||||
- without any account belonging to the submitter;
|
||||
- under an unprivileged, per-job, ephemeral uid;
|
||||
- isolated from the host and from other jobs, including other jobs by the same
|
||||
submitter;
|
||||
- with its input and output directories, or a temporary workspace, mounted in.
|
||||
|
||||
Today the worker can only run a job as a named account. There is no isolation,
|
||||
and a jailed process could not even get its output back to the user.
|
||||
|
||||
This is the worker half of the job-sandbox initiative. It covers single-node
|
||||
execution, including single-node MPI in one unit. Multi-node is handled by
|
||||
`job-sandbox-multinode`.
|
||||
|
||||
## Current State
|
||||
Code exploration, 2026-09-29.
|
||||
|
||||
**Launch**
|
||||
- `xcecd` must run as root (`worker/src/main.c:300-303`).
|
||||
- In the production `DBUS_MANAGER` build, each job starts as a systemd
|
||||
**transient unit** `xcecd-worker-<jobid>` via D-Bus `StartTransientUnit`
|
||||
(`xcecd_cgroups.c` `start_systemd_job`, `xcecd_dbus_manager.c:73-76`).
|
||||
- The unit runs `ExecStart=/bin/sh -c "<path>/xcecd_worker ..."` with
|
||||
`RemainAfterExit=1`, `StandardInput=null`, `KillMode=control-group` and
|
||||
`User=<execuser>` (`xcecd_cgroups.c:300-316`). Optional cgroup limits come from
|
||||
exdata.
|
||||
- The identity switch happens only through `User=`. There is no
|
||||
`setuid`/`initgroups` anywhere in `src/`.
|
||||
- No sandboxing properties are set: no `DynamicUser`, `Protect*`, `PrivateTmp`,
|
||||
`NoNewPrivileges`, `WorkingDirectory` or `BindPaths`. The job runs in `/`.
|
||||
|
||||
**Job helper**
|
||||
- `xcecd_worker` receives the command over `/var/run/.xcecd.sock`, which is mode
|
||||
**0666** (`main.c:641`).
|
||||
- It `uv_spawn`s `/bin/sh -c cmd` (`uv_proc.c:270-288`).
|
||||
- `exechome`/`execcwd` are copied but never applied
|
||||
(`xcecd_worker_utils.c:146-157`).
|
||||
|
||||
**Output**
|
||||
- Written to `/tmp/job_<id>_task<N>.out` (`job_output_path.h:24-41`), created
|
||||
`O_EXCL|O_NOFOLLOW` 0600 by the helper.
|
||||
- Root `xcecd` serves it through `handle_fetchfile`, which accepts only that exact
|
||||
path (bug-1508). See also bug-1868, the `/tmp` symlink hazard.
|
||||
|
||||
**Execution backends**
|
||||
- There is no abstraction; the launch code is tied to systemd, with a
|
||||
`#ifndef DBUS_MANAGER` process-manager fallback.
|
||||
- The user wants to support systems without systemd in future, so the backend
|
||||
must be pluggable.
|
||||
|
||||
**Known bug (fix first, separately)**
|
||||
- Remote-task units start with `limits == NULL`, so no `User=` is set and they
|
||||
**run as root** (`controller_connector.c:1204`, `:1388`;
|
||||
`xcecd_cgroups.c:296-312`).
|
||||
|
||||
## Desired Outcome
|
||||
- The worker has a **pluggable execution-backend interface**:
|
||||
- `direct`, today's behaviour, refactored behind the interface;
|
||||
- `systemd-sandbox`, new.
|
||||
|
||||
A backend without systemd (for example namespaces) can be added later without
|
||||
touching callers. The interface stays usable for future `container`/`vm`
|
||||
modes.
|
||||
- A sandbox job runs in its transient unit under a **systemd `DynamicUser`
|
||||
ephemeral uid**:
|
||||
- The unit's user name is a new, short, per-job name (for example
|
||||
`xj<jobid>`, 31 characters or fewer). It must never be the submitter's name.
|
||||
If a static account with that name existed, systemd would silently use it,
|
||||
and jobs from the same submitter would share a uid.
|
||||
- `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp` and an explicit
|
||||
`NoNewPrivileges=yes`.
|
||||
- `HOME` and `WorkingDirectory` are set to the workspace.
|
||||
- One unit per job per node, so that MPI shared memory works within a node.
|
||||
- **Directories**:
|
||||
- Job-specified input directories are mounted read-only
|
||||
(`BindReadOnlyPaths`), and output directories read-write (`BindPaths`).
|
||||
- If none are specified, the worker creates a private temporary per-job
|
||||
workspace and mounts it.
|
||||
- The workspace can't be handed to the uid in advance, because the uid does
|
||||
not exist until the unit starts. The worker learns the uid when the helper
|
||||
connects (`SO_PEERCRED`) and hands the workspace to it. It must never be made
|
||||
world-writable.
|
||||
- **Output**:
|
||||
- stdout and stderr are opened by systemd outside the sandbox
|
||||
(`StandardOutput=file:<host path>`). Avoid PrivateTmp invisibility and
|
||||
ownership problems.
|
||||
- Existing output retrieval (`handle_fetchfile`, the `jobs output` API) serves
|
||||
sandbox job output, including files left in the workspace.
|
||||
- **Helper authentication**: `xcecd` identifies a genuine job helper by its unit
|
||||
or cgroup, not by uid. A sandboxed job must not be able to pose as a helper
|
||||
over the 0666 socket.
|
||||
- **Mandatory cleanup after the job**:
|
||||
- Remove the workspace, or re-own it (policy decides).
|
||||
- Normalise ownership of everything the job's uid wrote in any writable mount.
|
||||
|
||||
Dynamic uids are reused, and systemd does not check who owns files on disk.
|
||||
Leftovers would otherwise let a later job that gets the same uid read or change
|
||||
them.
|
||||
- **Capability**:
|
||||
- The worker detects its sandbox capability (systemd 242 or later, `systemd`
|
||||
in `nsswitch.conf` passwd/group, and D-Bus property support) and advertises
|
||||
it to `application-profile`'s placement.
|
||||
- It refuses, with a clear error, a sandbox job it cannot honour.
|
||||
- **Direct mode is unchanged.**
|
||||
|
||||
## Approach
|
||||
**Chosen: approach A, systemd sandbox properties on the existing transient unit,
|
||||
behind a pluggable backend interface** (user decision, 2026-09-29).
|
||||
|
||||
Rejected alternatives:
|
||||
- **B, XCE-built namespaces.** Amounts to writing a container runtime in C as
|
||||
root; a possible later backend.
|
||||
- **C, container runtime (Apptainer or OCI).** Adds a dependency and image
|
||||
management; a possible later `container` mode.
|
||||
|
||||
Viability check (2026-09-29, systemd NEWS, systemd.exec(5), v239 sources):
|
||||
- All properties can be set on transient units on 239 and later.
|
||||
- `BindPaths` has D-Bus signature `a(ssbt)`, with flags 0 or `MS_REC`.
|
||||
`ProtectSystem`/`ProtectHome` are strings.
|
||||
- **The minimum is systemd 242** (user decision). From 242, `RestrictSUIDSGID`
|
||||
and `NoNewPrivileges` are implied, which closes CVE-2019-3843/3844-style
|
||||
takeover of a reused uid through SUID binaries. This excludes stock RHEL 8
|
||||
(239).
|
||||
- Uid capacity is about 4,336 per node, shared with other DynamicUser services.
|
||||
`pick_uid` gives up after 100 tries. Stale `RemainAfterExit` units hold their
|
||||
uid, so units must be stopped reliably.
|
||||
- There is no `idmap` option for BindPaths or BindReadOnlyPaths. Ownership of
|
||||
mounted directories is XCE's problem (handover plus cleanup).
|
||||
- DynamicUser is **not a full sandbox**:
|
||||
- the host network is shared (MPI needs it);
|
||||
- `/proc` still shows other jobs (`ProtectProc` needs 247+);
|
||||
- there is no syscall filter.
|
||||
|
||||
The requirements should decide whether to add `ProtectProc` where available,
|
||||
`PrivateDevices`, or a baseline `SystemCallFilter`, and must document the
|
||||
remaining exposure.
|
||||
- Tmpfs pages count against the job's `MemoryMax`. RDMA needs
|
||||
`LimitMEMLOCK=infinity`.
|
||||
|
||||
## Scope
|
||||
- **In**:
|
||||
- The backend interface, with `direct` refactored behind it and
|
||||
`systemd-sandbox` added.
|
||||
- DynamicUser unit properties.
|
||||
- The workspace lifecycle: create, hand over, mount, clean up or re-own.
|
||||
- Input/output mounts.
|
||||
- `StandardOutput=file:` output and retrieval.
|
||||
- Helper authentication by unit or cgroup.
|
||||
- Capability detection and advertisement.
|
||||
- Single-node MPI in one unit.
|
||||
- Tests: unit tests plus a live test on a systemd 242+ node.
|
||||
- **Out**:
|
||||
- The submission side: mode field, policy, placement (`application-profile`).
|
||||
- Multi-node sandbox launch (`job-sandbox-multinode`).
|
||||
- Non-systemd backends and container/vm modes (the interface only, not the
|
||||
implementations).
|
||||
- The remote-task root bug, which is fixed separately, first.
|
||||
|
||||
## Boundary Candidates
|
||||
- The execution-backend interface and `direct` refactor.
|
||||
- Building the systemd-sandbox unit (properties, naming, capability probe).
|
||||
- The workspace and mount lifecycle (create, hand over via `SO_PEERCRED`, clean
|
||||
up or normalise ownership).
|
||||
- Output capture and retrieval.
|
||||
- Helper authentication.
|
||||
|
||||
## Out of Boundary
|
||||
- Job mode selection and policy.
|
||||
- Cross-node uid consistency and remote launch.
|
||||
- Worker cgroup resource limits, which are unchanged and simply apply to sandbox
|
||||
units too.
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream**:
|
||||
- `application-profile`: mode and directory fields on `job_startinfo_t`, and
|
||||
capability-based placement.
|
||||
- The remote-task root fix.
|
||||
- **Downstream**:
|
||||
- `job-sandbox-multinode`.
|
||||
- Future non-systemd or container backends.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends**: none.
|
||||
- **Adjacent**:
|
||||
- `xce-node-manager` (worker node privileges)
|
||||
- `worker-rejoin-resync` (a sandbox job's state across a worker rejoin)
|
||||
- output retrieval (bug-1508 and bug-1868 history)
|
||||
|
||||
## Constraints
|
||||
- Nodes need systemd 242 or later, with `nss-systemd` in `nsswitch.conf`. This is
|
||||
a documented deployment prerequisite, and capability detection must enforce
|
||||
it.
|
||||
- Direct-mode behaviour must be byte-for-byte unchanged (regression tests).
|
||||
- Root code must fail closed. On any failure to set up the sandbox, the job is
|
||||
refused; it never falls back to running unsandboxed or as root.
|
||||
- Cleanup is mandatory and must be retried and survive a worker crash or restart
|
||||
(for example by sweeping orphaned workspaces at startup).
|
||||
- TDD: `testsuite/` first. ASAN must be clean.
|
||||
107
.kiro/specs/job-sandbox-multinode/brief.md
Normal file
107
.kiro/specs/job-sandbox-multinode/brief.md
Normal file
|
|
@ -0,0 +1,107 @@
|
|||
# Brief: job-sandbox-multinode
|
||||
|
||||
## Problem
|
||||
Multi-node jobs (MPI via `openmpi`/`mpich` launch types, or XCE fan-out remote
|
||||
tasks) must also run in sandbox mode (user decision, 2026-09-29: "include
|
||||
multi-node"). A per-node, per-job DynamicUser sandbox (`job-sandbox-execution`)
|
||||
does not work across nodes as it is:
|
||||
- **The uid is not consistent across nodes.** systemd tries a hash of the unit's
|
||||
user name first but falls back to a random uid on collision, and there is no
|
||||
way to pin it. MPI ranks on different nodes, and files on shared filesystems,
|
||||
see different uids.
|
||||
- **ssh-based launchers cannot work.** A sandbox user has no account, no keys and
|
||||
no usable home. Today MPI goes through `xrun -host(s)`
|
||||
(`controller_connector.c:752-756`, `:993-997`), and other launch types go
|
||||
through `job_start_remote_task` → `xcecli_runtask`.
|
||||
- **Shared filesystems.** Writes over NFS/AUTH_SYS land under raw uids
|
||||
61184–65519 cluster-wide:
|
||||
- the submitter cannot read them;
|
||||
- they can collide with real LDAP or AD uids in that range (collision checks
|
||||
only cover what NSS resolves on that node);
|
||||
- `root_squash` does not help.
|
||||
- **MPI transports.** Intra-node shared memory (vader/sm with CMA, XPMEM, KNEM)
|
||||
needs the ranks on a node to share a uid, so one unit per job per node is
|
||||
required. The PMIx session directory also needs a writable, private location.
|
||||
|
||||
## Current State
|
||||
- **Remote tasks.** The worker receives `xce_taskhost_t` run messages and starts
|
||||
a transient unit per task. That unit currently **runs as root**, because
|
||||
`limits` is never set on that path (bug; fixed separately before this spec).
|
||||
- MPI launch wraps the command with `xrun` and runs it once locally with
|
||||
ntasks=1.
|
||||
- **No cross-node identity coordination exists.** Each worker decides its unit
|
||||
independently.
|
||||
|
||||
## Desired Outcome
|
||||
- A sandbox job spanning N nodes runs with **one sandbox unit per node**, all
|
||||
under the **same execution uid**, so MPI and shared-filesystem access behave
|
||||
consistently.
|
||||
- All remote launches go through XCE workers, never ssh. MPI integration works
|
||||
through the worker, for example a PMIx server or daemon-launched `prted`
|
||||
(design decides).
|
||||
- Files the job writes to shared storage have a well-defined owner and a
|
||||
well-defined post-job normalisation. They never collide with real directory
|
||||
uids. The submitter can retrieve outputs through XCE.
|
||||
- All of `job-sandbox-execution`'s guarantees hold on every node: workspace,
|
||||
cleanup, helper authentication and fail-closed behaviour.
|
||||
|
||||
## Approach
|
||||
Design must research and decide. The core problem is **cluster-consistent
|
||||
ephemeral uids**. Candidates:
|
||||
- **(a) XCE-managed uid pool.** The controller (primary) allocates a uid from a
|
||||
cluster-reserved range that must not overlap LDAP/AD, and replicates the
|
||||
lease through the WAL. Each worker runs the unit with that fixed uid, using a
|
||||
static-uid transient user rather than DynamicUser (for example `User=` a
|
||||
numeric uid with `nss-systemd` or a worker NSS module), keeping the same
|
||||
protections.
|
||||
- **(b) DynamicUser with a deterministic name.** Hope the hash matches, and
|
||||
detect and handle mismatches. Fragile.
|
||||
- **(c) A different backend for multi-node.** For example the namespace backend
|
||||
with explicit uid mapping, using the pluggable interface from
|
||||
`job-sandbox-execution`.
|
||||
|
||||
Option (a) is expected, but it adds a controller-replicated lease (HA and
|
||||
persistence), so this spec must follow the full workflow.
|
||||
|
||||
## Scope
|
||||
- **In**:
|
||||
- Consistent uid allocation across nodes, and release on job end or failure.
|
||||
- Sandbox remote-task and MPI launch through workers.
|
||||
- Shared-filesystem ownership policy and post-job normalisation.
|
||||
- Intra-node MPI with one unit per job per node.
|
||||
- Failure handling: a node fails mid-job, or the uid lease is lost on
|
||||
controller failover.
|
||||
- **Out**:
|
||||
- Single-node sandbox mechanics (`job-sandbox-execution`).
|
||||
- The submission option and policy (`application-profile`).
|
||||
- Container or vm modes.
|
||||
|
||||
## Boundary Candidates
|
||||
- Cluster uid lease allocation (controller, WAL-replicated).
|
||||
- Worker multi-node sandbox launch (remote tasks and MPI/PMIx).
|
||||
- Shared-storage ownership normalisation.
|
||||
|
||||
## Out of Boundary
|
||||
- The per-node sandbox unit construction (reused from `job-sandbox-execution`).
|
||||
- The scheduler's host selection for multi-node jobs (unchanged apart from
|
||||
capability filtering).
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream**:
|
||||
- `job-sandbox-execution` (HARD).
|
||||
- `application-profile` (HARD).
|
||||
- The remote-task root bug fix (HARD).
|
||||
- The WAL/replication machinery, if option (a) is chosen.
|
||||
- **Downstream**: future container or vm multi-node modes.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends**: `job-sandbox-execution` (reuses its backend interface).
|
||||
- **Adjacent**:
|
||||
- `controller-job-cleanup` (lease release relative to job removal)
|
||||
- `ha-wal-replication`
|
||||
|
||||
## Constraints
|
||||
- The uid range must be configurable and must not overlap directory-service uids.
|
||||
- Leases must survive controller failover, and nothing may leak on worker loss.
|
||||
- Fail closed. Never fall back to ssh, root or direct mode.
|
||||
- HA/persistence rules apply if leases are replicated.
|
||||
139
.kiro/specs/job-status-detail-display/brief.md
Normal file
139
.kiro/specs/job-status-detail-display/brief.md
Normal file
|
|
@ -0,0 +1,139 @@
|
|||
# Brief: job-status-detail-display
|
||||
|
||||
## Problem
|
||||
End users check on their jobs through `xcecli` and the REST API. Two questions
|
||||
never get an answer, for any job:
|
||||
- A pending job: why can't it run?
|
||||
- A running job: how much has it reserved, and how much is it really using?
|
||||
|
||||
The controller holds both answers (`xce_job_t.pend_reasons` and `run_usage` /
|
||||
`allocs`, `src/include/xce/types.h:499-517`), but neither reaches the user. Only
|
||||
the `xce_test` harness and the scheduler simulator show pend reasons, and those
|
||||
are developer tools.
|
||||
|
||||
(Renamed from `job-pend-reason-display` on 2026-09-29 when running-job usage came
|
||||
into scope, by user decision.)
|
||||
|
||||
## Current State
|
||||
Codebase exploration, 2026-09-29.
|
||||
|
||||
**How users see jobs**
|
||||
- REST: `GET /xce/v1/jobs` (`GetJobsInfo`) and `GET /xce/v1/jobs/:jobids`
|
||||
(`GetJobByIDs`), registered in `apiserver/router/routers.go:131-132`.
|
||||
- xcecli: only `xce jobs list` (`cmd/jobs/list.go:291`).
|
||||
- Even a lookup by job id goes through `GET /xce/v1/jobs` with query params.
|
||||
- `-l`/`--verbose` detail (`printJobsVerboseOutput`,
|
||||
`common/output/page_print_jobs.go:249`) already has a `PENDING REASONS:` block
|
||||
(:321-327), printed only for PEND/PSUSP jobs with a non-empty value. The
|
||||
array-element printer reuses it (`page_print_array_elements_jobs.go:123`).
|
||||
- `--json` and the `-o pending_reason` field exist.
|
||||
|
||||
**Where the pend reason is lost**
|
||||
- The apiserver gets jobs by controller push, not by querying:
|
||||
- a full dataset push on connect (`controller/apiserver.c:70` → `backend/clib.go`
|
||||
`handleDataset`);
|
||||
- `updateJob` notifies (`apiserver.c:349`).
|
||||
- Both carry the wire `Job.pendreason @10` (`controller.capnp:335`), but
|
||||
`cJobToModel` (`backend/convert.go:79-142`) never reads it. `PendingReason`
|
||||
(`apiserver/models/jobs.go:553`) stays empty.
|
||||
- The store replaces the whole job entry on each update (`backend/handler.go:79-81`),
|
||||
so any field the conversion skips is wiped every time.
|
||||
- A pend-reason change is never pushed on its own. `set_preason` (`job.c:3646`)
|
||||
fires `HOOK_DETAIL_SETPREASON`, but the apiserver hook dispatcher
|
||||
(`apiserver.c:399-418`) ignores it. The reason only arrives with the next full
|
||||
push or unrelated `updateJob`.
|
||||
|
||||
**Run usage and reservation**
|
||||
- Reserved: `allocs` (`Job @7`) reaches the apiserver. Today it is used only for
|
||||
`ExecHosts`.
|
||||
- Real usage: `xce_job_runusage_t` (`types.h:457-468`) holds job totals (max, avg
|
||||
and used mem, cpu time, efficiencies) and a per-host `host_runusage` list
|
||||
(cpu_time, mem_usage, mem_swap, tasks_current). It is maintained on the
|
||||
controller in `update_job_rusages` (`job.c:3572`) and merged on WAL replay.
|
||||
- The wire `Job` struct has **no** run-usage field. The per-report
|
||||
`schedJobRusages` notify (`apiserver.c:367-395`) is dropped by the apiserver
|
||||
(`clib.go:513-534`). So `JobRusage` (`models/jobs.go:555`) is always empty.
|
||||
|
||||
**Reason codes**
|
||||
- There are only three (`src/include/preason.h:7-9`): `PREASON_BAD_RESREQ=1`,
|
||||
`PREASON_NO_RESOURCE=2`, `PREASON_MATCH_FAIL=3`. No code-to-text mapping exists
|
||||
anywhere.
|
||||
- The level is decided by the arguments, not the code: a main reason, a host
|
||||
reason (host), or a resource reason (resource, with an optional host).
|
||||
|
||||
**Type mismatch**
|
||||
- xcecli declares `PendingReason string` and `JobRusage string`
|
||||
(`xcecli/models/jobs.go:165, 120`).
|
||||
- The apiserver `Jsonb` passes valid JSON through raw.
|
||||
- A structured value needs the xcecli types changed.
|
||||
|
||||
## Desired Outcome
|
||||
- For a pending job, REST and `xcecli` show why it can't run: the main reason,
|
||||
host-level reasons and resource-level reasons, all in readable text. They also
|
||||
show every alternative's reasons, labelled, once `scheduler-alter-pend-reasons`
|
||||
lands.
|
||||
- For a running job, they show reserved amounts per host and resource beside the
|
||||
reported real usage. If nothing has been reported yet, they say "resource usage
|
||||
has not been reported yet".
|
||||
- The information stays current: a changed reason or a new usage report shows up
|
||||
without waiting for an unrelated job change, and it survives an apiserver
|
||||
reconnect or a controller failover.
|
||||
- `pending_reason` and `job_rusage` are documented, structured JSON.
|
||||
|
||||
## Approach
|
||||
Design decides the details. The expected shape:
|
||||
- Add a code-to-text mapping, with one source of truth, shared by every surface.
|
||||
- Deliver pend-reason changes and run usage to the apiserver. This needs a
|
||||
controller hook/notify for `SETPREASON`, and run usage has to reach the
|
||||
apiserver (a wire field or handling the rusage notify).
|
||||
- Map both in the pure conversion layer (`convert_pure.go` `rawJobToModel`,
|
||||
following the `ExecHosts` pattern), and test it in `convert_test.go`.
|
||||
- Change the xcecli model types and render the data in the `-l` output and the
|
||||
JSON output.
|
||||
|
||||
## Scope
|
||||
- **In**:
|
||||
- Pend-reason and run-usage delivery from the controller to the apiserver.
|
||||
- Conversion and the JSON schema.
|
||||
- The reason-code text.
|
||||
- xcecli verbose and JSON rendering, including array elements.
|
||||
- Freshness and failover behaviour.
|
||||
- Tests.
|
||||
- **Out**:
|
||||
- Producing pend reasons or changing their content (the scheduler and
|
||||
`scheduler-alter-pend-reasons`).
|
||||
- The simulator REPL (`scheduler-sim-diagnostics`).
|
||||
- `xceadmin`.
|
||||
- RBAC or middleware changes.
|
||||
- Usage summaries for finished jobs.
|
||||
|
||||
## Boundary Candidates
|
||||
- Controller → apiserver delivery (hook dispatch, wire).
|
||||
- apiserver conversion and JSON contract.
|
||||
- xcecli rendering.
|
||||
- Reason-code text (one shared source).
|
||||
|
||||
## Out of Boundary
|
||||
- Scheduler, controller and WAL pend-reason content (`scheduler-alter-pend-reasons`).
|
||||
- Deciding which jobs a user may see (existing RBAC).
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream**: the existing flat `pend_reasons`, `allocs` and `run_usage` data.
|
||||
Per-alternative rendering depends on `scheduler-alter-pend-reasons`; the JSON
|
||||
shape includes `alternatives` from the start.
|
||||
- **Downstream**: other REST clients.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends**: none.
|
||||
- **Adjacent**:
|
||||
- `scheduler-alter-pend-reasons` (data producer)
|
||||
- `scheduler-sim-diagnostics` (the matching simulator `info job` display)
|
||||
|
||||
## Constraints
|
||||
- Middleware chain unchanged: WhitelistURL → WhitelistIP → RateLimit → JWT → CORS
|
||||
→ RBAC.
|
||||
- Go error pattern: `ReturnError(code, msg, nil); return`.
|
||||
- TDD: `_test.go` before Go changes, `testsuite/` before C changes.
|
||||
- Controller push must not block. Async I/O rules apply (`ha-async-io.md`).
|
||||
- Any wire change must be an additive Cap'n Proto field.
|
||||
- The reason-code text has one source of truth.
|
||||
129
.kiro/specs/job-status-detail-display/requirements.md
Normal file
129
.kiro/specs/job-status-detail-display/requirements.md
Normal file
|
|
@ -0,0 +1,129 @@
|
|||
# Requirements Document
|
||||
|
||||
## Project Description (Input)
|
||||
**Who has the problem:** End users of XCE who submit jobs and check on them through `xcecli` or the REST API.
|
||||
- When a job is pending, they need to know why it cannot run, so they can wait, change their resource requirements, or pick a different queue.
|
||||
- When a job is running, they need to see what it has reserved and what it actually uses, so they can size future requests.
|
||||
|
||||
**Current situation:** Users see neither a pending reason nor running-job usage, for any job.
|
||||
- The controller holds each pending job's pending reasons and each running job's reported usage. Its reserved allocation also reaches the user tier, but the pending reason is dropped when the apiserver converts the job, and run usage never reaches the apiserver at all.
|
||||
- A changed pending reason is not delivered on its own; it arrives only with the next unrelated job change.
|
||||
- No readable text exists for the three reason codes.
|
||||
- Only developer tools (the `xce_test` harness and the scheduler simulator) show pending reasons.
|
||||
|
||||
**What should change:**
|
||||
- REST and `xcecli` show why a pending job cannot run, in readable text, including each alternative's reasons once `scheduler-alter-pend-reasons` delivers them.
|
||||
- They show a running job's reserved amounts beside its reported real usage, or "resource usage has not been reported yet".
|
||||
- The information stays current and survives a controller failover.
|
||||
- The spec was renamed from `job-pend-reason-display` when running-job usage was brought into scope (user decision, 2026-09-29).
|
||||
|
||||
See `brief.md` for the full discovery context and code references.
|
||||
|
||||
## Introduction
|
||||
XCE end users check on their jobs through the `xcecli` command-line client and the XCE REST API. Today neither surface can answer the two questions a user asks about a job that is not finishing:
|
||||
- "Why is my pending job not running?"
|
||||
- "How much is my running job using compared with what it reserved?"
|
||||
|
||||
The cluster already knows both answers, but they never reach the user. This feature makes the job-detail views of the REST API and `xcecli` show:
|
||||
- a pending job's pending reasons in readable text;
|
||||
- a running job's reserved resources beside its reported real usage.
|
||||
|
||||
It also keeps that information current as the scheduler and workers report changes.
|
||||
|
||||
## Boundary Context
|
||||
- **In scope**:
|
||||
- Showing pending reasons for pending jobs and reserved-vs-used resources for running jobs in the REST API job queries and in `xce jobs list`'s detailed (`-l`) and JSON (`--json`) output, including array-job elements.
|
||||
- Readable descriptions for reason codes.
|
||||
- Keeping the displayed information current, including across controller failover and apiserver reconnection.
|
||||
- **Out of scope**:
|
||||
- Deciding or changing *what* the pending reasons are. The scheduler produces them, and `scheduler-alter-pend-reasons` owns per-alternative reasons.
|
||||
- The scheduler simulator REPL (`scheduler-sim-diagnostics`).
|
||||
- `xceadmin`.
|
||||
- Usage summaries for finished (DONE/EXIT) jobs.
|
||||
- Any change to who may view which jobs.
|
||||
- Any change to the non-detailed `xce jobs list` table output.
|
||||
- **Adjacent expectations**:
|
||||
- This feature relies on the scheduler recording pending reasons, and on workers reporting job resource usage, as they do today.
|
||||
- Per-alternative display (Requirement 3) becomes visible only once `scheduler-alter-pend-reasons` supplies per-alternative reasons. Until then, jobs show the reasons that exist today.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Pending Reasons for Pending Jobs
|
||||
**Objective:** As a user with a pending job, I want to see why the job cannot run, so that I can decide whether to wait or change my request.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a user requests job details for a job in a pending state (PEND or PSUSP) through the REST API or through `xce jobs list -l`, the XCE Job Detail Service shall show the job's pending reasons.
|
||||
2. The XCE Job Detail Service shall show each pending reason at its level:
|
||||
- a job-level reason, with no host or resource;
|
||||
- a host-level reason, with the host name;
|
||||
- a resource-level reason, with the resource name and, where recorded, the host name.
|
||||
3. When several hosts share the same reason (and, for resource-level reasons, the same resource), the `xce jobs list -l` output shall show them as one entry giving the reason, the number of hosts, and the host names.
|
||||
4. When several hosts share the same reason, the REST API and `xce jobs list --json` output shall still list every host-level and resource-level reason individually.
|
||||
5. If a job is pending and no pending reason has been recorded for it yet, the XCE Job Detail Service shall state that no pending reason has been reported yet, rather than show an empty reason section.
|
||||
6. While a job is in any state other than PEND or PSUSP, the XCE Job Detail Service shall not show a pending reason, including one recorded while the job was pending.
|
||||
|
||||
### Requirement 2: Readable Reason Descriptions
|
||||
**Objective:** As a user, I want each pending reason explained in words, so that I don't need to know internal reason codes.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. The XCE Job Detail Service shall describe each known reason code in readable text:
|
||||
- BAD_RESREQ: the job's resource requirement is invalid.
|
||||
- NO_RESOURCE: not enough of a requested resource is available.
|
||||
- MATCH_FAIL: the host does not satisfy the job's resource requirement.
|
||||
2. The REST API and `xce jobs list --json` output shall give both the reason code and its readable description for every pending reason.
|
||||
3. If a pending reason carries a code with no known description, the XCE Job Detail Service shall show it as an unknown reason together with its numeric code, rather than omit it or fail the request.
|
||||
4. The XCE Job Detail Service shall use the same description for a given reason code on every surface where the reason is shown.
|
||||
|
||||
### Requirement 3: Per-Alternative Pending Reasons
|
||||
**Objective:** As a user whose job lists alternative resource requirements, I want to see why each alternative could not be satisfied, so that I know which alternative to adjust.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. Where a pending job's pending reasons include per-alternative reasons, the XCE Job Detail Service shall show each alternative's reasons as a separate group.
|
||||
2. The XCE Job Detail Service shall label each group so that the user can tell which of the job's alternative resource requirements it belongs to.
|
||||
3. Where a pending job's pending reasons include per-alternative reasons, the XCE Job Detail Service shall present the groups in the order the alternatives appear in the job's resource requirement.
|
||||
4. While a pending job's pending reasons include no per-alternative reasons, the XCE Job Detail Service shall show them as defined in Requirement 1, without an alternatives grouping.
|
||||
5. The REST API output format for pending reasons shall define how per-alternative reasons are represented from the first release of this feature, so that adding per-alternative data later needs no change to the output format.
|
||||
|
||||
### Requirement 4: Reserved and Used Resources for Running Jobs
|
||||
**Objective:** As a user with a running job, I want to see what the job reserved next to what it actually uses, so that I can tell whether my resource request is sized correctly.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a user requests job details for a job in a running or suspended-while-running state (RUN, USUSP, or SSUSP) through the REST API or through `xce jobs list -l`, the XCE Job Detail Service shall show:
|
||||
- the resources reserved for the job on each host, per resource, including any shared (cluster-wide) resources reserved for it;
|
||||
- the job's most recently reported real usage, both per host and as job totals.
|
||||
2. The per-host reported usage shall include CPU time, memory used, swap used, and the current number of tasks.
|
||||
3. The job totals shall include maximum memory, average memory, CPU time, memory efficiency, and CPU efficiency.
|
||||
4. Where a host has both a memory reservation and a reported memory usage, the XCE Job Detail Service shall show the two values side by side for that host.
|
||||
5. If a running job has not yet reported any resource usage, the XCE Job Detail Service shall still show the reserved resources and shall state that resource usage has not been reported yet, rather than showing zero usage.
|
||||
6. If a running job has reported usage for some of its hosts but not others, the XCE Job Detail Service shall mark each host without a report as not yet reported.
|
||||
7. The `xce jobs list -l` output shall show memory and other byte amounts in human-readable units (for example KB, MB, GB).
|
||||
8. The REST API and `xce jobs list --json` output shall give byte amounts as numbers with a documented unit.
|
||||
9. While a job is in any state other than RUN, USUSP, or SSUSP, the XCE Job Detail Service shall not show the running-job usage section.
|
||||
|
||||
### Requirement 5: Freshness and Continuity
|
||||
**Objective:** As a user, I want the displayed reasons and usage to reflect the cluster's current view of the job, so that I do not act on stale information.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When the scheduler records a new or changed pending reason for a job, the XCE Job Detail Service shall show it on the next job-detail request after the change reaches the apiserver, without requiring any other change to the job. It shall reach the apiserver within the same delay as a job state change.
|
||||
2. When a worker reports new resource usage for a running job, the XCE Job Detail Service shall reflect it on the next job-detail request after the report reaches the apiserver. It shall reach the apiserver within the same delay as a job state change.
|
||||
3. When a job leaves the pending state, the XCE Job Detail Service shall stop showing its pending reasons from the next job-detail request after the state change reaches the apiserver.
|
||||
4. When the controller's primary changes (failover), the XCE Job Detail Service shall continue to show each job's latest pending reasons and latest reported usage once it has reconnected to the new primary. It shall not show them as empty or not yet reported unless they genuinely are.
|
||||
5. When the apiserver reconnects to the controller, the XCE Job Detail Service shall show the same pending reasons and reported usage as the controller currently holds.
|
||||
|
||||
### Requirement 6: Output Surfaces and Compatibility
|
||||
**Objective:** As a user or REST client developer, I want the new information in the views I already use, without breaking existing output, so that adopting it needs no new commands and existing scripts keep working.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. The `xce jobs list -l` output shall show the pending reasons (Requirements 1 to 3) and running-job resources (Requirement 4) for each job it displays, including elements of array jobs.
|
||||
2. The REST API job queries (listing jobs and fetching jobs by ID) and `xce jobs list --json` output shall include the pending reasons and running-job resources as structured fields, not pre-formatted text.
|
||||
3. The format of those structured fields shall be documented.
|
||||
4. The XCE Job Detail Service shall leave the non-detailed `xce jobs list` table output unchanged.
|
||||
5. The XCE Job Detail Service shall leave every existing field of the REST API job output unchanged in name and meaning.
|
||||
6. If `xce jobs list` receives job data that carries no pending reason or usage information (for example from an apiserver without this feature), it shall display the job without that information and without reporting an error.
|
||||
|
||||
### Requirement 7: Access Control
|
||||
**Objective:** As a cluster operator, I want the new job details to follow existing job visibility rules, so that this feature exposes nothing a user could not already see.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. The XCE Job Detail Service shall show pending reasons and resource usage only for jobs the requesting user is already permitted to view.
|
||||
2. The XCE Job Detail Service shall not change which users may view which jobs.
|
||||
111
.kiro/specs/rbac-persistence/brief.md
Normal file
111
.kiro/specs/rbac-persistence/brief.md
Normal file
|
|
@ -0,0 +1,111 @@
|
|||
# Brief: rbac-persistence
|
||||
|
||||
## Problem
|
||||
Cluster admins manage roles and permissions through the apiserver's REST API:
|
||||
- create and delete roles;
|
||||
- add permissions to a role;
|
||||
- assign roles to users and remove them.
|
||||
|
||||
Those changes live only in apiserver memory. When the apiserver restarts, every
|
||||
change made through the API since startup **silently reverts** to whatever
|
||||
`config/rbac.yaml` says. There is no error, warning or audit trail. An admin who
|
||||
granted a user access (or, worse, **revoked** it) finds the change gone after a
|
||||
routine restart or upgrade. A revoked permission silently coming back is a
|
||||
security problem.
|
||||
|
||||
This matters more now:
|
||||
- `application-profile` adds `app:use`/`app:view` grants, which admins will
|
||||
manage per profile.
|
||||
- The job-sandbox initiative relies on RBAC to decide who may run which
|
||||
applications, including sandbox versus direct execution.
|
||||
|
||||
## Current State
|
||||
Documented in `.kiro/steering/security.md` as a known, deferred gap (flagged
|
||||
2026-08-27 during the `scheduler-snapshot-job-allocation` persistence survey).
|
||||
|
||||
- **Loading.** `store.Roles` and `store.UserRoles` (`apiserver/store/stores.go`)
|
||||
are loaded from `config/rbac.yaml` at startup (`LoadRBACConfig()`,
|
||||
`controllers/rbac_check.go`).
|
||||
- **Changes made at runtime with no write-back:**
|
||||
- `RoleController.CreateRole` / `AddPermissionsForRole` /
|
||||
`DeleteRoleAndPermissions` (`controllers/roles.go:499`, `:540`, `:594`);
|
||||
- `store.AddRoleForUser` / `DeleteRoleForUser`, called from the automatic role
|
||||
assignment in `AuthController.Logon` and from the user-role endpoints in
|
||||
`controllers/users.go`.
|
||||
- **Model.** Permission strings are `action:scope` with domain types `all`,
|
||||
`self`, `self_ug`, `queue` (with values) and `hostgroup` (with values).
|
||||
`cluster_admin` bypasses checks through one early return.
|
||||
- **Failure class.** The same save/restore field-completeness failure that
|
||||
`persistence.md` describes, but in the apiserver's in-memory RBAC store.
|
||||
- **HA.** The apiserver is a separate process fed by controller push. Nothing
|
||||
covers how several apiserver instances would share RBAC state; the design must
|
||||
settle this.
|
||||
|
||||
## Desired Outcome
|
||||
- Every RBAC change accepted by the API is **durable**. It survives an apiserver
|
||||
restart, or it is rejected; never silently lost.
|
||||
- Revocations are durable exactly like grants.
|
||||
- Every RBAC change is **audited**: who changed what, when, and old versus new
|
||||
value.
|
||||
- On startup, the effective RBAC state equals the last successfully persisted
|
||||
state. Conflicts between the shipped `rbac.yaml` defaults and API changes are
|
||||
resolved by a documented, predictable rule.
|
||||
- If there are several apiserver instances, they converge on the same RBAC state
|
||||
(the design decides how).
|
||||
|
||||
## Approach
|
||||
Design decides. Candidates noted at discovery:
|
||||
- **(a) Write-through to `rbac.yaml`.** Atomic write (temp file plus rename),
|
||||
preserving the builtin roles. Simple, but a single node's file, and it merges
|
||||
badly with shipped defaults.
|
||||
- **(b) Store RBAC state in the controller's replicated state.** A new cluster
|
||||
entity persisted through the WAL and snapshots. HA-consistent and shared by
|
||||
every apiserver, but a much larger HA/persistence change.
|
||||
- **(c) A small embedded store** (for example SQLite or bbolt) in the apiserver.
|
||||
Durable, but local to the node.
|
||||
- **(d) Declare API changes runtime-only and reject them.** Make `rbac.yaml` (or
|
||||
xceadmin-managed config) the only source, and turn the mutating endpoints into
|
||||
errors or read-only. The smallest change, but it removes a capability.
|
||||
|
||||
## Scope
|
||||
- **In**:
|
||||
- Durability of every mutating RBAC API: roles, permissions, user-role
|
||||
assignments, and automatic role assignment at login.
|
||||
- The audit trail.
|
||||
- The rule that reconciles shipped defaults with API changes at startup.
|
||||
- Multi-instance consistency (whatever the design chooses).
|
||||
- Tests, including restart round-trips of grants and revocations.
|
||||
- **Out**:
|
||||
- New permission types (`application-profile` owns `app:*`).
|
||||
- Auth backends and JWT.
|
||||
- Changing the middleware chain or the `cluster_admin` bypass.
|
||||
|
||||
## Boundary Candidates
|
||||
- RBAC state storage and load (the persistence mechanism).
|
||||
- Mutating-API write path (durability, error on failure).
|
||||
- Audit log.
|
||||
- Reconciling defaults with runtime changes.
|
||||
|
||||
## Out of Boundary
|
||||
- The permission model's semantics (actions, domain types).
|
||||
- Profile, queue and other resource-specific permission checks.
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream**: the existing RBAC store and APIs. If approach (b) is chosen, the
|
||||
controller WAL/snapshot machinery.
|
||||
- **Downstream**: `application-profile` (durable `app:use` grants) and any
|
||||
future RBAC-governed feature.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends**: none (closes a gap recorded in `security.md`).
|
||||
- **Adjacent**: `application-profile`, `admin-config-tool` (if RBAC moves into
|
||||
xceadmin-managed config).
|
||||
|
||||
## Constraints
|
||||
- Middleware chain unchanged: WhitelistURL → WhitelistIP → RateLimit → JWT →
|
||||
CORS → RBAC.
|
||||
- Fail closed. If a change cannot be persisted, the API returns an error and the
|
||||
in-memory state is left unchanged.
|
||||
- Never log secrets (`security.md`).
|
||||
- If approach (b): HA/persistence rules, and the full 3-phase workflow.
|
||||
- When done, update `security.md` to remove the "Known gap" note.
|
||||
169
.kiro/specs/scheduler-alter-pend-reasons/brief.md
Normal file
169
.kiro/specs/scheduler-alter-pend-reasons/brief.md
Normal file
|
|
@ -0,0 +1,169 @@
|
|||
# Brief: scheduler-alter-pend-reasons
|
||||
|
||||
## Problem
|
||||
Pend reasons exist to tell the user **why their job cannot run**. For a job whose
|
||||
resource requirement is an ALTER (a list of alternative resreqs), each alternative
|
||||
is a separate way the job could have run. So the answer has to cover every
|
||||
alternative. Without that, the user cannot tell which of their alternatives is
|
||||
close to fitting, or which one to change. Today the user sees the pend reason of
|
||||
only the **last** alternative tried. Why the earlier alternatives failed is lost. A job can look
|
||||
like it pends for `MATCH_FAIL` when its preferred alternative actually failed on
|
||||
`NO_RESOURCE`, which sends the investigation the wrong way. This affects every
|
||||
surface that shows pend reasons: the controller job record, `xcecli`, the harness,
|
||||
the simulator REPL (`info job`), recordings, and `--sim-verify`.
|
||||
|
||||
## Current State
|
||||
- `sched_make_alter_allocation()` (`src/scheduler/src/scheduler.c:403-427`) tries
|
||||
each alternative through `sched_make_compound_alloc_()`, which ends with
|
||||
`MOVE_PTR(obj->reason, ctx.reason)` (`scheduler.c:388-390`). Each failing
|
||||
alternative overwrites the previous one. The code carries
|
||||
`/* FIXME: merge pend reason from all children */` at `scheduler.c:424`.
|
||||
- **Confirmed leak:** `MOVE_PTR` (`src/include/utils.h:135`) doesn't free the old
|
||||
pointer, and nothing clears `obj->reason` between alternatives, so every
|
||||
overwritten reason is orphaned. The same overwrite happens across scheduling
|
||||
cycles in the primary and compound paths (`scheduler.c:324-326`).
|
||||
`make_pend_decision` (`decision.c:16`) shares the pointer without clearing
|
||||
`schedobj->reason`, so it is unclear who owns it.
|
||||
- Side effects of the current overwrite:
|
||||
- An alternative that succeeds after earlier failures leaves a stale reason in
|
||||
`obj->reason`.
|
||||
- A later alternative that produces no reason lets an earlier one silently win.
|
||||
- **End users see no pend reason at all today, for any job.**
|
||||
- `apiserver/models/jobs.go:553` declares `PendingReason` (JSONB), and
|
||||
`xcecli/common/output/page_print_jobs.go:323` prints it if it is present.
|
||||
- No non-test Go code fills it in, so `xcecli` / REST always shows it empty.
|
||||
- Pend reasons are visible only through the harness (`xce_test`) and the
|
||||
simulator.
|
||||
- Wire: `JobPendReason` (`src/schema/controller.capnp:75`, maps to
|
||||
`xce_job_preason_t` at `src/include/xce/types.h:491-497`) is flat: `mainReason`,
|
||||
`hostReasons`, `resReasons`. It has no per-alternative structure.
|
||||
|
||||
## Desired Outcome
|
||||
- A user asking why their pending ALTER job isn't running gets the reason for
|
||||
every alternative, each labelled with the alternative it belongs to.
|
||||
- A pending ALTER job carries one pend-reason entry per alternative tried, in try
|
||||
order, each attributed to its alternative. The top-level reason keeps a defined
|
||||
meaning, so existing readers still work.
|
||||
- The per-alternative reasons survive every hop:
|
||||
- scheduler → controller
|
||||
- the controller job record and WAL preason fact (a WAL replay or failover loses
|
||||
no alternative)
|
||||
- recordings (`ReplayDecision`) and `--sim-verify`
|
||||
- harness display and parse
|
||||
- Reasons are neither leaked nor stale. A dispatched job carries no leftover pend
|
||||
reason from a failed alternative.
|
||||
- Primary and compound (non-ALTER) jobs produce byte-identical pend reasons to
|
||||
today.
|
||||
|
||||
## Approach
|
||||
Per-alternative wire shape (user decision, 2026-09-28): add an optional,
|
||||
self-referential field to `JobPendReason`:
|
||||
|
||||
```
|
||||
alternatives @3: List(JobPendReason) $C.mapname("alternatives") $C.maplistcount("nalternatives");
|
||||
```
|
||||
|
||||
Why:
|
||||
- It keeps which alternative failed and why, which is the purpose of the request.
|
||||
- Cap'n Proto additive fields are backward compatible. Old WAL facts, snapshots
|
||||
and recordings decode with `nalternatives=0`.
|
||||
- The C codegen already handles self-reference. Precedent: `WalCatchupEntry.entries`
|
||||
(`controller.capnp:1090`, tagged struct at `typemap.h:20`).
|
||||
- Generated encode/decode/free (`capnpc-c`, pinned fork) pick up the field
|
||||
automatically.
|
||||
|
||||
Rejected alternative: merge every alternative's reasons into the existing flat
|
||||
struct. It needs no schema change, but loses attribution and makes `mainReason`
|
||||
ambiguous.
|
||||
|
||||
Viability check (2026-09-28): no blockers. Required groundwork:
|
||||
- Give `xce_job_preason_t` a struct tag so it can reference itself.
|
||||
- Hand-written helpers in `src/lib/preason.c` need updating:
|
||||
- `erase_job_preason` (13-50) must free alternatives recursively.
|
||||
- `add_job_preason` (a main-reason-only call erases, at 152) must not wipe
|
||||
alternatives.
|
||||
- `job_preason_eq` (317-356) must compare alternatives recursively.
|
||||
- No copy function exists, generated or hand-written. Every transfer is a
|
||||
`MOVE_PTR`.
|
||||
|
||||
## Scope
|
||||
- **In**:
|
||||
- Schema field and C struct tag.
|
||||
- `preason.c` erase, add, eq and any new append helper.
|
||||
- The ALTER path in `scheduler.c`: collect each alternative's reason into
|
||||
`alternatives[]` and define the top-level reason.
|
||||
- Fix the `obj->reason` overwrite leak and the stale-reason cases: ALTER, and the
|
||||
primary/compound cross-cycle overwrite that shares the same ownership seam.
|
||||
- A defined top-level `main_reason` for readers that use only the top level:
|
||||
- `scheduler.c:284` (unsatisfiable cache key)
|
||||
- `sim_verify.c:356-358` and `:415-417`
|
||||
- A deliberate `--sim-verify` decision: `sim_verify_pend_main_reason_eq`
|
||||
(`sim_verify.c:153-168`) compares `main_reason` only (bug-1845). Choose between
|
||||
keeping that and comparing alternatives, and document the choice.
|
||||
- Harness pend-reason display and parse (`src/harness/test.c:3246-3270`,
|
||||
`:4988`).
|
||||
- Free-path audit of the `erase_job_preason` callers:
|
||||
- `controller/job.c:3673` and `:3750`
|
||||
- `lib/jobj.c:678`, which also has an existing leak of the struct itself
|
||||
- `harness/test.c:5104` and `:5133`
|
||||
- WAL round-trip and snapshot field-completeness tests (`persistence.md`).
|
||||
- **Out**:
|
||||
- The `info job` REPL display. It belongs to the `scheduler-sim-diagnostics`
|
||||
update and consumes this spec's output.
|
||||
- Decision-trace detail about why each host was rejected
|
||||
(`scheduler-sim-decision-trace`).
|
||||
- Showing pend reasons to users through REST and `xcecli`. That is owned by the
|
||||
follow-up spec `job-status-detail-display` (user decision, 2026-09-28). This
|
||||
spec makes the data complete and durable up to the controller's `xce_job_t`.
|
||||
- Any change to how alternatives are tried or ordered.
|
||||
|
||||
## Boundary Candidates
|
||||
- libxce preason model: struct tag, schema field, erase/add/eq, recursive
|
||||
semantics.
|
||||
- Scheduler ALTER collection and `obj->reason` ownership (leak and stale-reason
|
||||
fix).
|
||||
- Persistence and replay seams: controller WAL fact round-trip, recordings,
|
||||
`--sim-verify` comparison policy.
|
||||
- Harness display and parse.
|
||||
|
||||
## Out of Boundary
|
||||
- The allocation algorithm and `my_alloc()` structure (`scheduler-allocator-runner`).
|
||||
- The decision-trace data model (`scheduler-sim-decision-trace`).
|
||||
- The REPL `info job` formatting (`scheduler-sim-diagnostics`).
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream**:
|
||||
- `libxce-alloc-preason-equality` (HARD dependency). It introduces
|
||||
`job_preason_eq` in `preason.{c,h}`, which this spec extends to recurse into
|
||||
alternatives. Its tasks are 0/8 done.
|
||||
- `scheduler-allocator-runner`: sequence this spec after it, because the runner
|
||||
requires pend-reason content to stay unchanged while it is implemented.
|
||||
- **Downstream**:
|
||||
- `job-status-detail-display`, which shows these reasons to users through REST
|
||||
and `xcecli`.
|
||||
- The `scheduler-sim-diagnostics` `info job` per-alternative display.
|
||||
- `scheduler-sim-decision-trace`, whose pend-reason display (its Req "same
|
||||
content as the pend decision") then includes alternatives automatically.
|
||||
- A possible Go/`xcecli` pend-reason rendering follow-up.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends**: `libxce-alloc-preason-equality` (`job_preason_eq` semantics now
|
||||
cover alternatives).
|
||||
- **Adjacent**:
|
||||
- `scheduler-sim-diagnostics`
|
||||
- `scheduler-sim-decision-trace`
|
||||
- `scheduler-allocator-runner`
|
||||
- `ha-wal-replication` / `persistence.md` (WAL preason fact)
|
||||
|
||||
## Constraints
|
||||
- TDD: tests in `testsuite/` come first. ASAN must be clean, including the leak
|
||||
fix.
|
||||
- Backward compatibility: WAL facts, snapshots and recordings written before this
|
||||
change must still decode and replay. A mixed-version HA cluster must not misread
|
||||
the new field.
|
||||
- Save/restore field-completeness (`persistence.md`): alternatives must round-trip
|
||||
through both the write side and every apply seam.
|
||||
- Recursion: only the scheduler produces this data, so nesting depth is one level
|
||||
in practice. The requirements should say whether nested `alternatives` are
|
||||
rejected or bounded.
|
||||
- Non-ALTER jobs: pend-reason output byte-identical to today.
|
||||
|
|
@ -59,6 +59,17 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
|
|||
- [x] ha-single-leader-role — promotion no-op + own-term gate, transitional-state (the former fourth role) removal. Dependencies: ha-commit-engine, ha-commit-barrier-rollout (HARD: barrier before state removal). (specs done 2026-06-11: 2 reqs / 7 components / 11 sub-tasks)
|
||||
- [x] ha-convergence-recovery — demotion truncate+self-restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum logging. Dependencies: ha-commit-engine (E2E additionally needs ha-single-leader-role). (specs done 2026-06-11: 9 reqs / 7 components / 13 sub-tasks)
|
||||
- [x] ha-commit-durability — closes bug-661 (acknowledgment did not imply durability; committed jobs lost across failover). Makes acknowledged ⇒ durable on a majority: the primary counts its own copy toward the commit majority only at its durable LSN (`majority_candidate_locked` caps self at `persist_durable_lsn`, not the appended tip); `fsync_worker` repurposed timer→demand-driven group-commit flusher (one `fdatasync` over pending appends, advance `durable_lsn`, wake commit-eval) — absorbs the former "WAL group-commit batching" backlog item (see below); recovery integrity verdict (clean vs corruption-truncated WAL) + durable-backed freshness floor (candidate advertises bare `persist_durable_floor`, voter floor = `max(durable_floor, learned_committed)`) + pre-promotion eligibility fence. Establishes the **controllers-never-self-terminate** invariant: a recovery-truncated/behind member that cannot catch up is a LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — never `exit`/`abort`), self-heals on peer return (see `ha-async-io.md` process-lifecycle invariant + `src/controller/CLAUDE.md`). Dependencies: ha-commit-engine (consumes commit-position semantics), ha-wal-replication (standby ACK/catch-up), ha-commit-barrier-rollout (reply gate, unchanged). (specs done 2026-06-18; **IMPLEMENTED 2026-06-18**: 17/17 tasks (1.1–5.3), validate-impl GO, controller suite 0 failures under ASAN; deterministic ack-before-fdatasync E2E RED→GREEN + rolling-SIGKILL soak zero acked-job loss. Verified by tests + Protocol Verification Checklist, NOT by raft.tla (which assumes a durable log). Fixed bug-760 (adjacent `test_fact_hook_append` raced the async flusher — test made durable-deterministic via `persist_flush`). raft.tla durability-model extension deferred unless explicitly opted in.)
|
||||
- [ ] scheduler-allocator-runner — NEW 2026-09-28 (`/kiro-discovery`, Path C). The framework owns `job_alloc_fn` through a new `xce_sched_allocator_iface_t` (accepts guard + plan iterator) and a single `sched_allocator_run()`; both `my_alloc()`s are deleted (seqjob/parjob), and `alloc_iter.h` `init` is retyped to `xce_sched_core_t*`. Behaviour must stay identical (bug-1882/1767/1768). Closes `issues/new_requirements.md` item 4. Brief: `.kiro/specs/scheduler-allocator-runner/brief.md`. Dependencies: scheduler-allocation, scheduler-core (both done).
|
||||
- [ ] scheduler-alter-pend-reasons — NEW 2026-09-28 (`/kiro-discovery`, Path E). An ALTER-resreq job keeps a pend reason for EVERY alternative instead of only the last (`scheduler.c:424` FIXME). Wire shape (user decision): additive `alternatives @3: List(JobPendReason)` on `JobPendReason`, per-alternative attribution; carried end-to-end through controller job record, WAL preason fact, recordings and `job_preason_eq`. Also fixes the confirmed `obj->reason` overwrite leak (`scheduler.c:388`). Brief: `.kiro/specs/scheduler-alter-pend-reasons/brief.md`. Dependencies: libxce-alloc-preason-equality (HARD: extends its `job_preason_eq`); sequence after scheduler-allocator-runner (its "no pend-reason content change" rule).
|
||||
- [ ] application-profile — NEW 2026-09-29 (`/kiro-discovery`, Job Sandbox Execution initiative, Path D; reshaped from `job-execution-mode` by user direction: execution mode is a job attribute determined by the APPLICATION). Admin-defined named application profiles (LSF `-app` precedent) in cluster config: execution mode + options (`direct`|`sandbox`, room for `container`/`vm`), launch type, env + pre/post exec, resource defaults/limits (extensible). Users pick a profile at submit (`--app`), cluster default profile otherwise; queues (scheduling policy) and profiles (application config) are orthogonal — neither restricts the other; only profile-marked overridable attributes may be overridden; access by RBAC permission (new `app:use`/`app:view` with `app` domain type, folded into this spec); apiserver checks profile existence; controller resolves at submit and persists effective values on the job (wire additive, WAL/snapshot field-completeness). apiserver never checks node accounts; worker fails a `direct` job with a clear reason if the exec user is missing. Worker capability advertisement + scheduler placement. Brief: `.kiro/specs/application-profile/brief.md`. Dependencies: none.
|
||||
- [ ] job-sandbox-execution — NEW 2026-09-29 (Job Sandbox Execution). Worker: pluggable execution-backend interface (`direct` refactored + `systemd-sandbox`; non-systemd backends later). Sandbox = existing transient unit + `DynamicUser` with fresh per-job name (never the submitter's), ProtectSystem=strict/ProtectHome/PrivateTmp/NoNewPrivileges, Bind(ReadOnly)Paths for job I/O dirs or worker-created temp workspace (handed to the uid via SO_PEERCRED), `StandardOutput=file:` output, helper auth by unit/cgroup, MANDATORY post-job cleanup/ownership normalization (dynamic uid reuse), systemd ≥ 242 + nss-systemd capability probe; single-node incl. single-node MPI. Viability-checked 2026-09-29. Brief: `.kiro/specs/job-sandbox-execution/brief.md`. Dependencies: application-profile; remote-task-root fix (Direct-Implementation).
|
||||
- [ ] job-sandbox-multinode — NEW 2026-09-29 (Job Sandbox Execution). Multi-node sandbox jobs: cluster-consistent ephemeral uid (expected: controller-allocated, WAL-replicated uid lease from a reserved range), worker-driven remote/MPI launch (no ssh; PMIx/prted via worker), shared-FS ownership normalization, one unit per job per node. Brief: `.kiro/specs/job-sandbox-multinode/brief.md`. Dependencies: job-sandbox-execution, application-profile.
|
||||
- [ ] rbac-persistence — NEW 2026-09-29 (`/kiro-discovery`, raised during application-profile discovery). Closes the deferred `security.md` gap: runtime RBAC mutations (roles, permissions, user-role assignments, Logon auto-assignment) are in-memory only and silently revert to `rbac.yaml` on apiserver restart — including revocations. Durable-or-rejected mutations, audit trail, defaults-vs-runtime reconciliation, multi-apiserver consistency; storage approach (rbac.yaml write-through / controller-replicated entity / embedded store / read-only) decided in design. Brief: `.kiro/specs/rbac-persistence/brief.md`. Dependencies: none (independent of the sandbox initiative; application-profile's API-made `app:use` grants rely on it for durability).
|
||||
- [ ] controller-job-cleanup — NEW 2026-09-29 (`/kiro-discovery`, Path C). Primary-only periodic sweep (configurable `job_cleanup_period`, default 300 s) removes DONE/EXIT jobs finished longer than configurable `job_retention` (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever `jobmgr` delete; refcount-safe `xkvmgr` delete), propagated to scheduler and apiserver (`OnJobDeleted`). Retires the node-local snapshot-only `job_should_retain` finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: `.kiro/specs/controller-job-cleanup/brief.md`. Dependencies: none (extends ha-entity-snapshots).
|
||||
- [ ] job-status-detail-display — NEW 2026-09-28 (`/kiro-discovery`, Path E). End users (REST/`xcecli`) see NO pend reason for ANY job today: the apiserver cgo conversion (`backend/convert.go`) never maps `xce_job_t.pend_reasons`, so `pending_reason` is always empty. Maps it to a structured JSON (including `alternatives` from day one) using libxce's code→text mapping, and renders it in `xcecli` job detail. Open for requirements: also map `run_usage` → `JobRusage` (same gap; reserved vs. used for RUN jobs). Brief: `.kiro/specs/job-status-detail-display/brief.md`. Dependencies: none for flat reasons; scheduler-alter-pend-reasons for per-alternative rendering.
|
||||
|
||||
## Existing Spec Updates (2026-09-28 `info job` discovery)
|
||||
- [ ] scheduler-sim-diagnostics — `info job <jobid>` enhancement: a PEND job shows its pend reason set (per alternative once scheduler-alter-pend-reasons lands; shows the flat reason until then); a RUN job shows per-host/per-resource reserved (`job->allocs`) vs. real used (`job->rusages`), and when no rusage has been reported yet says "resource usage not reported yet" (no fabricated zeros; reserved still shown). Byte units per its Req 6. Read-only. Next: `/kiro-spec-requirements scheduler-sim-diagnostics`. Dependencies: none (alternatives display depends on scheduler-alter-pend-reasons).
|
||||
|
||||
## Future Specs / Backlog (dependency order)
|
||||
|
||||
|
|
@ -80,6 +91,8 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
|
|||
|
||||
Smaller items that likely don't warrant a full spec; pick up opportunistically or bundle into adjacent spec work.
|
||||
|
||||
- [ ] **remote-task units run as root (SECURITY, found 2026-09-29 during job-sandbox discovery)** — `job_start_remote_task` path allocates `task_ctx` (`src/worker/src/controller_connector.c:1204`) and submits it (`:1388`) without ever setting `task_ctx->limits`; `start_systemd_job()` only appends `User=` when `limits != NULL` (`xcecd_cgroups.c:296-312`), so remote tasks of multi-host jobs run in a system unit as root. Confirmed by code reading (not yet live-reproduced). Fix: build `limits` (at least `user_name` = taskhost `execuser`) on the remote-task path; refuse to start with an empty user. Log via `buglog` before fixing. HARD prereq for job-sandbox-execution / job-sandbox-multinode.
|
||||
|
||||
- [x] ~~eliminate src/frontend/Makefile, consolidate Go builds onto meson~~ — **PROMOTED to spec 2026-08-27** (`/kiro-discovery`; brief at `.kiro/specs/eliminate-frontend-makefile/brief.md`). Investigation (recorded 2026-08-21, from `dirty-content-fingerprint` spec's task 4.3) confirmed the risk is real and independently corroborated: `build-server` is separately confirmed BROKEN today (bug-1597, bare `go build` with no CGO flags, `fatal error: client.h: No such file or directory`; `src/docs/deployment.md:243-247` already tells developers to use `/xce-build apiserver` instead). Blast radius is narrower than originally framed: `/xce-build` already builds both binaries via meson exclusively (`build-cli`/`build-server` are dead weight from that path today); only 2 live call sites (`xce-build.sh:240,287`) depend on `make test`, with an already-proven direct-`go test` replacement 3 lines away in the same file. One genuine gap found: no meson `test()` exists for the CGO backend suite (`test-backend`) — that must be added before the Makefile can be safely removed, or backend CGO coverage is lost, not just inconvenienced. Next: `/kiro-spec-init eliminate-frontend-makefile`.
|
||||
- [ ] **job_ctrl direct-mutate path — migrate to commit-then-apply** — **recorded 2026-07-13, investigated 2026-08-19 (finding: real risk, wider scope than originally framed).** `raft-apply-decoupling` deliberately scoped `job_ctrl`'s `mark_job_update`/`mark_job_fail` (admin SIGCONT/SIGSTOP) OUT-OF-BOUNDARY (task 4.1/6.1 Implementation Notes). The 2026-08-19 investigation found the original framing wrong in two ways: (1) **NOT the only remaining call site** — `on_job_stop`/`on_job_resume`/`on_job_ckpt` (`job.c:2067-2222`) share the identical unconverted mutate-then-notify-then-WAL-append pattern, fed by ordinary worker/scheduler traffic, not "rare, admin-only" as originally described; (2) **blast radius (a) is confirmed real, not theoretical** — the pre-commit mutation is synchronously pushed live to BOTH the scheduler (`sched_job_update_fun`) AND every connected apiserver/client (`apiserver_job_update_fun`) before the commit barrier resolves; on barrier timeout the admin gets a failure reply but `job->job.state` is never rolled back and no retraction is ever sent — matching the bug-795/828/851/1060 shape. Live-apply vs replay-apply (b) is narrower than feared for `job_ctrl` specifically (the `allocs != NULL` gate mutually excludes it from the converted deferred-apply fact types), but the investigation surfaced a SEPARATE bug underpinning that safety argument: `job_ctrl` (`job.c:999`) holds only a read lock while mutating job state — every other mutator in this file takes a write lock — so two concurrent admin commands on the same job aren't mutually excluded, undermining the "same lock, no staleness window" claim the replay-safety argument depends on (already flagged by a `raft-apply-decoupling` task 2.1 reviewer, never fixed). **Recommendation**: promote to a small follow-up spec covering all four call sites together (bounded, well-precedented fix — same shape as the fact-receipt conversion `raft-apply-decoupling` already built for job submit); the lock-type bug is independently fixable and low-risk regardless of when/whether the larger migration happens. Not yet promoted — user decision 2026-08-19 to correct scope/framing here first, promote later.
|
||||
- [x] **WAL-trim cursor-bound guard (bug-1147; surfaced 2026-06-25 by raft-drive-async design review)** — **IMPLEMENTED** as part of `raft-drive-async` task 3.2 (`raft-drive-async` is IMPLEMENTED, validate-impl GO 2026-07-18; see above). `trim_wal_segments` (`persist.c` ~2597-2644) now computes `floor = repl_trim_lsn(&p->repl, active_seg_lsn, p->config.snap_interval_s)` — folding in active-standby `acked_lsn` and disconnected-slot `last_disconnected_lsn` — and only unlinks segments below that floor; the code comment at `persist.c:2634` explicitly labels this the "(bug-1147 guard)". Far-behind/empty join (cursor=0) is additionally covered now that `entity-bearing snapshots` (strategy B) has landed as `ha-entity-snapshots` (see above), with a live decision path (`ha.c:4413`, "< oldest_wal_lsn → requesting snapshot transfer"). Related: bug-795, bug-828, bug-1060.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue