Rongsong Shen
7fe9b22381
New specs from /kiro-discovery (2026-09-28/29), added to roadmap.md: - job-status-detail-display: show pending reasons and reserved-vs-used resources in REST and xcecli. Requirements generated (7 requirements). - scheduler-alter-pend-reasons: keep a pending reason for every alternative resreq (per-alternative JobPendReason) and fix the obj->reason leak. - controller-job-cleanup: primary-only sweep that removes finished jobs past a configurable retention, via a replicated WAL removal fact. - application-profile: admin-defined application profiles (execution mode, launch type, env, resource defaults) chosen per job, with app:use/app:view RBAC. Queues and profiles stay orthogonal. - job-sandbox-execution / job-sandbox-multinode: run jobs in a systemd DynamicUser sandbox behind a pluggable execution backend. - rbac-persistence: make RBAC changes made through the API survive an apiserver restart (closes the security.md gap). roadmap.md also records the scheduler-sim-diagnostics info job update and a security item: remote-task units appear to run as root (found by code reading). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
107 lines
5 KiB
Markdown
107 lines
5 KiB
Markdown
# Brief: job-sandbox-multinode
|
||
|
||
## Problem
|
||
Multi-node jobs (MPI via `openmpi`/`mpich` launch types, or XCE fan-out remote
|
||
tasks) must also run in sandbox mode (user decision, 2026-09-29: "include
|
||
multi-node"). A per-node, per-job DynamicUser sandbox (`job-sandbox-execution`)
|
||
does not work across nodes as it is:
|
||
- **The uid is not consistent across nodes.** systemd tries a hash of the unit's
|
||
user name first but falls back to a random uid on collision, and there is no
|
||
way to pin it. MPI ranks on different nodes, and files on shared filesystems,
|
||
see different uids.
|
||
- **ssh-based launchers cannot work.** A sandbox user has no account, no keys and
|
||
no usable home. Today MPI goes through `xrun -host(s)`
|
||
(`controller_connector.c:752-756`, `:993-997`), and other launch types go
|
||
through `job_start_remote_task` → `xcecli_runtask`.
|
||
- **Shared filesystems.** Writes over NFS/AUTH_SYS land under raw uids
|
||
61184–65519 cluster-wide:
|
||
- the submitter cannot read them;
|
||
- they can collide with real LDAP or AD uids in that range (collision checks
|
||
only cover what NSS resolves on that node);
|
||
- `root_squash` does not help.
|
||
- **MPI transports.** Intra-node shared memory (vader/sm with CMA, XPMEM, KNEM)
|
||
needs the ranks on a node to share a uid, so one unit per job per node is
|
||
required. The PMIx session directory also needs a writable, private location.
|
||
|
||
## Current State
|
||
- **Remote tasks.** The worker receives `xce_taskhost_t` run messages and starts
|
||
a transient unit per task. That unit currently **runs as root**, because
|
||
`limits` is never set on that path (bug; fixed separately before this spec).
|
||
- MPI launch wraps the command with `xrun` and runs it once locally with
|
||
ntasks=1.
|
||
- **No cross-node identity coordination exists.** Each worker decides its unit
|
||
independently.
|
||
|
||
## Desired Outcome
|
||
- A sandbox job spanning N nodes runs with **one sandbox unit per node**, all
|
||
under the **same execution uid**, so MPI and shared-filesystem access behave
|
||
consistently.
|
||
- All remote launches go through XCE workers, never ssh. MPI integration works
|
||
through the worker, for example a PMIx server or daemon-launched `prted`
|
||
(design decides).
|
||
- Files the job writes to shared storage have a well-defined owner and a
|
||
well-defined post-job normalisation. They never collide with real directory
|
||
uids. The submitter can retrieve outputs through XCE.
|
||
- All of `job-sandbox-execution`'s guarantees hold on every node: workspace,
|
||
cleanup, helper authentication and fail-closed behaviour.
|
||
|
||
## Approach
|
||
Design must research and decide. The core problem is **cluster-consistent
|
||
ephemeral uids**. Candidates:
|
||
- **(a) XCE-managed uid pool.** The controller (primary) allocates a uid from a
|
||
cluster-reserved range that must not overlap LDAP/AD, and replicates the
|
||
lease through the WAL. Each worker runs the unit with that fixed uid, using a
|
||
static-uid transient user rather than DynamicUser (for example `User=` a
|
||
numeric uid with `nss-systemd` or a worker NSS module), keeping the same
|
||
protections.
|
||
- **(b) DynamicUser with a deterministic name.** Hope the hash matches, and
|
||
detect and handle mismatches. Fragile.
|
||
- **(c) A different backend for multi-node.** For example the namespace backend
|
||
with explicit uid mapping, using the pluggable interface from
|
||
`job-sandbox-execution`.
|
||
|
||
Option (a) is expected, but it adds a controller-replicated lease (HA and
|
||
persistence), so this spec must follow the full workflow.
|
||
|
||
## Scope
|
||
- **In**:
|
||
- Consistent uid allocation across nodes, and release on job end or failure.
|
||
- Sandbox remote-task and MPI launch through workers.
|
||
- Shared-filesystem ownership policy and post-job normalisation.
|
||
- Intra-node MPI with one unit per job per node.
|
||
- Failure handling: a node fails mid-job, or the uid lease is lost on
|
||
controller failover.
|
||
- **Out**:
|
||
- Single-node sandbox mechanics (`job-sandbox-execution`).
|
||
- The submission option and policy (`application-profile`).
|
||
- Container or vm modes.
|
||
|
||
## Boundary Candidates
|
||
- Cluster uid lease allocation (controller, WAL-replicated).
|
||
- Worker multi-node sandbox launch (remote tasks and MPI/PMIx).
|
||
- Shared-storage ownership normalisation.
|
||
|
||
## Out of Boundary
|
||
- The per-node sandbox unit construction (reused from `job-sandbox-execution`).
|
||
- The scheduler's host selection for multi-node jobs (unchanged apart from
|
||
capability filtering).
|
||
|
||
## Upstream / Downstream
|
||
- **Upstream**:
|
||
- `job-sandbox-execution` (HARD).
|
||
- `application-profile` (HARD).
|
||
- The remote-task root bug fix (HARD).
|
||
- The WAL/replication machinery, if option (a) is chosen.
|
||
- **Downstream**: future container or vm multi-node modes.
|
||
|
||
## Existing Spec Touchpoints
|
||
- **Extends**: `job-sandbox-execution` (reuses its backend interface).
|
||
- **Adjacent**:
|
||
- `controller-job-cleanup` (lease release relative to job removal)
|
||
- `ha-wal-replication`
|
||
|
||
## Constraints
|
||
- The uid range must be configurable and must not overlap directory-service uids.
|
||
- Leases must survive controller failover, and nothing may leak on worker loss.
|
||
- Fail closed. Never fall back to ssh, root or direct mode.
|
||
- HA/persistence rules apply if leases are replicated.
|