myxce/.kiro/specs/job-sandbox-multinode/brief.md

Brief: job-sandbox-multinode

Problem

Multi-node jobs (MPI via openmpi/mpich launch types, or XCE fan-out remote tasks) must also run in sandbox mode (user decision, 2026-09-29: "include multi-node"). A per-node, per-job DynamicUser sandbox (job-sandbox-execution) does not work across nodes as it is:

• The uid is not consistent across nodes. systemd tries a hash of the unit's user name first but falls back to a random uid on collision, and there is no way to pin it. MPI ranks on different nodes, and files on shared filesystems, see different uids.
• ssh-based launchers cannot work. A sandbox user has no account, no keys and no usable home. Today MPI goes through xrun -host(s) (controller_connector.c:752-756, :993-997), and other launch types go through job_start_remote_task → xcecli_runtask.
• Shared filesystems. Writes over NFS/AUTH_SYS land under raw uids 61184–65519 cluster-wide:
  • the submitter cannot read them;
  • they can collide with real LDAP or AD uids in that range (collision checks only cover what NSS resolves on that node);
  • root_squash does not help.
• MPI transports. Intra-node shared memory (vader/sm with CMA, XPMEM, KNEM) needs the ranks on a node to share a uid, so one unit per job per node is required. The PMIx session directory also needs a writable, private location.

Current State

• Remote tasks. The worker receives xce_taskhost_t run messages and starts a transient unit per task. That unit currently runs as root, because limits is never set on that path (bug; fixed separately before this spec).
• MPI launch wraps the command with xrun and runs it once locally with ntasks=1.
• No cross-node identity coordination exists. Each worker decides its unit independently.

Desired Outcome

• A sandbox job spanning N nodes runs with one sandbox unit per node, all under the same execution uid, so MPI and shared-filesystem access behave consistently.
• All remote launches go through XCE workers, never ssh. MPI integration works through the worker, for example a PMIx server or daemon-launched prted (design decides).
• Files the job writes to shared storage have a well-defined owner and a well-defined post-job normalisation. They never collide with real directory uids. The submitter can retrieve outputs through XCE.
• All of job-sandbox-execution's guarantees hold on every node: workspace, cleanup, helper authentication and fail-closed behaviour.

Approach

Design must research and decide. The core problem is cluster-consistent ephemeral uids. Candidates:

• (a) XCE-managed uid pool. The controller (primary) allocates a uid from a cluster-reserved range that must not overlap LDAP/AD, and replicates the lease through the WAL. Each worker runs the unit with that fixed uid, using a static-uid transient user rather than DynamicUser (for example User= a numeric uid with nss-systemd or a worker NSS module), keeping the same protections.
• (b) DynamicUser with a deterministic name. Hope the hash matches, and detect and handle mismatches. Fragile.
• (c) A different backend for multi-node. For example the namespace backend with explicit uid mapping, using the pluggable interface from job-sandbox-execution.

Option (a) is expected, but it adds a controller-replicated lease (HA and persistence), so this spec must follow the full workflow.

Scope

• In:
  • Consistent uid allocation across nodes, and release on job end or failure.
  • Sandbox remote-task and MPI launch through workers.
  • Shared-filesystem ownership policy and post-job normalisation.
  • Intra-node MPI with one unit per job per node.
  • Failure handling: a node fails mid-job, or the uid lease is lost on controller failover.
• Out:
  • Single-node sandbox mechanics (job-sandbox-execution).
  • The submission option and policy (application-profile).
  • Container or vm modes.

Boundary Candidates

• Cluster uid lease allocation (controller, WAL-replicated).
• Worker multi-node sandbox launch (remote tasks and MPI/PMIx).
• Shared-storage ownership normalisation.

Out of Boundary

• The per-node sandbox unit construction (reused from job-sandbox-execution).
• The scheduler's host selection for multi-node jobs (unchanged apart from capability filtering).

Upstream / Downstream

• Upstream:
  • job-sandbox-execution (HARD).
  • application-profile (HARD).
  • The remote-task root bug fix (HARD).
  • The WAL/replication machinery, if option (a) is chosen.
• Downstream: future container or vm multi-node modes.

Existing Spec Touchpoints

• Extends: job-sandbox-execution (reuses its backend interface).
• Adjacent:
  • controller-job-cleanup (lease release relative to job removal)
  • ha-wal-replication

Constraints

• The uid range must be configurable and must not overlap directory-service uids.
• Leases must survive controller failover, and nothing may leak on worker loss.
• Fail closed. Never fall back to ssh, root or direct mode.
• HA/persistence rules apply if leases are replicated.