myxce/.kiro/specs/job-sandbox-execution/brief.md

# Brief: job-sandbox-execution

## Problem
A job whose execution mode is `sandbox` (from `application-profile`) must run on a
worker node:
- without any account belonging to the submitter;
- under an unprivileged, per-job, ephemeral uid;
- isolated from the host and from other jobs, including other jobs by the same
  submitter;
- with its input and output directories, or a temporary workspace, mounted in.

Today the worker can only run a job as a named account. There is no isolation,
and a jailed process could not even get its output back to the user.

This is the worker half of the job-sandbox initiative. It covers single-node
execution, including single-node MPI in one unit. Multi-node is handled by
`job-sandbox-multinode`.

## Current State
Code exploration, 2026-09-29.

**Launch**
- `xcecd` must run as root (`worker/src/main.c:300-303`).
- In the production `DBUS_MANAGER` build, each job starts as a systemd
  **transient unit** `xcecd-worker-<jobid>` via D-Bus `StartTransientUnit`
  (`xcecd_cgroups.c` `start_systemd_job`, `xcecd_dbus_manager.c:73-76`).
- The unit runs `ExecStart=/bin/sh -c "<path>/xcecd_worker ..."` with
  `RemainAfterExit=1`, `StandardInput=null`, `KillMode=control-group` and
  `User=<execuser>` (`xcecd_cgroups.c:300-316`). Optional cgroup limits come from
  exdata.
- The identity switch happens only through `User=`. There is no
  `setuid`/`initgroups` anywhere in `src/`.
- No sandboxing properties are set: no `DynamicUser`, `Protect*`, `PrivateTmp`,
  `NoNewPrivileges`, `WorkingDirectory` or `BindPaths`. The job runs in `/`.

**Job helper**
- `xcecd_worker` receives the command over `/var/run/.xcecd.sock`, which is mode
  **0666** (`main.c:641`).
- It `uv_spawn`s `/bin/sh -c cmd` (`uv_proc.c:270-288`).
- `exechome`/`execcwd` are copied but never applied
  (`xcecd_worker_utils.c:146-157`).

**Output**
- Written to `/tmp/job_<id>_task<N>.out` (`job_output_path.h:24-41`), created
  `O_EXCL|O_NOFOLLOW` 0600 by the helper.
- Root `xcecd` serves it through `handle_fetchfile`, which accepts only that exact
  path (bug-1508). See also bug-1868, the `/tmp` symlink hazard.

**Execution backends**
- There is no abstraction; the launch code is tied to systemd, with a
  `#ifndef DBUS_MANAGER` process-manager fallback.
- The user wants to support systems without systemd in future, so the backend
  must be pluggable.

**Known bug (fix first, separately)**
- Remote-task units start with `limits == NULL`, so no `User=` is set and they
  **run as root** (`controller_connector.c:1204`, `:1388`;
  `xcecd_cgroups.c:296-312`).

## Desired Outcome
- The worker has a **pluggable execution-backend interface**:
  - `direct`, today's behaviour, refactored behind the interface;
  - `systemd-sandbox`, new.

  A backend without systemd (for example namespaces) can be added later without
  touching callers. The interface stays usable for future `container`/`vm`
  modes.
- A sandbox job runs in its transient unit under a **systemd `DynamicUser`
  ephemeral uid**:
  - The unit's user name is a new, short, per-job name (for example
    `xj<jobid>`, 31 characters or fewer). It must never be the submitter's name.
    If a static account with that name existed, systemd would silently use it,
    and jobs from the same submitter would share a uid.
  - `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp` and an explicit
    `NoNewPrivileges=yes`.
  - `HOME` and `WorkingDirectory` are set to the workspace.
  - One unit per job per node, so that MPI shared memory works within a node.
- **Directories**:
  - Job-specified input directories are mounted read-only
    (`BindReadOnlyPaths`), and output directories read-write (`BindPaths`).
  - If none are specified, the worker creates a private temporary per-job
    workspace and mounts it.
  - The workspace can't be handed to the uid in advance, because the uid does
    not exist until the unit starts. The worker learns the uid when the helper
    connects (`SO_PEERCRED`) and hands the workspace to it. It must never be made
    world-writable.
- **Output**:
  - stdout and stderr are opened by systemd outside the sandbox
    (`StandardOutput=file:<host path>`). Avoid PrivateTmp invisibility and
    ownership problems.
  - Existing output retrieval (`handle_fetchfile`, the `jobs output` API) serves
    sandbox job output, including files left in the workspace.
- **Helper authentication**: `xcecd` identifies a genuine job helper by its unit
  or cgroup, not by uid. A sandboxed job must not be able to pose as a helper
  over the 0666 socket.
- **Mandatory cleanup after the job**:
  - Remove the workspace, or re-own it (policy decides).
  - Normalise ownership of everything the job's uid wrote in any writable mount.

  Dynamic uids are reused, and systemd does not check who owns files on disk.
  Leftovers would otherwise let a later job that gets the same uid read or change
  them.
- **Capability**:
  - The worker detects its sandbox capability (systemd 242 or later, `systemd`
    in `nsswitch.conf` passwd/group, and D-Bus property support) and advertises
    it to `application-profile`'s placement.
  - It refuses, with a clear error, a sandbox job it cannot honour.
- **Direct mode is unchanged.**

## Approach
**Chosen: approach A, systemd sandbox properties on the existing transient unit,
behind a pluggable backend interface** (user decision, 2026-09-29).

Rejected alternatives:
- **B, XCE-built namespaces.** Amounts to writing a container runtime in C as
  root; a possible later backend.
- **C, container runtime (Apptainer or OCI).** Adds a dependency and image
  management; a possible later `container` mode.

Viability check (2026-09-29, systemd NEWS, systemd.exec(5), v239 sources):
- All properties can be set on transient units on 239 and later.
- `BindPaths` has D-Bus signature `a(ssbt)`, with flags 0 or `MS_REC`.
  `ProtectSystem`/`ProtectHome` are strings.
- **The minimum is systemd 242** (user decision). From 242, `RestrictSUIDSGID`
  and `NoNewPrivileges` are implied, which closes CVE-2019-3843/3844-style
  takeover of a reused uid through SUID binaries. This excludes stock RHEL 8
  (239).
- Uid capacity is about 4,336 per node, shared with other DynamicUser services.
  `pick_uid` gives up after 100 tries. Stale `RemainAfterExit` units hold their
  uid, so units must be stopped reliably.
- There is no `idmap` option for BindPaths or BindReadOnlyPaths. Ownership of
  mounted directories is XCE's problem (handover plus cleanup).
- DynamicUser is **not a full sandbox**:
  - the host network is shared (MPI needs it);
  - `/proc` still shows other jobs (`ProtectProc` needs 247+);
  - there is no syscall filter.

  The requirements should decide whether to add `ProtectProc` where available,
  `PrivateDevices`, or a baseline `SystemCallFilter`, and must document the
  remaining exposure.
- Tmpfs pages count against the job's `MemoryMax`. RDMA needs
  `LimitMEMLOCK=infinity`.

## Scope
- **In**:
  - The backend interface, with `direct` refactored behind it and
    `systemd-sandbox` added.
  - DynamicUser unit properties.
  - The workspace lifecycle: create, hand over, mount, clean up or re-own.
  - Input/output mounts.
  - `StandardOutput=file:` output and retrieval.
  - Helper authentication by unit or cgroup.
  - Capability detection and advertisement.
  - Single-node MPI in one unit.
  - Tests: unit tests plus a live test on a systemd 242+ node.
- **Out**:
  - The submission side: mode field, policy, placement (`application-profile`).
  - Multi-node sandbox launch (`job-sandbox-multinode`).
  - Non-systemd backends and container/vm modes (the interface only, not the
    implementations).
  - The remote-task root bug, which is fixed separately, first.

## Boundary Candidates
- The execution-backend interface and `direct` refactor.
- Building the systemd-sandbox unit (properties, naming, capability probe).
- The workspace and mount lifecycle (create, hand over via `SO_PEERCRED`, clean
  up or normalise ownership).
- Output capture and retrieval.
- Helper authentication.

## Out of Boundary
- Job mode selection and policy.
- Cross-node uid consistency and remote launch.
- Worker cgroup resource limits, which are unchanged and simply apply to sandbox
  units too.

## Upstream / Downstream
- **Upstream**:
  - `application-profile`: mode and directory fields on `job_startinfo_t`, and
    capability-based placement.
  - The remote-task root fix.
- **Downstream**:
  - `job-sandbox-multinode`.
  - Future non-systemd or container backends.

## Existing Spec Touchpoints
- **Extends**: none.
- **Adjacent**:
  - `xce-node-manager` (worker node privileges)
  - `worker-rejoin-resync` (a sandbox job's state across a worker rejoin)
  - output retrieval (bug-1508 and bug-1868 history)

## Constraints
- Nodes need systemd 242 or later, with `nss-systemd` in `nsswitch.conf`. This is
  a documented deployment prerequisite, and capability detection must enforce
  it.
- Direct-mode behaviour must be byte-for-byte unchanged (regression tests).
- Root code must fail closed. On any failure to set up the sandbox, the job is
  refused; it never falls back to running unsandboxed or as root.
- Cleanup is mandatory and must be retried and survive a worker crash or restart
  (for example by sweeping orphaned workspaces at startup).
- TDD: `testsuite/` first. ASAN must be clean.