myxce/.kiro/specs/job-sandbox-execution/brief.md

Brief: job-sandbox-execution

Problem

A job whose execution mode is sandbox (from application-profile) must run on a worker node:

• without any account belonging to the submitter;
• under an unprivileged, per-job, ephemeral uid;
• isolated from the host and from other jobs, including other jobs by the same submitter;
• with its input and output directories, or a temporary workspace, mounted in.

Today the worker can only run a job as a named account. There is no isolation, and a jailed process could not even get its output back to the user.

This is the worker half of the job-sandbox initiative. It covers single-node execution, including single-node MPI in one unit. Multi-node is handled by job-sandbox-multinode.

Current State

Code exploration, 2026-09-29.

Launch

• xcecd must run as root (worker/src/main.c:300-303).
• In the production DBUS_MANAGER build, each job starts as a systemd transient unit xcecd-worker-<jobid> via D-Bus StartTransientUnit (xcecd_cgroups.c start_systemd_job, xcecd_dbus_manager.c:73-76).
• The unit runs ExecStart=/bin/sh -c "<path>/xcecd_worker ..." with RemainAfterExit=1, StandardInput=null, KillMode=control-group and User=<execuser> (xcecd_cgroups.c:300-316). Optional cgroup limits come from exdata.
• The identity switch happens only through User=. There is no setuid/initgroups anywhere in src/.
• No sandboxing properties are set: no DynamicUser, Protect*, PrivateTmp, NoNewPrivileges, WorkingDirectory or BindPaths. The job runs in /.

Job helper

• xcecd_worker receives the command over /var/run/.xcecd.sock, which is mode 0666 (main.c:641).
• It uv_spawns /bin/sh -c cmd (uv_proc.c:270-288).
• exechome/execcwd are copied but never applied (xcecd_worker_utils.c:146-157).

Output

• Written to /tmp/job_<id>_task<N>.out (job_output_path.h:24-41), created O_EXCL|O_NOFOLLOW 0600 by the helper.
• Root xcecd serves it through handle_fetchfile, which accepts only that exact path (bug-1508). See also bug-1868, the /tmp symlink hazard.

Execution backends

• There is no abstraction; the launch code is tied to systemd, with a #ifndef DBUS_MANAGER process-manager fallback.
• The user wants to support systems without systemd in future, so the backend must be pluggable.

Known bug (fix first, separately)

• Remote-task units start with limits == NULL, so no User= is set and they run as root (controller_connector.c:1204, :1388; xcecd_cgroups.c:296-312).

Desired Outcome

• The worker has a pluggable execution-backend interface:

  • direct, today's behaviour, refactored behind the interface;
  • systemd-sandbox, new.

  A backend without systemd (for example namespaces) can be added later without touching callers. The interface stays usable for future container/vm modes.

• A sandbox job runs in its transient unit under a systemd DynamicUser ephemeral uid:

  • The unit's user name is a new, short, per-job name (for example xj<jobid>, 31 characters or fewer). It must never be the submitter's name. If a static account with that name existed, systemd would silently use it, and jobs from the same submitter would share a uid.
  • ProtectSystem=strict, ProtectHome, PrivateTmp and an explicit NoNewPrivileges=yes.
  • HOME and WorkingDirectory are set to the workspace.
  • One unit per job per node, so that MPI shared memory works within a node.

• Directories:

  • Job-specified input directories are mounted read-only (BindReadOnlyPaths), and output directories read-write (BindPaths).
  • If none are specified, the worker creates a private temporary per-job workspace and mounts it.
  • The workspace can't be handed to the uid in advance, because the uid does not exist until the unit starts. The worker learns the uid when the helper connects (SO_PEERCRED) and hands the workspace to it. It must never be made world-writable.

• Output:

  • stdout and stderr are opened by systemd outside the sandbox (StandardOutput=file:<host path>). Avoid PrivateTmp invisibility and ownership problems.
  • Existing output retrieval (handle_fetchfile, the jobs output API) serves sandbox job output, including files left in the workspace.

• Helper authentication: xcecd identifies a genuine job helper by its unit or cgroup, not by uid. A sandboxed job must not be able to pose as a helper over the 0666 socket.

• Mandatory cleanup after the job:

  • Remove the workspace, or re-own it (policy decides).
  • Normalise ownership of everything the job's uid wrote in any writable mount.

  Dynamic uids are reused, and systemd does not check who owns files on disk. Leftovers would otherwise let a later job that gets the same uid read or change them.

• Capability:

  • The worker detects its sandbox capability (systemd 242 or later, systemd in nsswitch.conf passwd/group, and D-Bus property support) and advertises it to application-profile's placement.
  • It refuses, with a clear error, a sandbox job it cannot honour.

• Direct mode is unchanged.

Approach

Chosen: approach A, systemd sandbox properties on the existing transient unit, behind a pluggable backend interface (user decision, 2026-09-29).

Rejected alternatives:

• B, XCE-built namespaces. Amounts to writing a container runtime in C as root; a possible later backend.
• C, container runtime (Apptainer or OCI). Adds a dependency and image management; a possible later container mode.

Viability check (2026-09-29, systemd NEWS, systemd.exec(5), v239 sources):

• All properties can be set on transient units on 239 and later.

• BindPaths has D-Bus signature a(ssbt), with flags 0 or MS_REC. ProtectSystem/ProtectHome are strings.

• The minimum is systemd 242 (user decision). From 242, RestrictSUIDSGID and NoNewPrivileges are implied, which closes CVE-2019-3843/3844-style takeover of a reused uid through SUID binaries. This excludes stock RHEL 8 (239).

• Uid capacity is about 4,336 per node, shared with other DynamicUser services. pick_uid gives up after 100 tries. Stale RemainAfterExit units hold their uid, so units must be stopped reliably.

• There is no idmap option for BindPaths or BindReadOnlyPaths. Ownership of mounted directories is XCE's problem (handover plus cleanup).

• DynamicUser is not a full sandbox:

  • the host network is shared (MPI needs it);
  • /proc still shows other jobs (ProtectProc needs 247+);
  • there is no syscall filter.

  The requirements should decide whether to add ProtectProc where available, PrivateDevices, or a baseline SystemCallFilter, and must document the remaining exposure.

• Tmpfs pages count against the job's MemoryMax. RDMA needs LimitMEMLOCK=infinity.

Scope

• In:
  • The backend interface, with direct refactored behind it and systemd-sandbox added.
  • DynamicUser unit properties.
  • The workspace lifecycle: create, hand over, mount, clean up or re-own.
  • Input/output mounts.
  • StandardOutput=file: output and retrieval.
  • Helper authentication by unit or cgroup.
  • Capability detection and advertisement.
  • Single-node MPI in one unit.
  • Tests: unit tests plus a live test on a systemd 242+ node.
• Out:
  • The submission side: mode field, policy, placement (application-profile).
  • Multi-node sandbox launch (job-sandbox-multinode).
  • Non-systemd backends and container/vm modes (the interface only, not the implementations).
  • The remote-task root bug, which is fixed separately, first.

Boundary Candidates

• The execution-backend interface and direct refactor.
• Building the systemd-sandbox unit (properties, naming, capability probe).
• The workspace and mount lifecycle (create, hand over via SO_PEERCRED, clean up or normalise ownership).
• Output capture and retrieval.
• Helper authentication.

Out of Boundary

• Job mode selection and policy.
• Cross-node uid consistency and remote launch.
• Worker cgroup resource limits, which are unchanged and simply apply to sandbox units too.

Upstream / Downstream

• Upstream:
  • application-profile: mode and directory fields on job_startinfo_t, and capability-based placement.
  • The remote-task root fix.
• Downstream:
  • job-sandbox-multinode.
  • Future non-systemd or container backends.

Existing Spec Touchpoints

• Extends: none.
• Adjacent:
  • xce-node-manager (worker node privileges)
  • worker-rejoin-resync (a sandbox job's state across a worker rejoin)
  • output retrieval (bug-1508 and bug-1868 history)

Constraints

• Nodes need systemd 242 or later, with nss-systemd in nsswitch.conf. This is a documented deployment prerequisite, and capability detection must enforce it.
• Direct-mode behaviour must be byte-for-byte unchanged (regression tests).
• Root code must fail closed. On any failure to set up the sandbox, the job is refused; it never falls back to running unsandboxed or as root.
• Cleanup is mandatory and must be retried and survive a worker crash or restart (for example by sweeping orphaned workspaces at startup).
• TDD: testsuite/ first. ASAN must be clean.