diff --git a/.kiro/specs/application-profile/brief.md b/.kiro/specs/application-profile/brief.md index 0a95a2b5..80abd2e9 100644 --- a/.kiro/specs/application-profile/brief.md +++ b/.kiro/specs/application-profile/brief.md @@ -45,6 +45,18 @@ consumed by `service-supervision` (worker probes and restarts), `service-endpoint` (ports, endpoints). **Update 2026-10-12 (discussion; user decisions):** +- **Split: execution modes move to `job-sandbox-execution`.** This spec is + the profile core: config, access and submit, job state (name, dispatch + record), the outer pre/post-exec layer. Its schema has `mode:` but + accepts only `direct`. The `sandbox` mode with its options, I/O + directories, worker capability advertisement and scheduler placement by + mode belong to `job-sandbox-execution`, since nothing uses a non-`direct` + mode before the sandbox exists. The worker's clear launch error for a + missing `direct` exec user stays here. +- **Order with `job-config-merge`:** requirements and design of this spec + first (it fixes the profile schema), then `job-config-merge`'s design; + implementation tasks interleave (schema/config, merge library, submit + validation and dispatch record). - **Own configuration file.** Profiles live in a new `src/configs/applications.yaml` with its own JSON schema, mapped into `ClusterConfig` and `Conf4Sched` like queues and limits. See "Profile @@ -144,9 +156,9 @@ fits XCE's LSF-style job model. - **Admins define named application profiles** in the cluster configuration. The first version contains the following (user decision; the format must be extensible for "maybe others"): - 1. **Execution mode and its options.** `direct` or `sandbox` now, with room for - `container`/`vm`. Sandbox options include the allowed input/output - prefixes and isolation extras. + 1. **Execution mode.** The `mode:` field, accepting only `direct` in this + spec; `job-sandbox-execution` adds `sandbox` with its options (allowed + input/output prefixes, isolation extras), later `container`/`vm`. 2. **Launch type.** Serial, `openmpi` or `mpich`. 3. **Environment and pre/post-exec.** Environment variables the application needs, and the outer pre/post-exec commands. @@ -210,9 +222,9 @@ fits XCE's LSF-style job model. user doesn't exist on the node, the worker fails the job at launch with a clear, user-visible reason, instead of today's opaque systemd failure. - Account-less users are served by profiles whose mode is `sandbox`. -- **Placement.** A job is placed only on workers whose advertised capabilities - support its effective mode (sandbox needs systemd 242 or later and - `nss-systemd`). +- **Placement by mode** (moved to `job-sandbox-execution`): a job is placed + only on workers whose advertised capabilities support its mode (sandbox + needs systemd 242 or later and `nss-systemd`). - **Ownership.** The submitter remains the owner for RBAC, limits and accounting in every mode. @@ -248,9 +260,9 @@ fits XCE's LSF-style job model. launch type, env, profile pre/post-exec) are read from the profile at dispatch, as queue pre/post-exec is today (`get_queue_exec()`, `controller/queue.c:199`). -- **Workers advertise their supported execution modes.** The scheduler filters - hosts by the job's effective mode. The design picks the mechanism, for example - a host attribute or boolean resource. +- **Workers advertise their supported execution modes** and the scheduler + filters hosts by mode: `job-sandbox-execution` (the design picks the + mechanism, e.g. a host attribute or boolean resource). ### Profile configuration - **One file per entity type**, as for hosts, queues, resources, limits, @@ -527,9 +539,13 @@ Decided 2026-10-12 (user), so a rename never strands jobs: - RBAC permission for profile use. - Submit surfaces: REST, `xcecli --app`, harness, libxce. - The profile name and the job's own values persisted on the job. - - Execution-mode and I/O-directory fields end to end. - - Capability advertisement and placement. + - The `mode:` schema field (only `direct`). - A worker launch error when a `direct` exec user is missing. + - Dispatch record (scheduling and execution values, content hash) and the + retryable dispatch failure for a missing or changed profile. + - Profile pre/post-exec as the outer layer (moved from the queue), with + a clean environment; the reserved `XCE_` prefix. + - Aliases, live-upgrade warning, pend reason for a missing profile. - Profile name and mode shown in job details. - Listing available profiles (REST and xcecli). - Profile `--dry-run` support: the effective values a job would get @@ -552,12 +568,12 @@ Decided 2026-10-12 (user), so a rename never strands jobs: the outer layer and reserved `XCE_` instead of an env denylist; moving pending jobs deferred to `job-modify`, profile aliases in v1; override granularity (no markers; additive rule), limits a job may lower (profile limits are maximums), personal presets - (client side), `app:override` and inheritance (deferred). - - Splitting the spec: execution mode, I/O directories, capability - advertisement and placement as a separate spec (or folded into - `job-sandbox-execution`), leaving this spec the profile core. + (client side), `app:override` and inheritance (deferred); execution modes + split into `job-sandbox-execution`. - **Out**: - - The worker sandbox mechanics (`job-sandbox-execution`). + - The `sandbox` mode and its options, I/O directories, capability + advertisement, placement by mode and the worker sandbox mechanics + (`job-sandbox-execution`). - Multi-node sandbox (`job-sandbox-multinode`). - Container or vm execution itself (only room for them in the schema). - Auth backends and JWT claims. @@ -569,7 +585,6 @@ Decided 2026-10-12 (user), so a rename never strands jobs: - Submit and listing surfaces (REST, xcecli, harness, libxce). - Persisting the profile name and the job's own values (wire, WAL submit fact, snapshot). -- Capability advertisement and scheduler placement. ## Out of Boundary - Worker execution backends and sandbox mechanics. @@ -587,9 +602,10 @@ Decided 2026-10-12 (user), so a rename never strands jobs: - The config pipeline (`clusterconfig-schema-split`, `config-upgrade-snapshot`, `admin-config-tool`). - RBAC (`rbac_check.go`). - - Resource/attribute reporting from worker to scheduler. - **Downstream**: - - `job-sandbox-execution` consumes the effective mode, options and directories. + - `job-sandbox-execution` adds the `sandbox` mode, its options and I/O + directories, capability advertisement and placement, on top of this + spec's `mode:` field and dispatch record. - `job-sandbox-multinode`. - Future `container`/`vm` modes, which add profile attributes. - `job-status-detail-display`, which can show the profile and mode. @@ -618,6 +634,6 @@ Decided 2026-10-12 (user), so a rename never strands jobs: - Input validation (`security.md`): the job's additions and paths are validated server-side. No path traversal, and mount prefixes are restricted by the profile. -- Size: this is likely 20 or more tasks. If requirements confirm that, split - as listed under "Open for requirements". +- Size: still large after the split (config, access and submit, job state, + outer layer); one coherent feature, testable end to end with `direct`. - HA/persistence spec: use the full 3-phase workflow. diff --git a/.kiro/specs/job-sandbox-execution/brief.md b/.kiro/specs/job-sandbox-execution/brief.md index e1571f63..e5dbb406 100644 --- a/.kiro/specs/job-sandbox-execution/brief.md +++ b/.kiro/specs/job-sandbox-execution/brief.md @@ -1,5 +1,16 @@ # Brief: job-sandbox-execution +**Update 2026-10-12 (application-profile split; user decision):** this spec +now also owns the submission side of execution modes, moved out of +`application-profile` (whose schema keeps `mode:` accepting only `direct`): +the `sandbox` mode value and its options (allowed input/output prefixes, +isolation extras), the job's I/O-directory fields end to end (wire, WAL +submit fact and snapshot, field completeness), worker capability +advertisement, and scheduler placement by mode. The mode and options reach +the worker through `application-profile`'s dispatch record. Sandbox-specific +env exposure is here; the outer pre/post-exec clean environment and the +reserved `XCE_` prefix are `application-profile`'s. + ## Problem A job whose execution mode is `sandbox` (from `application-profile`) must run on a worker node: @@ -103,7 +114,8 @@ Code exploration, 2026-09-29. - **Capability**: - The worker detects its sandbox capability (systemd 242 or later, `systemd` in `nsswitch.conf` passwd/group, and D-Bus property support) and advertises - it to `application-profile`'s placement. + it; the scheduler places sandbox jobs only on capable workers (this + spec). - It refuses, with a clear error, a sandbox job it cannot honour. - **Direct mode is unchanged.** @@ -150,11 +162,14 @@ Viability check (2026-09-29, systemd NEWS, systemd.exec(5), v239 sources): - Input/output mounts. - `StandardOutput=file:` output and retrieval. - Helper authentication by unit or cgroup. - - Capability detection and advertisement. + - Capability detection and advertisement, and scheduler placement by mode. + - The `sandbox` mode value and options in the profile schema, and the + job's I/O-directory fields end to end (wire, WAL, snapshot). - Single-node MPI in one unit. - Tests: unit tests plus a live test on a systemd 242+ node. - **Out**: - - The submission side: mode field, policy, placement (`application-profile`). + - The profile core (`application-profile`): config, `mode:` field, + access, dispatch record. - Multi-node sandbox launch (`job-sandbox-multinode`). - Non-systemd backends and container/vm modes (the interface only, not the implementations). @@ -169,15 +184,15 @@ Viability check (2026-09-29, systemd NEWS, systemd.exec(5), v239 sources): - Helper authentication. ## Out of Boundary -- Job mode selection and policy. +- Profile selection and access (`application-profile`). - Cross-node uid consistency and remote launch. - Worker cgroup resource limits, which are unchanged and simply apply to sandbox units too. ## Upstream / Downstream - **Upstream**: - - `application-profile`: mode and directory fields on `job_startinfo_t`, and - capability-based placement. + - `application-profile`: the profile core, `mode:` field and the dispatch + record that carries mode and options to the worker. - The remote-task root fix. - **Downstream**: - `job-sandbox-multinode`. diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index 3a931d3a..c04d8391 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -63,8 +63,8 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han - [x] ha-commit-durability — closes bug-661 (acknowledgment did not imply durability; committed jobs lost across failover). Makes acknowledged ⇒ durable on a majority: the primary counts its own copy toward the commit majority only at its durable LSN (`majority_candidate_locked` caps self at `persist_durable_lsn`, not the appended tip); `fsync_worker` repurposed timer→demand-driven group-commit flusher (one `fdatasync` over pending appends, advance `durable_lsn`, wake commit-eval) — absorbs the former "WAL group-commit batching" backlog item (see below); recovery integrity verdict (clean vs corruption-truncated WAL) + durable-backed freshness floor (candidate advertises bare `persist_durable_floor`, voter floor = `max(durable_floor, learned_committed)`) + pre-promotion eligibility fence. Establishes the **controllers-never-self-terminate** invariant: a recovery-truncated/behind member that cannot catch up is a LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — never `exit`/`abort`), self-heals on peer return (see `ha-async-io.md` process-lifecycle invariant + `src/controller/CLAUDE.md`). Dependencies: ha-commit-engine (consumes commit-position semantics), ha-wal-replication (standby ACK/catch-up), ha-commit-barrier-rollout (reply gate, unchanged). (specs done 2026-06-18; **IMPLEMENTED 2026-06-18**: 17/17 tasks (1.1–5.3), validate-impl GO, controller suite 0 failures under ASAN; deterministic ack-before-fdatasync E2E RED→GREEN + rolling-SIGKILL soak zero acked-job loss. Verified by tests + Protocol Verification Checklist, NOT by raft.tla (which assumes a durable log). Fixed bug-760 (adjacent `test_fact_hook_append` raced the async flusher — test made durable-deterministic via `persist_flush`). raft.tla durability-model extension deferred unless explicitly opted in.) - [x] scheduler-allocator-runner — NEW 2026-09-28 (`/kiro-discovery`, Path C). The framework owns `job_alloc_fn` through a new `xce_sched_allocator_iface_t` (accepts guard + plan iterator) and a single `sched_allocator_run()`; both `my_alloc()`s are deleted (seqjob/parjob), and `alloc_iter.h` `init` is retyped to `xce_sched_core_t*`. Behaviour must stay identical (bug-1882/1767/1768). Closes `issues/new_requirements.md` item 4. Brief: `.kiro/specs/scheduler-allocator-runner/brief.md`. Dependencies: scheduler-allocation, scheduler-core (both done). **IMPLEMENTED 2026-09-30** (11/11 tasks, e2266582..28cff179; validate-impl GO; scheduler ASAN suite 96/96): runner in `src/scheduler/src/alloc_runner.c`; decided along the way: the runner clears `alloc_req.host` before every candidate so every short host gets pend reasons (both allocators), and fixes bug-1893 (shared reasons were never recorded). Follow-ups: publish pend reasons only when changed; sim-mode parjob allocation parity fixture; bug-1892; bug-1897. - [ ] scheduler-alter-pend-reasons — NEW 2026-09-28 (`/kiro-discovery`, Path E). An ALTER-resreq job keeps a pend reason for EVERY alternative instead of only the last (`scheduler.c:424` FIXME). Wire shape (user decision): additive `alternatives @3: List(JobPendReason)` on `JobPendReason`, per-alternative attribution; carried end-to-end through controller job record, WAL preason fact, recordings and `job_preason_eq`. Also fixes the confirmed `obj->reason` overwrite leak (`scheduler.c:388`). Brief: `.kiro/specs/scheduler-alter-pend-reasons/brief.md`. Dependencies: libxce-alloc-preason-equality (HARD: extends its `job_preason_eq`); sequence after scheduler-allocator-runner (its "no pend-reason content change" rule). -- [ ] application-profile — NEW 2026-09-29 (`/kiro-discovery`, Job Sandbox Execution initiative, Path D; reshaped from `job-execution-mode` by user direction: execution mode is a job attribute determined by the APPLICATION). Admin-defined named application profiles (LSF `-app` precedent) in cluster config: execution mode + options (`direct`|`sandbox`, room for `container`/`vm`), launch type, env + pre/post exec, resource defaults/limits (extensible). Users pick a profile at submit (`--app`), cluster default profile otherwise; queues (scheduling policy) and profiles (application config) are orthogonal — neither restricts the other; a profile holds only essentials and the job may add to them but never change them (additive rule, 2026-10-12; convenience defaults are client-side presets, `xcecli-presets`); access by RBAC permission (new `app:use`/`app:view` with `app` domain type, folded into this spec); apiserver checks profile existence; controller validates at submit and persists the profile name + the job's own values (live merge via job-config-merge; wire additive, WAL/snapshot field-completeness); own config file `applications.yaml`, cluster default `default_application` in `xce.yaml` (undefined name = config error, never a fallback; updated 2026-10-12). apiserver never checks node accounts; worker fails a `direct` job with a clear reason if the exec user is missing. Worker capability advertisement + scheduler placement. Brief: `.kiro/specs/application-profile/brief.md`. Dependencies: none. -- [ ] job-sandbox-execution — NEW 2026-09-29 (Job Sandbox Execution). Worker: pluggable execution-backend interface (`direct` refactored + `systemd-sandbox`; non-systemd backends later). Sandbox = existing transient unit + `DynamicUser` with fresh per-job name (never the submitter's), ProtectSystem=strict/ProtectHome/PrivateTmp/NoNewPrivileges, Bind(ReadOnly)Paths for job I/O dirs or worker-created temp workspace (handed to the uid via SO_PEERCRED), `StandardOutput=file:` output, helper auth by unit/cgroup, MANDATORY post-job cleanup/ownership normalization (dynamic uid reuse), systemd ≥ 242 + nss-systemd capability probe; single-node incl. single-node MPI. Viability-checked 2026-09-29. Brief: `.kiro/specs/job-sandbox-execution/brief.md`. Dependencies: application-profile; remote-task-root fix (Direct-Implementation). +- [ ] application-profile — NEW 2026-09-29 (`/kiro-discovery`, Job Sandbox Execution initiative, Path D; reshaped from `job-execution-mode` by user direction: execution mode is a job attribute determined by the APPLICATION). Admin-defined named application profiles (LSF `-app` precedent) in cluster config: execution mode + options (`direct`|`sandbox`, room for `container`/`vm`), launch type, env + pre/post exec, resource defaults/limits (extensible). Users pick a profile at submit (`--app`), cluster default profile otherwise; queues (scheduling policy) and profiles (application config) are orthogonal — neither restricts the other; a profile holds only essentials and the job may add to them but never change them (additive rule, 2026-10-12; convenience defaults are client-side presets, `xcecli-presets`); access by RBAC permission (new `app:use`/`app:view` with `app` domain type, folded into this spec); apiserver checks profile existence; controller validates at submit and persists the profile name + the job's own values (live merge via job-config-merge; wire additive, WAL/snapshot field-completeness); own config file `applications.yaml`, cluster default `default_application` in `xce.yaml` (undefined name = config error, never a fallback; updated 2026-10-12). apiserver never checks node accounts; worker fails a `direct` job with a clear reason if the exec user is missing. **Split 2026-10-12 (user):** execution modes (`sandbox` value + options, I/O dirs, worker capability advertisement, placement by mode) moved to job-sandbox-execution; this spec keeps `mode: direct`, the dispatch record (resreq + execution values + profile content hash; retryable dispatch failure when the profile is gone/changed), aliases, outer pre/post-exec with clean env, reserved `XCE_`. Order: this spec's requirements+design before job-config-merge's design. Brief: `.kiro/specs/application-profile/brief.md`. Dependencies: none. +- [ ] job-sandbox-execution — NEW 2026-09-29 (Job Sandbox Execution). Worker: pluggable execution-backend interface (`direct` refactored + `systemd-sandbox`; non-systemd backends later). Sandbox = existing transient unit + `DynamicUser` with fresh per-job name (never the submitter's), ProtectSystem=strict/ProtectHome/PrivateTmp/NoNewPrivileges, Bind(ReadOnly)Paths for job I/O dirs or worker-created temp workspace (handed to the uid via SO_PEERCRED), `StandardOutput=file:` output, helper auth by unit/cgroup, MANDATORY post-job cleanup/ownership normalization (dynamic uid reuse), systemd ≥ 242 + nss-systemd capability probe; single-node incl. single-node MPI. **2026-10-12:** also owns the submission side of execution modes moved from application-profile (`sandbox` mode + options, job I/O-directory fields end to end, capability advertisement, scheduler placement by mode). Viability-checked 2026-09-29. Brief: `.kiro/specs/job-sandbox-execution/brief.md`. Dependencies: application-profile; remote-task-root fix (Direct-Implementation). - [ ] job-sandbox-multinode — NEW 2026-09-29 (Job Sandbox Execution). Multi-node sandbox jobs: cluster-consistent ephemeral uid (expected: controller-allocated, WAL-replicated uid lease from a reserved range), worker-driven remote/MPI launch (no ssh; PMIx/prted via worker), shared-FS ownership normalization, one unit per job per node. Brief: `.kiro/specs/job-sandbox-multinode/brief.md`. Dependencies: job-sandbox-execution, application-profile. - [ ] rbac-persistence — NEW 2026-09-29 (`/kiro-discovery`, raised during application-profile discovery). Closes the deferred `security.md` gap: runtime RBAC mutations (roles, permissions, user-role assignments, Logon auto-assignment) are in-memory only and silently revert to `rbac.yaml` on apiserver restart — including revocations. Durable-or-rejected mutations, audit trail, defaults-vs-runtime reconciliation, multi-apiserver consistency; storage approach (rbac.yaml write-through / controller-replicated entity / embedded store / read-only) decided in design. Brief: `.kiro/specs/rbac-persistence/brief.md`. Dependencies: none (independent of the sandbox initiative; application-profile's API-made `app:use` grants rely on it for durability). - [ ] job-time-window — NEW 2026-09-29 (`/kiro-discovery`, Path C). Job begin time (scheduler gate in `sched_job_is_ready()` via `xce_core_now`, pend reason BEGIN_TIME_NOT_REACHED) and end time (WORKER enforces running jobs: helper SIGTERM→grace→SIGKILL, exit reason END_TIME_REACHED persisted, unit RuntimeMaxUSec backstop; PEND jobs whose end time can't be met stay PEND with END_TIME_UNMEETABLE until user kills/modifies — no controller sweep, primary not involved). Adds runtime estimate (`-We/--estimated_run_time`, scheduling hint only, never enforced; estimate > run limit rejected); unmeetable rule `now + (estimate ‖ run_limit ‖ 0) > end`. Times/estimate modifiable only while PEND (new `xce jobs modify` + harness `jobmod`). Both xcecli and harness `jobsub` gain all options (`-b`/`-t`/`-W`/estimate). Also fixes the silent drop of xcecli `-b`/`-t`/`-W` and REST `run_time_estimation` (apiserver `modelJobToSubmit` never sets them nor `exdata`). Brief: `.kiro/specs/job-time-window/brief.md`. Dependencies: none (new reason codes need text in job-status-detail-display).