From f6cab0ab87c615462d6ca2dbf73daf91ba15171e Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 29 Sep 2026 09:40:52 +0000 Subject: [PATCH] docs(spec): job-time-window discovery brief (begin/end time, runtime estimate) New spec from /kiro-discovery (2026-09-29), added to roadmap.md: - Begin time: the scheduler won't start a job before it (gate in sched_job_is_ready() using the core clock). New pend reason BEGIN_TIME_NOT_REACHED. - End time: the worker enforces it for running jobs (SIGTERM, grace, SIGKILL) and records a persisted END_TIME_REACHED exit reason. The primary is not involved. A pending job whose end time can't be met stays PEND with END_TIME_UNMEETABLE until the user kills or modifies it. - Runtime estimate (-We): a scheduling hint only. It sets the rule now + (estimate, else run limit, else 0) > end; an estimate larger than the run limit is rejected. - Times and estimate can be changed only while the job is pending, through a new `xce jobs modify` in xcecli and a new `jobmod` in the harness. Harness jobsub gains all four options. - Fixes the silent drop of xcecli -b/-t/-W and REST run_time_estimation, which the apiserver's modelJobToSubmit never passes to the controller. Co-Authored-By: Claude Opus 5.5 (1M context) --- .kiro/specs/job-time-window/brief.md | 268 +++++++++++++++++++++++++++ .kiro/steering/roadmap.md | 1 + 2 files changed, 269 insertions(+) create mode 100644 .kiro/specs/job-time-window/brief.md diff --git a/.kiro/specs/job-time-window/brief.md b/.kiro/specs/job-time-window/brief.md new file mode 100644 index 00000000..334a4d5d --- /dev/null +++ b/.kiro/specs/job-time-window/brief.md @@ -0,0 +1,268 @@ +# Brief: job-time-window + +## Problem +XCE users can't constrain *when* a job runs: +- **Begin time.** "Don't start this job before 22:00", for example to wait for + off-peak hours, input data, or a maintenance window to end. +- **End time.** "This job must be finished by 06:00", for example when results + are useless after a deadline, or a reservation or maintenance starts. + +`xcecli jobs submit` already offers `-b/--begin_time`, `-t/--term_time` and +`-W/--run_limit`, and the apiserver accepts them. They are **silently dropped** +before reaching the controller, so users believe they are protected when they +are not. + +## Current State +Codebase exploration, 2026-09-29. + +**Client and apiserver** +- xcecli: `-b` (`cmd/jobs/submit.go:682`), `-t` (:686) and `-W` (:688), parsed + into `jobReq.BeginTime` / `TermTime` / `RunLimit` (:278, :312-330). +- apiserver submit request: `BeginTime`, `TermTime`, `RunTimeEstimation` and + `RunLimit` (`models/jobs.go:641-674`). They are copied only into the Go job + model (`controllers/jobs.go:1475-1487`, restart path :1287-1318). + - `run_limit` is folded into the `ExtendData` JSON under `limits` + (`controllers/jobs.go:2195-2211`). +- **The gap:** `modelJobToSubmit` (`backend/convert.go:272-318`) sets none of + begin time, term time, runtime estimate, or `ExtendData` → `start_info.exdata`. + So `-b`, `-t` and `-W` never reach the controller. Only the `xce_test` + harness's `-x ` delivers a run limit today. + +**C side** +- `job_submit_data_t` (`include/xce/types.h:356-370`), `JobSubmitData` + (`schema/controller.capnp:579-591`, last ordinal @10) and `xce_sched_job_t` + (`scheduler/include/sched/types.h:122-142`) have no begin, end or limit field. + +**Scheduler** +- The FCFS picker (`plugins/builtin_fcfs.c:837-887`) takes the first job for which + `sched_queue_is_ready()` and `sched_job_is_ready()` both hold. **Both are stubs + returning 1** (`core.c:490`, `:493`), which is the natural eligibility hook. +- Sessions run about every 1 s (`main.c:508-545`), so the scheduler notices a + begin time within about a second, with no extra wakeup. +- Time must come from `xce_core_now(core)` / `core->session_start` + (`core_clock.c`), never `time()`. That keeps the simulator's discrete clock and + `--sim-verify` deterministic, and `test_core_isolation.c` forbids wall-clock + calls in `core*.c`. + +**Pend reasons** +- Only resource reasons exist: `PREASON_BAD_RESREQ`, `PREASON_NO_RESOURCE` and + `PREASON_MATCH_FAIL` (`include/preason.h:7-9`). +- There is no non-resource reason, and no publish path for a job the picker + skips. Publishing goes through `sched_publish_pend_reason` + (`scheduler.c:1821`). + +**Worker** +- The run limit is read from `exdata.limits.run_limit` (`xcecd_exdata.c:306-309`) + and set as `RuntimeMaxUSec` on the transient unit (`xcecd_cgroups.c:753-770`). + That is relative to unit start. +- The kill path is `xcecd_worker.c:1115-1124` (`signal_gid_except_self`, + TERM→KILL escalation). +- The job helper `xcecd_worker` runs *inside* the unit and reports the job's exit + (`uv_ipc.c:1000`). If systemd times out the unit, the helper is killed too, and + the exit shows only as a plain signal exit. + +**Exit info** +- `job_exitinfo_t` is `{exitcode}` only (`types.h:378-380`; capnp `JobExitInfo`). +- `job_failinfo_t.detail` is not persisted (WAL_JOB_FAIL carries code plus stage + only, `persist.c:4224-4257`). +- There is no exit-reason field. + +**Persistence** +- The WAL_JOB_SUBMIT payload is the whole capnp `xce_job_t`, so new + `JobSubmitData` fields ride along automatically. +- These must be updated by hand: + - the libxce job-submit interning table (`lib/jobj.c:29-55`, `:123-182`). Its + equality and hash must include the new fields, or submits that differ only + in time would be merged; + - the scheduler's `schedobj.c:253-279` and update path `:733`; + - the scheduler snapshot `snapshot_wire.c:134-166`, which rebuilds `submit` + field by field. + +## Desired Outcome +- **Begin time.** + - A job is not started before its begin time. + - Until then it pends with a clear pend reason: "begin time not reached (starts + at T)". +- **End time, running jobs** (user decision, 2026-09-29: the worker handles + running jobs, and the primary is not involved). + - A running job still running at its end time is terminated by its worker: + - a warning signal (SIGTERM); + - a configurable grace period; + - then SIGKILL. + - The job ends as EXIT with a recorded exit reason, "end time reached", visible + to users and persisted. +- **End time, pending jobs** (user decision). + - A pending job whose end time can no longer be met is **not started** and + **stays PEND** with a clear pend reason ("end time cannot be met"). + - It stays that way until the user kills it or changes its end time. It is + never auto-expired, and the controller runs no sweep. +- **Runtime estimate** (user decision, 2026-09-29, folded into this spec). + - A new submit option `xcecli jobs submit --estimated_run_time` / `-We`. The + REST API already accepts `run_time_estimation` (`models/jobs.go:659`) but + drops it, like `-b`/`-t`/`-W`. + - It is **a scheduling hint only and is never enforced**. That is the + difference from the run limit `-W`, which is a hard cap the worker kills at; + this mirrors LSF `-We` versus `-W`. + - It is carried end to end to the scheduler (object and snapshot) and can be + changed while the job is pending. + - If both are given and **estimate > run limit, the submit is rejected** with a + clear error (user decision). + - An `application-profile` may later supply a default estimate. +- **"End time cannot be met" rule** (settled). A pending job is not started when + `now + expected_runtime > end`, where `expected_runtime` is the estimate if + given, else the run limit if given, else 0 (plain `now ≥ end`). `now` comes from + the scheduler core clock. +- **Changes.** Begin time, end time and the runtime estimate can be changed + **only while the job is pending** (user decision). Once the job runs, its end time is fixed, so the + worker never needs live updates. +- **Wiring fix** (user decision: fixed in this spec). `-b`, `-t` and `-W` + (`exdata` / `run_limit`) actually reach the controller, scheduler and worker. + Log it in the buglog as the drop defect. +- **Validation at submit.** + - Reject begin ≥ end. + - Reject an end time already in the past. + - Reject a time that cannot be parsed. +- **Both client tools support every new option** (user direction, 2026-09-29). + - **xcecli** (`cmd/jobs/submit.go`): + - `-b/--begin_time`, `-t/--term_time` and `-W/--run_limit` exist today but + are dropped downstream; this spec makes them work end to end; + - add `-We/--estimated_run_time`; + - `xcecli` has no command for changing a submitted job, so add one, for + example `xce jobs modify [-b] [-t] [-We]`, pending jobs only. + - **Harness `xce_test`** (`src/harness/test.c`): + - `jobsub` (`jobsub_flags[]` at :3575, usage at :602) today takes only `-u`, + `-q`, `-c`, `-R`, `-n`, `-x` and `-a`. Add begin time, end time, run limit + and estimate (for example `-b/--begin-time`, `-t/--end-time`, + `-W/--run-limit`, `-e/--estimated-run-time`; short letters are settled in + design). `-W` must be a first-class flag rather than hand-written `-x` + exdata JSON, and it must compose with a user-supplied `-x`. + - Add a matching `jobmod` command for pending jobs. + - Update the harness help (`test_repl_help`-style help tests) and the + `xce_test` rc-file or script docs. + - Both tools accept the same time formats, validated identically: shared rules + for absolute times, relative offsets and time zones, settled in requirements. + - Both show clear errors for begin ≥ end, an end time in the past, and + estimate > run limit. +- **Visibility.** Both times appear in job details (`job-status-detail-display` + renders the new pend and exit reasons). + +## Approach +**Chosen: approach B′, the worker enforces running jobs and the scheduler gates +pending ones** (user direction, 2026-09-29). This keeps the primary out of it. +The rejected option (A) was a primary-only controller sweep with the worker as a +backstop. + +1. **Fields.** Add `beginTime` and `endTime` (absolute epoch seconds, 0 = unset) + and `runEstimate` (seconds, 0 = unset) as additive fields on `JobSubmitData`. + The run limit stays in `exdata.limits.run_limit`, but the scheduler also needs + it for the unmeetable check, so expose it on the scheduler job object. Carry them through: + - `job_submit_data_t`; + - the libxce interning equality and hash; + - the apiserver `modelJobToSubmit`, which also passes `ExtendData` → `exdata`; + - `xce_sched_job_t` plus `schedobj.c`; + - the scheduler snapshot; + - the recording and replay path. +2. **Scheduler gate** in `sched_job_is_ready()`, or an equivalent eligibility + step, using `xce_core_now(core)`: + - `now < begin` → not eligible, reason BEGIN_TIME_NOT_REACHED; + - "end cannot be met" → not eligible, reason END_TIME_UNMEETABLE. + + Two new non-resource pend-reason codes, published for skipped jobs. +3. **Worker enforcement.** The job helper runs the deadline itself: at `end`, it + sends SIGTERM to the job's process group, waits the grace period, then sends + SIGKILL, and reports the exit with reason END_TIME_REACHED. Belt and braces: + the unit's `RuntimeMaxUSec` is set to `end + grace + margin − now` at launch, + as a last-resort backstop if the helper itself hangs. Every node of a + multi-host job enforces its own tasks. +4. **Exit reason.** An additive exit-reason field on the job's exit info, + persisted through WAL_JOB_FAIL/FINISH and the snapshot (field completeness). +5. **Changing times on a pending job** goes through the existing job + modify/update path (WAL_JOB_UPDATE) and is refused for non-pending jobs. + +## Scope +- **In**: + - The begin and end fields and the runtime estimate end to end (new + `-We/--estimated_run_time` xcecli flag), and the `-W` / `exdata` wiring fix. + - Submit validation, including estimate > run limit. + - Submit validation. + - The scheduler gate and the two new pend reasons. + - Worker end-time enforcement with grace, plus the backstop. + - The new, persisted exit reason. + - Changing the times of pending jobs, through a new `xce jobs modify` in + xcecli and a new `jobmod` in the harness. + - Client support in **both** xcecli (the `-We` flag, with `-b`/`-t`/`-W` made + to work) and the harness `jobsub` (all four options), with help text and + docs. + - Display in xcecli and REST. + - Simulator determinism (the gate uses the core clock). + - Tests: unit, sim, and a live kill-at-deadline test. +- **Open for requirements**: + - The grace-period default, and whether it is cluster config or per job. + - The time zone and format accepted by `-b`/`-t` (xcecli `utils.ParseTime`). + - Whether a job array's elements share the window. + - Clock-skew tolerance between the scheduler, controller and worker hosts + (NTP assumed). +- **Out**: + - Controller-side enforcement or sweeps (user decision). + - Automatic expiry of unmeetable pending jobs (user decision: they stay PEND). + - Changing the end time of a running job. + - Backfill or advance reservation built on end times. + - Deadline-based preemption (`scheduler-preemption` lists it as out of scope). + - Queue run and dispatch windows (`types.h:570-573`, `lib/window.c`, unused). + +## Boundary Candidates +- Field plumbing: capnp, C structs, libxce interning, apiserver conversion, + scheduler object and snapshot, sim recording. +- Client surfaces: xcecli submit and modify, harness `jobsub` and `jobmod`, + shared time parsing and validation. +- The scheduler eligibility gate and the new pend-reason codes. +- Worker deadline enforcement (helper timer plus unit backstop). +- The exit-reason field and its persistence. +- The pending-job change path and submit validation. + +## Out of Boundary +- The display of reasons (`job-status-detail-display`). This spec adds the codes; + that spec's code-to-text table needs the new entries. +- How pend reasons are grouped for jobs with alternative resource requirements + (`scheduler-alter-pend-reasons`). +- Deciding which jobs a user may see or control (existing RBAC). + +## Upstream / Downstream +- **Upstream**: + - The existing submit, modify and kill paths. + - The REST `run_time_estimation` field, which exists but is dropped today. + - The scheduler FCFS picker and `sched_job_is_ready()`. + - The worker transient unit (`RuntimeMaxUSec` support already exists). + - The WAL submit and update facts. +- **Downstream**: + - `job-status-detail-display`, which shows the times and the new reason texts. + - Future backfill or reservation work, which can use the runtime estimate. + - `application-profile`, where resource defaults might supply a default run + limit. + +## Existing Spec Touchpoints +- **Extends**: none. +- **Adjacent**: + - `job-status-detail-display` (new pend and exit reason codes need text) + - `scheduler-alter-pend-reasons` (pend-reason model) + - `scheduler-preemption` + - `controller-job-cleanup` (an EXIT from the end time becomes eligible for + cleanup as usual) + - `application-profile` + - `scheduler-sim-discrete-clock` (the core clock) + +## Constraints +- The primary does no end-time work. Enforcement lives on the worker, and gating + lives in the scheduler. +- All wire and schema changes are additive Cap'n Proto. Old WAL records and + snapshots decode with no window, which is today's behaviour. +- Save/restore field completeness (`persistence.md`): the begin and end fields + and the exit reason must round trip through the WAL submit, update and fail + facts, the controller snapshot, the scheduler snapshot and the recordings. Run + `/xce-audit-invariants` (controller dimension 7, scheduler dimension 4). +- The scheduler's core time only comes from `xce_core_now`, so `--sim-verify` + stays deterministic. +- The worker's non-systemd process-manager build (`#ifndef DBUS_MANAGER`) must + enforce the deadline too; the helper timer covers it. +- TDD: `testsuite/` first, then `_test.go`. ASAN must be clean. +- HA/persistence touch: use the full 3-phase workflow. diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index 95c4aae7..0b67e12a 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -65,6 +65,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han - [ ] job-sandbox-execution — NEW 2026-09-29 (Job Sandbox Execution). Worker: pluggable execution-backend interface (`direct` refactored + `systemd-sandbox`; non-systemd backends later). Sandbox = existing transient unit + `DynamicUser` with fresh per-job name (never the submitter's), ProtectSystem=strict/ProtectHome/PrivateTmp/NoNewPrivileges, Bind(ReadOnly)Paths for job I/O dirs or worker-created temp workspace (handed to the uid via SO_PEERCRED), `StandardOutput=file:` output, helper auth by unit/cgroup, MANDATORY post-job cleanup/ownership normalization (dynamic uid reuse), systemd ≥ 242 + nss-systemd capability probe; single-node incl. single-node MPI. Viability-checked 2026-09-29. Brief: `.kiro/specs/job-sandbox-execution/brief.md`. Dependencies: application-profile; remote-task-root fix (Direct-Implementation). - [ ] job-sandbox-multinode — NEW 2026-09-29 (Job Sandbox Execution). Multi-node sandbox jobs: cluster-consistent ephemeral uid (expected: controller-allocated, WAL-replicated uid lease from a reserved range), worker-driven remote/MPI launch (no ssh; PMIx/prted via worker), shared-FS ownership normalization, one unit per job per node. Brief: `.kiro/specs/job-sandbox-multinode/brief.md`. Dependencies: job-sandbox-execution, application-profile. - [ ] rbac-persistence — NEW 2026-09-29 (`/kiro-discovery`, raised during application-profile discovery). Closes the deferred `security.md` gap: runtime RBAC mutations (roles, permissions, user-role assignments, Logon auto-assignment) are in-memory only and silently revert to `rbac.yaml` on apiserver restart — including revocations. Durable-or-rejected mutations, audit trail, defaults-vs-runtime reconciliation, multi-apiserver consistency; storage approach (rbac.yaml write-through / controller-replicated entity / embedded store / read-only) decided in design. Brief: `.kiro/specs/rbac-persistence/brief.md`. Dependencies: none (independent of the sandbox initiative; application-profile's API-made `app:use` grants rely on it for durability). +- [ ] job-time-window — NEW 2026-09-29 (`/kiro-discovery`, Path C). Job begin time (scheduler gate in `sched_job_is_ready()` via `xce_core_now`, pend reason BEGIN_TIME_NOT_REACHED) and end time (WORKER enforces running jobs: helper SIGTERM→grace→SIGKILL, exit reason END_TIME_REACHED persisted, unit RuntimeMaxUSec backstop; PEND jobs whose end time can't be met stay PEND with END_TIME_UNMEETABLE until user kills/modifies — no controller sweep, primary not involved). Adds runtime estimate (`-We/--estimated_run_time`, scheduling hint only, never enforced; estimate > run limit rejected); unmeetable rule `now + (estimate ‖ run_limit ‖ 0) > end`. Times/estimate modifiable only while PEND (new `xce jobs modify` + harness `jobmod`). Both xcecli and harness `jobsub` gain all options (`-b`/`-t`/`-W`/estimate). Also fixes the silent drop of xcecli `-b`/`-t`/`-W` and REST `run_time_estimation` (apiserver `modelJobToSubmit` never sets them nor `exdata`). Brief: `.kiro/specs/job-time-window/brief.md`. Dependencies: none (new reason codes need text in job-status-detail-display). - [ ] controller-job-cleanup — NEW 2026-09-29 (`/kiro-discovery`, Path C). Primary-only periodic sweep (configurable `job_cleanup_period`, default 300 s) removes DONE/EXIT jobs finished longer than configurable `job_retention` (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever `jobmgr` delete; refcount-safe `xkvmgr` delete), propagated to scheduler and apiserver (`OnJobDeleted`). Retires the node-local snapshot-only `job_should_retain` finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: `.kiro/specs/controller-job-cleanup/brief.md`. Dependencies: none (extends ha-entity-snapshots). - [ ] job-status-detail-display — NEW 2026-09-28 (`/kiro-discovery`, Path E). End users (REST/`xcecli`) see NO pend reason for ANY job today: the apiserver cgo conversion (`backend/convert.go`) never maps `xce_job_t.pend_reasons`, so `pending_reason` is always empty. Maps it to a structured JSON (including `alternatives` from day one) using libxce's code→text mapping, and renders it in `xcecli` job detail. Open for requirements: also map `run_usage` → `JobRusage` (same gap; reserved vs. used for RUN jobs). Brief: `.kiro/specs/job-status-detail-display/brief.md`. Dependencies: none for flat reasons; scheduler-alter-pend-reasons for per-alternative rendering.