docs(spec): job-time-window discovery brief (begin/end time, runtime estimate)
New spec from /kiro-discovery (2026-09-29), added to roadmap.md: - Begin time: the scheduler won't start a job before it (gate in sched_job_is_ready() using the core clock). New pend reason BEGIN_TIME_NOT_REACHED. - End time: the worker enforces it for running jobs (SIGTERM, grace, SIGKILL) and records a persisted END_TIME_REACHED exit reason. The primary is not involved. A pending job whose end time can't be met stays PEND with END_TIME_UNMEETABLE until the user kills or modifies it. - Runtime estimate (-We): a scheduling hint only. It sets the rule now + (estimate, else run limit, else 0) > end; an estimate larger than the run limit is rejected. - Times and estimate can be changed only while the job is pending, through a new `xce jobs modify` in xcecli and a new `jobmod` in the harness. Harness jobsub gains all four options. - Fixes the silent drop of xcecli -b/-t/-W and REST run_time_estimation, which the apiserver's modelJobToSubmit never passes to the controller. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
c1cd8f917d
commit
f6cab0ab87
GPG key ID:
80533536B0EB651C
2 changed files with 269 additions and 0 deletions
268
.kiro/specs/job-time-window/brief.md
Normal file
268
.kiro/specs/job-time-window/brief.md
Normal file
|
|
@ -0,0 +1,268 @@
|
|||
# Brief: job-time-window
|
||||
|
||||
## Problem
|
||||
XCE users can't constrain *when* a job runs:
|
||||
- **Begin time.** "Don't start this job before 22:00", for example to wait for
|
||||
off-peak hours, input data, or a maintenance window to end.
|
||||
- **End time.** "This job must be finished by 06:00", for example when results
|
||||
are useless after a deadline, or a reservation or maintenance starts.
|
||||
|
||||
`xcecli jobs submit` already offers `-b/--begin_time`, `-t/--term_time` and
|
||||
`-W/--run_limit`, and the apiserver accepts them. They are **silently dropped**
|
||||
before reaching the controller, so users believe they are protected when they
|
||||
are not.
|
||||
|
||||
## Current State
|
||||
Codebase exploration, 2026-09-29.
|
||||
|
||||
**Client and apiserver**
|
||||
- xcecli: `-b` (`cmd/jobs/submit.go:682`), `-t` (:686) and `-W` (:688), parsed
|
||||
into `jobReq.BeginTime` / `TermTime` / `RunLimit` (:278, :312-330).
|
||||
- apiserver submit request: `BeginTime`, `TermTime`, `RunTimeEstimation` and
|
||||
`RunLimit` (`models/jobs.go:641-674`). They are copied only into the Go job
|
||||
model (`controllers/jobs.go:1475-1487`, restart path :1287-1318).
|
||||
- `run_limit` is folded into the `ExtendData` JSON under `limits`
|
||||
(`controllers/jobs.go:2195-2211`).
|
||||
- **The gap:** `modelJobToSubmit` (`backend/convert.go:272-318`) sets none of
|
||||
begin time, term time, runtime estimate, or `ExtendData` → `start_info.exdata`.
|
||||
So `-b`, `-t` and `-W` never reach the controller. Only the `xce_test`
|
||||
harness's `-x <exdata>` delivers a run limit today.
|
||||
|
||||
**C side**
|
||||
- `job_submit_data_t` (`include/xce/types.h:356-370`), `JobSubmitData`
|
||||
(`schema/controller.capnp:579-591`, last ordinal @10) and `xce_sched_job_t`
|
||||
(`scheduler/include/sched/types.h:122-142`) have no begin, end or limit field.
|
||||
|
||||
**Scheduler**
|
||||
- The FCFS picker (`plugins/builtin_fcfs.c:837-887`) takes the first job for which
|
||||
`sched_queue_is_ready()` and `sched_job_is_ready()` both hold. **Both are stubs
|
||||
returning 1** (`core.c:490`, `:493`), which is the natural eligibility hook.
|
||||
- Sessions run about every 1 s (`main.c:508-545`), so the scheduler notices a
|
||||
begin time within about a second, with no extra wakeup.
|
||||
- Time must come from `xce_core_now(core)` / `core->session_start`
|
||||
(`core_clock.c`), never `time()`. That keeps the simulator's discrete clock and
|
||||
`--sim-verify` deterministic, and `test_core_isolation.c` forbids wall-clock
|
||||
calls in `core*.c`.
|
||||
|
||||
**Pend reasons**
|
||||
- Only resource reasons exist: `PREASON_BAD_RESREQ`, `PREASON_NO_RESOURCE` and
|
||||
`PREASON_MATCH_FAIL` (`include/preason.h:7-9`).
|
||||
- There is no non-resource reason, and no publish path for a job the picker
|
||||
skips. Publishing goes through `sched_publish_pend_reason`
|
||||
(`scheduler.c:1821`).
|
||||
|
||||
**Worker**
|
||||
- The run limit is read from `exdata.limits.run_limit` (`xcecd_exdata.c:306-309`)
|
||||
and set as `RuntimeMaxUSec` on the transient unit (`xcecd_cgroups.c:753-770`).
|
||||
That is relative to unit start.
|
||||
- The kill path is `xcecd_worker.c:1115-1124` (`signal_gid_except_self`,
|
||||
TERM→KILL escalation).
|
||||
- The job helper `xcecd_worker` runs *inside* the unit and reports the job's exit
|
||||
(`uv_ipc.c:1000`). If systemd times out the unit, the helper is killed too, and
|
||||
the exit shows only as a plain signal exit.
|
||||
|
||||
**Exit info**
|
||||
- `job_exitinfo_t` is `{exitcode}` only (`types.h:378-380`; capnp `JobExitInfo`).
|
||||
- `job_failinfo_t.detail` is not persisted (WAL_JOB_FAIL carries code plus stage
|
||||
only, `persist.c:4224-4257`).
|
||||
- There is no exit-reason field.
|
||||
|
||||
**Persistence**
|
||||
- The WAL_JOB_SUBMIT payload is the whole capnp `xce_job_t`, so new
|
||||
`JobSubmitData` fields ride along automatically.
|
||||
- These must be updated by hand:
|
||||
- the libxce job-submit interning table (`lib/jobj.c:29-55`, `:123-182`). Its
|
||||
equality and hash must include the new fields, or submits that differ only
|
||||
in time would be merged;
|
||||
- the scheduler's `schedobj.c:253-279` and update path `:733`;
|
||||
- the scheduler snapshot `snapshot_wire.c:134-166`, which rebuilds `submit`
|
||||
field by field.
|
||||
|
||||
## Desired Outcome
|
||||
- **Begin time.**
|
||||
- A job is not started before its begin time.
|
||||
- Until then it pends with a clear pend reason: "begin time not reached (starts
|
||||
at T)".
|
||||
- **End time, running jobs** (user decision, 2026-09-29: the worker handles
|
||||
running jobs, and the primary is not involved).
|
||||
- A running job still running at its end time is terminated by its worker:
|
||||
- a warning signal (SIGTERM);
|
||||
- a configurable grace period;
|
||||
- then SIGKILL.
|
||||
- The job ends as EXIT with a recorded exit reason, "end time reached", visible
|
||||
to users and persisted.
|
||||
- **End time, pending jobs** (user decision).
|
||||
- A pending job whose end time can no longer be met is **not started** and
|
||||
**stays PEND** with a clear pend reason ("end time cannot be met").
|
||||
- It stays that way until the user kills it or changes its end time. It is
|
||||
never auto-expired, and the controller runs no sweep.
|
||||
- **Runtime estimate** (user decision, 2026-09-29, folded into this spec).
|
||||
- A new submit option `xcecli jobs submit --estimated_run_time` / `-We`. The
|
||||
REST API already accepts `run_time_estimation` (`models/jobs.go:659`) but
|
||||
drops it, like `-b`/`-t`/`-W`.
|
||||
- It is **a scheduling hint only and is never enforced**. That is the
|
||||
difference from the run limit `-W`, which is a hard cap the worker kills at;
|
||||
this mirrors LSF `-We` versus `-W`.
|
||||
- It is carried end to end to the scheduler (object and snapshot) and can be
|
||||
changed while the job is pending.
|
||||
- If both are given and **estimate > run limit, the submit is rejected** with a
|
||||
clear error (user decision).
|
||||
- An `application-profile` may later supply a default estimate.
|
||||
- **"End time cannot be met" rule** (settled). A pending job is not started when
|
||||
`now + expected_runtime > end`, where `expected_runtime` is the estimate if
|
||||
given, else the run limit if given, else 0 (plain `now ≥ end`). `now` comes from
|
||||
the scheduler core clock.
|
||||
- **Changes.** Begin time, end time and the runtime estimate can be changed
|
||||
**only while the job is pending** (user decision). Once the job runs, its end time is fixed, so the
|
||||
worker never needs live updates.
|
||||
- **Wiring fix** (user decision: fixed in this spec). `-b`, `-t` and `-W`
|
||||
(`exdata` / `run_limit`) actually reach the controller, scheduler and worker.
|
||||
Log it in the buglog as the drop defect.
|
||||
- **Validation at submit.**
|
||||
- Reject begin ≥ end.
|
||||
- Reject an end time already in the past.
|
||||
- Reject a time that cannot be parsed.
|
||||
- **Both client tools support every new option** (user direction, 2026-09-29).
|
||||
- **xcecli** (`cmd/jobs/submit.go`):
|
||||
- `-b/--begin_time`, `-t/--term_time` and `-W/--run_limit` exist today but
|
||||
are dropped downstream; this spec makes them work end to end;
|
||||
- add `-We/--estimated_run_time`;
|
||||
- `xcecli` has no command for changing a submitted job, so add one, for
|
||||
example `xce jobs modify <jobid> [-b] [-t] [-We]`, pending jobs only.
|
||||
- **Harness `xce_test`** (`src/harness/test.c`):
|
||||
- `jobsub` (`jobsub_flags[]` at :3575, usage at :602) today takes only `-u`,
|
||||
`-q`, `-c`, `-R`, `-n`, `-x` and `-a`. Add begin time, end time, run limit
|
||||
and estimate (for example `-b/--begin-time`, `-t/--end-time`,
|
||||
`-W/--run-limit`, `-e/--estimated-run-time`; short letters are settled in
|
||||
design). `-W` must be a first-class flag rather than hand-written `-x`
|
||||
exdata JSON, and it must compose with a user-supplied `-x`.
|
||||
- Add a matching `jobmod` command for pending jobs.
|
||||
- Update the harness help (`test_repl_help`-style help tests) and the
|
||||
`xce_test` rc-file or script docs.
|
||||
- Both tools accept the same time formats, validated identically: shared rules
|
||||
for absolute times, relative offsets and time zones, settled in requirements.
|
||||
- Both show clear errors for begin ≥ end, an end time in the past, and
|
||||
estimate > run limit.
|
||||
- **Visibility.** Both times appear in job details (`job-status-detail-display`
|
||||
renders the new pend and exit reasons).
|
||||
|
||||
## Approach
|
||||
**Chosen: approach B′, the worker enforces running jobs and the scheduler gates
|
||||
pending ones** (user direction, 2026-09-29). This keeps the primary out of it.
|
||||
The rejected option (A) was a primary-only controller sweep with the worker as a
|
||||
backstop.
|
||||
|
||||
1. **Fields.** Add `beginTime` and `endTime` (absolute epoch seconds, 0 = unset)
|
||||
and `runEstimate` (seconds, 0 = unset) as additive fields on `JobSubmitData`.
|
||||
The run limit stays in `exdata.limits.run_limit`, but the scheduler also needs
|
||||
it for the unmeetable check, so expose it on the scheduler job object. Carry them through:
|
||||
- `job_submit_data_t`;
|
||||
- the libxce interning equality and hash;
|
||||
- the apiserver `modelJobToSubmit`, which also passes `ExtendData` → `exdata`;
|
||||
- `xce_sched_job_t` plus `schedobj.c`;
|
||||
- the scheduler snapshot;
|
||||
- the recording and replay path.
|
||||
2. **Scheduler gate** in `sched_job_is_ready()`, or an equivalent eligibility
|
||||
step, using `xce_core_now(core)`:
|
||||
- `now < begin` → not eligible, reason BEGIN_TIME_NOT_REACHED;
|
||||
- "end cannot be met" → not eligible, reason END_TIME_UNMEETABLE.
|
||||
|
||||
Two new non-resource pend-reason codes, published for skipped jobs.
|
||||
3. **Worker enforcement.** The job helper runs the deadline itself: at `end`, it
|
||||
sends SIGTERM to the job's process group, waits the grace period, then sends
|
||||
SIGKILL, and reports the exit with reason END_TIME_REACHED. Belt and braces:
|
||||
the unit's `RuntimeMaxUSec` is set to `end + grace + margin − now` at launch,
|
||||
as a last-resort backstop if the helper itself hangs. Every node of a
|
||||
multi-host job enforces its own tasks.
|
||||
4. **Exit reason.** An additive exit-reason field on the job's exit info,
|
||||
persisted through WAL_JOB_FAIL/FINISH and the snapshot (field completeness).
|
||||
5. **Changing times on a pending job** goes through the existing job
|
||||
modify/update path (WAL_JOB_UPDATE) and is refused for non-pending jobs.
|
||||
|
||||
## Scope
|
||||
- **In**:
|
||||
- The begin and end fields and the runtime estimate end to end (new
|
||||
`-We/--estimated_run_time` xcecli flag), and the `-W` / `exdata` wiring fix.
|
||||
- Submit validation, including estimate > run limit.
|
||||
- Submit validation.
|
||||
- The scheduler gate and the two new pend reasons.
|
||||
- Worker end-time enforcement with grace, plus the backstop.
|
||||
- The new, persisted exit reason.
|
||||
- Changing the times of pending jobs, through a new `xce jobs modify` in
|
||||
xcecli and a new `jobmod` in the harness.
|
||||
- Client support in **both** xcecli (the `-We` flag, with `-b`/`-t`/`-W` made
|
||||
to work) and the harness `jobsub` (all four options), with help text and
|
||||
docs.
|
||||
- Display in xcecli and REST.
|
||||
- Simulator determinism (the gate uses the core clock).
|
||||
- Tests: unit, sim, and a live kill-at-deadline test.
|
||||
- **Open for requirements**:
|
||||
- The grace-period default, and whether it is cluster config or per job.
|
||||
- The time zone and format accepted by `-b`/`-t` (xcecli `utils.ParseTime`).
|
||||
- Whether a job array's elements share the window.
|
||||
- Clock-skew tolerance between the scheduler, controller and worker hosts
|
||||
(NTP assumed).
|
||||
- **Out**:
|
||||
- Controller-side enforcement or sweeps (user decision).
|
||||
- Automatic expiry of unmeetable pending jobs (user decision: they stay PEND).
|
||||
- Changing the end time of a running job.
|
||||
- Backfill or advance reservation built on end times.
|
||||
- Deadline-based preemption (`scheduler-preemption` lists it as out of scope).
|
||||
- Queue run and dispatch windows (`types.h:570-573`, `lib/window.c`, unused).
|
||||
|
||||
## Boundary Candidates
|
||||
- Field plumbing: capnp, C structs, libxce interning, apiserver conversion,
|
||||
scheduler object and snapshot, sim recording.
|
||||
- Client surfaces: xcecli submit and modify, harness `jobsub` and `jobmod`,
|
||||
shared time parsing and validation.
|
||||
- The scheduler eligibility gate and the new pend-reason codes.
|
||||
- Worker deadline enforcement (helper timer plus unit backstop).
|
||||
- The exit-reason field and its persistence.
|
||||
- The pending-job change path and submit validation.
|
||||
|
||||
## Out of Boundary
|
||||
- The display of reasons (`job-status-detail-display`). This spec adds the codes;
|
||||
that spec's code-to-text table needs the new entries.
|
||||
- How pend reasons are grouped for jobs with alternative resource requirements
|
||||
(`scheduler-alter-pend-reasons`).
|
||||
- Deciding which jobs a user may see or control (existing RBAC).
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream**:
|
||||
- The existing submit, modify and kill paths.
|
||||
- The REST `run_time_estimation` field, which exists but is dropped today.
|
||||
- The scheduler FCFS picker and `sched_job_is_ready()`.
|
||||
- The worker transient unit (`RuntimeMaxUSec` support already exists).
|
||||
- The WAL submit and update facts.
|
||||
- **Downstream**:
|
||||
- `job-status-detail-display`, which shows the times and the new reason texts.
|
||||
- Future backfill or reservation work, which can use the runtime estimate.
|
||||
- `application-profile`, where resource defaults might supply a default run
|
||||
limit.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends**: none.
|
||||
- **Adjacent**:
|
||||
- `job-status-detail-display` (new pend and exit reason codes need text)
|
||||
- `scheduler-alter-pend-reasons` (pend-reason model)
|
||||
- `scheduler-preemption`
|
||||
- `controller-job-cleanup` (an EXIT from the end time becomes eligible for
|
||||
cleanup as usual)
|
||||
- `application-profile`
|
||||
- `scheduler-sim-discrete-clock` (the core clock)
|
||||
|
||||
## Constraints
|
||||
- The primary does no end-time work. Enforcement lives on the worker, and gating
|
||||
lives in the scheduler.
|
||||
- All wire and schema changes are additive Cap'n Proto. Old WAL records and
|
||||
snapshots decode with no window, which is today's behaviour.
|
||||
- Save/restore field completeness (`persistence.md`): the begin and end fields
|
||||
and the exit reason must round trip through the WAL submit, update and fail
|
||||
facts, the controller snapshot, the scheduler snapshot and the recordings. Run
|
||||
`/xce-audit-invariants` (controller dimension 7, scheduler dimension 4).
|
||||
- The scheduler's core time only comes from `xce_core_now`, so `--sim-verify`
|
||||
stays deterministic.
|
||||
- The worker's non-systemd process-manager build (`#ifndef DBUS_MANAGER`) must
|
||||
enforce the deadline too; the helper timer covers it.
|
||||
- TDD: `testsuite/` first, then `_test.go`. ASAN must be clean.
|
||||
- HA/persistence touch: use the full 3-phase workflow.
|
||||
|
|
@ -65,6 +65,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
|
|||
- [ ] job-sandbox-execution — NEW 2026-09-29 (Job Sandbox Execution). Worker: pluggable execution-backend interface (`direct` refactored + `systemd-sandbox`; non-systemd backends later). Sandbox = existing transient unit + `DynamicUser` with fresh per-job name (never the submitter's), ProtectSystem=strict/ProtectHome/PrivateTmp/NoNewPrivileges, Bind(ReadOnly)Paths for job I/O dirs or worker-created temp workspace (handed to the uid via SO_PEERCRED), `StandardOutput=file:` output, helper auth by unit/cgroup, MANDATORY post-job cleanup/ownership normalization (dynamic uid reuse), systemd ≥ 242 + nss-systemd capability probe; single-node incl. single-node MPI. Viability-checked 2026-09-29. Brief: `.kiro/specs/job-sandbox-execution/brief.md`. Dependencies: application-profile; remote-task-root fix (Direct-Implementation).
|
||||
- [ ] job-sandbox-multinode — NEW 2026-09-29 (Job Sandbox Execution). Multi-node sandbox jobs: cluster-consistent ephemeral uid (expected: controller-allocated, WAL-replicated uid lease from a reserved range), worker-driven remote/MPI launch (no ssh; PMIx/prted via worker), shared-FS ownership normalization, one unit per job per node. Brief: `.kiro/specs/job-sandbox-multinode/brief.md`. Dependencies: job-sandbox-execution, application-profile.
|
||||
- [ ] rbac-persistence — NEW 2026-09-29 (`/kiro-discovery`, raised during application-profile discovery). Closes the deferred `security.md` gap: runtime RBAC mutations (roles, permissions, user-role assignments, Logon auto-assignment) are in-memory only and silently revert to `rbac.yaml` on apiserver restart — including revocations. Durable-or-rejected mutations, audit trail, defaults-vs-runtime reconciliation, multi-apiserver consistency; storage approach (rbac.yaml write-through / controller-replicated entity / embedded store / read-only) decided in design. Brief: `.kiro/specs/rbac-persistence/brief.md`. Dependencies: none (independent of the sandbox initiative; application-profile's API-made `app:use` grants rely on it for durability).
|
||||
- [ ] job-time-window — NEW 2026-09-29 (`/kiro-discovery`, Path C). Job begin time (scheduler gate in `sched_job_is_ready()` via `xce_core_now`, pend reason BEGIN_TIME_NOT_REACHED) and end time (WORKER enforces running jobs: helper SIGTERM→grace→SIGKILL, exit reason END_TIME_REACHED persisted, unit RuntimeMaxUSec backstop; PEND jobs whose end time can't be met stay PEND with END_TIME_UNMEETABLE until user kills/modifies — no controller sweep, primary not involved). Adds runtime estimate (`-We/--estimated_run_time`, scheduling hint only, never enforced; estimate > run limit rejected); unmeetable rule `now + (estimate ‖ run_limit ‖ 0) > end`. Times/estimate modifiable only while PEND (new `xce jobs modify` + harness `jobmod`). Both xcecli and harness `jobsub` gain all options (`-b`/`-t`/`-W`/estimate). Also fixes the silent drop of xcecli `-b`/`-t`/`-W` and REST `run_time_estimation` (apiserver `modelJobToSubmit` never sets them nor `exdata`). Brief: `.kiro/specs/job-time-window/brief.md`. Dependencies: none (new reason codes need text in job-status-detail-display).
|
||||
- [ ] controller-job-cleanup — NEW 2026-09-29 (`/kiro-discovery`, Path C). Primary-only periodic sweep (configurable `job_cleanup_period`, default 300 s) removes DONE/EXIT jobs finished longer than configurable `job_retention` (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever `jobmgr` delete; refcount-safe `xkvmgr` delete), propagated to scheduler and apiserver (`OnJobDeleted`). Retires the node-local snapshot-only `job_should_retain` finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: `.kiro/specs/controller-job-cleanup/brief.md`. Dependencies: none (extends ha-entity-snapshots).
|
||||
- [ ] job-status-detail-display — NEW 2026-09-28 (`/kiro-discovery`, Path E). End users (REST/`xcecli`) see NO pend reason for ANY job today: the apiserver cgo conversion (`backend/convert.go`) never maps `xce_job_t.pend_reasons`, so `pending_reason` is always empty. Maps it to a structured JSON (including `alternatives` from day one) using libxce's code→text mapping, and renders it in `xcecli` job detail. Open for requirements: also map `run_usage` → `JobRusage` (same gap; reserved vs. used for RUN jobs). Brief: `.kiro/specs/job-status-detail-display/brief.md`. Dependencies: none for flat reasons; scheduler-alter-pend-reasons for per-alternative rendering.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue