myxce/.kiro/specs/job-time-window/brief.md

# Brief: job-time-window

## Problem
XCE users can't constrain *when* a job runs:
- **Begin time.** "Don't start this job before 22:00", for example to wait for
  off-peak hours, input data, or a maintenance window to end.
- **End time.** "This job must be finished by 06:00", for example when results
  are useless after a deadline, or a reservation or maintenance starts.

`xcecli jobs submit` already offers `-b/--begin_time`, `-t/--term_time` and
`-W/--run_limit`, and the apiserver accepts them. They are **silently dropped**
before reaching the controller, so users believe they are protected when they
are not.

## Current State
Codebase exploration, 2026-09-29.

**Client and apiserver**
- xcecli: `-b` (`cmd/jobs/submit.go:682`), `-t` (:686) and `-W` (:688), parsed
  into `jobReq.BeginTime` / `TermTime` / `RunLimit` (:278, :312-330).
- apiserver submit request: `BeginTime`, `TermTime`, `RunTimeEstimation` and
  `RunLimit` (`models/jobs.go:641-674`). They are copied only into the Go job
  model (`controllers/jobs.go:1475-1487`, restart path :1287-1318).
  - `run_limit` is folded into the `ExtendData` JSON under `limits`
    (`controllers/jobs.go:2195-2211`).
- **The gap:** `modelJobToSubmit` (`backend/convert.go:272-318`) sets none of
  begin time, term time, runtime estimate, or `ExtendData` → `start_info.exdata`.
  So `-b`, `-t` and `-W` never reach the controller. Only the `xce_test`
  harness's `-x <exdata>` delivers a run limit today.

**C side**
- `job_submit_data_t` (`include/xce/types.h:356-370`), `JobSubmitData`
  (`schema/controller.capnp:579-591`, last ordinal @10) and `xce_sched_job_t`
  (`scheduler/include/sched/types.h:122-142`) have no begin, end or limit field.

**Scheduler**
- The FCFS picker (`plugins/builtin_fcfs.c:837-887`) takes the first job for which
  `sched_queue_is_ready()` and `sched_job_is_ready()` both hold. **Both are stubs
  returning 1** (`core.c:490`, `:493`), which is the natural eligibility hook.
- Sessions run about every 1 s (`main.c:508-545`), so the scheduler notices a
  begin time within about a second, with no extra wakeup.
- Time must come from `xce_core_now(core)` / `core->session_start`
  (`core_clock.c`), never `time()`. That keeps the simulator's discrete clock and
  `--sim-verify` deterministic, and `test_core_isolation.c` forbids wall-clock
  calls in `core*.c`.

**Pend reasons**
- Only resource reasons exist: `PREASON_BAD_RESREQ`, `PREASON_NO_RESOURCE` and
  `PREASON_MATCH_FAIL` (`include/preason.h:7-9`).
- There is no non-resource reason, and no publish path for a job the picker
  skips. Publishing goes through `sched_publish_pend_reason`
  (`scheduler.c:1821`).

**Worker**
- The run limit is read from `exdata.limits.run_limit` (`xcecd_exdata.c:306-309`)
  and set as `RuntimeMaxUSec` on the transient unit (`xcecd_cgroups.c:753-770`).
  That is relative to unit start.
- The kill path is `xcecd_worker.c:1115-1124` (`signal_gid_except_self`,
  TERM→KILL escalation).
- The job helper `xcecd_worker` runs *inside* the unit and reports the job's exit
  (`uv_ipc.c:1000`). If systemd times out the unit, the helper is killed too, and
  the exit shows only as a plain signal exit.

**Exit info**
- `job_exitinfo_t` is `{exitcode}` only (`types.h:378-380`; capnp `JobExitInfo`).
- `job_failinfo_t.detail` is not persisted (WAL_JOB_FAIL carries code plus stage
  only, `persist.c:4224-4257`).
- There is no exit-reason field.

**Persistence**
- The WAL_JOB_SUBMIT payload is the whole capnp `xce_job_t`, so new
  `JobSubmitData` fields ride along automatically.
- These must be updated by hand:
  - the libxce job-submit interning table (`lib/jobj.c:29-55`, `:123-182`). Its
    equality and hash must include the new fields, or submits that differ only
    in time would be merged;
  - the scheduler's `schedobj.c:253-279` and update path `:733`;
  - the scheduler snapshot `snapshot_wire.c:134-166`, which rebuilds `submit`
    field by field.

## Desired Outcome
- **Begin time.**
  - A job is not started before its begin time.
  - Until then it pends with a clear pend reason: "begin time not reached (starts
    at T)".
- **End time, running jobs** (user decision, 2026-09-29: the worker handles
  running jobs, and the primary is not involved).
  - A running job still running at its end time is terminated by its worker:
    - a warning signal (SIGTERM);
    - a configurable grace period;
    - then SIGKILL.
  - The job ends as EXIT with a recorded exit reason, "end time reached", visible
    to users and persisted.
- **End time, pending jobs** (user decision).
  - A pending job whose end time can no longer be met is **not started** and
    **stays PEND** with a clear pend reason ("end time cannot be met").
  - It stays that way until the user kills it or changes its end time. It is
    never auto-expired, and the controller runs no sweep.
- **Runtime estimate** (user decision, 2026-09-29, folded into this spec).
  - A new submit option `xcecli jobs submit --estimated_run_time` / `-We`. The
    REST API already accepts `run_time_estimation` (`models/jobs.go:659`) but
    drops it, like `-b`/`-t`/`-W`.
  - It is **a scheduling hint only and is never enforced**. That is the
    difference from the run limit `-W`, which is a hard cap the worker kills at;
    this mirrors LSF `-We` versus `-W`.
  - It is carried end to end to the scheduler (object and snapshot) and can be
    changed while the job is pending.
  - If both are given and **estimate > run limit, the submit is rejected** with a
    clear error (user decision).
  - An `application-profile` may later supply a default estimate.
- **"End time cannot be met" rule** (settled). A pending job is not started when
  `now + expected_runtime > end`, where `expected_runtime` is the estimate if
  given, else the run limit if given, else 0 (plain `now ≥ end`). `now` comes from
  the scheduler core clock.
- **Changes.** Begin time, end time and the runtime estimate can be changed
  **only while the job is pending** (user decision). Once the job runs, its end time is fixed, so the
  worker never needs live updates.
- **Wiring fix** (user decision: fixed in this spec). `-b`, `-t` and `-W`
  (`exdata` / `run_limit`) actually reach the controller, scheduler and worker.
  Log it in the buglog as the drop defect.
- **Validation at submit.**
  - Reject begin ≥ end.
  - Reject an end time already in the past.
  - Reject a time that cannot be parsed.
- **Both client tools support every new option** (user direction, 2026-09-29).
  - **xcecli** (`cmd/jobs/submit.go`):
    - `-b/--begin_time`, `-t/--term_time` and `-W/--run_limit` exist today but
      are dropped downstream; this spec makes them work end to end;
    - add `-We/--estimated_run_time`;
    - `xcecli` has no command for changing a submitted job, so add one, for
      example `xce jobs modify <jobid> [-b] [-t] [-We]`, pending jobs only.
  - **Harness `xce_test`** (`src/harness/test.c`):
    - `jobsub` (`jobsub_flags[]` at :3575, usage at :602) today takes only `-u`,
      `-q`, `-c`, `-R`, `-n`, `-x` and `-a`. Add begin time, end time, run limit
      and estimate (for example `-b/--begin-time`, `-t/--end-time`,
      `-W/--run-limit`, `-e/--estimated-run-time`; short letters are settled in
      design). `-W` must be a first-class flag rather than hand-written `-x`
      exdata JSON, and it must compose with a user-supplied `-x`.
    - Add a matching `jobmod` command for pending jobs.
    - Update the harness help (`test_repl_help`-style help tests) and the
      `xce_test` rc-file or script docs.
  - Both tools accept the same time formats, validated identically: shared rules
    for absolute times, relative offsets and time zones, settled in requirements.
  - Both show clear errors for begin ≥ end, an end time in the past, and
    estimate > run limit.
- **Visibility.** Both times appear in job details (`job-status-detail-display`
  renders the new pend and exit reasons).

## Approach
**Chosen: approach B′, the worker enforces running jobs and the scheduler gates
pending ones** (user direction, 2026-09-29). This keeps the primary out of it.
The rejected option (A) was a primary-only controller sweep with the worker as a
backstop.

1. **Fields.** Add `beginTime` and `endTime` (absolute epoch seconds, 0 = unset)
   and `runEstimate` (seconds, 0 = unset) as additive fields on `JobSubmitData`.
   The run limit stays in `exdata.limits.run_limit`, but the scheduler also needs
   it for the unmeetable check, so expose it on the scheduler job object. Carry them through:
   - `job_submit_data_t`;
   - the libxce interning equality and hash;
   - the apiserver `modelJobToSubmit`, which also passes `ExtendData` → `exdata`;
   - `xce_sched_job_t` plus `schedobj.c`;
   - the scheduler snapshot;
   - the recording and replay path.
2. **Scheduler gate** in `sched_job_is_ready()`, or an equivalent eligibility
   step, using `xce_core_now(core)`:
   - `now < begin` → not eligible, reason BEGIN_TIME_NOT_REACHED;
   - "end cannot be met" → not eligible, reason END_TIME_UNMEETABLE.

   Two new non-resource pend-reason codes, published for skipped jobs.
3. **Worker enforcement.** The job helper runs the deadline itself: at `end`, it
   sends SIGTERM to the job's process group, waits the grace period, then sends
   SIGKILL, and reports the exit with reason END_TIME_REACHED. Belt and braces:
   the unit's `RuntimeMaxUSec` is set to `end + grace + margin − now` at launch,
   as a last-resort backstop if the helper itself hangs. Every node of a
   multi-host job enforces its own tasks.
4. **Exit reason.** An additive exit-reason field on the job's exit info,
   persisted through WAL_JOB_FAIL/FINISH and the snapshot (field completeness).
5. **Changing times on a pending job** goes through the existing job
   modify/update path (WAL_JOB_UPDATE) and is refused for non-pending jobs.

## Scope
- **In**:
  - The begin and end fields and the runtime estimate end to end (new
    `-We/--estimated_run_time` xcecli flag), and the `-W` / `exdata` wiring fix.
  - Submit validation, including estimate > run limit.
  - Submit validation.
  - The scheduler gate and the two new pend reasons.
  - Worker end-time enforcement with grace, plus the backstop.
  - The new, persisted exit reason.
  - Changing the times of pending jobs, through a new `xce jobs modify` in
    xcecli and a new `jobmod` in the harness.
  - Client support in **both** xcecli (the `-We` flag, with `-b`/`-t`/`-W` made
    to work) and the harness `jobsub` (all four options), with help text and
    docs.
  - Display in xcecli and REST.
  - Simulator determinism (the gate uses the core clock).
  - Tests: unit, sim, and a live kill-at-deadline test.
- **Open for requirements**:
  - The grace-period default, and whether it is cluster config or per job.
  - The time zone and format accepted by `-b`/`-t` (xcecli `utils.ParseTime`).
  - Whether a job array's elements share the window.
  - Clock-skew tolerance between the scheduler, controller and worker hosts
    (NTP assumed).
- **Out**:
  - Controller-side enforcement or sweeps (user decision).
  - Automatic expiry of unmeetable pending jobs (user decision: they stay PEND).
  - Changing the end time of a running job.
  - Backfill or advance reservation built on end times.
  - Deadline-based preemption (`scheduler-preemption` lists it as out of scope).
  - Queue run and dispatch windows (`types.h:570-573`, `lib/window.c`, unused).

## Boundary Candidates
- Field plumbing: capnp, C structs, libxce interning, apiserver conversion,
  scheduler object and snapshot, sim recording.
- Client surfaces: xcecli submit and modify, harness `jobsub` and `jobmod`,
  shared time parsing and validation.
- The scheduler eligibility gate and the new pend-reason codes.
- Worker deadline enforcement (helper timer plus unit backstop).
- The exit-reason field and its persistence.
- The pending-job change path and submit validation.

## Out of Boundary
- The display of reasons (`job-status-detail-display`). This spec adds the codes;
  that spec's code-to-text table needs the new entries.
- How pend reasons are grouped for jobs with alternative resource requirements
  (`scheduler-alter-pend-reasons`).
- Deciding which jobs a user may see or control (existing RBAC).

## Upstream / Downstream
- **Upstream**:
  - The existing submit, modify and kill paths.
  - The REST `run_time_estimation` field, which exists but is dropped today.
  - The scheduler FCFS picker and `sched_job_is_ready()`.
  - The worker transient unit (`RuntimeMaxUSec` support already exists).
  - The WAL submit and update facts.
- **Downstream**:
  - `job-status-detail-display`, which shows the times and the new reason texts.
  - Future backfill or reservation work, which can use the runtime estimate.
  - `application-profile`, where resource defaults might supply a default run
    limit.

## Existing Spec Touchpoints
- **Extends**: none.
- **Adjacent**:
  - `job-status-detail-display` (new pend and exit reason codes need text)
  - `scheduler-alter-pend-reasons` (pend-reason model)
  - `scheduler-preemption`
  - `controller-job-cleanup` (an EXIT from the end time becomes eligible for
    cleanup as usual)
  - `application-profile`
  - `scheduler-sim-discrete-clock` (the core clock)

## Constraints
- The primary does no end-time work. Enforcement lives on the worker, and gating
  lives in the scheduler.
- All wire and schema changes are additive Cap'n Proto. Old WAL records and
  snapshots decode with no window, which is today's behaviour.
- Save/restore field completeness (`persistence.md`): the begin and end fields
  and the exit reason must round trip through the WAL submit, update and fail
  facts, the controller snapshot, the scheduler snapshot and the recordings. Run
  `/xce-audit-invariants` (controller dimension 7, scheduler dimension 4).
- The scheduler's core time only comes from `xce_core_now`, so `--sim-verify`
  stays deterministic.
- The worker's non-systemd process-manager build (`#ifndef DBUS_MANAGER`) must
  enforce the deadline too; the helper timer covers it.
- TDD: `testsuite/` first, then `_test.go`. ASAN must be clean.
- HA/persistence touch: use the full 3-phase workflow.