myxce/.kiro/specs/job-time-window/brief.md

Brief: job-time-window

Problem

XCE users can't constrain when a job runs:

• Begin time. "Don't start this job before 22:00", for example to wait for off-peak hours, input data, or a maintenance window to end.
• End time. "This job must be finished by 06:00", for example when results are useless after a deadline, or a reservation or maintenance starts.

xcecli jobs submit already offers -b/--begin_time, -t/--term_time and -W/--run_limit, and the apiserver accepts them. They are silently dropped before reaching the controller, so users believe they are protected when they are not.

Current State

Codebase exploration, 2026-09-29.

Client and apiserver

• xcecli: -b (cmd/jobs/submit.go:682), -t (:686) and -W (:688), parsed into jobReq.BeginTime / TermTime / RunLimit (:278, :312-330).
• apiserver submit request: BeginTime, TermTime, RunTimeEstimation and RunLimit (models/jobs.go:641-674). They are copied only into the Go job model (controllers/jobs.go:1475-1487, restart path :1287-1318).
  • run_limit is folded into the ExtendData JSON under limits (controllers/jobs.go:2195-2211).
• The gap: modelJobToSubmit (backend/convert.go:272-318) sets none of begin time, term time, runtime estimate, or ExtendData → start_info.exdata. So -b, -t and -W never reach the controller. Only the xce_test harness's -x <exdata> delivers a run limit today.

C side

• job_submit_data_t (include/xce/types.h:356-370), JobSubmitData (schema/controller.capnp:579-591, last ordinal @10) and xce_sched_job_t (scheduler/include/sched/types.h:122-142) have no begin, end or limit field.

Scheduler

• The FCFS picker (plugins/builtin_fcfs.c:837-887) takes the first job for which sched_queue_is_ready() and sched_job_is_ready() both hold. Both are stubs returning 1 (core.c:490, :493), which is the natural eligibility hook.
• Sessions run about every 1 s (main.c:508-545), so the scheduler notices a begin time within about a second, with no extra wakeup.
• Time must come from xce_core_now(core) / core->session_start (core_clock.c), never time(). That keeps the simulator's discrete clock and --sim-verify deterministic, and test_core_isolation.c forbids wall-clock calls in core*.c.

Pend reasons

• Only resource reasons exist: PREASON_BAD_RESREQ, PREASON_NO_RESOURCE and PREASON_MATCH_FAIL (include/preason.h:7-9).
• There is no non-resource reason, and no publish path for a job the picker skips. Publishing goes through sched_publish_pend_reason (scheduler.c:1821).

Worker

• The run limit is read from exdata.limits.run_limit (xcecd_exdata.c:306-309) and set as RuntimeMaxUSec on the transient unit (xcecd_cgroups.c:753-770). That is relative to unit start.
• The kill path is xcecd_worker.c:1115-1124 (signal_gid_except_self, TERM→KILL escalation).
• The job helper xcecd_worker runs inside the unit and reports the job's exit (uv_ipc.c:1000). If systemd times out the unit, the helper is killed too, and the exit shows only as a plain signal exit.

Exit info

• job_exitinfo_t is {exitcode} only (types.h:378-380; capnp JobExitInfo).
• job_failinfo_t.detail is not persisted (WAL_JOB_FAIL carries code plus stage only, persist.c:4224-4257).
• There is no exit-reason field.

Persistence

• The WAL_JOB_SUBMIT payload is the whole capnp xce_job_t, so new JobSubmitData fields ride along automatically.
• These must be updated by hand:
  • the libxce job-submit interning table (lib/jobj.c:29-55, :123-182). Its equality and hash must include the new fields, or submits that differ only in time would be merged;
  • the scheduler's schedobj.c:253-279 and update path :733;
  • the scheduler snapshot snapshot_wire.c:134-166, which rebuilds submit field by field.

Desired Outcome

• Begin time.
  • A job is not started before its begin time.
  • Until then it pends with a clear pend reason: "begin time not reached (starts at T)".
• End time, running jobs (user decision, 2026-09-29: the worker handles running jobs, and the primary is not involved).
  • A running job still running at its end time is terminated by its worker:
    • a warning signal (SIGTERM);
    • a configurable grace period;
    • then SIGKILL.
  • The job ends as EXIT with a recorded exit reason, "end time reached", visible to users and persisted.
• End time, pending jobs (user decision).
  • A pending job whose end time can no longer be met is not started and stays PEND with a clear pend reason ("end time cannot be met").
  • It stays that way until the user kills it or changes its end time. It is never auto-expired, and the controller runs no sweep.
• Runtime estimate (user decision, 2026-09-29, folded into this spec).
  • A new submit option xcecli jobs submit --estimated_run_time / -We. The REST API already accepts run_time_estimation (models/jobs.go:659) but drops it, like -b/-t/-W.
  • It is a scheduling hint only and is never enforced. That is the difference from the run limit -W, which is a hard cap the worker kills at; this mirrors LSF -We versus -W.
  • It is carried end to end to the scheduler (object and snapshot) and can be changed while the job is pending.
  • If both are given and estimate > run limit, the submit is rejected with a clear error (user decision).
  • An application-profile may later supply a default estimate.
• "End time cannot be met" rule (settled). A pending job is not started when now + expected_runtime > end, where expected_runtime is the estimate if given, else the run limit if given, else 0 (plain now ≥ end). now comes from the scheduler core clock.
• Changes. Begin time, end time and the runtime estimate can be changed only while the job is pending (user decision). Once the job runs, its end time is fixed, so the worker never needs live updates.
• Wiring fix (user decision: fixed in this spec). -b, -t and -W (exdata / run_limit) actually reach the controller, scheduler and worker. Log it in the buglog as the drop defect.
• Validation at submit.
  • Reject begin ≥ end.
  • Reject an end time already in the past.
  • Reject a time that cannot be parsed.
• Both client tools support every new option (user direction, 2026-09-29).
  • xcecli (cmd/jobs/submit.go):
    • -b/--begin_time, -t/--term_time and -W/--run_limit exist today but are dropped downstream; this spec makes them work end to end;
    • add -We/--estimated_run_time;
    • xcecli has no command for changing a submitted job, so add one, for example xce jobs modify <jobid> [-b] [-t] [-We], pending jobs only.
  • Harness xce_test (src/harness/test.c):
    • jobsub (jobsub_flags[] at :3575, usage at :602) today takes only -u, -q, -c, -R, -n, -x and -a. Add begin time, end time, run limit and estimate (for example -b/--begin-time, -t/--end-time, -W/--run-limit, -e/--estimated-run-time; short letters are settled in design). -W must be a first-class flag rather than hand-written -x exdata JSON, and it must compose with a user-supplied -x.
    • Add a matching jobmod command for pending jobs.
    • Update the harness help (test_repl_help-style help tests) and the xce_test rc-file or script docs.
  • Both tools accept the same time formats, validated identically: shared rules for absolute times, relative offsets and time zones, settled in requirements.
  • Both show clear errors for begin ≥ end, an end time in the past, and estimate > run limit.
• Visibility. Both times appear in job details (job-status-detail-display renders the new pend and exit reasons).

Approach

Chosen: approach B′, the worker enforces running jobs and the scheduler gates pending ones (user direction, 2026-09-29). This keeps the primary out of it. The rejected option (A) was a primary-only controller sweep with the worker as a backstop.

1. Fields. Add beginTime and endTime (absolute epoch seconds, 0 = unset) and runEstimate (seconds, 0 = unset) as additive fields on JobSubmitData. The run limit stays in exdata.limits.run_limit, but the scheduler also needs it for the unmeetable check, so expose it on the scheduler job object. Carry them through:

   • job_submit_data_t;
   • the libxce interning equality and hash;
   • the apiserver modelJobToSubmit, which also passes ExtendData → exdata;
   • xce_sched_job_t plus schedobj.c;
   • the scheduler snapshot;
   • the recording and replay path.

2. Scheduler gate in sched_job_is_ready(), or an equivalent eligibility step, using xce_core_now(core):

   • now < begin → not eligible, reason BEGIN_TIME_NOT_REACHED;
   • "end cannot be met" → not eligible, reason END_TIME_UNMEETABLE.

   Two new non-resource pend-reason codes, published for skipped jobs.

3. Worker enforcement. The job helper runs the deadline itself: at end, it sends SIGTERM to the job's process group, waits the grace period, then sends SIGKILL, and reports the exit with reason END_TIME_REACHED. Belt and braces: the unit's RuntimeMaxUSec is set to end + grace + margin − now at launch, as a last-resort backstop if the helper itself hangs. Every node of a multi-host job enforces its own tasks.

4. Exit reason. An additive exit-reason field on the job's exit info, persisted through WAL_JOB_FAIL/FINISH and the snapshot (field completeness).

5. Changing times on a pending job goes through the existing job modify/update path (WAL_JOB_UPDATE) and is refused for non-pending jobs.

Scope

• In:
  • The begin and end fields and the runtime estimate end to end (new -We/--estimated_run_time xcecli flag), and the -W / exdata wiring fix.
  • Submit validation, including estimate > run limit.
  • Submit validation.
  • The scheduler gate and the two new pend reasons.
  • Worker end-time enforcement with grace, plus the backstop.
  • The new, persisted exit reason.
  • Changing the times of pending jobs, through a new xce jobs modify in xcecli and a new jobmod in the harness.
  • Client support in both xcecli (the -We flag, with -b/-t/-W made to work) and the harness jobsub (all four options), with help text and docs.
  • Display in xcecli and REST.
  • Simulator determinism (the gate uses the core clock).
  • Tests: unit, sim, and a live kill-at-deadline test.
• Open for requirements:
  • The grace-period default, and whether it is cluster config or per job.
  • The time zone and format accepted by -b/-t (xcecli utils.ParseTime).
  • Whether a job array's elements share the window.
  • Clock-skew tolerance between the scheduler, controller and worker hosts (NTP assumed).
• Out:
  • Controller-side enforcement or sweeps (user decision).
  • Automatic expiry of unmeetable pending jobs (user decision: they stay PEND).
  • Changing the end time of a running job.
  • Backfill or advance reservation built on end times.
  • Deadline-based preemption (scheduler-preemption lists it as out of scope).
  • Queue run and dispatch windows (types.h:570-573, lib/window.c, unused).

Boundary Candidates

• Field plumbing: capnp, C structs, libxce interning, apiserver conversion, scheduler object and snapshot, sim recording.
• Client surfaces: xcecli submit and modify, harness jobsub and jobmod, shared time parsing and validation.
• The scheduler eligibility gate and the new pend-reason codes.
• Worker deadline enforcement (helper timer plus unit backstop).
• The exit-reason field and its persistence.
• The pending-job change path and submit validation.

Out of Boundary

• The display of reasons (job-status-detail-display). This spec adds the codes; that spec's code-to-text table needs the new entries.
• How pend reasons are grouped for jobs with alternative resource requirements (scheduler-alter-pend-reasons).
• Deciding which jobs a user may see or control (existing RBAC).

Upstream / Downstream

• Upstream:
  • The existing submit, modify and kill paths.
  • The REST run_time_estimation field, which exists but is dropped today.
  • The scheduler FCFS picker and sched_job_is_ready().
  • The worker transient unit (RuntimeMaxUSec support already exists).
  • The WAL submit and update facts.
• Downstream:
  • job-status-detail-display, which shows the times and the new reason texts.
  • Future backfill or reservation work, which can use the runtime estimate.
  • application-profile, where resource defaults might supply a default run limit.

Existing Spec Touchpoints

• Extends: none.
• Adjacent:
  • job-status-detail-display (new pend and exit reason codes need text)
  • scheduler-alter-pend-reasons (pend-reason model)
  • scheduler-preemption
  • controller-job-cleanup (an EXIT from the end time becomes eligible for cleanup as usual)
  • application-profile
  • scheduler-sim-discrete-clock (the core clock)

Constraints

• The primary does no end-time work. Enforcement lives on the worker, and gating lives in the scheduler.
• All wire and schema changes are additive Cap'n Proto. Old WAL records and snapshots decode with no window, which is today's behaviour.
• Save/restore field completeness (persistence.md): the begin and end fields and the exit reason must round trip through the WAL submit, update and fail facts, the controller snapshot, the scheduler snapshot and the recordings. Run /xce-audit-invariants (controller dimension 7, scheduler dimension 4).
• The scheduler's core time only comes from xce_core_now, so --sim-verify stays deterministic.
• The worker's non-systemd process-manager build (#ifndef DBUS_MANAGER) must enforce the deadline too; the helper timer covers it.
• TDD: testsuite/ first, then _test.go. ASAN must be clean.
• HA/persistence touch: use the full 3-phase workflow.