myxce/.kiro/specs/job-dependency/brief.md

Brief: job-dependency

Problem

XCE users run workflows: preprocess → solve → postprocess, or "run B only if A failed". Today they must babysit each step and submit the next job by hand when the previous one finishes, or write external polling scripts. XCE has no job dependency concept at all. job_submit_data_t has no dependency field, and nothing in the controller, scheduler, libxce or clients handles one.

controller-job-cleanup also needs a real "does any job depend on this one?" check; it is a stub today.

Reference: IBM Spectrum LSF 10.1 Administering LSF, section "Job dependency and job priority" (printed pp. 459-465, arrays p. 522, orphans pp. 91-94). The local extracted text is in the discovery scratchpad. User scope decision (2026-09-29): job dependency only. Job priority (user priority, priority escalation, absolute priority scheduling) is a separate, later discovery.

Current State

Codebase exploration, 2026-09-29.

• No dependency fields in job_submit_data_t (include/xce/types.h:356-370) or capnp JobSubmitData (schema/controller.capnp:579-591, last ordinal @10).
• The scheduler gate is a stub. The FCFS picker skips a job unless sched_job_is_ready() holds (scheduler/src/core.c:493, returns 1 today). job-time-window uses the same gate for begin and end times.
  • Sessions run about every 1 s.
  • Time and state must come from the scheduler core (xce_core_now) so that --sim-verify stays deterministic.
• The scheduler sees parent state through XCE_CORE_EVENT_JOB_STATE_CHANGE {jobid, old_state, new_state, exit_code} (scheduler/include/core.h:550-555).
  • Gap: xce_sched_job_t does not store the exit code.
  • Gap: the full-push wire SchedJob (controller.capnp:347-357) carries no exit info, and neither does the scheduler snapshot (snapshot_wire.c).
  • So after a scheduler restart or failover, exit(id, op code) could not be evaluated. This must be closed (save/restore field completeness).
• Job states (types.h:382-429): DONE, EXIT, PDONE (post-exec done OK), PERR (post-exec error), RUN, USUSP, SSUSP, and so on. LSF post_done / post_err map onto PDONE / PERR directly.
• Pend reasons. Only resource codes exist (include/preason.h:7-9). A new non-resource code, and a publish path for skipped jobs, is also needed by job-time-window.
• Controller job table. c->jobmgr holds jobs. The scheduler's core->jobs also keeps every job forever today, which controller-job-cleanup changes.
• Array jobs. There is no parent object. Each element is an independent job with its own jobid. Elements are numbered 1..N per job-array-one-based-index (2026-10-01) and are referenced as <array-id>[k], k = 1..N; a bare element jobid is rejected by the CLI/REST listing and control commands (use <array-id>[k]; peek and restart still take the element's own jobid). The earlier note here, jobid base+i with i from 0, described numbering from 0.
• Submit paths: REST (apiserver, fed every job by controller push, cgo into libxce), xcecli jobs submit, the harness xce_test jobsub (harness/test.c:3575), and libxce directly.
• The job-modify command (xce jobs modify, harness jobmod) is being introduced by job-time-window.

Desired Outcome

• A user submits a job with a dependency expression, for example xcecli jobs submit -w 'done(1001) && (started(1002) || exit(1003, != 0))' .... The job is not started until the expression is TRUE.
• v1 conditions (user decision), all by job ID:
  • done(id)
  • ended(id) (DONE or EXIT)
  • exit(id [, op code]), where op is one of > >= < <= == !=; the default is ==, and with no code any exit code matches
  • started(id) (RUN, USUSP, SSUSP, DONE or EXIT)
  • post_done(id) (PDONE) and post_err(id) (PERR)
  • a bare id, which means done(id)
  • operators &&, ||, !, with parentheses
• Validation at submit, rejected with a precise error, following LSF:
  • bad syntax;
  • a referenced job that does not exist (no forward references);
  • a referenced job the submitter may not view (reuse RBAC job:view);
  • self-reference or a cycle (impossible with existing parents at submit, but it must be checked on modify).
• While waiting, the job is PEND with the pend reason "dependency not satisfied".
• When the condition can never be satisfied (an orphan, for example done(P) after P exited), the job stays PEND with the reason "dependency can never be satisfied" until the user kills it or changes the dependency. This is the default (user decision).
• Optional orphan termination is owned by job-dependency-insight.
• Changing a pending job's dependency: set or remove it (-w, and -wn to remove) through the modify path introduced by job-time-window. It is re-validated like a submit.
• Cleanup. A job that any pending job depends on is never removed by controller-job-cleanup. This spec implements that spec's job_has_dependents() stub.
• Both client tools, xcecli and the harness (jobsub -w, jobmod -w/-wn), support the options, and job details show the dependency expression.

Approach

Chosen (user decision, 2026-09-29): the apiserver validates early, the controller is authoritative for validation and owns the dependency graph, and the scheduler evaluates.

1. One shared grammar in libxce. A parser, AST and printer for the expression, used by the controller, the scheduler, the harness and (via cgo) the apiserver. No second copy in Go.

2. apiserver, early validation (like application-profile's existence check), before anything goes downstream:

   • parse;
   • check each referenced job exists in its job store;
   • check the submitter holds job:view on it.

   It rejects with a clear 4xx.

3. Controller, authoritative.

   • On submit and modify it re-validates: the apiserver's store can be stale, a parent may have been cleaned up meanwhile, and the harness and libxce bypass the apiserver.
   • It stores the canonical expression in a new additive JobSubmitData field, carried by the WAL submit, update and snapshot paths (field completeness).
   • It keeps an in-memory reverse index (parent → dependents), rebuilt from the stored expressions on replay and snapshot load. There is no new persisted state.
   • The index serves job_has_dependents() (for cleanup) and the queries in job-dependency-insight.

4. Scheduler, evaluation. Each session, sched_job_is_ready() evaluates the AST against the scheduler's own job states (deterministic under the core clock). It yields satisfied, not yet, or never, and publishes the corresponding pend reason.

   • The scheduler must keep each parent's final state and exit code, so add the exit code to xce_sched_job_t, to SchedJob, and to the scheduler snapshot.
   • It must not drop finished parents while pending dependents exist (coordinate with controller-job-cleanup).

5. Primary work. The primary does no per-session evaluation. Its only dependency work is validation and the index updates at submit, modify and remove.

Rejected alternatives:

• The controller evaluates on each state change (LSF mbatchd style): more primary work.
• Scheduler-only: no validation at submit, and no authoritative reverse index.
• The apiserver owns the graph: libxce and the harness bypass it, its store is non-authoritative and lost on restart, and cleanup runs on the controller.

Scope

• In:
  • The shared libxce expression grammar.
  • The new submit field and its persistence.
  • apiserver and controller validation.
  • The controller reverse index and job_has_dependents().
  • Scheduler evaluation plus the two pend reasons.
  • The scheduler exit-code field-completeness fix.
  • Modifying pending dependencies (set and remove).
  • xcecli -w / modify -w -wn, harness jobsub -w / jobmod.
  • Showing the expression in job details.
  • Simulator determinism.
  • Tests.
• Open for requirements:
  • What happens when a dependent's parent is later removed by cleanup. It can't happen while the dependent is pending, but it can after the dependent itself has finished.
  • Requeue and rerun semantics. LSF: requeuing a finished parent can orphan its dependents.
  • A limit on expression length or complexity.
  • Whether whole-array references (all elements) are wanted in v1 without the num* conditions.
• Out:
  • Referring to jobs by name, and wildcards (not selected for v1).
  • The array count conditions num* (not selected for v1).
  • external().
  • The LSF JOB_DEP_LAST_SUB and EVALUATE_JOB_DEPENDENCY tuning.
  • Job priority features.
  • A bjdepinfo-style query, dependency-graph display, and orphan termination (job-dependency-insight).
  • Cross-cluster dependencies.

Boundary Candidates

• The libxce grammar (parse, validate, print, evaluate on an abstract state-lookup interface).
• apiserver early validation.
• Controller authoritative validation, storage and the reverse index.
• Scheduler evaluation, pend reasons, and keeping parent state and exit code.
• Client surfaces: xcecli, harness, REST.

Out of Boundary

• Showing dependency trees and levels (job-dependency-insight).
• Automatic orphan termination (job-dependency-insight).
• How pend reasons are rendered as text (job-status-detail-display, which needs the new codes).

Upstream / Downstream

• Upstream:
  • The scheduler eligibility gate (shared with job-time-window; sequence or coordinate the two).
  • The job-modify path introduced by job-time-window.
  • RBAC job:view.
  • The WAL submit and update facts.
• Downstream:
  • job-dependency-insight.
  • controller-job-cleanup (the real job_has_dependents()).
  • job-status-detail-display (text for the new reasons, showing the expression).
  • Future name-based references, array conditions, and job priority.

Existing Spec Touchpoints

• Extends: controller-job-cleanup (replaces its stub dependency check).
• Adjacent:
  • job-time-window (the same gate, the modify path, the pend-reason infrastructure)
  • job-status-detail-display
  • scheduler-alter-pend-reasons
  • scheduler-sim-discrete-clock

Constraints

• The primary does no per-session evaluation.

• Every wire change is an additive Cap'n Proto field. Old WAL records and snapshots decode as "no dependency".

• Save/restore field completeness:

  • the expression round trips through the WAL submit and update facts, the controller snapshot, the scheduler snapshot and the recordings;
  • the scheduler job's exit code round trips through the full push and the snapshot;
  • the reverse index is rebuilt identically after replay.

  Run /xce-audit-invariants (controller dimension 7, scheduler dimension 4).

• Scheduler evaluation uses only core state and clock (--sim-verify determinism).

• One grammar implementation (libxce), used from Go through cgo.

• Middleware chain unchanged. RBAC job:view governs which jobs a user may reference.

• HA/persistence touch: use the full 3-phase workflow.