myxce/.kiro/specs/job-dependency/brief.md

# Brief: job-dependency

## Problem
XCE users run workflows: preprocess → solve → postprocess, or "run B only if A
failed". Today they must babysit each step and submit the next job by hand when
the previous one finishes, or write external polling scripts. XCE has **no job
dependency concept at all**. `job_submit_data_t` has no dependency field, and
nothing in the controller, scheduler, libxce or clients handles one.

`controller-job-cleanup` also needs a real "does any job depend on this one?"
check; it is a stub today.

Reference: *IBM Spectrum LSF 10.1 Administering LSF*, section "Job dependency and
job priority" (printed pp. 459-465, arrays p. 522, orphans pp. 91-94). The local
extracted text is in the discovery scratchpad. User scope decision (2026-09-29):
**job dependency only**. Job priority (user priority, priority escalation,
absolute priority scheduling) is a separate, later discovery.

## Current State
Codebase exploration, 2026-09-29.

- **No dependency fields** in `job_submit_data_t` (`include/xce/types.h:356-370`)
  or capnp `JobSubmitData` (`schema/controller.capnp:579-591`, last ordinal @10).
- **The scheduler gate is a stub.** The FCFS picker skips a job unless
  `sched_job_is_ready()` holds (`scheduler/src/core.c:493`, returns 1 today).
  `job-time-window` uses the same gate for begin and end times.
  - Sessions run about every 1 s.
  - Time and state must come from the scheduler core (`xce_core_now`) so that
    `--sim-verify` stays deterministic.
- **The scheduler sees parent state** through `XCE_CORE_EVENT_JOB_STATE_CHANGE`
  `{jobid, old_state, new_state, exit_code}` (`scheduler/include/core.h:550-555`).
  - **Gap:** `xce_sched_job_t` does not store the exit code.
  - **Gap:** the full-push wire `SchedJob` (`controller.capnp:347-357`) carries
    no exit info, and neither does the scheduler snapshot (`snapshot_wire.c`).
  - So after a scheduler restart or failover, `exit(id, op code)` could not be
    evaluated. This must be closed (save/restore field completeness).
- **Job states** (`types.h:382-429`): DONE, EXIT, PDONE (post-exec done OK),
  PERR (post-exec error), RUN, USUSP, SSUSP, and so on. LSF `post_done` /
  `post_err` map onto PDONE / PERR directly.
- **Pend reasons.** Only resource codes exist (`include/preason.h:7-9`). A new
  non-resource code, and a publish path for skipped jobs, is also needed by
  `job-time-window`.
- **Controller job table.** `c->jobmgr` holds jobs. The scheduler's `core->jobs`
  also keeps every job forever today, which `controller-job-cleanup` changes.
- **Array jobs.** There is no parent object. Each element is an independent job
  with its own jobid. Elements are numbered 1..N per `job-array-one-based-index`
  (2026-10-01) and are referenced as `<array-id>[k]`, k = 1..N; a bare element
  jobid is rejected by the CLI/REST listing and control commands (use
  `<array-id>[k]`; peek and restart still take the element's own jobid). The earlier note here,
  jobid `base+i` with i from 0, described numbering from 0.
- **Submit paths:** REST (apiserver, fed every job by controller push,
  cgo into libxce), `xcecli jobs submit`, the harness `xce_test jobsub`
  (`harness/test.c:3575`), and libxce directly.
- **The job-modify command** (`xce jobs modify`, harness `jobmod`) is being
  introduced by `job-time-window`.

## Desired Outcome
- A user submits a job with a **dependency expression**, for example
  `xcecli jobs submit -w 'done(1001) && (started(1002) || exit(1003, != 0))' ...`.
  The job is not started until the expression is TRUE.
- **v1 conditions** (user decision), all by **job ID**:
  - `done(id)`
  - `ended(id)` (DONE or EXIT)
  - `exit(id [, op code])`, where `op` is one of `> >= < <= == !=`; the default is
    `==`, and with no code any exit code matches
  - `started(id)` (RUN, USUSP, SSUSP, DONE or EXIT)
  - `post_done(id)` (PDONE) and `post_err(id)` (PERR)
  - a bare `id`, which means `done(id)`
  - operators `&&`, `||`, `!`, with parentheses
- **Validation at submit**, rejected with a precise error, following LSF:
  - bad syntax;
  - a referenced job that does not exist (no forward references);
  - a referenced job the submitter may not view (reuse RBAC `job:view`);
  - self-reference or a cycle (impossible with existing parents at submit, but
    it must be checked on modify).
- **While waiting**, the job is PEND with the pend reason "dependency not
  satisfied".
- **When the condition can never be satisfied** (an orphan, for example
  `done(P)` after P exited), the job stays PEND with the reason "dependency can
  never be satisfied" until the user kills it or changes the dependency. This is
  the default (user decision).
- **Optional orphan termination** is owned by `job-dependency-insight`.
- **Changing a pending job's dependency**: set or remove it (`-w`, and `-wn` to
  remove) through the modify path introduced by `job-time-window`. It is
  re-validated like a submit.
- **Cleanup.** A job that any **pending** job depends on is never removed by
  `controller-job-cleanup`. This spec implements that spec's
  `job_has_dependents()` stub.
- **Both client tools**, xcecli and the harness (`jobsub -w`, `jobmod -w/-wn`),
  support the options, and job details show the dependency expression.

## Approach
**Chosen (user decision, 2026-09-29):** the apiserver validates early, the
controller is authoritative for validation and owns the dependency graph, and
the scheduler evaluates.

1. **One shared grammar in libxce.** A parser, AST and printer for the
   expression, used by the controller, the scheduler, the harness and (via cgo)
   the apiserver. No second copy in Go.
2. **apiserver, early validation** (like `application-profile`'s existence
   check), before anything goes downstream:
   - parse;
   - check each referenced job exists in its job store;
   - check the submitter holds `job:view` on it.

   It rejects with a clear 4xx.
3. **Controller, authoritative.**
   - On submit and modify it re-validates: the apiserver's store can be stale, a
     parent may have been cleaned up meanwhile, and the harness and libxce bypass
     the apiserver.
   - It stores the canonical expression in a new additive `JobSubmitData` field,
     carried by the WAL submit, update and snapshot paths (field completeness).
   - It keeps an **in-memory reverse index** (parent → dependents), rebuilt from
     the stored expressions on replay and snapshot load. There is no new
     persisted state.
   - The index serves `job_has_dependents()` (for cleanup) and the queries in
     `job-dependency-insight`.
4. **Scheduler, evaluation.** Each session, `sched_job_is_ready()` evaluates the
   AST against the scheduler's own job states (deterministic under the core
   clock). It yields satisfied, not yet, or never, and publishes the
   corresponding pend reason.
   - The scheduler must keep each parent's final state **and exit code**, so add
     the exit code to `xce_sched_job_t`, to `SchedJob`, and to the scheduler
     snapshot.
   - It must not drop finished parents while pending dependents exist (coordinate
     with `controller-job-cleanup`).
5. **Primary work.** The primary does no per-session evaluation. Its only
   dependency work is validation and the index updates at submit, modify and
   remove.

Rejected alternatives:
- **The controller evaluates on each state change** (LSF mbatchd style): more
  primary work.
- **Scheduler-only**: no validation at submit, and no authoritative reverse
  index.
- **The apiserver owns the graph**: libxce and the harness bypass it, its store
  is non-authoritative and lost on restart, and cleanup runs on the controller.

## Scope
- **In**:
  - The shared libxce expression grammar.
  - The new submit field and its persistence.
  - apiserver and controller validation.
  - The controller reverse index and `job_has_dependents()`.
  - Scheduler evaluation plus the two pend reasons.
  - The scheduler exit-code field-completeness fix.
  - Modifying pending dependencies (set and remove).
  - xcecli `-w` / `modify -w -wn`, harness `jobsub -w` / `jobmod`.
  - Showing the expression in job details.
  - Simulator determinism.
  - Tests.
- **Open for requirements**:
  - What happens when a dependent's parent is later removed by cleanup. It can't
    happen while the dependent is pending, but it can after the dependent itself
    has finished.
  - Requeue and rerun semantics. LSF: requeuing a finished parent can orphan its
    dependents.
  - A limit on expression length or complexity.
  - Whether whole-array references (all elements) are wanted in v1 without the
    `num*` conditions.
- **Out**:
  - Referring to jobs by name, and wildcards (not selected for v1).
  - The array count conditions `num*` (not selected for v1).
  - `external()`.
  - The LSF `JOB_DEP_LAST_SUB` and `EVALUATE_JOB_DEPENDENCY` tuning.
  - Job priority features.
  - A `bjdepinfo`-style query, dependency-graph display, and orphan termination
    (`job-dependency-insight`).
  - Cross-cluster dependencies.

## Boundary Candidates
- The libxce grammar (parse, validate, print, evaluate on an abstract
  state-lookup interface).
- apiserver early validation.
- Controller authoritative validation, storage and the reverse index.
- Scheduler evaluation, pend reasons, and keeping parent state and exit code.
- Client surfaces: xcecli, harness, REST.

## Out of Boundary
- Showing dependency trees and levels (`job-dependency-insight`).
- Automatic orphan termination (`job-dependency-insight`).
- How pend reasons are rendered as text (`job-status-detail-display`, which needs
  the new codes).

## Upstream / Downstream
- **Upstream**:
  - The scheduler eligibility gate (shared with `job-time-window`; sequence or
    coordinate the two).
  - The job-modify path introduced by `job-time-window`.
  - RBAC `job:view`.
  - The WAL submit and update facts.
- **Downstream**:
  - `job-dependency-insight`.
  - `controller-job-cleanup` (the real `job_has_dependents()`).
  - `job-status-detail-display` (text for the new reasons, showing the
    expression).
  - Future name-based references, array conditions, and job priority.

## Existing Spec Touchpoints
- **Extends**: `controller-job-cleanup` (replaces its stub dependency check).
- **Adjacent**:
  - `job-time-window` (the same gate, the modify path, the pend-reason
    infrastructure)
  - `job-status-detail-display`
  - `scheduler-alter-pend-reasons`
  - `scheduler-sim-discrete-clock`

## Constraints
- The primary does no per-session evaluation.
- Every wire change is an additive Cap'n Proto field. Old WAL records and
  snapshots decode as "no dependency".
- Save/restore field completeness:
  - the expression round trips through the WAL submit and update facts, the
    controller snapshot, the scheduler snapshot and the recordings;
  - the scheduler job's exit code round trips through the full push and the
    snapshot;
  - the reverse index is rebuilt identically after replay.

  Run `/xce-audit-invariants` (controller dimension 7, scheduler dimension 4).
- Scheduler evaluation uses only core state and clock (`--sim-verify`
  determinism).
- One grammar implementation (libxce), used from Go through cgo.
- Middleware chain unchanged. RBAC `job:view` governs which jobs a user may
  reference.
- HA/persistence touch: use the full 3-phase workflow.