docs(spec): job-dependency and job-dependency-insight discovery briefs

New specs from /kiro-discovery (2026-09-29), based on the LSF 10.1 Admin
Guide section "Job dependency and job priority". Dependency only; job priority
is deferred to a later discovery. Both are added to roadmap.md.

- job-dependency: -w expressions by job ID (done, ended, exit(id[,op code]),
  started, post_done, post_err, bare id; && || ! and parentheses). One libxce
  grammar, used from Go through cgo. The apiserver validates early (syntax,
  parent exists, job:view). The controller re-validates, stores the
  expression, and keeps an in-memory reverse index rebuilt on replay. The
  scheduler evaluates in sched_job_is_ready(); unsatisfiable dependencies stay
  PEND. Also closes the scheduler exit-code field-completeness gap. Adds
  xcecli -w / modify -w -wn and harness jobsub -w / jobmod.
- job-dependency-insight: bjdepinfo-style query, dependency info in job
  details, and optional orphan termination (-ti or a cluster grace period).
- controller-job-cleanup brief: job-dependency will replace the
  job_has_dependents() stub.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.kiro/specs/controller-job-cleanup/brief.md

@@ -179,7 +179,10 @@ Rejected alternatives:
   - The config pipeline (`clusterconfig-schema-split`, `config-upgrade-snapshot`,
     `admin-config-tool`).
 - **Downstream**:
-  - A future job-dependency feature, which replaces the stub.
+  - `job-dependency` (2026-09-29) replaces the stub `job_has_dependents()` with a
+    real check against the controller's in-memory reverse index. Keep the stub's
+    signature stable so that swap is drop-in. A job is never removed while any
+    **pending** job depends on it.
   - A future accounting or history feature, which archives before removal.
   - `job-status-detail-display` (a removed job simply stops appearing).
 

.kiro/specs/job-dependency-insight/brief.md

@@ -0,0 +1,113 @@
+# Brief: job-dependency-insight
+
+## Problem
+Once jobs can depend on each other (`job-dependency`), users and admins need to
+**see** and **manage** the resulting graphs:
+- Why is my job still pending, and which parent is holding it?
+- What will run after this job?
+- Is this chain stuck forever?
+
+Without insight, a user whose dependency can never be satisfied (an orphan) only
+sees "dependency can never be satisfied" and must work out the cause by hand.
+Stuck orphans also pile up pending forever unless someone kills them.
+
+Reference: *IBM Spectrum LSF 10.1 Administering LSF*:
+- `bjdepinfo` (printed pp. 463-465, 523);
+- `bjobs -l` / `bhist -l` dependency display (p. 93);
+- orphan job termination, `ORPHAN_JOB_TERM_GRACE_PERIOD` and `bsub/bmod -ti`
+  (pp. 91-94).
+
+## Current State
+- Nothing exists yet. `job-dependency` provides:
+  - the stored expression per job;
+  - the controller's in-memory reverse index (parent → dependents);
+  - scheduler-side evaluation with pend reasons "dependency not satisfied" and
+    "dependency can never be satisfied".
+- XCE's REST and xcecli have no dependency query command. Job detail output
+  (`job-status-detail-display`) renders pend reasons but not dependency
+  structure.
+
+## Desired Outcome
+- **A dependency query** (REST, plus an xcecli command such as
+  `xce jobs deps <jobid>` and a harness equivalent) that shows, like
+  `bjdepinfo`:
+  - the parents (default), the children (`-c`), or only the parents currently
+    blocking the job (`-p`);
+  - for each: job ID, status, name and level;
+  - `-r N` to follow indirect dependencies up to N levels;
+  - `-l` for the condition text, and whether each condition and the whole
+    expression is satisfied, not yet satisfied, or never satisfiable.
+
+  Results respect RBAC `job:view`: jobs the user can't view are hidden or
+  redacted.
+- **Job details** (`xce jobs list -l`, REST) show the dependency condition and
+  its current verdict.
+- **Optional orphan termination** (user decision: default PEND plus optional
+  termination):
+  - A per-job flag (`-ti` on submit and modify) terminates the job as soon as its
+    dependency becomes unsatisfiable.
+  - A cluster-wide grace period (configurable, 0 = off, the default) terminates
+    orphans after it expires. Elapsed time survives failover, and once the period
+    expires the whole orphan subtree may be terminated.
+  - Terminated orphans end as **EXIT with the reason ORPHAN** (persisted exit
+    reason; reuses `job-time-window`'s exit-reason field).
+
+## Approach
+- The query is served by the **controller** from the reverse index and stored
+  expressions (authoritative, no scheduler round trip).
+- Per-condition verdicts come from evaluating the expression with the shared
+  libxce evaluator against controller job states.
+- **Orphan termination.** The scheduler already detects "never satisfiable"
+  each session, so it is the natural detector. The design decides who carries
+  out the termination:
+  - (a) the scheduler emits a terminate decision that the controller commits
+    (keeps the primary's periodic work at zero); or
+  - (b) the controller acts when it records a pend-reason change to "never".
+
+  The user has preferred keeping the primary out of periodic work.
+- The grace-period start (when the orphan was first detected) must be persisted
+  so it survives failover, as LSF does.
+
+## Scope
+- **In**:
+  - The dependency query API and commands (xcecli, harness, REST).
+  - Dependency info in job details.
+  - Per-job `-ti` and the cluster grace period.
+  - Orphan termination with the ORPHAN exit reason.
+  - Tests.
+- **Out**:
+  - The dependency grammar, validation, storage and evaluation
+    (`job-dependency`).
+  - Name references, `num*` array conditions, `external()`.
+  - Job priority.
+  - Graphical DAG rendering.
+
+## Boundary Candidates
+- The controller-side query engine (parents, children, levels, verdicts).
+- Client surfaces (xcecli, harness, REST, job-detail rendering).
+- Orphan detection → termination pipeline plus grace-period persistence.
+
+## Out of Boundary
+- Changing how dependencies are evaluated for scheduling.
+- The pend-reason text catalogue (`job-status-detail-display`).
+
+## Upstream / Downstream
+- **Upstream**:
+  - `job-dependency` (HARD).
+  - `job-time-window` (the exit-reason field, and the modify path for `-ti`).
+  - `job-status-detail-display` (job detail rendering).
+- **Downstream**: future workflow tooling.
+
+## Existing Spec Touchpoints
+- **Extends**: `job-dependency`.
+- **Adjacent**:
+  - `job-status-detail-display`
+  - `controller-job-cleanup` (terminated orphans become eligible for cleanup)
+
+## Constraints
+- RBAC: never reveal jobs the requester may not view.
+- Orphan termination must be deterministic and failover-safe: a persisted
+  grace-period start, and no double termination after failover.
+- Keep the primary's periodic work minimal (user preference).
+- All wire changes additive. HA/persistence touch: use the full 3-phase
+  workflow.

.kiro/specs/job-dependency/brief.md

@@ -0,0 +1,221 @@
+# Brief: job-dependency
+
+## Problem
+XCE users run workflows: preprocess → solve → postprocess, or "run B only if A
+failed". Today they must babysit each step and submit the next job by hand when
+the previous one finishes, or write external polling scripts. XCE has **no job
+dependency concept at all**. `job_submit_data_t` has no dependency field, and
+nothing in the controller, scheduler, libxce or clients handles one.
+
+`controller-job-cleanup` also needs a real "does any job depend on this one?"
+check; it is a stub today.
+
+Reference: *IBM Spectrum LSF 10.1 Administering LSF*, section "Job dependency and
+job priority" (printed pp. 459-465, arrays p. 522, orphans pp. 91-94). The local
+extracted text is in the discovery scratchpad. User scope decision (2026-09-29):
+**job dependency only**. Job priority (user priority, priority escalation,
+absolute priority scheduling) is a separate, later discovery.
+
+## Current State
+Codebase exploration, 2026-09-29.
+
+- **No dependency fields** in `job_submit_data_t` (`include/xce/types.h:356-370`)
+  or capnp `JobSubmitData` (`schema/controller.capnp:579-591`, last ordinal @10).
+- **The scheduler gate is a stub.** The FCFS picker skips a job unless
+  `sched_job_is_ready()` holds (`scheduler/src/core.c:493`, returns 1 today).
+  `job-time-window` uses the same gate for begin and end times.
+  - Sessions run about every 1 s.
+  - Time and state must come from the scheduler core (`xce_core_now`) so that
+    `--sim-verify` stays deterministic.
+- **The scheduler sees parent state** through `XCE_CORE_EVENT_JOB_STATE_CHANGE`
+  `{jobid, old_state, new_state, exit_code}` (`scheduler/include/core.h:550-555`).
+  - **Gap:** `xce_sched_job_t` does not store the exit code.
+  - **Gap:** the full-push wire `SchedJob` (`controller.capnp:347-357`) carries
+    no exit info, and neither does the scheduler snapshot (`snapshot_wire.c`).
+  - So after a scheduler restart or failover, `exit(id, op code)` could not be
+    evaluated. This must be closed (save/restore field completeness).
+- **Job states** (`types.h:382-429`): DONE, EXIT, PDONE (post-exec done OK),
+  PERR (post-exec error), RUN, USUSP, SSUSP, and so on. LSF `post_done` /
+  `post_err` map onto PDONE / PERR directly.
+- **Pend reasons.** Only resource codes exist (`include/preason.h:7-9`). A new
+  non-resource code, and a publish path for skipped jobs, is also needed by
+  `job-time-window`.
+- **Controller job table.** `c->jobmgr` holds jobs. The scheduler's `core->jobs`
+  also keeps every job forever today, which `controller-job-cleanup` changes.
+- **Array jobs.** There is no parent object. Each element is an independent job
+  with jobid `base+i`, so an element can be referenced by its own ID.
+- **Submit paths:** REST (apiserver, fed every job by controller push,
+  cgo into libxce), `xcecli jobs submit`, the harness `xce_test jobsub`
+  (`harness/test.c:3575`), and libxce directly.
+- **The job-modify command** (`xce jobs modify`, harness `jobmod`) is being
+  introduced by `job-time-window`.
+
+## Desired Outcome
+- A user submits a job with a **dependency expression**, for example
+  `xcecli jobs submit -w 'done(1001) && (started(1002) || exit(1003, != 0))' ...`.
+  The job is not started until the expression is TRUE.
+- **v1 conditions** (user decision), all by **job ID**:
+  - `done(id)`
+  - `ended(id)` (DONE or EXIT)
+  - `exit(id [, op code])`, where `op` is one of `> >= < <= == !=`; the default is
+    `==`, and with no code any exit code matches
+  - `started(id)` (RUN, USUSP, SSUSP, DONE or EXIT)
+  - `post_done(id)` (PDONE) and `post_err(id)` (PERR)
+  - a bare `id`, which means `done(id)`
+  - operators `&&`, `||`, `!`, with parentheses
+- **Validation at submit**, rejected with a precise error, following LSF:
+  - bad syntax;
+  - a referenced job that does not exist (no forward references);
+  - a referenced job the submitter may not view (reuse RBAC `job:view`);
+  - self-reference or a cycle (impossible with existing parents at submit, but
+    it must be checked on modify).
+- **While waiting**, the job is PEND with the pend reason "dependency not
+  satisfied".
+- **When the condition can never be satisfied** (an orphan, for example
+  `done(P)` after P exited), the job stays PEND with the reason "dependency can
+  never be satisfied" until the user kills it or changes the dependency. This is
+  the default (user decision).
+- **Optional orphan termination** is owned by `job-dependency-insight`.
+- **Changing a pending job's dependency**: set or remove it (`-w`, and `-wn` to
+  remove) through the modify path introduced by `job-time-window`. It is
+  re-validated like a submit.
+- **Cleanup.** A job that any **pending** job depends on is never removed by
+  `controller-job-cleanup`. This spec implements that spec's
+  `job_has_dependents()` stub.
+- **Both client tools**, xcecli and the harness (`jobsub -w`, `jobmod -w/-wn`),
+  support the options, and job details show the dependency expression.
+
+## Approach
+**Chosen (user decision, 2026-09-29):** the apiserver validates early, the
+controller is authoritative for validation and owns the dependency graph, and
+the scheduler evaluates.
+
+1. **One shared grammar in libxce.** A parser, AST and printer for the
+   expression, used by the controller, the scheduler, the harness and (via cgo)
+   the apiserver. No second copy in Go.
+2. **apiserver, early validation** (like `application-profile`'s existence
+   check), before anything goes downstream:
+   - parse;
+   - check each referenced job exists in its job store;
+   - check the submitter holds `job:view` on it.
+
+   It rejects with a clear 4xx.
+3. **Controller, authoritative.**
+   - On submit and modify it re-validates: the apiserver's store can be stale, a
+     parent may have been cleaned up meanwhile, and the harness and libxce bypass
+     the apiserver.
+   - It stores the canonical expression in a new additive `JobSubmitData` field,
+     carried by the WAL submit, update and snapshot paths (field completeness).
+   - It keeps an **in-memory reverse index** (parent → dependents), rebuilt from
+     the stored expressions on replay and snapshot load. There is no new
+     persisted state.
+   - The index serves `job_has_dependents()` (for cleanup) and the queries in
+     `job-dependency-insight`.
+4. **Scheduler, evaluation.** Each session, `sched_job_is_ready()` evaluates the
+   AST against the scheduler's own job states (deterministic under the core
+   clock). It yields satisfied, not yet, or never, and publishes the
+   corresponding pend reason.
+   - The scheduler must keep each parent's final state **and exit code**, so add
+     the exit code to `xce_sched_job_t`, to `SchedJob`, and to the scheduler
+     snapshot.
+   - It must not drop finished parents while pending dependents exist (coordinate
+     with `controller-job-cleanup`).
+5. **Primary work.** The primary does no per-session evaluation. Its only
+   dependency work is validation and the index updates at submit, modify and
+   remove.
+
+Rejected alternatives:
+- **The controller evaluates on each state change** (LSF mbatchd style): more
+  primary work.
+- **Scheduler-only**: no validation at submit, and no authoritative reverse
+  index.
+- **The apiserver owns the graph**: libxce and the harness bypass it, its store
+  is non-authoritative and lost on restart, and cleanup runs on the controller.
+
+## Scope
+- **In**:
+  - The shared libxce expression grammar.
+  - The new submit field and its persistence.
+  - apiserver and controller validation.
+  - The controller reverse index and `job_has_dependents()`.
+  - Scheduler evaluation plus the two pend reasons.
+  - The scheduler exit-code field-completeness fix.
+  - Modifying pending dependencies (set and remove).
+  - xcecli `-w` / `modify -w -wn`, harness `jobsub -w` / `jobmod`.
+  - Showing the expression in job details.
+  - Simulator determinism.
+  - Tests.
+- **Open for requirements**:
+  - What happens when a dependent's parent is later removed by cleanup. It can't
+    happen while the dependent is pending, but it can after the dependent itself
+    has finished.
+  - Requeue and rerun semantics. LSF: requeuing a finished parent can orphan its
+    dependents.
+  - A limit on expression length or complexity.
+  - Whether whole-array references (all elements) are wanted in v1 without the
+    `num*` conditions.
+- **Out**:
+  - Referring to jobs by name, and wildcards (not selected for v1).
+  - The array count conditions `num*` (not selected for v1).
+  - `external()`.
+  - The LSF `JOB_DEP_LAST_SUB` and `EVALUATE_JOB_DEPENDENCY` tuning.
+  - Job priority features.
+  - A `bjdepinfo`-style query, dependency-graph display, and orphan termination
+    (`job-dependency-insight`).
+  - Cross-cluster dependencies.
+
+## Boundary Candidates
+- The libxce grammar (parse, validate, print, evaluate on an abstract
+  state-lookup interface).
+- apiserver early validation.
+- Controller authoritative validation, storage and the reverse index.
+- Scheduler evaluation, pend reasons, and keeping parent state and exit code.
+- Client surfaces: xcecli, harness, REST.
+
+## Out of Boundary
+- Showing dependency trees and levels (`job-dependency-insight`).
+- Automatic orphan termination (`job-dependency-insight`).
+- How pend reasons are rendered as text (`job-status-detail-display`, which needs
+  the new codes).
+
+## Upstream / Downstream
+- **Upstream**:
+  - The scheduler eligibility gate (shared with `job-time-window`; sequence or
+    coordinate the two).
+  - The job-modify path introduced by `job-time-window`.
+  - RBAC `job:view`.
+  - The WAL submit and update facts.
+- **Downstream**:
+  - `job-dependency-insight`.
+  - `controller-job-cleanup` (the real `job_has_dependents()`).
+  - `job-status-detail-display` (text for the new reasons, showing the
+    expression).
+  - Future name-based references, array conditions, and job priority.
+
+## Existing Spec Touchpoints
+- **Extends**: `controller-job-cleanup` (replaces its stub dependency check).
+- **Adjacent**:
+  - `job-time-window` (the same gate, the modify path, the pend-reason
+    infrastructure)
+  - `job-status-detail-display`
+  - `scheduler-alter-pend-reasons`
+  - `scheduler-sim-discrete-clock`
+
+## Constraints
+- The primary does no per-session evaluation.
+- Every wire change is an additive Cap'n Proto field. Old WAL records and
+  snapshots decode as "no dependency".
+- Save/restore field completeness:
+  - the expression round trips through the WAL submit and update facts, the
+    controller snapshot, the scheduler snapshot and the recordings;
+  - the scheduler job's exit code round trips through the full push and the
+    snapshot;
+  - the reverse index is rebuilt identically after replay.
+
+  Run `/xce-audit-invariants` (controller dimension 7, scheduler dimension 4).
+- Scheduler evaluation uses only core state and clock (`--sim-verify`
+  determinism).
+- One grammar implementation (libxce), used from Go through cgo.
+- Middleware chain unchanged. RBAC `job:view` governs which jobs a user may
+  reference.
+- HA/persistence touch: use the full 3-phase workflow.

.kiro/steering/roadmap.md

@@ -66,6 +66,8 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
 - [ ] job-sandbox-multinode — NEW 2026-09-29 (Job Sandbox Execution). Multi-node sandbox jobs: cluster-consistent ephemeral uid (expected: controller-allocated, WAL-replicated uid lease from a reserved range), worker-driven remote/MPI launch (no ssh; PMIx/prted via worker), shared-FS ownership normalization, one unit per job per node. Brief: `.kiro/specs/job-sandbox-multinode/brief.md`. Dependencies: job-sandbox-execution, application-profile.
 - [ ] rbac-persistence — NEW 2026-09-29 (`/kiro-discovery`, raised during application-profile discovery). Closes the deferred `security.md` gap: runtime RBAC mutations (roles, permissions, user-role assignments, Logon auto-assignment) are in-memory only and silently revert to `rbac.yaml` on apiserver restart — including revocations. Durable-or-rejected mutations, audit trail, defaults-vs-runtime reconciliation, multi-apiserver consistency; storage approach (rbac.yaml write-through / controller-replicated entity / embedded store / read-only) decided in design. Brief: `.kiro/specs/rbac-persistence/brief.md`. Dependencies: none (independent of the sandbox initiative; application-profile's API-made `app:use` grants rely on it for durability).
 - [ ] job-time-window — NEW 2026-09-29 (`/kiro-discovery`, Path C). Job begin time (scheduler gate in `sched_job_is_ready()` via `xce_core_now`, pend reason BEGIN_TIME_NOT_REACHED) and end time (WORKER enforces running jobs: helper SIGTERM→grace→SIGKILL, exit reason END_TIME_REACHED persisted, unit RuntimeMaxUSec backstop; PEND jobs whose end time can't be met stay PEND with END_TIME_UNMEETABLE until user kills/modifies — no controller sweep, primary not involved). Adds runtime estimate (`-We/--estimated_run_time`, scheduling hint only, never enforced; estimate > run limit rejected); unmeetable rule `now + (estimate ‖ run_limit ‖ 0) > end`. Times/estimate modifiable only while PEND (new `xce jobs modify` + harness `jobmod`). Both xcecli and harness `jobsub` gain all options (`-b`/`-t`/`-W`/estimate). Also fixes the silent drop of xcecli `-b`/`-t`/`-W` and REST `run_time_estimation` (apiserver `modelJobToSubmit` never sets them nor `exdata`). Brief: `.kiro/specs/job-time-window/brief.md`. Dependencies: none (new reason codes need text in job-status-detail-display).
+- [ ] job-dependency — NEW 2026-09-29 (`/kiro-discovery`, Path D; ref LSF 10.1 Admin Guide "Job dependency and job priority" pp. 459-465 — dependency only, priority deferred). `-w` expressions by job ID: done/ended/exit(id[,op code])/started/post_done/post_err (PDONE/PERR) + bare id, `&& || !` (). Shared libxce grammar (Go via cgo); apiserver early validation (syntax, parent exists, submitter has `job:view`); controller authoritative re-validation, stores expression (additive JobSubmitData field) + in-memory reverse index rebuilt on replay (implements controller-job-cleanup's `job_has_dependents()`); scheduler evaluates in `sched_job_is_ready()` (pend reasons DEP_NOT_SATISFIED / DEP_NEVER_SATISFIED; orphans stay PEND). Fixes scheduler exit-code field-completeness gap (xce_sched_job_t/SchedJob/snapshot lack exit code). xcecli `-w`, `modify -w/-wn`; harness `jobsub -w`/`jobmod`. Brief: `.kiro/specs/job-dependency/brief.md`. Dependencies: job-time-window (shared gate + modify path) — coordinate.
+- [ ] job-dependency-insight — NEW 2026-09-29. `bjdepinfo`-style query (parents/children/-p blocking/-r levels/-l verdicts, RBAC-filtered) via REST/xcecli/harness, dependency info in job details, optional orphan termination (per-job `-ti` + cluster grace period, default off; EXIT reason ORPHAN; failover-safe persisted grace start; minimal primary periodic work). Brief: `.kiro/specs/job-dependency-insight/brief.md`. Dependencies: job-dependency, job-time-window (exit-reason field).
 - [ ] controller-job-cleanup — NEW 2026-09-29 (`/kiro-discovery`, Path C). Primary-only periodic sweep (configurable `job_cleanup_period`, default 300 s) removes DONE/EXIT jobs finished longer than configurable `job_retention` (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever `jobmgr` delete; refcount-safe `xkvmgr` delete), propagated to scheduler and apiserver (`OnJobDeleted`). Retires the node-local snapshot-only `job_should_retain` finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: `.kiro/specs/controller-job-cleanup/brief.md`. Dependencies: none (extends ha-entity-snapshots).
 - [ ] job-status-detail-display — NEW 2026-09-28 (`/kiro-discovery`, Path E). End users (REST/`xcecli`) see NO pend reason for ANY job today: the apiserver cgo conversion (`backend/convert.go`) never maps `xce_job_t.pend_reasons`, so `pending_reason` is always empty. Maps it to a structured JSON (including `alternatives` from day one) using libxce's code→text mapping, and renders it in `xcecli` job detail. Open for requirements: also map `run_usage` → `JobRusage` (same gap; reserved vs. used for RUN jobs). Brief: `.kiro/specs/job-status-detail-display/brief.md`. Dependencies: none for flat reasons; scheduler-alter-pend-reasons for per-alternative rendering.