docs(spec): job-dependency and job-dependency-insight discovery briefs

New specs from /kiro-discovery (2026-09-29), based on the LSF 10.1 Admin
Guide section "Job dependency and job priority". Dependency only; job priority
is deferred to a later discovery. Both are added to roadmap.md.

- job-dependency: -w expressions by job ID (done, ended, exit(id[,op code]),
  started, post_done, post_err, bare id; && || ! and parentheses). One libxce
  grammar, used from Go through cgo. The apiserver validates early (syntax,
  parent exists, job:view). The controller re-validates, stores the
  expression, and keeps an in-memory reverse index rebuilt on replay. The
  scheduler evaluates in sched_job_is_ready(); unsatisfiable dependencies stay
  PEND. Also closes the scheduler exit-code field-completeness gap. Adds
  xcecli -w / modify -w -wn and harness jobsub -w / jobmod.
- job-dependency-insight: bjdepinfo-style query, dependency info in job
  details, and optional orphan termination (-ti or a cluster grace period).
- controller-job-cleanup brief: job-dependency will replace the
  job_has_dependents() stub.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-09-29 13:08:26 +00:00

4.8 KiB

Raw Blame History

Brief: job-dependency-insight

Problem

Once jobs can depend on each other (job-dependency), users and admins need to see and manage the resulting graphs:

Why is my job still pending, and which parent is holding it?
What will run after this job?
Is this chain stuck forever?

Without insight, a user whose dependency can never be satisfied (an orphan) only sees "dependency can never be satisfied" and must work out the cause by hand. Stuck orphans also pile up pending forever unless someone kills them.

Reference: IBM Spectrum LSF 10.1 Administering LSF:

bjdepinfo (printed pp. 463-465, 523);
bjobs -l / bhist -l dependency display (p. 93);
orphan job termination, ORPHAN_JOB_TERM_GRACE_PERIOD and bsub/bmod -ti (pp. 91-94).

Current State

Nothing exists yet. job-dependency provides:
- the stored expression per job;
- the controller's in-memory reverse index (parent → dependents);
- scheduler-side evaluation with pend reasons "dependency not satisfied" and "dependency can never be satisfied".
XCE's REST and xcecli have no dependency query command. Job detail output (job-status-detail-display) renders pend reasons but not dependency structure.

Desired Outcome

A dependency query (REST, plus an xcecli command such as xce jobs deps <jobid> and a harness equivalent) that shows, like bjdepinfo:
- the parents (default), the children (-c), or only the parents currently blocking the job (-p);
- for each: job ID, status, name and level;
- -r N to follow indirect dependencies up to N levels;
- -l for the condition text, and whether each condition and the whole expression is satisfied, not yet satisfied, or never satisfiable.
Results respect RBAC job:view: jobs the user can't view are hidden or redacted.
Job details (xce jobs list -l, REST) show the dependency condition and its current verdict.
Optional orphan termination (user decision: default PEND plus optional termination):
- A per-job flag (-ti on submit and modify) terminates the job as soon as its dependency becomes unsatisfiable.
- A cluster-wide grace period (configurable, 0 = off, the default) terminates orphans after it expires. Elapsed time survives failover, and once the period expires the whole orphan subtree may be terminated.
- Terminated orphans end as EXIT with the reason ORPHAN (persisted exit reason; reuses job-time-window's exit-reason field).

Approach

The query is served by the controller from the reverse index and stored expressions (authoritative, no scheduler round trip).
Per-condition verdicts come from evaluating the expression with the shared libxce evaluator against controller job states.
Orphan termination. The scheduler already detects "never satisfiable" each session, so it is the natural detector. The design decides who carries out the termination:
- (a) the scheduler emits a terminate decision that the controller commits (keeps the primary's periodic work at zero); or
- (b) the controller acts when it records a pend-reason change to "never".
The user has preferred keeping the primary out of periodic work.
The grace-period start (when the orphan was first detected) must be persisted so it survives failover, as LSF does.

Scope

In:
- The dependency query API and commands (xcecli, harness, REST).
- Dependency info in job details.
- Per-job -ti and the cluster grace period.
- Orphan termination with the ORPHAN exit reason.
- Tests.
Out:
- The dependency grammar, validation, storage and evaluation (job-dependency).
- Name references, num* array conditions, external().
- Job priority.
- Graphical DAG rendering.

Boundary Candidates

The controller-side query engine (parents, children, levels, verdicts).
Client surfaces (xcecli, harness, REST, job-detail rendering).
Orphan detection → termination pipeline plus grace-period persistence.

Out of Boundary

Changing how dependencies are evaluated for scheduling.
The pend-reason text catalogue (job-status-detail-display).

Upstream / Downstream

Upstream:
- job-dependency (HARD).
- job-time-window (the exit-reason field, and the modify path for -ti).
- job-status-detail-display (job detail rendering).
Downstream: future workflow tooling.

Existing Spec Touchpoints

Extends: job-dependency.
Adjacent:
- job-status-detail-display
- controller-job-cleanup (terminated orphans become eligible for cleanup)

Constraints

RBAC: never reveal jobs the requester may not view.
Orphan termination must be deterministic and failover-safe: a persisted grace-period start, and no double termination after failover.
Keep the primary's periodic work minimal (user preference).
All wire changes additive. HA/persistence touch: use the full 3-phase workflow.

4.8 KiB Raw Blame History