Rongsong Shen
3444cf1657
New specs from /kiro-discovery (2026-09-29), based on the LSF 10.1 Admin Guide section "Job dependency and job priority". Dependency only; job priority is deferred to a later discovery. Both are added to roadmap.md. - job-dependency: -w expressions by job ID (done, ended, exit(id[,op code]), started, post_done, post_err, bare id; && || ! and parentheses). One libxce grammar, used from Go through cgo. The apiserver validates early (syntax, parent exists, job:view). The controller re-validates, stores the expression, and keeps an in-memory reverse index rebuilt on replay. The scheduler evaluates in sched_job_is_ready(); unsatisfiable dependencies stay PEND. Also closes the scheduler exit-code field-completeness gap. Adds xcecli -w / modify -w -wn and harness jobsub -w / jobmod. - job-dependency-insight: bjdepinfo-style query, dependency info in job details, and optional orphan termination (-ti or a cluster grace period). - controller-job-cleanup brief: job-dependency will replace the job_has_dependents() stub. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
113 lines
4.8 KiB
Markdown
113 lines
4.8 KiB
Markdown
# Brief: job-dependency-insight
|
|
|
|
## Problem
|
|
Once jobs can depend on each other (`job-dependency`), users and admins need to
|
|
**see** and **manage** the resulting graphs:
|
|
- Why is my job still pending, and which parent is holding it?
|
|
- What will run after this job?
|
|
- Is this chain stuck forever?
|
|
|
|
Without insight, a user whose dependency can never be satisfied (an orphan) only
|
|
sees "dependency can never be satisfied" and must work out the cause by hand.
|
|
Stuck orphans also pile up pending forever unless someone kills them.
|
|
|
|
Reference: *IBM Spectrum LSF 10.1 Administering LSF*:
|
|
- `bjdepinfo` (printed pp. 463-465, 523);
|
|
- `bjobs -l` / `bhist -l` dependency display (p. 93);
|
|
- orphan job termination, `ORPHAN_JOB_TERM_GRACE_PERIOD` and `bsub/bmod -ti`
|
|
(pp. 91-94).
|
|
|
|
## Current State
|
|
- Nothing exists yet. `job-dependency` provides:
|
|
- the stored expression per job;
|
|
- the controller's in-memory reverse index (parent → dependents);
|
|
- scheduler-side evaluation with pend reasons "dependency not satisfied" and
|
|
"dependency can never be satisfied".
|
|
- XCE's REST and xcecli have no dependency query command. Job detail output
|
|
(`job-status-detail-display`) renders pend reasons but not dependency
|
|
structure.
|
|
|
|
## Desired Outcome
|
|
- **A dependency query** (REST, plus an xcecli command such as
|
|
`xce jobs deps <jobid>` and a harness equivalent) that shows, like
|
|
`bjdepinfo`:
|
|
- the parents (default), the children (`-c`), or only the parents currently
|
|
blocking the job (`-p`);
|
|
- for each: job ID, status, name and level;
|
|
- `-r N` to follow indirect dependencies up to N levels;
|
|
- `-l` for the condition text, and whether each condition and the whole
|
|
expression is satisfied, not yet satisfied, or never satisfiable.
|
|
|
|
Results respect RBAC `job:view`: jobs the user can't view are hidden or
|
|
redacted.
|
|
- **Job details** (`xce jobs list -l`, REST) show the dependency condition and
|
|
its current verdict.
|
|
- **Optional orphan termination** (user decision: default PEND plus optional
|
|
termination):
|
|
- A per-job flag (`-ti` on submit and modify) terminates the job as soon as its
|
|
dependency becomes unsatisfiable.
|
|
- A cluster-wide grace period (configurable, 0 = off, the default) terminates
|
|
orphans after it expires. Elapsed time survives failover, and once the period
|
|
expires the whole orphan subtree may be terminated.
|
|
- Terminated orphans end as **EXIT with the reason ORPHAN** (persisted exit
|
|
reason; reuses `job-time-window`'s exit-reason field).
|
|
|
|
## Approach
|
|
- The query is served by the **controller** from the reverse index and stored
|
|
expressions (authoritative, no scheduler round trip).
|
|
- Per-condition verdicts come from evaluating the expression with the shared
|
|
libxce evaluator against controller job states.
|
|
- **Orphan termination.** The scheduler already detects "never satisfiable"
|
|
each session, so it is the natural detector. The design decides who carries
|
|
out the termination:
|
|
- (a) the scheduler emits a terminate decision that the controller commits
|
|
(keeps the primary's periodic work at zero); or
|
|
- (b) the controller acts when it records a pend-reason change to "never".
|
|
|
|
The user has preferred keeping the primary out of periodic work.
|
|
- The grace-period start (when the orphan was first detected) must be persisted
|
|
so it survives failover, as LSF does.
|
|
|
|
## Scope
|
|
- **In**:
|
|
- The dependency query API and commands (xcecli, harness, REST).
|
|
- Dependency info in job details.
|
|
- Per-job `-ti` and the cluster grace period.
|
|
- Orphan termination with the ORPHAN exit reason.
|
|
- Tests.
|
|
- **Out**:
|
|
- The dependency grammar, validation, storage and evaluation
|
|
(`job-dependency`).
|
|
- Name references, `num*` array conditions, `external()`.
|
|
- Job priority.
|
|
- Graphical DAG rendering.
|
|
|
|
## Boundary Candidates
|
|
- The controller-side query engine (parents, children, levels, verdicts).
|
|
- Client surfaces (xcecli, harness, REST, job-detail rendering).
|
|
- Orphan detection → termination pipeline plus grace-period persistence.
|
|
|
|
## Out of Boundary
|
|
- Changing how dependencies are evaluated for scheduling.
|
|
- The pend-reason text catalogue (`job-status-detail-display`).
|
|
|
|
## Upstream / Downstream
|
|
- **Upstream**:
|
|
- `job-dependency` (HARD).
|
|
- `job-time-window` (the exit-reason field, and the modify path for `-ti`).
|
|
- `job-status-detail-display` (job detail rendering).
|
|
- **Downstream**: future workflow tooling.
|
|
|
|
## Existing Spec Touchpoints
|
|
- **Extends**: `job-dependency`.
|
|
- **Adjacent**:
|
|
- `job-status-detail-display`
|
|
- `controller-job-cleanup` (terminated orphans become eligible for cleanup)
|
|
|
|
## Constraints
|
|
- RBAC: never reveal jobs the requester may not view.
|
|
- Orphan termination must be deterministic and failover-safe: a persisted
|
|
grace-period start, and no double termination after failover.
|
|
- Keep the primary's periodic work minimal (user preference).
|
|
- All wire changes additive. HA/persistence touch: use the full 3-phase
|
|
workflow.
|