From 3444cf1657178b4c33ceef5a2e64f5206a88d66a Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 29 Sep 2026 13:08:26 +0000 Subject: [PATCH] docs(spec): job-dependency and job-dependency-insight discovery briefs New specs from /kiro-discovery (2026-09-29), based on the LSF 10.1 Admin Guide section "Job dependency and job priority". Dependency only; job priority is deferred to a later discovery. Both are added to roadmap.md. - job-dependency: -w expressions by job ID (done, ended, exit(id[,op code]), started, post_done, post_err, bare id; && || ! and parentheses). One libxce grammar, used from Go through cgo. The apiserver validates early (syntax, parent exists, job:view). The controller re-validates, stores the expression, and keeps an in-memory reverse index rebuilt on replay. The scheduler evaluates in sched_job_is_ready(); unsatisfiable dependencies stay PEND. Also closes the scheduler exit-code field-completeness gap. Adds xcecli -w / modify -w -wn and harness jobsub -w / jobmod. - job-dependency-insight: bjdepinfo-style query, dependency info in job details, and optional orphan termination (-ti or a cluster grace period). - controller-job-cleanup brief: job-dependency will replace the job_has_dependents() stub. Co-Authored-By: Claude Opus 5.5 (1M context) --- .kiro/specs/controller-job-cleanup/brief.md | 5 +- .kiro/specs/job-dependency-insight/brief.md | 113 ++++++++++ .kiro/specs/job-dependency/brief.md | 221 ++++++++++++++++++++ .kiro/steering/roadmap.md | 2 + 4 files changed, 340 insertions(+), 1 deletion(-) create mode 100644 .kiro/specs/job-dependency-insight/brief.md create mode 100644 .kiro/specs/job-dependency/brief.md diff --git a/.kiro/specs/controller-job-cleanup/brief.md b/.kiro/specs/controller-job-cleanup/brief.md index adc6d76d..b6f5697e 100644 --- a/.kiro/specs/controller-job-cleanup/brief.md +++ b/.kiro/specs/controller-job-cleanup/brief.md @@ -179,7 +179,10 @@ Rejected alternatives: - The config pipeline (`clusterconfig-schema-split`, `config-upgrade-snapshot`, `admin-config-tool`). - **Downstream**: - - A future job-dependency feature, which replaces the stub. + - `job-dependency` (2026-09-29) replaces the stub `job_has_dependents()` with a + real check against the controller's in-memory reverse index. Keep the stub's + signature stable so that swap is drop-in. A job is never removed while any + **pending** job depends on it. - A future accounting or history feature, which archives before removal. - `job-status-detail-display` (a removed job simply stops appearing). diff --git a/.kiro/specs/job-dependency-insight/brief.md b/.kiro/specs/job-dependency-insight/brief.md new file mode 100644 index 00000000..004d4eeb --- /dev/null +++ b/.kiro/specs/job-dependency-insight/brief.md @@ -0,0 +1,113 @@ +# Brief: job-dependency-insight + +## Problem +Once jobs can depend on each other (`job-dependency`), users and admins need to +**see** and **manage** the resulting graphs: +- Why is my job still pending, and which parent is holding it? +- What will run after this job? +- Is this chain stuck forever? + +Without insight, a user whose dependency can never be satisfied (an orphan) only +sees "dependency can never be satisfied" and must work out the cause by hand. +Stuck orphans also pile up pending forever unless someone kills them. + +Reference: *IBM Spectrum LSF 10.1 Administering LSF*: +- `bjdepinfo` (printed pp. 463-465, 523); +- `bjobs -l` / `bhist -l` dependency display (p. 93); +- orphan job termination, `ORPHAN_JOB_TERM_GRACE_PERIOD` and `bsub/bmod -ti` + (pp. 91-94). + +## Current State +- Nothing exists yet. `job-dependency` provides: + - the stored expression per job; + - the controller's in-memory reverse index (parent → dependents); + - scheduler-side evaluation with pend reasons "dependency not satisfied" and + "dependency can never be satisfied". +- XCE's REST and xcecli have no dependency query command. Job detail output + (`job-status-detail-display`) renders pend reasons but not dependency + structure. + +## Desired Outcome +- **A dependency query** (REST, plus an xcecli command such as + `xce jobs deps ` and a harness equivalent) that shows, like + `bjdepinfo`: + - the parents (default), the children (`-c`), or only the parents currently + blocking the job (`-p`); + - for each: job ID, status, name and level; + - `-r N` to follow indirect dependencies up to N levels; + - `-l` for the condition text, and whether each condition and the whole + expression is satisfied, not yet satisfied, or never satisfiable. + + Results respect RBAC `job:view`: jobs the user can't view are hidden or + redacted. +- **Job details** (`xce jobs list -l`, REST) show the dependency condition and + its current verdict. +- **Optional orphan termination** (user decision: default PEND plus optional + termination): + - A per-job flag (`-ti` on submit and modify) terminates the job as soon as its + dependency becomes unsatisfiable. + - A cluster-wide grace period (configurable, 0 = off, the default) terminates + orphans after it expires. Elapsed time survives failover, and once the period + expires the whole orphan subtree may be terminated. + - Terminated orphans end as **EXIT with the reason ORPHAN** (persisted exit + reason; reuses `job-time-window`'s exit-reason field). + +## Approach +- The query is served by the **controller** from the reverse index and stored + expressions (authoritative, no scheduler round trip). +- Per-condition verdicts come from evaluating the expression with the shared + libxce evaluator against controller job states. +- **Orphan termination.** The scheduler already detects "never satisfiable" + each session, so it is the natural detector. The design decides who carries + out the termination: + - (a) the scheduler emits a terminate decision that the controller commits + (keeps the primary's periodic work at zero); or + - (b) the controller acts when it records a pend-reason change to "never". + + The user has preferred keeping the primary out of periodic work. +- The grace-period start (when the orphan was first detected) must be persisted + so it survives failover, as LSF does. + +## Scope +- **In**: + - The dependency query API and commands (xcecli, harness, REST). + - Dependency info in job details. + - Per-job `-ti` and the cluster grace period. + - Orphan termination with the ORPHAN exit reason. + - Tests. +- **Out**: + - The dependency grammar, validation, storage and evaluation + (`job-dependency`). + - Name references, `num*` array conditions, `external()`. + - Job priority. + - Graphical DAG rendering. + +## Boundary Candidates +- The controller-side query engine (parents, children, levels, verdicts). +- Client surfaces (xcecli, harness, REST, job-detail rendering). +- Orphan detection → termination pipeline plus grace-period persistence. + +## Out of Boundary +- Changing how dependencies are evaluated for scheduling. +- The pend-reason text catalogue (`job-status-detail-display`). + +## Upstream / Downstream +- **Upstream**: + - `job-dependency` (HARD). + - `job-time-window` (the exit-reason field, and the modify path for `-ti`). + - `job-status-detail-display` (job detail rendering). +- **Downstream**: future workflow tooling. + +## Existing Spec Touchpoints +- **Extends**: `job-dependency`. +- **Adjacent**: + - `job-status-detail-display` + - `controller-job-cleanup` (terminated orphans become eligible for cleanup) + +## Constraints +- RBAC: never reveal jobs the requester may not view. +- Orphan termination must be deterministic and failover-safe: a persisted + grace-period start, and no double termination after failover. +- Keep the primary's periodic work minimal (user preference). +- All wire changes additive. HA/persistence touch: use the full 3-phase + workflow. diff --git a/.kiro/specs/job-dependency/brief.md b/.kiro/specs/job-dependency/brief.md new file mode 100644 index 00000000..657740c5 --- /dev/null +++ b/.kiro/specs/job-dependency/brief.md @@ -0,0 +1,221 @@ +# Brief: job-dependency + +## Problem +XCE users run workflows: preprocess → solve → postprocess, or "run B only if A +failed". Today they must babysit each step and submit the next job by hand when +the previous one finishes, or write external polling scripts. XCE has **no job +dependency concept at all**. `job_submit_data_t` has no dependency field, and +nothing in the controller, scheduler, libxce or clients handles one. + +`controller-job-cleanup` also needs a real "does any job depend on this one?" +check; it is a stub today. + +Reference: *IBM Spectrum LSF 10.1 Administering LSF*, section "Job dependency and +job priority" (printed pp. 459-465, arrays p. 522, orphans pp. 91-94). The local +extracted text is in the discovery scratchpad. User scope decision (2026-09-29): +**job dependency only**. Job priority (user priority, priority escalation, +absolute priority scheduling) is a separate, later discovery. + +## Current State +Codebase exploration, 2026-09-29. + +- **No dependency fields** in `job_submit_data_t` (`include/xce/types.h:356-370`) + or capnp `JobSubmitData` (`schema/controller.capnp:579-591`, last ordinal @10). +- **The scheduler gate is a stub.** The FCFS picker skips a job unless + `sched_job_is_ready()` holds (`scheduler/src/core.c:493`, returns 1 today). + `job-time-window` uses the same gate for begin and end times. + - Sessions run about every 1 s. + - Time and state must come from the scheduler core (`xce_core_now`) so that + `--sim-verify` stays deterministic. +- **The scheduler sees parent state** through `XCE_CORE_EVENT_JOB_STATE_CHANGE` + `{jobid, old_state, new_state, exit_code}` (`scheduler/include/core.h:550-555`). + - **Gap:** `xce_sched_job_t` does not store the exit code. + - **Gap:** the full-push wire `SchedJob` (`controller.capnp:347-357`) carries + no exit info, and neither does the scheduler snapshot (`snapshot_wire.c`). + - So after a scheduler restart or failover, `exit(id, op code)` could not be + evaluated. This must be closed (save/restore field completeness). +- **Job states** (`types.h:382-429`): DONE, EXIT, PDONE (post-exec done OK), + PERR (post-exec error), RUN, USUSP, SSUSP, and so on. LSF `post_done` / + `post_err` map onto PDONE / PERR directly. +- **Pend reasons.** Only resource codes exist (`include/preason.h:7-9`). A new + non-resource code, and a publish path for skipped jobs, is also needed by + `job-time-window`. +- **Controller job table.** `c->jobmgr` holds jobs. The scheduler's `core->jobs` + also keeps every job forever today, which `controller-job-cleanup` changes. +- **Array jobs.** There is no parent object. Each element is an independent job + with jobid `base+i`, so an element can be referenced by its own ID. +- **Submit paths:** REST (apiserver, fed every job by controller push, + cgo into libxce), `xcecli jobs submit`, the harness `xce_test jobsub` + (`harness/test.c:3575`), and libxce directly. +- **The job-modify command** (`xce jobs modify`, harness `jobmod`) is being + introduced by `job-time-window`. + +## Desired Outcome +- A user submits a job with a **dependency expression**, for example + `xcecli jobs submit -w 'done(1001) && (started(1002) || exit(1003, != 0))' ...`. + The job is not started until the expression is TRUE. +- **v1 conditions** (user decision), all by **job ID**: + - `done(id)` + - `ended(id)` (DONE or EXIT) + - `exit(id [, op code])`, where `op` is one of `> >= < <= == !=`; the default is + `==`, and with no code any exit code matches + - `started(id)` (RUN, USUSP, SSUSP, DONE or EXIT) + - `post_done(id)` (PDONE) and `post_err(id)` (PERR) + - a bare `id`, which means `done(id)` + - operators `&&`, `||`, `!`, with parentheses +- **Validation at submit**, rejected with a precise error, following LSF: + - bad syntax; + - a referenced job that does not exist (no forward references); + - a referenced job the submitter may not view (reuse RBAC `job:view`); + - self-reference or a cycle (impossible with existing parents at submit, but + it must be checked on modify). +- **While waiting**, the job is PEND with the pend reason "dependency not + satisfied". +- **When the condition can never be satisfied** (an orphan, for example + `done(P)` after P exited), the job stays PEND with the reason "dependency can + never be satisfied" until the user kills it or changes the dependency. This is + the default (user decision). +- **Optional orphan termination** is owned by `job-dependency-insight`. +- **Changing a pending job's dependency**: set or remove it (`-w`, and `-wn` to + remove) through the modify path introduced by `job-time-window`. It is + re-validated like a submit. +- **Cleanup.** A job that any **pending** job depends on is never removed by + `controller-job-cleanup`. This spec implements that spec's + `job_has_dependents()` stub. +- **Both client tools**, xcecli and the harness (`jobsub -w`, `jobmod -w/-wn`), + support the options, and job details show the dependency expression. + +## Approach +**Chosen (user decision, 2026-09-29):** the apiserver validates early, the +controller is authoritative for validation and owns the dependency graph, and +the scheduler evaluates. + +1. **One shared grammar in libxce.** A parser, AST and printer for the + expression, used by the controller, the scheduler, the harness and (via cgo) + the apiserver. No second copy in Go. +2. **apiserver, early validation** (like `application-profile`'s existence + check), before anything goes downstream: + - parse; + - check each referenced job exists in its job store; + - check the submitter holds `job:view` on it. + + It rejects with a clear 4xx. +3. **Controller, authoritative.** + - On submit and modify it re-validates: the apiserver's store can be stale, a + parent may have been cleaned up meanwhile, and the harness and libxce bypass + the apiserver. + - It stores the canonical expression in a new additive `JobSubmitData` field, + carried by the WAL submit, update and snapshot paths (field completeness). + - It keeps an **in-memory reverse index** (parent → dependents), rebuilt from + the stored expressions on replay and snapshot load. There is no new + persisted state. + - The index serves `job_has_dependents()` (for cleanup) and the queries in + `job-dependency-insight`. +4. **Scheduler, evaluation.** Each session, `sched_job_is_ready()` evaluates the + AST against the scheduler's own job states (deterministic under the core + clock). It yields satisfied, not yet, or never, and publishes the + corresponding pend reason. + - The scheduler must keep each parent's final state **and exit code**, so add + the exit code to `xce_sched_job_t`, to `SchedJob`, and to the scheduler + snapshot. + - It must not drop finished parents while pending dependents exist (coordinate + with `controller-job-cleanup`). +5. **Primary work.** The primary does no per-session evaluation. Its only + dependency work is validation and the index updates at submit, modify and + remove. + +Rejected alternatives: +- **The controller evaluates on each state change** (LSF mbatchd style): more + primary work. +- **Scheduler-only**: no validation at submit, and no authoritative reverse + index. +- **The apiserver owns the graph**: libxce and the harness bypass it, its store + is non-authoritative and lost on restart, and cleanup runs on the controller. + +## Scope +- **In**: + - The shared libxce expression grammar. + - The new submit field and its persistence. + - apiserver and controller validation. + - The controller reverse index and `job_has_dependents()`. + - Scheduler evaluation plus the two pend reasons. + - The scheduler exit-code field-completeness fix. + - Modifying pending dependencies (set and remove). + - xcecli `-w` / `modify -w -wn`, harness `jobsub -w` / `jobmod`. + - Showing the expression in job details. + - Simulator determinism. + - Tests. +- **Open for requirements**: + - What happens when a dependent's parent is later removed by cleanup. It can't + happen while the dependent is pending, but it can after the dependent itself + has finished. + - Requeue and rerun semantics. LSF: requeuing a finished parent can orphan its + dependents. + - A limit on expression length or complexity. + - Whether whole-array references (all elements) are wanted in v1 without the + `num*` conditions. +- **Out**: + - Referring to jobs by name, and wildcards (not selected for v1). + - The array count conditions `num*` (not selected for v1). + - `external()`. + - The LSF `JOB_DEP_LAST_SUB` and `EVALUATE_JOB_DEPENDENCY` tuning. + - Job priority features. + - A `bjdepinfo`-style query, dependency-graph display, and orphan termination + (`job-dependency-insight`). + - Cross-cluster dependencies. + +## Boundary Candidates +- The libxce grammar (parse, validate, print, evaluate on an abstract + state-lookup interface). +- apiserver early validation. +- Controller authoritative validation, storage and the reverse index. +- Scheduler evaluation, pend reasons, and keeping parent state and exit code. +- Client surfaces: xcecli, harness, REST. + +## Out of Boundary +- Showing dependency trees and levels (`job-dependency-insight`). +- Automatic orphan termination (`job-dependency-insight`). +- How pend reasons are rendered as text (`job-status-detail-display`, which needs + the new codes). + +## Upstream / Downstream +- **Upstream**: + - The scheduler eligibility gate (shared with `job-time-window`; sequence or + coordinate the two). + - The job-modify path introduced by `job-time-window`. + - RBAC `job:view`. + - The WAL submit and update facts. +- **Downstream**: + - `job-dependency-insight`. + - `controller-job-cleanup` (the real `job_has_dependents()`). + - `job-status-detail-display` (text for the new reasons, showing the + expression). + - Future name-based references, array conditions, and job priority. + +## Existing Spec Touchpoints +- **Extends**: `controller-job-cleanup` (replaces its stub dependency check). +- **Adjacent**: + - `job-time-window` (the same gate, the modify path, the pend-reason + infrastructure) + - `job-status-detail-display` + - `scheduler-alter-pend-reasons` + - `scheduler-sim-discrete-clock` + +## Constraints +- The primary does no per-session evaluation. +- Every wire change is an additive Cap'n Proto field. Old WAL records and + snapshots decode as "no dependency". +- Save/restore field completeness: + - the expression round trips through the WAL submit and update facts, the + controller snapshot, the scheduler snapshot and the recordings; + - the scheduler job's exit code round trips through the full push and the + snapshot; + - the reverse index is rebuilt identically after replay. + + Run `/xce-audit-invariants` (controller dimension 7, scheduler dimension 4). +- Scheduler evaluation uses only core state and clock (`--sim-verify` + determinism). +- One grammar implementation (libxce), used from Go through cgo. +- Middleware chain unchanged. RBAC `job:view` governs which jobs a user may + reference. +- HA/persistence touch: use the full 3-phase workflow. diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index 0b67e12a..42f4bc93 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -66,6 +66,8 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han - [ ] job-sandbox-multinode — NEW 2026-09-29 (Job Sandbox Execution). Multi-node sandbox jobs: cluster-consistent ephemeral uid (expected: controller-allocated, WAL-replicated uid lease from a reserved range), worker-driven remote/MPI launch (no ssh; PMIx/prted via worker), shared-FS ownership normalization, one unit per job per node. Brief: `.kiro/specs/job-sandbox-multinode/brief.md`. Dependencies: job-sandbox-execution, application-profile. - [ ] rbac-persistence — NEW 2026-09-29 (`/kiro-discovery`, raised during application-profile discovery). Closes the deferred `security.md` gap: runtime RBAC mutations (roles, permissions, user-role assignments, Logon auto-assignment) are in-memory only and silently revert to `rbac.yaml` on apiserver restart — including revocations. Durable-or-rejected mutations, audit trail, defaults-vs-runtime reconciliation, multi-apiserver consistency; storage approach (rbac.yaml write-through / controller-replicated entity / embedded store / read-only) decided in design. Brief: `.kiro/specs/rbac-persistence/brief.md`. Dependencies: none (independent of the sandbox initiative; application-profile's API-made `app:use` grants rely on it for durability). - [ ] job-time-window — NEW 2026-09-29 (`/kiro-discovery`, Path C). Job begin time (scheduler gate in `sched_job_is_ready()` via `xce_core_now`, pend reason BEGIN_TIME_NOT_REACHED) and end time (WORKER enforces running jobs: helper SIGTERM→grace→SIGKILL, exit reason END_TIME_REACHED persisted, unit RuntimeMaxUSec backstop; PEND jobs whose end time can't be met stay PEND with END_TIME_UNMEETABLE until user kills/modifies — no controller sweep, primary not involved). Adds runtime estimate (`-We/--estimated_run_time`, scheduling hint only, never enforced; estimate > run limit rejected); unmeetable rule `now + (estimate ‖ run_limit ‖ 0) > end`. Times/estimate modifiable only while PEND (new `xce jobs modify` + harness `jobmod`). Both xcecli and harness `jobsub` gain all options (`-b`/`-t`/`-W`/estimate). Also fixes the silent drop of xcecli `-b`/`-t`/`-W` and REST `run_time_estimation` (apiserver `modelJobToSubmit` never sets them nor `exdata`). Brief: `.kiro/specs/job-time-window/brief.md`. Dependencies: none (new reason codes need text in job-status-detail-display). +- [ ] job-dependency — NEW 2026-09-29 (`/kiro-discovery`, Path D; ref LSF 10.1 Admin Guide "Job dependency and job priority" pp. 459-465 — dependency only, priority deferred). `-w` expressions by job ID: done/ended/exit(id[,op code])/started/post_done/post_err (PDONE/PERR) + bare id, `&& || !` (). Shared libxce grammar (Go via cgo); apiserver early validation (syntax, parent exists, submitter has `job:view`); controller authoritative re-validation, stores expression (additive JobSubmitData field) + in-memory reverse index rebuilt on replay (implements controller-job-cleanup's `job_has_dependents()`); scheduler evaluates in `sched_job_is_ready()` (pend reasons DEP_NOT_SATISFIED / DEP_NEVER_SATISFIED; orphans stay PEND). Fixes scheduler exit-code field-completeness gap (xce_sched_job_t/SchedJob/snapshot lack exit code). xcecli `-w`, `modify -w/-wn`; harness `jobsub -w`/`jobmod`. Brief: `.kiro/specs/job-dependency/brief.md`. Dependencies: job-time-window (shared gate + modify path) — coordinate. +- [ ] job-dependency-insight — NEW 2026-09-29. `bjdepinfo`-style query (parents/children/-p blocking/-r levels/-l verdicts, RBAC-filtered) via REST/xcecli/harness, dependency info in job details, optional orphan termination (per-job `-ti` + cluster grace period, default off; EXIT reason ORPHAN; failover-safe persisted grace start; minimal primary periodic work). Brief: `.kiro/specs/job-dependency-insight/brief.md`. Dependencies: job-dependency, job-time-window (exit-reason field). - [ ] controller-job-cleanup — NEW 2026-09-29 (`/kiro-discovery`, Path C). Primary-only periodic sweep (configurable `job_cleanup_period`, default 300 s) removes DONE/EXIT jobs finished longer than configurable `job_retention` (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever `jobmgr` delete; refcount-safe `xkvmgr` delete), propagated to scheduler and apiserver (`OnJobDeleted`). Retires the node-local snapshot-only `job_should_retain` finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: `.kiro/specs/controller-job-cleanup/brief.md`. Dependencies: none (extends ha-entity-snapshots). - [ ] job-status-detail-display — NEW 2026-09-28 (`/kiro-discovery`, Path E). End users (REST/`xcecli`) see NO pend reason for ANY job today: the apiserver cgo conversion (`backend/convert.go`) never maps `xce_job_t.pend_reasons`, so `pending_reason` is always empty. Maps it to a structured JSON (including `alternatives` from day one) using libxce's code→text mapping, and renders it in `xcecli` job detail. Open for requirements: also map `run_usage` → `JobRusage` (same gap; reserved vs. used for RUN jobs). Brief: `.kiro/specs/job-status-detail-display/brief.md`. Dependencies: none for flat reasons; scheduler-alter-pend-reasons for per-alternative rendering.