docs(steering): one owner per job attribute; pre/post-exec moves to the app profile
New steering job-config-layering.md: queue = scheduling policy, application profile = what the application needs and how it runs (resreq, task range, execution settings, pre/post-exec, limits), project = resource ownership. Only two interactions: the job overrides a profile attribute marked default (fixed: not at all; select: only ANDed terms), and queue ceilings or project quotas check effective values (reject if never satisfiable, pend if full now). Replaces the simplified job > app > queue rules. Queue pre_exec/post_exec are deprecated with queue res_req: the profile's pre/post-exec fills the worker's existing outer layer, the job's own stays the inner layer, allowed or forbidden by the profile. job-config-merge and application-profile briefs and the roadmap follow; CLAUDE.md lists the new steering file. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
b761878b58
commit
25cdbc79b4
5 changed files with 123 additions and 21 deletions
82
.kiro/steering/job-config-layering.md
Normal file
82
.kiro/steering/job-config-layering.md
Normal file
|
|
@ -0,0 +1,82 @@
|
|||
# Job Configuration Layering
|
||||
|
||||
How a job's settings are split between the job, its application profile, its
|
||||
queue and its project, and how they interact. Decided 2026-10-09
|
||||
(`/kiro-discovery`, configuration layering). Specs that follow this:
|
||||
`job-config-merge`, `application-profile`, `project-config`,
|
||||
`project-guarantee`, `parjob-placement`, `lsf-config-migration`.
|
||||
|
||||
## Why not LSF's rules
|
||||
LSF lets the queue, the application profile and the job set the same
|
||||
attributes (`RES_REQ`, limits, pre/post-exec, job starter, ...) and then needs
|
||||
a rule per section to decide which wins (select ANDed, order/span overridden,
|
||||
rusage merged with the queue as ceiling, two-stage compound/alternative
|
||||
merges, queue pre-exec before app pre-exec, ...; Admin Guide pp. 314-316,
|
||||
435-437, 581). The roles of queue, application profile and project overlap.
|
||||
XCE instead gives **every attribute exactly one owning level**.
|
||||
|
||||
## Ownership
|
||||
|
||||
| Level | Owns | Never owns |
|
||||
|---|---|---|
|
||||
| **Queue**: scheduling policy | priority and ordering, host partition, dispatch/run windows, preemption policy, policy ceilings (max run time, max tasks, max hosts, ...) | resource requirement, anything about how a job executes (pre/post-exec, env, launch, limits enforced on the job) |
|
||||
| **Application profile**: what the application needs and how it runs | resreq (select / rusage / span / same / order), task range (min / default / max), execution mode and options, launch type, env, **pre/post-exec**, execution limits (mem, runtime, CPU, tasks) | priority, queue choice, quotas |
|
||||
| **Project**: resource ownership | membership, quotas / limits, later guarantees, accounting | how a job runs or is ordered |
|
||||
| **Job** | its own values, only for attributes its profile allows | n/a |
|
||||
|
||||
Access to each is separate: queue and profile by RBAC permission, project by
|
||||
membership (`project-config`). Any allowed queue combines with any allowed
|
||||
profile and project.
|
||||
|
||||
## The only two interactions
|
||||
|
||||
1. **Override: job vs. application profile.** Each profile attribute is
|
||||
either
|
||||
- **default**: the job's value replaces it, or
|
||||
- **fixed**: the job cannot change it. Exception: for `select`, the job may
|
||||
add terms (narrow the host set, ANDed) but never drop the profile's.
|
||||
2. **Ceiling: queue or project vs. the job's effective values.** A queue
|
||||
ceiling or project quota never changes the job's values, it only checks
|
||||
them:
|
||||
- never satisfiable (runtime above the queue's maximum, task range above
|
||||
its max tasks): rejected at submit, never silently clipped;
|
||||
- not satisfiable right now (project quota in use): the job pends with a
|
||||
reason naming the queue or project.
|
||||
|
||||
There is no third case: no queue resreq, no rusage merge, no ordering of
|
||||
queue against profile settings, no compound/alternative merge stages.
|
||||
|
||||
## Pre/post-exec
|
||||
- Moves from the queue to the application profile. Queue `pre_exec` /
|
||||
`post_exec` are deprecated (`xceadmin config build` warns, then rejects).
|
||||
- Two layers remain, each with one owner:
|
||||
1. **Profile pre/post-exec**: admin-defined, always runs (fixed); the job
|
||||
cannot replace or skip it.
|
||||
2. **Job pre/post-exec**: the user's own (`start_info` today); allowed or
|
||||
forbidden by the profile (e.g. a sandbox profile may forbid it).
|
||||
- Order: profile pre → job pre → job → job post → profile post. A failed
|
||||
profile pre-exec skips the rest; profile post-exec still runs for cleanup.
|
||||
- The worker already runs exactly two layers (`queue_pre_exec()` /
|
||||
`queue_post_exec()`, `worker/src/xcecd_worker.c`; stages
|
||||
`JOB_EXEC_STAGE_Q_PRE_EXEC` / `Q_POST_EXEC`), so the worker keeps its
|
||||
structure; the controller fills the outer layer from the profile instead of
|
||||
the queue (`get_queue_exec()`, `controller/queue.c:199`, called at dispatch
|
||||
`controller/job.c:1827`) and the stages are renamed.
|
||||
|
||||
## Timing
|
||||
Live (`job-config-merge`): the WAL keeps the job's own values plus the names
|
||||
of its queue, profile and project. The controller and apiserver validate at
|
||||
submit; the scheduler computes effective values and re-evaluates pending jobs
|
||||
when config changes; running jobs are unaffected. Execution-side profile
|
||||
values (pre/post-exec, env, mode) are read at dispatch, as queue pre/post-exec
|
||||
is today.
|
||||
|
||||
## Edge cases
|
||||
- Overlapping limits: the profile's runtime limit (what the application
|
||||
needs, enforced on the job) and the queue's max run time (policy ceiling)
|
||||
stay separate; the ceiling checks the profile/job value.
|
||||
- Site-wide host restrictions for a queue use the queue's host partition, not
|
||||
a resreq.
|
||||
- LSF migration (`lsf-config-migration`): an LSF queue `RES_REQ` that only
|
||||
selects hosts becomes a queue host partition; anything else moves into a
|
||||
profile or is reported. LSF queue `PRE_EXEC`/`POST_EXEC` move into profiles.
|
||||
|
|
@ -70,7 +70,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
|
|||
- [ ] job-dependency-insight — NEW 2026-09-29. `bjdepinfo`-style query (parents/children/-p blocking/-r levels/-l verdicts, RBAC-filtered) via REST/xcecli/harness, dependency info in job details, optional orphan termination (per-job `-ti` + cluster grace period, default off; EXIT reason ORPHAN; failover-safe persisted grace start; minimal primary periodic work). Brief: `.kiro/specs/job-dependency-insight/brief.md`. Dependencies: job-dependency, job-time-window (exit-reason field).
|
||||
- [ ] controller-job-cleanup — NEW 2026-09-29 (`/kiro-discovery`, Path C). Primary-only periodic sweep (configurable `job_cleanup_period`, default 300 s) removes DONE/EXIT jobs finished longer than configurable `job_retention` (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever `jobmgr` delete; refcount-safe `xkvmgr` delete), propagated to scheduler and apiserver (`OnJobDeleted`). Retires the node-local snapshot-only `job_should_retain` finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: `.kiro/specs/controller-job-cleanup/brief.md`. Dependencies: none (extends ha-entity-snapshots).
|
||||
- [ ] job-status-detail-display — NEW 2026-09-28 (`/kiro-discovery`, Path E). End users (REST/`xcecli`) see NO pend reason for ANY job today: the apiserver cgo conversion (`backend/convert.go`) never maps `xce_job_t.pend_reasons`, so `pending_reason` is always empty. Maps it to a structured JSON (including `alternatives` from day one) using libxce's code→text mapping, and renders it in `xcecli` job detail. Open for requirements: also map `run_usage` → `JobRusage` (same gap; reserved vs. used for RUN jobs). Brief: `.kiro/specs/job-status-detail-display/brief.md`. Dependencies: none for flat reasons; scheduler-alter-pend-reasons for per-alternative rendering.
|
||||
- [ ] job-config-merge — NEW 2026-10-09 (`/kiro-discovery`, Path E "configuration layering"; ref LSF 10.1 Admin Guide pp. 314-316, 435-437, 561-563, 643, 660). One merge function in `src/lib` turning the job's own values + application profile + queue policy + project into effective values (resreq incl. `span`, task range, limits, pre/post-exec, env). Simplified rules (user decision): resreq is two-level (profile, job), job overrides section by section, `select` ANDed; queue hard limits and task limits are ceilings that reject. **Live** merge (user decision): WAL keeps inputs + names; controller/apiserver validate at submit; scheduler computes effective values and re-evaluates pending jobs on config change. Queue is policy only: queue `res_req` deprecated, dropped queue limits (cpulimit/memlimit/runlimit/task limit) wired. Brief: `.kiro/specs/job-config-merge/brief.md`. Dependencies: none.
|
||||
- [ ] job-config-merge — NEW 2026-10-09 (`/kiro-discovery`, Path E "configuration layering"; ref LSF 10.1 Admin Guide pp. 314-316, 435-437, 561-563, 643, 660). One merge function in `src/lib` turning the job's own values + application profile + queue policy + project into effective values (resreq incl. `span`, task range, limits, pre/post-exec, env). One owner per attribute (user decision; `.kiro/steering/job-config-layering.md`): job overrides a profile attribute only if it is *default* (for `select` only by adding ANDed terms); queue ceilings and project quotas only check (reject if never satisfiable, pend if full now). **Live** merge (user decision): WAL keeps inputs + names; controller/apiserver validate at submit; scheduler computes effective values and re-evaluates pending jobs on config change. Queue is policy only: queue `res_req` and `pre_exec`/`post_exec` deprecated (pre/post-exec moves to the profile), dropped queue limits (cpulimit/memlimit/runlimit/task limit) wired. Brief: `.kiro/specs/job-config-merge/brief.md`. Dependencies: none.
|
||||
- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (members, priority, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config so `PROJECTS`/`PER_PROJECT` limits work end to end, per-project usage display. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none.
|
||||
|
||||
## Existing Spec Updates (2026-10-09 configuration layering discovery)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue