docs(steering): one owner per job attribute; pre/post-exec moves to the app profile

New steering job-config-layering.md: queue = scheduling policy, application
profile = what the application needs and how it runs (resreq, task range,
execution settings, pre/post-exec, limits), project = resource ownership.
Only two interactions: the job overrides a profile attribute marked
default (fixed: not at all; select: only ANDed terms), and queue ceilings
or project quotas check effective values (reject if never satisfiable,
pend if full now). Replaces the simplified job > app > queue rules.

Queue pre_exec/post_exec are deprecated with queue res_req: the profile's
pre/post-exec fills the worker's existing outer layer, the job's own stays
the inner layer, allowed or forbidden by the profile.

job-config-merge and application-profile briefs and the roadmap follow;
CLAUDE.md lists the new steering file.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-10 03:19:20 +00:00
parent b761878b58
commit 25cdbc79b4
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
5 changed files with 123 additions and 21 deletions

View file

@ -13,10 +13,18 @@ decisions):**
requirement* (select / rusage / `span[...]` / same) and task limits requirement* (select / rusage / `span[...]` / same) and task limits
(min / default / max, LSF TASKLIMIT), **project** = resource ownership (min / default / max, LSF TASKLIMIT), **project** = resource ownership
(`project-config`). (`project-config`).
- How the levels combine is owned by **`job-config-merge`**: job overrides - Every attribute has one owning level (`.kiro/steering/job-config-layering.md`).
profile section by section, `select` is ANDed, queue hard and task limits How the levels interact is owned by **`job-config-merge`**: each profile
are ceilings that reject. This answers the open question "how resource attribute is *default* (the job may replace it) or *fixed* (the job may
defaults combine with a job's own resreq" below. not; for `select` it may only add ANDed terms); queue ceilings and project
quotas only check, never change, the job's values. This answers the open
question "how resource defaults combine with a job's own resreq" below.
- **Pre/post-exec moves here from the queue** (queue `pre_exec`/`post_exec`
deprecated). The profile's pre/post-exec is admin-defined and always runs
as the outer layer; the job's own pre/post-exec is the inner layer, allowed
or forbidden by the profile. Order: profile pre → job pre → job → job post
→ profile post (the worker's existing two-layer structure,
`xcecd_worker.c` `queue_pre_exec()`/`queue_post_exec()`).
- **Replaces "Resolution at submit time"** below: the merge is *live*. The WAL - **Replaces "Resolution at submit time"** below: the merge is *live*. The WAL
keeps the job's own values plus the profile name; the controller and keeps the job's own values plus the profile name; the controller and
apiserver validate at submit, and the scheduler computes effective values apiserver validate at submit, and the scheduler computes effective values

View file

@ -45,18 +45,23 @@ apiserver and display would disagree about what a job actually asked for.
application profile, the queue's policy and the project; output is the application profile, the queue's policy and the project; output is the
job's **effective** values: resreq (select / rusage / span / same / job's **effective** values: resreq (select / rusage / span / same /
order), task range, execution limits, pre/post-exec, environment. order), task range, execution limits, pre/post-exec, environment.
- **Merge rules (simplified, user decision 2026-10-09; deliberately not - **Merge rules: one owner per attribute** (user decisions 2026-10-09; the
LSF's full per-section rules):** full rule set is `.kiro/steering/job-config-layering.md`). Every attribute
- The resreq has two levels only, app profile and job (queues carry no has exactly one owning level (queue = scheduling policy, application
resreq, see below). The job overrides the app section by section; profile = what the application needs and how it runs, project = resource
`select` is the exception: both apply (ANDed). ownership), so only two interactions exist:
- Everything else: job overrides app overrides queue defaults. - **Override (job vs. profile):** each profile attribute is *default* (the
- Queue **hard limits** and queue **task limits** are ceilings: a job (or job's value replaces it) or *fixed* (the job cannot change it; for
app) value outside them is rejected at submit, never silently clipped. `select` the job may only add ANDed terms). This replaces LSF's
- The application profile's task limits (min/default/max) also bound the per-section merge table.
job's `-n`; with no `-n`, the app default applies (LSF p660 semantics). - **Ceiling (queue or project vs. effective values):** never changes a
- Where the simplified rules diverge from LSF, the requirements list the value; a never-satisfiable value (runtime above the queue maximum, task
divergence explicitly (input for `lsf-config-migration`). range above the queue's max tasks) is rejected at submit, a quota that is
only full now makes the job pend with a reason naming the queue/project.
- The profile's task limits (min/default/max) bound the job's `-n`; with no
`-n`, the profile default applies.
- Where this diverges from LSF, the requirements list the divergence
explicitly (input for `lsf-config-migration`).
- **Live merge (user decision 2026-10-09).** The WAL keeps the job's own - **Live merge (user decision 2026-10-09).** The WAL keeps the job's own
values plus the names of its queue, profile and project. The merge runs: values plus the names of its queue, profile and project. The merge runs:
- at submit, in the controller (authoritative for every submit path) and - at submit, in the controller (authoritative for every submit path) and
@ -66,8 +71,13 @@ apiserver and display would disagree about what a job actually asked for.
upgrade). A pending job that no longer satisfies the merged values pends upgrade). A pending job that no longer satisfies the merged values pends
with a clear reason (LSF: "Job no longer satisfies TASKLIMIT with a clear reason (LSF: "Job no longer satisfies TASKLIMIT
configuration"); running jobs are unaffected. configuration"); running jobs are unaffected.
- **Queue is policy only (user decision 2026-10-09).** Queue `res_req` is - **Queue is policy only (user decision 2026-10-09).** Queue `res_req`,
deprecated: warned on by `xceadmin config build` first, then rejected. `pre_exec` and `post_exec` are deprecated: warned on by `xceadmin config
build` first, then rejected. Pre/post-exec moves to the application
profile: the controller fills the worker's outer pre/post-exec layer (today
the queue's, `get_queue_exec()`, `controller/queue.c:199`, called at
dispatch `job.c:1827`) from the profile; the job's own pre/post-exec stays
the inner layer, allowed or forbidden by the profile.
The queue limits `xceadmin` drops today (cpulimit, memlimit, runlimit, The queue limits `xceadmin` drops today (cpulimit, memlimit, runlimit,
task limit) are wired through config, capnp and the merge as hard limits. task limit) are wired through config, capnp and the merge as hard limits.
- **Visibility.** Job details (REST, `xcecli`, REPL `info job`, decision - **Visibility.** Job details (REST, `xcecli`, REPL `info job`, decision
@ -86,7 +96,8 @@ values and the three names must round-trip).
- **In**: the merge library and its rules; controller and apiserver submit - **In**: the merge library and its rules; controller and apiserver submit
validation; scheduler effective-value computation and re-evaluation on validation; scheduler effective-value computation and re-evaluation on
config change; queue hard/task limit wiring (YAML, schema, `mapping.py`, config change; queue hard/task limit wiring (YAML, schema, `mapping.py`,
capnp, C); queue `res_req` deprecation; effective values in job details and capnp, C); queue `res_req` and pre/post-exec deprecation; outer
pre/post-exec sourced from the profile; effective values in job details and
the decision trace; tests. the decision trace; tests.
- **Out**: the profile model itself (`application-profile`); project - **Out**: the profile model itself (`application-profile`); project
definitions (`project-config`); `span` placement semantics definitions (`project-config`); `span` placement semantics

View file

@ -0,0 +1,82 @@
# Job Configuration Layering
How a job's settings are split between the job, its application profile, its
queue and its project, and how they interact. Decided 2026-10-09
(`/kiro-discovery`, configuration layering). Specs that follow this:
`job-config-merge`, `application-profile`, `project-config`,
`project-guarantee`, `parjob-placement`, `lsf-config-migration`.
## Why not LSF's rules
LSF lets the queue, the application profile and the job set the same
attributes (`RES_REQ`, limits, pre/post-exec, job starter, ...) and then needs
a rule per section to decide which wins (select ANDed, order/span overridden,
rusage merged with the queue as ceiling, two-stage compound/alternative
merges, queue pre-exec before app pre-exec, ...; Admin Guide pp. 314-316,
435-437, 581). The roles of queue, application profile and project overlap.
XCE instead gives **every attribute exactly one owning level**.
## Ownership
| Level | Owns | Never owns |
|---|---|---|
| **Queue**: scheduling policy | priority and ordering, host partition, dispatch/run windows, preemption policy, policy ceilings (max run time, max tasks, max hosts, ...) | resource requirement, anything about how a job executes (pre/post-exec, env, launch, limits enforced on the job) |
| **Application profile**: what the application needs and how it runs | resreq (select / rusage / span / same / order), task range (min / default / max), execution mode and options, launch type, env, **pre/post-exec**, execution limits (mem, runtime, CPU, tasks) | priority, queue choice, quotas |
| **Project**: resource ownership | membership, quotas / limits, later guarantees, accounting | how a job runs or is ordered |
| **Job** | its own values, only for attributes its profile allows | n/a |
Access to each is separate: queue and profile by RBAC permission, project by
membership (`project-config`). Any allowed queue combines with any allowed
profile and project.
## The only two interactions
1. **Override: job vs. application profile.** Each profile attribute is
either
- **default**: the job's value replaces it, or
- **fixed**: the job cannot change it. Exception: for `select`, the job may
add terms (narrow the host set, ANDed) but never drop the profile's.
2. **Ceiling: queue or project vs. the job's effective values.** A queue
ceiling or project quota never changes the job's values, it only checks
them:
- never satisfiable (runtime above the queue's maximum, task range above
its max tasks): rejected at submit, never silently clipped;
- not satisfiable right now (project quota in use): the job pends with a
reason naming the queue or project.
There is no third case: no queue resreq, no rusage merge, no ordering of
queue against profile settings, no compound/alternative merge stages.
## Pre/post-exec
- Moves from the queue to the application profile. Queue `pre_exec` /
`post_exec` are deprecated (`xceadmin config build` warns, then rejects).
- Two layers remain, each with one owner:
1. **Profile pre/post-exec**: admin-defined, always runs (fixed); the job
cannot replace or skip it.
2. **Job pre/post-exec**: the user's own (`start_info` today); allowed or
forbidden by the profile (e.g. a sandbox profile may forbid it).
- Order: profile pre → job pre → job → job post → profile post. A failed
profile pre-exec skips the rest; profile post-exec still runs for cleanup.
- The worker already runs exactly two layers (`queue_pre_exec()` /
`queue_post_exec()`, `worker/src/xcecd_worker.c`; stages
`JOB_EXEC_STAGE_Q_PRE_EXEC` / `Q_POST_EXEC`), so the worker keeps its
structure; the controller fills the outer layer from the profile instead of
the queue (`get_queue_exec()`, `controller/queue.c:199`, called at dispatch
`controller/job.c:1827`) and the stages are renamed.
## Timing
Live (`job-config-merge`): the WAL keeps the job's own values plus the names
of its queue, profile and project. The controller and apiserver validate at
submit; the scheduler computes effective values and re-evaluates pending jobs
when config changes; running jobs are unaffected. Execution-side profile
values (pre/post-exec, env, mode) are read at dispatch, as queue pre/post-exec
is today.
## Edge cases
- Overlapping limits: the profile's runtime limit (what the application
needs, enforced on the job) and the queue's max run time (policy ceiling)
stay separate; the ceiling checks the profile/job value.
- Site-wide host restrictions for a queue use the queue's host partition, not
a resreq.
- LSF migration (`lsf-config-migration`): an LSF queue `RES_REQ` that only
selects hosts becomes a queue host partition; anything else moves into a
profile or is reported. LSF queue `PRE_EXEC`/`POST_EXEC` move into profiles.

View file

@ -70,7 +70,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
- [ ] job-dependency-insight — NEW 2026-09-29. `bjdepinfo`-style query (parents/children/-p blocking/-r levels/-l verdicts, RBAC-filtered) via REST/xcecli/harness, dependency info in job details, optional orphan termination (per-job `-ti` + cluster grace period, default off; EXIT reason ORPHAN; failover-safe persisted grace start; minimal primary periodic work). Brief: `.kiro/specs/job-dependency-insight/brief.md`. Dependencies: job-dependency, job-time-window (exit-reason field). - [ ] job-dependency-insight — NEW 2026-09-29. `bjdepinfo`-style query (parents/children/-p blocking/-r levels/-l verdicts, RBAC-filtered) via REST/xcecli/harness, dependency info in job details, optional orphan termination (per-job `-ti` + cluster grace period, default off; EXIT reason ORPHAN; failover-safe persisted grace start; minimal primary periodic work). Brief: `.kiro/specs/job-dependency-insight/brief.md`. Dependencies: job-dependency, job-time-window (exit-reason field).
- [ ] controller-job-cleanup — NEW 2026-09-29 (`/kiro-discovery`, Path C). Primary-only periodic sweep (configurable `job_cleanup_period`, default 300 s) removes DONE/EXIT jobs finished longer than configurable `job_retention` (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever `jobmgr` delete; refcount-safe `xkvmgr` delete), propagated to scheduler and apiserver (`OnJobDeleted`). Retires the node-local snapshot-only `job_should_retain` finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: `.kiro/specs/controller-job-cleanup/brief.md`. Dependencies: none (extends ha-entity-snapshots). - [ ] controller-job-cleanup — NEW 2026-09-29 (`/kiro-discovery`, Path C). Primary-only periodic sweep (configurable `job_cleanup_period`, default 300 s) removes DONE/EXIT jobs finished longer than configurable `job_retention` (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever `jobmgr` delete; refcount-safe `xkvmgr` delete), propagated to scheduler and apiserver (`OnJobDeleted`). Retires the node-local snapshot-only `job_should_retain` finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: `.kiro/specs/controller-job-cleanup/brief.md`. Dependencies: none (extends ha-entity-snapshots).
- [ ] job-status-detail-display — NEW 2026-09-28 (`/kiro-discovery`, Path E). End users (REST/`xcecli`) see NO pend reason for ANY job today: the apiserver cgo conversion (`backend/convert.go`) never maps `xce_job_t.pend_reasons`, so `pending_reason` is always empty. Maps it to a structured JSON (including `alternatives` from day one) using libxce's code→text mapping, and renders it in `xcecli` job detail. Open for requirements: also map `run_usage` → `JobRusage` (same gap; reserved vs. used for RUN jobs). Brief: `.kiro/specs/job-status-detail-display/brief.md`. Dependencies: none for flat reasons; scheduler-alter-pend-reasons for per-alternative rendering. - [ ] job-status-detail-display — NEW 2026-09-28 (`/kiro-discovery`, Path E). End users (REST/`xcecli`) see NO pend reason for ANY job today: the apiserver cgo conversion (`backend/convert.go`) never maps `xce_job_t.pend_reasons`, so `pending_reason` is always empty. Maps it to a structured JSON (including `alternatives` from day one) using libxce's code→text mapping, and renders it in `xcecli` job detail. Open for requirements: also map `run_usage` → `JobRusage` (same gap; reserved vs. used for RUN jobs). Brief: `.kiro/specs/job-status-detail-display/brief.md`. Dependencies: none for flat reasons; scheduler-alter-pend-reasons for per-alternative rendering.
- [ ] job-config-merge — NEW 2026-10-09 (`/kiro-discovery`, Path E "configuration layering"; ref LSF 10.1 Admin Guide pp. 314-316, 435-437, 561-563, 643, 660). One merge function in `src/lib` turning the job's own values + application profile + queue policy + project into effective values (resreq incl. `span`, task range, limits, pre/post-exec, env). Simplified rules (user decision): resreq is two-level (profile, job), job overrides section by section, `select` ANDed; queue hard limits and task limits are ceilings that reject. **Live** merge (user decision): WAL keeps inputs + names; controller/apiserver validate at submit; scheduler computes effective values and re-evaluates pending jobs on config change. Queue is policy only: queue `res_req` deprecated, dropped queue limits (cpulimit/memlimit/runlimit/task limit) wired. Brief: `.kiro/specs/job-config-merge/brief.md`. Dependencies: none. - [ ] job-config-merge — NEW 2026-10-09 (`/kiro-discovery`, Path E "configuration layering"; ref LSF 10.1 Admin Guide pp. 314-316, 435-437, 561-563, 643, 660). One merge function in `src/lib` turning the job's own values + application profile + queue policy + project into effective values (resreq incl. `span`, task range, limits, pre/post-exec, env). One owner per attribute (user decision; `.kiro/steering/job-config-layering.md`): job overrides a profile attribute only if it is *default* (for `select` only by adding ANDed terms); queue ceilings and project quotas only check (reject if never satisfiable, pend if full now). **Live** merge (user decision): WAL keeps inputs + names; controller/apiserver validate at submit; scheduler computes effective values and re-evaluates pending jobs on config change. Queue is policy only: queue `res_req` and `pre_exec`/`post_exec` deprecated (pre/post-exec moves to the profile), dropped queue limits (cpulimit/memlimit/runlimit/task limit) wired. Brief: `.kiro/specs/job-config-merge/brief.md`. Dependencies: none.
- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (members, priority, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config so `PROJECTS`/`PER_PROJECT` limits work end to end, per-project usage display. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none. - [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (members, priority, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config so `PROJECTS`/`PER_PROJECT` limits work end to end, per-project usage display. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none.
## Existing Spec Updates (2026-10-09 configuration layering discovery) ## Existing Spec Updates (2026-10-09 configuration layering discovery)

View file

@ -27,6 +27,7 @@ For details on any topic below, read the corresponding steering file in `.kiro/s
| RBAC, JWT, auth backends, input validation, secrets | `security.md` | | RBAC, JWT, auth backends, input validation, secrets | `security.md` |
| WAL durability, recovery, save/restore field-completeness | `persistence.md` | | WAL durability, recovery, save/restore field-completeness | `persistence.md` |
| Real vs. virtual resources, reservation-vs-usage accounting (mem) | `resource-accounting.md` | | Real vs. virtual resources, reservation-vs-usage accounting (mem) | `resource-accounting.md` |
| Job/app-profile/queue/project ownership and merge rules | `job-config-layering.md` |
## Key Rules (from steering — summarized for quick reference) ## Key Rules (from steering — summarized for quick reference)
@ -105,5 +106,5 @@ Kiro-style Spec-Driven Development on an agentic SDLC
## Steering Configuration ## Steering Configuration
- Load entire `.kiro/steering/` as project memory — all `.md` files are authoritative - Load entire `.kiro/steering/` as project memory — all `.md` files are authoritative
- Core: `product.md`, `tech.md`, `structure.md` - Core: `product.md`, `tech.md`, `structure.md`
- Custom: `testing.md`, `error-handling.md`, `security.md`, `ha-async-io.md`, `api-standards.md`, `build-deployment.md`, `config-design.md`, `persistence.md`, `resource-accounting.md`, `roadmap.md` - Custom: `testing.md`, `error-handling.md`, `security.md`, `ha-async-io.md`, `api-standards.md`, `build-deployment.md`, `config-design.md`, `persistence.md`, `resource-accounting.md`, `job-config-layering.md`, `roadmap.md`
- Add new custom files via `/kiro-steering-custom`; sync existing via `/kiro-steering` - Add new custom files via `/kiro-steering-custom`; sync existing via `/kiro-steering`