diff --git a/.kiro/specs/application-profile/brief.md b/.kiro/specs/application-profile/brief.md index b6a207b7..9a6dd3f4 100644 --- a/.kiro/specs/application-profile/brief.md +++ b/.kiro/specs/application-profile/brief.md @@ -13,10 +13,18 @@ decisions):** requirement* (select / rusage / `span[...]` / same) and task limits (min / default / max, LSF TASKLIMIT), **project** = resource ownership (`project-config`). -- How the levels combine is owned by **`job-config-merge`**: job overrides - profile section by section, `select` is ANDed, queue hard and task limits - are ceilings that reject. This answers the open question "how resource - defaults combine with a job's own resreq" below. +- Every attribute has one owning level (`.kiro/steering/job-config-layering.md`). + How the levels interact is owned by **`job-config-merge`**: each profile + attribute is *default* (the job may replace it) or *fixed* (the job may + not; for `select` it may only add ANDed terms); queue ceilings and project + quotas only check, never change, the job's values. This answers the open + question "how resource defaults combine with a job's own resreq" below. +- **Pre/post-exec moves here from the queue** (queue `pre_exec`/`post_exec` + deprecated). The profile's pre/post-exec is admin-defined and always runs + as the outer layer; the job's own pre/post-exec is the inner layer, allowed + or forbidden by the profile. Order: profile pre → job pre → job → job post + → profile post (the worker's existing two-layer structure, + `xcecd_worker.c` `queue_pre_exec()`/`queue_post_exec()`). - **Replaces "Resolution at submit time"** below: the merge is *live*. The WAL keeps the job's own values plus the profile name; the controller and apiserver validate at submit, and the scheduler computes effective values diff --git a/.kiro/specs/job-config-merge/brief.md b/.kiro/specs/job-config-merge/brief.md index 45d8a9df..b4a014ad 100644 --- a/.kiro/specs/job-config-merge/brief.md +++ b/.kiro/specs/job-config-merge/brief.md @@ -45,18 +45,23 @@ apiserver and display would disagree about what a job actually asked for. application profile, the queue's policy and the project; output is the job's **effective** values: resreq (select / rusage / span / same / order), task range, execution limits, pre/post-exec, environment. -- **Merge rules (simplified, user decision 2026-10-09; deliberately not - LSF's full per-section rules):** - - The resreq has two levels only, app profile and job (queues carry no - resreq, see below). The job overrides the app section by section; - `select` is the exception: both apply (ANDed). - - Everything else: job overrides app overrides queue defaults. - - Queue **hard limits** and queue **task limits** are ceilings: a job (or - app) value outside them is rejected at submit, never silently clipped. - - The application profile's task limits (min/default/max) also bound the - job's `-n`; with no `-n`, the app default applies (LSF p660 semantics). - - Where the simplified rules diverge from LSF, the requirements list the - divergence explicitly (input for `lsf-config-migration`). +- **Merge rules: one owner per attribute** (user decisions 2026-10-09; the + full rule set is `.kiro/steering/job-config-layering.md`). Every attribute + has exactly one owning level (queue = scheduling policy, application + profile = what the application needs and how it runs, project = resource + ownership), so only two interactions exist: + - **Override (job vs. profile):** each profile attribute is *default* (the + job's value replaces it) or *fixed* (the job cannot change it; for + `select` the job may only add ANDed terms). This replaces LSF's + per-section merge table. + - **Ceiling (queue or project vs. effective values):** never changes a + value; a never-satisfiable value (runtime above the queue maximum, task + range above the queue's max tasks) is rejected at submit, a quota that is + only full now makes the job pend with a reason naming the queue/project. + - The profile's task limits (min/default/max) bound the job's `-n`; with no + `-n`, the profile default applies. + - Where this diverges from LSF, the requirements list the divergence + explicitly (input for `lsf-config-migration`). - **Live merge (user decision 2026-10-09).** The WAL keeps the job's own values plus the names of its queue, profile and project. The merge runs: - at submit, in the controller (authoritative for every submit path) and @@ -66,8 +71,13 @@ apiserver and display would disagree about what a job actually asked for. upgrade). A pending job that no longer satisfies the merged values pends with a clear reason (LSF: "Job no longer satisfies TASKLIMIT configuration"); running jobs are unaffected. -- **Queue is policy only (user decision 2026-10-09).** Queue `res_req` is - deprecated: warned on by `xceadmin config build` first, then rejected. +- **Queue is policy only (user decision 2026-10-09).** Queue `res_req`, + `pre_exec` and `post_exec` are deprecated: warned on by `xceadmin config + build` first, then rejected. Pre/post-exec moves to the application + profile: the controller fills the worker's outer pre/post-exec layer (today + the queue's, `get_queue_exec()`, `controller/queue.c:199`, called at + dispatch `job.c:1827`) from the profile; the job's own pre/post-exec stays + the inner layer, allowed or forbidden by the profile. The queue limits `xceadmin` drops today (cpulimit, memlimit, runlimit, task limit) are wired through config, capnp and the merge as hard limits. - **Visibility.** Job details (REST, `xcecli`, REPL `info job`, decision @@ -86,7 +96,8 @@ values and the three names must round-trip). - **In**: the merge library and its rules; controller and apiserver submit validation; scheduler effective-value computation and re-evaluation on config change; queue hard/task limit wiring (YAML, schema, `mapping.py`, - capnp, C); queue `res_req` deprecation; effective values in job details and + capnp, C); queue `res_req` and pre/post-exec deprecation; outer + pre/post-exec sourced from the profile; effective values in job details and the decision trace; tests. - **Out**: the profile model itself (`application-profile`); project definitions (`project-config`); `span` placement semantics diff --git a/.kiro/steering/job-config-layering.md b/.kiro/steering/job-config-layering.md new file mode 100644 index 00000000..2520c34a --- /dev/null +++ b/.kiro/steering/job-config-layering.md @@ -0,0 +1,82 @@ +# Job Configuration Layering + +How a job's settings are split between the job, its application profile, its +queue and its project, and how they interact. Decided 2026-10-09 +(`/kiro-discovery`, configuration layering). Specs that follow this: +`job-config-merge`, `application-profile`, `project-config`, +`project-guarantee`, `parjob-placement`, `lsf-config-migration`. + +## Why not LSF's rules +LSF lets the queue, the application profile and the job set the same +attributes (`RES_REQ`, limits, pre/post-exec, job starter, ...) and then needs +a rule per section to decide which wins (select ANDed, order/span overridden, +rusage merged with the queue as ceiling, two-stage compound/alternative +merges, queue pre-exec before app pre-exec, ...; Admin Guide pp. 314-316, +435-437, 581). The roles of queue, application profile and project overlap. +XCE instead gives **every attribute exactly one owning level**. + +## Ownership + +| Level | Owns | Never owns | +|---|---|---| +| **Queue**: scheduling policy | priority and ordering, host partition, dispatch/run windows, preemption policy, policy ceilings (max run time, max tasks, max hosts, ...) | resource requirement, anything about how a job executes (pre/post-exec, env, launch, limits enforced on the job) | +| **Application profile**: what the application needs and how it runs | resreq (select / rusage / span / same / order), task range (min / default / max), execution mode and options, launch type, env, **pre/post-exec**, execution limits (mem, runtime, CPU, tasks) | priority, queue choice, quotas | +| **Project**: resource ownership | membership, quotas / limits, later guarantees, accounting | how a job runs or is ordered | +| **Job** | its own values, only for attributes its profile allows | n/a | + +Access to each is separate: queue and profile by RBAC permission, project by +membership (`project-config`). Any allowed queue combines with any allowed +profile and project. + +## The only two interactions + +1. **Override: job vs. application profile.** Each profile attribute is + either + - **default**: the job's value replaces it, or + - **fixed**: the job cannot change it. Exception: for `select`, the job may + add terms (narrow the host set, ANDed) but never drop the profile's. +2. **Ceiling: queue or project vs. the job's effective values.** A queue + ceiling or project quota never changes the job's values, it only checks + them: + - never satisfiable (runtime above the queue's maximum, task range above + its max tasks): rejected at submit, never silently clipped; + - not satisfiable right now (project quota in use): the job pends with a + reason naming the queue or project. + +There is no third case: no queue resreq, no rusage merge, no ordering of +queue against profile settings, no compound/alternative merge stages. + +## Pre/post-exec +- Moves from the queue to the application profile. Queue `pre_exec` / + `post_exec` are deprecated (`xceadmin config build` warns, then rejects). +- Two layers remain, each with one owner: + 1. **Profile pre/post-exec**: admin-defined, always runs (fixed); the job + cannot replace or skip it. + 2. **Job pre/post-exec**: the user's own (`start_info` today); allowed or + forbidden by the profile (e.g. a sandbox profile may forbid it). +- Order: profile pre → job pre → job → job post → profile post. A failed + profile pre-exec skips the rest; profile post-exec still runs for cleanup. +- The worker already runs exactly two layers (`queue_pre_exec()` / + `queue_post_exec()`, `worker/src/xcecd_worker.c`; stages + `JOB_EXEC_STAGE_Q_PRE_EXEC` / `Q_POST_EXEC`), so the worker keeps its + structure; the controller fills the outer layer from the profile instead of + the queue (`get_queue_exec()`, `controller/queue.c:199`, called at dispatch + `controller/job.c:1827`) and the stages are renamed. + +## Timing +Live (`job-config-merge`): the WAL keeps the job's own values plus the names +of its queue, profile and project. The controller and apiserver validate at +submit; the scheduler computes effective values and re-evaluates pending jobs +when config changes; running jobs are unaffected. Execution-side profile +values (pre/post-exec, env, mode) are read at dispatch, as queue pre/post-exec +is today. + +## Edge cases +- Overlapping limits: the profile's runtime limit (what the application + needs, enforced on the job) and the queue's max run time (policy ceiling) + stay separate; the ceiling checks the profile/job value. +- Site-wide host restrictions for a queue use the queue's host partition, not + a resreq. +- LSF migration (`lsf-config-migration`): an LSF queue `RES_REQ` that only + selects hosts becomes a queue host partition; anything else moves into a + profile or is reported. LSF queue `PRE_EXEC`/`POST_EXEC` move into profiles. diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index 4fe8e62e..9f195751 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -70,7 +70,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han - [ ] job-dependency-insight — NEW 2026-09-29. `bjdepinfo`-style query (parents/children/-p blocking/-r levels/-l verdicts, RBAC-filtered) via REST/xcecli/harness, dependency info in job details, optional orphan termination (per-job `-ti` + cluster grace period, default off; EXIT reason ORPHAN; failover-safe persisted grace start; minimal primary periodic work). Brief: `.kiro/specs/job-dependency-insight/brief.md`. Dependencies: job-dependency, job-time-window (exit-reason field). - [ ] controller-job-cleanup — NEW 2026-09-29 (`/kiro-discovery`, Path C). Primary-only periodic sweep (configurable `job_cleanup_period`, default 300 s) removes DONE/EXIT jobs finished longer than configurable `job_retention` (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever `jobmgr` delete; refcount-safe `xkvmgr` delete), propagated to scheduler and apiserver (`OnJobDeleted`). Retires the node-local snapshot-only `job_should_retain` finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: `.kiro/specs/controller-job-cleanup/brief.md`. Dependencies: none (extends ha-entity-snapshots). - [ ] job-status-detail-display — NEW 2026-09-28 (`/kiro-discovery`, Path E). End users (REST/`xcecli`) see NO pend reason for ANY job today: the apiserver cgo conversion (`backend/convert.go`) never maps `xce_job_t.pend_reasons`, so `pending_reason` is always empty. Maps it to a structured JSON (including `alternatives` from day one) using libxce's code→text mapping, and renders it in `xcecli` job detail. Open for requirements: also map `run_usage` → `JobRusage` (same gap; reserved vs. used for RUN jobs). Brief: `.kiro/specs/job-status-detail-display/brief.md`. Dependencies: none for flat reasons; scheduler-alter-pend-reasons for per-alternative rendering. -- [ ] job-config-merge — NEW 2026-10-09 (`/kiro-discovery`, Path E "configuration layering"; ref LSF 10.1 Admin Guide pp. 314-316, 435-437, 561-563, 643, 660). One merge function in `src/lib` turning the job's own values + application profile + queue policy + project into effective values (resreq incl. `span`, task range, limits, pre/post-exec, env). Simplified rules (user decision): resreq is two-level (profile, job), job overrides section by section, `select` ANDed; queue hard limits and task limits are ceilings that reject. **Live** merge (user decision): WAL keeps inputs + names; controller/apiserver validate at submit; scheduler computes effective values and re-evaluates pending jobs on config change. Queue is policy only: queue `res_req` deprecated, dropped queue limits (cpulimit/memlimit/runlimit/task limit) wired. Brief: `.kiro/specs/job-config-merge/brief.md`. Dependencies: none. +- [ ] job-config-merge — NEW 2026-10-09 (`/kiro-discovery`, Path E "configuration layering"; ref LSF 10.1 Admin Guide pp. 314-316, 435-437, 561-563, 643, 660). One merge function in `src/lib` turning the job's own values + application profile + queue policy + project into effective values (resreq incl. `span`, task range, limits, pre/post-exec, env). One owner per attribute (user decision; `.kiro/steering/job-config-layering.md`): job overrides a profile attribute only if it is *default* (for `select` only by adding ANDed terms); queue ceilings and project quotas only check (reject if never satisfiable, pend if full now). **Live** merge (user decision): WAL keeps inputs + names; controller/apiserver validate at submit; scheduler computes effective values and re-evaluates pending jobs on config change. Queue is policy only: queue `res_req` and `pre_exec`/`post_exec` deprecated (pre/post-exec moves to the profile), dropped queue limits (cpulimit/memlimit/runlimit/task limit) wired. Brief: `.kiro/specs/job-config-merge/brief.md`. Dependencies: none. - [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (members, priority, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config so `PROJECTS`/`PER_PROJECT` limits work end to end, per-project usage display. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none. ## Existing Spec Updates (2026-10-09 configuration layering discovery) diff --git a/CLAUDE.md b/CLAUDE.md index 9089aba5..97c8849e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,6 +27,7 @@ For details on any topic below, read the corresponding steering file in `.kiro/s | RBAC, JWT, auth backends, input validation, secrets | `security.md` | | WAL durability, recovery, save/restore field-completeness | `persistence.md` | | Real vs. virtual resources, reservation-vs-usage accounting (mem) | `resource-accounting.md` | +| Job/app-profile/queue/project ownership and merge rules | `job-config-layering.md` | ## Key Rules (from steering — summarized for quick reference) @@ -105,5 +106,5 @@ Kiro-style Spec-Driven Development on an agentic SDLC ## Steering Configuration - Load entire `.kiro/steering/` as project memory — all `.md` files are authoritative - Core: `product.md`, `tech.md`, `structure.md` -- Custom: `testing.md`, `error-handling.md`, `security.md`, `ha-async-io.md`, `api-standards.md`, `build-deployment.md`, `config-design.md`, `persistence.md`, `resource-accounting.md`, `roadmap.md` +- Custom: `testing.md`, `error-handling.md`, `security.md`, `ha-async-io.md`, `api-standards.md`, `build-deployment.md`, `config-design.md`, `persistence.md`, `resource-accounting.md`, `job-config-layering.md`, `roadmap.md` - Add new custom files via `/kiro-steering-custom`; sync existing via `/kiro-steering`