docs(specs): discovery for layered job configuration (queue/app/project)
New briefs: job-config-merge (one live merge of job, application profile, queue policy and project into effective values; simplified rules: job overrides profile, select ANDed, queue hard/task limits reject; queue is policy only, res_req deprecated), project-config (project quotas v1) and project-guarantee (backlog). application-profile: the profile carries the job's resreq (incl. span) and task limits; "frozen at submit" replaced by the live merge. parjob-placement: span constraints live in the fixed-N planner and the range search bisects over the planner's allowed counts; LSF 10.1 block, ptile, stripe and HOSTLIMIT_PER_JOB findings recorded. Roadmap: new specs, existing-spec updates, lsf-config-migration backlog. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
a219419e59
commit
b761878b58
6 changed files with 341 additions and 2 deletions
|
|
@ -5,6 +5,27 @@ mode is an attribute of the job, determined by the **application** the user runs
|
|||
so it is carried by a named application profile rather than by a bare mode
|
||||
option.)
|
||||
|
||||
**Update 2026-10-09 (`/kiro-discovery`, configuration layering; user
|
||||
decisions):**
|
||||
- Configuration is layered LSF-style: **queue** = scheduling policy only (no
|
||||
resreq; queue `res_req` is deprecated), **application profile** =
|
||||
application execution constraints *including the job's resource
|
||||
requirement* (select / rusage / `span[...]` / same) and task limits
|
||||
(min / default / max, LSF TASKLIMIT), **project** = resource ownership
|
||||
(`project-config`).
|
||||
- How the levels combine is owned by **`job-config-merge`**: job overrides
|
||||
profile section by section, `select` is ANDed, queue hard and task limits
|
||||
are ceilings that reject. This answers the open question "how resource
|
||||
defaults combine with a job's own resreq" below.
|
||||
- **Replaces "Resolution at submit time"** below: the merge is *live*. The WAL
|
||||
keeps the job's own values plus the profile name; the controller and
|
||||
apiserver validate at submit, and the scheduler computes effective values
|
||||
and recomputes them when a profile changes (a pending job that no longer
|
||||
satisfies its profile pends with a reason; running jobs are unaffected).
|
||||
Execution-side values (mode, options, launch type, env, pre/post-exec) are
|
||||
still taken from the profile at dispatch time; requirements decide whether
|
||||
dispatch freezes them on the job record.
|
||||
|
||||
## Problem
|
||||
XCE users run many kinds of applications: trusted in-house codes, untrusted
|
||||
scripts, vendor binaries, MPI solvers. Each needs a different way of running:
|
||||
|
|
@ -175,8 +196,8 @@ fits XCE's LSF-style job model.
|
|||
- Which extra attributes go into v1 beyond the four ("maybe others"). Examples:
|
||||
requeue exit values, runtime estimate, job starter or wrapper, output path
|
||||
template, checkpoint settings.
|
||||
- How resource defaults combine with a job's own resreq (replace, or fill in
|
||||
what's missing).
|
||||
- ~~How resource defaults combine with a job's own resreq~~ — settled
|
||||
2026-10-09, owned by `job-config-merge`.
|
||||
- Profile versioning and audit.
|
||||
- Whether the profile can be changed for a pending job.
|
||||
- **Out**:
|
||||
|
|
@ -215,6 +236,8 @@ fits XCE's LSF-style job model.
|
|||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends**: none directly.
|
||||
- **Depends on**: `job-config-merge` (merge rules and live re-evaluation).
|
||||
- **Feeds**: `parjob-placement` (the profile's `span[...]`).
|
||||
- **Adjacent**:
|
||||
- `job-status-detail-display`
|
||||
- `controller-job-cleanup`
|
||||
|
|
|
|||
126
.kiro/specs/job-config-merge/brief.md
Normal file
126
.kiro/specs/job-config-merge/brief.md
Normal file
|
|
@ -0,0 +1,126 @@
|
|||
# Brief: job-config-merge
|
||||
|
||||
(`/kiro-discovery`, 2026-10-09, Path E "configuration layering". Reference:
|
||||
IBM Spectrum LSF 10.1 Administering LSF, `../misc/lsf/`, pp. 314-316 (resreq
|
||||
merge), 435-437 (application profile vs queue), 561-563 (hard/soft limits),
|
||||
643 and 660 (TASKLIMIT). Page numbers are PDF pages.)
|
||||
|
||||
## Problem
|
||||
A job's settings will come from several places, as in LSF:
|
||||
- the **job** itself (submit options, `-R`, `-n min,max`, limits);
|
||||
- its **application profile** (`application-profile`): what the
|
||||
application needs to run, including its resource requirement (`span`
|
||||
included) and task limits;
|
||||
- its **queue**: scheduling policy only (priority, ordering, host partition,
|
||||
hard limits, task limits, windows);
|
||||
- its **project** (`project-config`): resource ownership (quotas now,
|
||||
guarantees later).
|
||||
|
||||
Nothing in XCE combines settings from more than one place today, so every
|
||||
level would invent its own precedence, and the scheduler, controller,
|
||||
apiserver and display would disagree about what a job actually asked for.
|
||||
|
||||
## Current State
|
||||
- No code merges two resreqs (`src/lib/resreq.c`). The only defaults:
|
||||
the scheduler turns an empty resreq into `"type == any"`
|
||||
(`scheduler/src/core_events.c:300-309`, also `schedobj.c:198`,
|
||||
`snapshot_load.c:344`) and `apply_job_task_count()` (`resreq.c:574-591`)
|
||||
overwrites the primary's min/max tasks from the job.
|
||||
- Queues carry a `resreq` string end to end (YAML `res_req`, capnp
|
||||
`QueueConf.resreq`, `xce_queue_t`, scheduler snapshot) but nothing reads
|
||||
it. `make_sched_queue()` (`core_events.c:377-388`) says wiring it is "a
|
||||
later task".
|
||||
- `xceadmin` drops queue cpulimit, memlimit, runlimit, runtimeest, fairshare,
|
||||
gpu_req, nice, job_starter, reservation and preemption
|
||||
(`xceadmin/src/xcecommon/mapping.py:80-97`).
|
||||
- The controller does not resolve anything at submit: `handle_jobsubmit`
|
||||
(`controller/job.c:2577`) checks user/queue/cmdline are non-null and
|
||||
`build_job_submit_payloads` (`job.c:347-371`) persists the client's
|
||||
`job_submit_data_t` unchanged in the WAL. The controller does not even
|
||||
check that the queue exists (the apiserver does, `jobs.go:1299-1322`).
|
||||
- The apiserver only checks resreq syntax (`backend/resreq_check.go:22`).
|
||||
|
||||
## Desired Outcome
|
||||
- **One merge function** in `src/lib`: inputs are the job's own values, the
|
||||
application profile, the queue's policy and the project; output is the
|
||||
job's **effective** values: resreq (select / rusage / span / same /
|
||||
order), task range, execution limits, pre/post-exec, environment.
|
||||
- **Merge rules (simplified, user decision 2026-10-09; deliberately not
|
||||
LSF's full per-section rules):**
|
||||
- The resreq has two levels only, app profile and job (queues carry no
|
||||
resreq, see below). The job overrides the app section by section;
|
||||
`select` is the exception: both apply (ANDed).
|
||||
- Everything else: job overrides app overrides queue defaults.
|
||||
- Queue **hard limits** and queue **task limits** are ceilings: a job (or
|
||||
app) value outside them is rejected at submit, never silently clipped.
|
||||
- The application profile's task limits (min/default/max) also bound the
|
||||
job's `-n`; with no `-n`, the app default applies (LSF p660 semantics).
|
||||
- Where the simplified rules diverge from LSF, the requirements list the
|
||||
divergence explicitly (input for `lsf-config-migration`).
|
||||
- **Live merge (user decision 2026-10-09).** The WAL keeps the job's own
|
||||
values plus the names of its queue, profile and project. The merge runs:
|
||||
- at submit, in the controller (authoritative for every submit path) and
|
||||
early in the apiserver, to **validate** and reject with a clear error;
|
||||
- in the scheduler, to compute the effective values it schedules with, and
|
||||
again whenever the profile, queue or project config changes (live config
|
||||
upgrade). A pending job that no longer satisfies the merged values pends
|
||||
with a clear reason (LSF: "Job no longer satisfies TASKLIMIT
|
||||
configuration"); running jobs are unaffected.
|
||||
- **Queue is policy only (user decision 2026-10-09).** Queue `res_req` is
|
||||
deprecated: warned on by `xceadmin config build` first, then rejected.
|
||||
The queue limits `xceadmin` drops today (cpulimit, memlimit, runlimit,
|
||||
task limit) are wired through config, capnp and the merge as hard limits.
|
||||
- **Visibility.** Job details (REST, `xcecli`, REPL `info job`, decision
|
||||
trace) show the effective values and, for each, which level it came from.
|
||||
|
||||
## Approach
|
||||
A pure, allocation-light C merge library in `src/lib` (shared by controller,
|
||||
scheduler and, through cgo, the apiserver) working on parsed resreqs, with a
|
||||
small resolved-values struct. The scheduler caches the effective values per
|
||||
job and invalidates them on config change; the controller calls the same
|
||||
function for validation only. Effective values are derived, never persisted,
|
||||
so WAL and snapshot keep only inputs (field completeness: the job's own
|
||||
values and the three names must round-trip).
|
||||
|
||||
## Scope
|
||||
- **In**: the merge library and its rules; controller and apiserver submit
|
||||
validation; scheduler effective-value computation and re-evaluation on
|
||||
config change; queue hard/task limit wiring (YAML, schema, `mapping.py`,
|
||||
capnp, C); queue `res_req` deprecation; effective values in job details and
|
||||
the decision trace; tests.
|
||||
- **Out**: the profile model itself (`application-profile`); project
|
||||
definitions (`project-config`); `span` placement semantics
|
||||
(`parjob-placement`); LSF compound/alternative merge stages; fairshare.
|
||||
|
||||
## Boundary Candidates
|
||||
- Merge library (pure functions, unit-tested in `src/lib/testsuite`).
|
||||
- Submit-time validation (controller, apiserver).
|
||||
- Scheduler effective-value cache and config-change re-evaluation.
|
||||
- Queue policy config wiring and `res_req` deprecation.
|
||||
- Display of effective values.
|
||||
|
||||
## Out of Boundary
|
||||
- Defining profiles or projects; enforcing execution limits on the worker
|
||||
(cgroup mapping already exists through exdata).
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream**: resreq parser (`src/lib/resreq.c`, `section.peg`), config
|
||||
pipeline (`admin-config-tool`, `config-upgrade-snapshot`), controller submit
|
||||
path, scheduler job intake (`core_events.c`).
|
||||
- **Downstream**: `application-profile` (plugs the profile level in),
|
||||
`project-config` (project defaults), `parjob-placement` (reads the effective
|
||||
`span`), `job-status-detail-display`, `lsf-config-migration`.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends**: none.
|
||||
- **Adjacent**: `resreq-maxtasks-wiring` (task counts), `scheduler-core`,
|
||||
`scheduler-snapshot-job-allocation`, `scheduler-sim-decision-trace`.
|
||||
|
||||
## Constraints
|
||||
- Additive Cap'n Proto only; old WAL records decode as "no profile, default
|
||||
project", which is today's behaviour.
|
||||
- Scheduler driver equivalence (`scheduler-driver-equivalence.md`): live and
|
||||
simulator must compute identical effective values.
|
||||
- Live and simulated re-evaluation on config change must not block the
|
||||
delaymgr (`ha-async-io.md`).
|
||||
- Touches persistence and scheduling: full 3-phase workflow.
|
||||
|
|
@ -60,6 +60,49 @@ At most about two plans plus the existing N. Configurable: the preference betwee
|
|||
- **Decision trace.** The trace should name the heuristic that produced each plan, e.g. `plan 1 (min-hosts) [4,1]`, so pend explanations stay readable.
|
||||
- **Pend reasons.** Still recorded once per job (`add_job_preason_once`, bug-1927).
|
||||
|
||||
## Task-range search and span constraints (discussion notes, 2026-10-09)
|
||||
|
||||
**Split (agreed).** bug-1931 (`e3b87d28`) already reuses the plan search for a
|
||||
fixed task count (`sched_alloc_count_()`, `alloc_runner.c`) and bisects the
|
||||
range (`sched_alloc_range_()`). Every span constraint (ptile, hosts=1,
|
||||
block, host-count limit, capacity-driven shapes) lives in the planner **for a
|
||||
fixed N**; the range search only chooses N. The search needs one contract
|
||||
from the planner: over the job's *allowed* counts, "fits with N ⇒ fits with
|
||||
every smaller allowed count". Each planned shape keeps it (balanced,
|
||||
capacity-greedy, ptile, hosts=1, nhosts max), provided the allowed counts are
|
||||
right. The range search asks the planner (optional allocator hook, default
|
||||
"every N in [min, max]") for the ordered allowed counts and bisects over that
|
||||
list.
|
||||
|
||||
**LSF 10.1 reference** (`../misc/lsf/IBM_Spectrum_LSF_10.1_Administering_LSF_Admin_Guide.pdf`,
|
||||
PDF pages):
|
||||
- `span[block=n]` (pp. 340-341): only multiples of n are allocated.
|
||||
`-n 2,13` → 4, 8 or 12; `-n 5`, `-n 2`, `-n 3`, `-n 2,3`, `-n 9,10` are
|
||||
rejected; the range minimum is silently rounded up (`-n 2,8` → `4,8`).
|
||||
Packing: as many blocks as fit on a host, then the next; each host checked
|
||||
once (`-n 9 block=3`, hosts with 6 free → 6 + 3). `block` cannot be combined
|
||||
with `ptile` or `hosts`. Allowed counts: multiples of n.
|
||||
- `span[ptile=n]` with a range (p. 339, written for resizable jobs, stated to
|
||||
apply to the first allocation as well): either the maximum (n per host, one
|
||||
host with fewer) or a multiple of n below it (`-n 1,14 ptile=4` → 14, or 4,
|
||||
8, 12). Allowed counts: multiples of n plus max (requirements decide whether
|
||||
to follow this or allow every N).
|
||||
- `span[stripe]` / `span[stripe=max]` (p. 337): even spread over the free
|
||||
candidate hosts, extra tasks to the first hosts; `stripe=max` caps per-host
|
||||
tasks. **Candidate scope item** (close to today's balanced plans).
|
||||
- Host-count limit: LSF has only `span[hosts=1]` per job; the host limit is the
|
||||
queue's `HOSTLIMIT_PER_JOB` (p. 663), which pends a job that needs more
|
||||
hosts and rejects a ptile job whose host count is known at submit (the
|
||||
guide's `-n 1,4096 ptile=1` example instead says the job is limited to 20
|
||||
hosts; pick one). A per-job min/max host count would be an XCE extension.
|
||||
- Submission must reject a task range containing no allowed count.
|
||||
|
||||
**Where `span` comes from** (`/kiro-discovery` 2026-10-09): the job's
|
||||
effective resreq produced by `job-config-merge`: the application profile's
|
||||
resreq (`application-profile`) overridden section by section by the job's
|
||||
`-R`; queues carry no resreq. A host-count limit from queue policy is a queue
|
||||
hard limit in that merge.
|
||||
|
||||
Status: initialized only; requirements to be discussed with the user before `/kiro-spec-requirements`.
|
||||
|
||||
## Requirements
|
||||
|
|
|
|||
85
.kiro/specs/project-config/brief.md
Normal file
85
.kiro/specs/project-config/brief.md
Normal file
|
|
@ -0,0 +1,85 @@
|
|||
# Brief: project-config
|
||||
|
||||
(`/kiro-discovery`, 2026-10-09, Path E "configuration layering". Reference:
|
||||
IBM Spectrum LSF 10.1 Administering LSF pp. 107 (default project), 446-453
|
||||
(resource allocation limits, PROJECTS / PER_PROJECT); Using LSF on UNIX p21
|
||||
(`bsub -P`).)
|
||||
|
||||
## Problem
|
||||
Projects are meant to control **resource ownership**: who may charge work to
|
||||
a project and how much of the cluster a project may use. Today a project is
|
||||
only a free-text label the user types; anyone can name any project, project
|
||||
limits cannot be configured end to end, and nothing shows usage per project.
|
||||
|
||||
## Current State
|
||||
- Submit data carries `project_name` (`JobSubmitData`, `controller.capnp:582`
|
||||
→ `job_submit_data_t`, `types.h:356-368`); `xcecli jobs submit -P`
|
||||
(`xcecli/cmd/jobs/submit.go:60`); the apiserver passes it through
|
||||
(`backend/convert.go:307-317`) without any check.
|
||||
- The scheduler maps an empty name to `"default"` (`sched_get_project`,
|
||||
`scheduler/src/core.c:617-623`). `SchedProject` (capnp:1260) has name and
|
||||
priority but no config source.
|
||||
- Limits already accept project scopes (`LimitConfig`, `limits.yaml`;
|
||||
`CONSUMER_PROJECT`, `lib/xcelimit.c:23-26`, vres key `/p=<project>`), but
|
||||
the scheduler's project-group manager `pgmgr` is created empty
|
||||
(`core.c:185`) and never loaded (`scheduler.c:1144-1145` loads only
|
||||
`ugmgr`/`hgmgr`), so project limits cannot be configured to work.
|
||||
- No project fairshare, accounting view or access check exists.
|
||||
|
||||
## Desired Outcome (v1 = quota; user decision 2026-10-09)
|
||||
- **Project definitions** in the cluster configuration: name, description,
|
||||
members (users and user groups), priority; plus a cluster default project.
|
||||
Schema, YAML, JSON schema, `xceadmin` mapping, live-upgrade replication.
|
||||
- **Project selection** at submit: `-P` > the user's default project (if
|
||||
defined) > the cluster default > `"default"` (LSF p107 order, without
|
||||
environment variables).
|
||||
- **Access check** at submit: the user must be a member of the project
|
||||
(apiserver early check, controller authoritative), with a clear error;
|
||||
`cluster_admin` keeps its existing bypass. Whether this is RBAC scope
|
||||
(`project` scope type) or config membership is settled in requirements.
|
||||
- **Quotas**: the scheduler loads project groups from config so
|
||||
`PROJECTS` / `PER_PROJECT` limits in `limits.yaml` work end to end
|
||||
(slots, jobs, mem, any resource), with pend reasons naming the project.
|
||||
- **Visibility**: project shown in job details and listings; per-project
|
||||
usage against its limits (REST, `xcecli`, REPL `info limit`).
|
||||
- **Merge input**: the project is one level for `job-config-merge` (default
|
||||
project; any project-level defaults decided in requirements).
|
||||
|
||||
## Approach
|
||||
Follow the queue/usergroup config pattern (`config-design.md`): projects are
|
||||
config entities pushed to controller and scheduler (`Conf4Sched`), the
|
||||
scheduler fills `pgmgr` from them, and the existing limit machinery does the
|
||||
rest. Projects already reach the scheduler on each job, so the WAL needs no
|
||||
new job field.
|
||||
|
||||
## Scope
|
||||
- **In**: project config model and pipeline; default project resolution;
|
||||
submit access check; `pgmgr` loading; project limits end to end; display;
|
||||
tests (unit, sim, live).
|
||||
- **Out**: guaranteed shares, owned pools, borrowing and reclaim
|
||||
(`project-guarantee`); fairshare; accounting history/billing.
|
||||
|
||||
## Boundary Candidates
|
||||
- Config model and pipeline.
|
||||
- Submit resolution and access check (apiserver, controller).
|
||||
- Scheduler project groups and limits.
|
||||
- Display surfaces.
|
||||
|
||||
## Out of Boundary
|
||||
- Scheduling-order policy (queues), application settings (profiles).
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream**: config pipeline (`admin-config-tool`, `config-upgrade-snapshot`),
|
||||
limit plugin (`plugins/limit.c`, `lib/xcelimit.c`), RBAC (`rbac_check.go`).
|
||||
- **Downstream**: `job-config-merge` (project level), `project-guarantee`,
|
||||
`lsf-config-migration`, `rbac-persistence` (if access uses RBAC scopes).
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends**: none.
|
||||
- **Adjacent**: `scheduler-sim-diagnostics` (`info limit`),
|
||||
`resource-distribution-mapping`, `rbac-persistence`.
|
||||
|
||||
## Constraints
|
||||
- Middleware chain unchanged; additive Cap'n Proto; old jobs keep their
|
||||
recorded project name even if the project is later removed from config
|
||||
(requirements decide what a pending job with a removed project does).
|
||||
54
.kiro/specs/project-guarantee/brief.md
Normal file
54
.kiro/specs/project-guarantee/brief.md
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
# Brief: project-guarantee
|
||||
|
||||
(`/kiro-discovery`, 2026-10-09, Path E "configuration layering". Backlog:
|
||||
not started until `project-config` and `scheduler-preemption` land.)
|
||||
|
||||
## Problem
|
||||
With `project-config`, projects can only be capped. Owners of a project (a
|
||||
group that funded hardware, a department with a committed share) need a
|
||||
**guarantee**: a minimum share or a pool of resources they get when they ask
|
||||
for it, while idle capacity is still lent to others.
|
||||
|
||||
## Current State
|
||||
- No guarantee, share or pool concept exists; fairshare is in the queue YAML
|
||||
schema only and dropped by `xceadmin` (`mapping.py:80-97`).
|
||||
- Preemption is specified but not implemented (`scheduler-preemption`,
|
||||
requirements generated).
|
||||
|
||||
## Desired Outcome
|
||||
- A project can own a guarantee: a resource amount (slots, hosts or any
|
||||
resource) or a pool of hosts.
|
||||
- Idle guaranteed capacity may be borrowed by other projects' jobs.
|
||||
- When an owner's pending job needs its guarantee back, borrowers are
|
||||
reclaimed (preempted, requeued or allowed to finish, by policy).
|
||||
- Guarantee state and borrowing are visible (REST, `xcecli`, REPL).
|
||||
|
||||
## Approach
|
||||
To be chosen in discovery when the spec starts. Candidates: LSF-style
|
||||
guarantee service classes (`lsb.resources` GuaranteedResourcePool), or
|
||||
guarantees as reserved virtual resources built on the limit/vres machinery
|
||||
plus preemption.
|
||||
|
||||
## Scope
|
||||
- **In**: guarantee model, borrowing, reclaim policy, visibility.
|
||||
- **Out**: quotas (`project-config`), fairshare ordering, accounting/billing.
|
||||
|
||||
## Boundary Candidates
|
||||
- Guarantee model and config.
|
||||
- Scheduler accounting of owned vs borrowed usage.
|
||||
- Reclaim through `scheduler-preemption`.
|
||||
|
||||
## Out of Boundary
|
||||
- Project definitions and membership (`project-config`).
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream**: `project-config`, `scheduler-preemption`, `job-config-merge`.
|
||||
- **Downstream**: `lsf-config-migration` (guarantee pools).
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends**: none.
|
||||
- **Adjacent**: `scheduler-preemption`, `resource-distribution-mapping`.
|
||||
|
||||
## Constraints
|
||||
- Scheduler driver equivalence (live and simulator identical).
|
||||
- Touches scheduling policy and HA state: full 3-phase workflow.
|
||||
|
|
@ -70,11 +70,19 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
|
|||
- [ ] job-dependency-insight — NEW 2026-09-29. `bjdepinfo`-style query (parents/children/-p blocking/-r levels/-l verdicts, RBAC-filtered) via REST/xcecli/harness, dependency info in job details, optional orphan termination (per-job `-ti` + cluster grace period, default off; EXIT reason ORPHAN; failover-safe persisted grace start; minimal primary periodic work). Brief: `.kiro/specs/job-dependency-insight/brief.md`. Dependencies: job-dependency, job-time-window (exit-reason field).
|
||||
- [ ] controller-job-cleanup — NEW 2026-09-29 (`/kiro-discovery`, Path C). Primary-only periodic sweep (configurable `job_cleanup_period`, default 300 s) removes DONE/EXIT jobs finished longer than configurable `job_retention` (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever `jobmgr` delete; refcount-safe `xkvmgr` delete), propagated to scheduler and apiserver (`OnJobDeleted`). Retires the node-local snapshot-only `job_should_retain` finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: `.kiro/specs/controller-job-cleanup/brief.md`. Dependencies: none (extends ha-entity-snapshots).
|
||||
- [ ] job-status-detail-display — NEW 2026-09-28 (`/kiro-discovery`, Path E). End users (REST/`xcecli`) see NO pend reason for ANY job today: the apiserver cgo conversion (`backend/convert.go`) never maps `xce_job_t.pend_reasons`, so `pending_reason` is always empty. Maps it to a structured JSON (including `alternatives` from day one) using libxce's code→text mapping, and renders it in `xcecli` job detail. Open for requirements: also map `run_usage` → `JobRusage` (same gap; reserved vs. used for RUN jobs). Brief: `.kiro/specs/job-status-detail-display/brief.md`. Dependencies: none for flat reasons; scheduler-alter-pend-reasons for per-alternative rendering.
|
||||
- [ ] job-config-merge — NEW 2026-10-09 (`/kiro-discovery`, Path E "configuration layering"; ref LSF 10.1 Admin Guide pp. 314-316, 435-437, 561-563, 643, 660). One merge function in `src/lib` turning the job's own values + application profile + queue policy + project into effective values (resreq incl. `span`, task range, limits, pre/post-exec, env). Simplified rules (user decision): resreq is two-level (profile, job), job overrides section by section, `select` ANDed; queue hard limits and task limits are ceilings that reject. **Live** merge (user decision): WAL keeps inputs + names; controller/apiserver validate at submit; scheduler computes effective values and re-evaluates pending jobs on config change. Queue is policy only: queue `res_req` deprecated, dropped queue limits (cpulimit/memlimit/runlimit/task limit) wired. Brief: `.kiro/specs/job-config-merge/brief.md`. Dependencies: none.
|
||||
- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (members, priority, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config so `PROJECTS`/`PER_PROJECT` limits work end to end, per-project usage display. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none.
|
||||
|
||||
## Existing Spec Updates (2026-10-09 configuration layering discovery)
|
||||
- [ ] application-profile — the profile carries the job's resource requirement (select/rusage/`span`/same) and task limits (min/default/max); "resolved and frozen at submit" replaced by the live merge of `job-config-merge` (brief updated). Dependencies: job-config-merge.
|
||||
- [ ] parjob-placement — `span` read from the effective resreq; span constraints in the fixed-N planner; range search bisects over the planner's allowed counts (block: multiples; ptile: multiples + max); LSF 10.1 findings recorded in its requirements.md notes; `span[stripe]` candidate. Dependencies: job-config-merge (for `span` from profiles; job-level `-R` span can land first).
|
||||
|
||||
## Existing Spec Updates (2026-09-28 `info job` discovery)
|
||||
- [x] scheduler-sim-diagnostics — `info job <jobid>` enhancement: a PEND job shows its pend reason set (per alternative once scheduler-alter-pend-reasons lands; shows the flat reason until then); a RUN job shows per-host/per-resource reserved (`job->allocs`) vs. real used (`job->rusages`), and when no rusage has been reported yet says "resource usage not reported yet" (no fabricated zeros; reserved still shown). Byte units per its Req 6. Read-only. Next: `/kiro-spec-requirements scheduler-sim-diagnostics`. Dependencies: none (alternatives display depends on scheduler-alter-pend-reasons).
|
||||
|
||||
## Future Specs / Backlog (dependency order)
|
||||
- [ ] project-guarantee — NEW 2026-10-09 (configuration layering, backlog). Project-owned guarantees (resource amount or host pool), borrowing of idle guaranteed capacity, reclaim through preemption. Brief: `.kiro/specs/project-guarantee/brief.md`. Dependencies: project-config, scheduler-preemption.
|
||||
- [ ] lsf-config-migration — NEW 2026-10-09 (user idea, backlog; no brief yet). A tool (likely an `xceadmin` subcommand) that converts LSF configuration (`lsb.queues`, `lsb.applications`, `lsb.resources` Limit sections, `lsb.users`/`lsb.hosts` groups, `lsb.params` defaults, project settings) into XCE `configs/*.yaml`, with a report of what does not map: queue `RES_REQ` (XCE queues are policy only; move into a profile or drop), LSF's full per-section merge rules vs XCE's simplified ones, compound/alternative resreqs, unsupported parameters. Dependencies: job-config-merge, application-profile, project-config, parjob-placement (span), admin-config-tool. Promote via `/kiro-discovery` when those land.
|
||||
|
||||
- [ ] **xce-harness-component** — **PROMOTED to spec 2026-09-13** (`/kiro-discovery`; brief at `.kiro/specs/xce-harness-component/brief.md`). Elevate `xce_test` (the REPL-driven wire-protocol test client used by HA soak / multi-host / resource-value verification) out of `src/controller/testsuite/` into its own first-class `src/harness/` component — own `meson.build`, `CLAUDE.md`, and unit-test binary (carved out of `controller_unit_test`, which today recompiles ~14 of xce_test's own files a second time paired with white-box tests) — fully decoupled from `/xce-build controller` (today's implicit "controller always builds xce_test" coupling, `build-deployment.md:24`). Verified finding: none of xce_test's 15 source files include any controller-private header — only libxce public headers + generated capnp schema — so this is a pure relocation + build-graph split, not a dependency untangling problem, and no xce_test behavior changes. Touches `/xce-test-ha`, `/xce-deploy-ha`, `/xce-test-multihost`, `/xce-test-resources` (each must build `harness` explicitly once decoupled) and is adjacent to (not overlapping) the three existing behavior specs `xce-test-cli-ux`/`xce-test-e2e-primitives`/`xce-test-rc-file`, whose file-path references go stale after the move. Next: `/kiro-spec-init xce-harness-component`.
|
||||
- [x] config-upgrade-snapshot — **descoped from `ha-term-safety` R8 on 2026-06-21 (requirements generated; `.kiro/specs/config-upgrade-snapshot/`).** Closes **bug-857**: a configuration upgrade can leave members with DIVERGENT materialized state because today it does config-push + restart + per-node WAL replay under the new config (`config_repl.c` `do_upgrade_config_with_bytes` ~1001-1163), with no snapshot pinning an identical post-upgrade state (replay is config-dependent — `lost_and_found` re-homes orphaned entities). Approach: an upgrade materializes a **full entity-bearing state snapshot at the config-change marker's `(term,lsn)` boundary**, embeds the new config, and installs it on every member via the `snapshot_transfer_s` transport — no per-node re-derivation; supersedes `ha-term-safety` R7's interim config-push byte delivery for upgrades. **Former HARD dependency `entity-bearing-snapshots` is CLEARED** — it shipped as spec `ha-entity-snapshots` (**IMPLEMENTATION-COMPLETE, validate-impl GO 2026-07-18**; see Future Specs/Backlog below), and `ha-term-safety` (the config-change marker it anchors on) is also implemented. `config-upgrade-snapshot`'s own `spec.json` now has `blocked_on: []`. Verify by reproducing bug-857 (upgrade during chaotic multi-kill, diff materialized state). **Ready to promote to design.**
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue