docs(specs): discovery for layered job configuration (queue/app/project)

New briefs: job-config-merge (one live merge of job, application profile,
queue policy and project into effective values; simplified rules: job
overrides profile, select ANDed, queue hard/task limits reject; queue is
policy only, res_req deprecated), project-config (project quotas v1) and
project-guarantee (backlog).

application-profile: the profile carries the job's resreq (incl. span) and
task limits; "frozen at submit" replaced by the live merge.
parjob-placement: span constraints live in the fixed-N planner and the
range search bisects over the planner's allowed counts; LSF 10.1 block,
ptile, stripe and HOSTLIMIT_PER_JOB findings recorded.
Roadmap: new specs, existing-spec updates, lsf-config-migration backlog.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-10 03:14:16 +00:00
parent a219419e59
commit b761878b58
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
6 changed files with 341 additions and 2 deletions

View file

@ -5,6 +5,27 @@ mode is an attribute of the job, determined by the **application** the user runs
so it is carried by a named application profile rather than by a bare mode
option.)
**Update 2026-10-09 (`/kiro-discovery`, configuration layering; user
decisions):**
- Configuration is layered LSF-style: **queue** = scheduling policy only (no
resreq; queue `res_req` is deprecated), **application profile** =
application execution constraints *including the job's resource
requirement* (select / rusage / `span[...]` / same) and task limits
(min / default / max, LSF TASKLIMIT), **project** = resource ownership
(`project-config`).
- How the levels combine is owned by **`job-config-merge`**: job overrides
profile section by section, `select` is ANDed, queue hard and task limits
are ceilings that reject. This answers the open question "how resource
defaults combine with a job's own resreq" below.
- **Replaces "Resolution at submit time"** below: the merge is *live*. The WAL
keeps the job's own values plus the profile name; the controller and
apiserver validate at submit, and the scheduler computes effective values
and recomputes them when a profile changes (a pending job that no longer
satisfies its profile pends with a reason; running jobs are unaffected).
Execution-side values (mode, options, launch type, env, pre/post-exec) are
still taken from the profile at dispatch time; requirements decide whether
dispatch freezes them on the job record.
## Problem
XCE users run many kinds of applications: trusted in-house codes, untrusted
scripts, vendor binaries, MPI solvers. Each needs a different way of running:
@ -175,8 +196,8 @@ fits XCE's LSF-style job model.
- Which extra attributes go into v1 beyond the four ("maybe others"). Examples:
requeue exit values, runtime estimate, job starter or wrapper, output path
template, checkpoint settings.
- How resource defaults combine with a job's own resreq (replace, or fill in
what's missing).
- ~~How resource defaults combine with a job's own resreq~~ — settled
2026-10-09, owned by `job-config-merge`.
- Profile versioning and audit.
- Whether the profile can be changed for a pending job.
- **Out**:
@ -215,6 +236,8 @@ fits XCE's LSF-style job model.
## Existing Spec Touchpoints
- **Extends**: none directly.
- **Depends on**: `job-config-merge` (merge rules and live re-evaluation).
- **Feeds**: `parjob-placement` (the profile's `span[...]`).
- **Adjacent**:
- `job-status-detail-display`
- `controller-job-cleanup`

View file

@ -0,0 +1,126 @@
# Brief: job-config-merge
(`/kiro-discovery`, 2026-10-09, Path E "configuration layering". Reference:
IBM Spectrum LSF 10.1 Administering LSF, `../misc/lsf/`, pp. 314-316 (resreq
merge), 435-437 (application profile vs queue), 561-563 (hard/soft limits),
643 and 660 (TASKLIMIT). Page numbers are PDF pages.)
## Problem
A job's settings will come from several places, as in LSF:
- the **job** itself (submit options, `-R`, `-n min,max`, limits);
- its **application profile** (`application-profile`): what the
application needs to run, including its resource requirement (`span`
included) and task limits;
- its **queue**: scheduling policy only (priority, ordering, host partition,
hard limits, task limits, windows);
- its **project** (`project-config`): resource ownership (quotas now,
guarantees later).
Nothing in XCE combines settings from more than one place today, so every
level would invent its own precedence, and the scheduler, controller,
apiserver and display would disagree about what a job actually asked for.
## Current State
- No code merges two resreqs (`src/lib/resreq.c`). The only defaults:
the scheduler turns an empty resreq into `"type == any"`
(`scheduler/src/core_events.c:300-309`, also `schedobj.c:198`,
`snapshot_load.c:344`) and `apply_job_task_count()` (`resreq.c:574-591`)
overwrites the primary's min/max tasks from the job.
- Queues carry a `resreq` string end to end (YAML `res_req`, capnp
`QueueConf.resreq`, `xce_queue_t`, scheduler snapshot) but nothing reads
it. `make_sched_queue()` (`core_events.c:377-388`) says wiring it is "a
later task".
- `xceadmin` drops queue cpulimit, memlimit, runlimit, runtimeest, fairshare,
gpu_req, nice, job_starter, reservation and preemption
(`xceadmin/src/xcecommon/mapping.py:80-97`).
- The controller does not resolve anything at submit: `handle_jobsubmit`
(`controller/job.c:2577`) checks user/queue/cmdline are non-null and
`build_job_submit_payloads` (`job.c:347-371`) persists the client's
`job_submit_data_t` unchanged in the WAL. The controller does not even
check that the queue exists (the apiserver does, `jobs.go:1299-1322`).
- The apiserver only checks resreq syntax (`backend/resreq_check.go:22`).
## Desired Outcome
- **One merge function** in `src/lib`: inputs are the job's own values, the
application profile, the queue's policy and the project; output is the
job's **effective** values: resreq (select / rusage / span / same /
order), task range, execution limits, pre/post-exec, environment.
- **Merge rules (simplified, user decision 2026-10-09; deliberately not
LSF's full per-section rules):**
- The resreq has two levels only, app profile and job (queues carry no
resreq, see below). The job overrides the app section by section;
`select` is the exception: both apply (ANDed).
- Everything else: job overrides app overrides queue defaults.
- Queue **hard limits** and queue **task limits** are ceilings: a job (or
app) value outside them is rejected at submit, never silently clipped.
- The application profile's task limits (min/default/max) also bound the
job's `-n`; with no `-n`, the app default applies (LSF p660 semantics).
- Where the simplified rules diverge from LSF, the requirements list the
divergence explicitly (input for `lsf-config-migration`).
- **Live merge (user decision 2026-10-09).** The WAL keeps the job's own
values plus the names of its queue, profile and project. The merge runs:
- at submit, in the controller (authoritative for every submit path) and
early in the apiserver, to **validate** and reject with a clear error;
- in the scheduler, to compute the effective values it schedules with, and
again whenever the profile, queue or project config changes (live config
upgrade). A pending job that no longer satisfies the merged values pends
with a clear reason (LSF: "Job no longer satisfies TASKLIMIT
configuration"); running jobs are unaffected.
- **Queue is policy only (user decision 2026-10-09).** Queue `res_req` is
deprecated: warned on by `xceadmin config build` first, then rejected.
The queue limits `xceadmin` drops today (cpulimit, memlimit, runlimit,
task limit) are wired through config, capnp and the merge as hard limits.
- **Visibility.** Job details (REST, `xcecli`, REPL `info job`, decision
trace) show the effective values and, for each, which level it came from.
## Approach
A pure, allocation-light C merge library in `src/lib` (shared by controller,
scheduler and, through cgo, the apiserver) working on parsed resreqs, with a
small resolved-values struct. The scheduler caches the effective values per
job and invalidates them on config change; the controller calls the same
function for validation only. Effective values are derived, never persisted,
so WAL and snapshot keep only inputs (field completeness: the job's own
values and the three names must round-trip).
## Scope
- **In**: the merge library and its rules; controller and apiserver submit
validation; scheduler effective-value computation and re-evaluation on
config change; queue hard/task limit wiring (YAML, schema, `mapping.py`,
capnp, C); queue `res_req` deprecation; effective values in job details and
the decision trace; tests.
- **Out**: the profile model itself (`application-profile`); project
definitions (`project-config`); `span` placement semantics
(`parjob-placement`); LSF compound/alternative merge stages; fairshare.
## Boundary Candidates
- Merge library (pure functions, unit-tested in `src/lib/testsuite`).
- Submit-time validation (controller, apiserver).
- Scheduler effective-value cache and config-change re-evaluation.
- Queue policy config wiring and `res_req` deprecation.
- Display of effective values.
## Out of Boundary
- Defining profiles or projects; enforcing execution limits on the worker
(cgroup mapping already exists through exdata).
## Upstream / Downstream
- **Upstream**: resreq parser (`src/lib/resreq.c`, `section.peg`), config
pipeline (`admin-config-tool`, `config-upgrade-snapshot`), controller submit
path, scheduler job intake (`core_events.c`).
- **Downstream**: `application-profile` (plugs the profile level in),
`project-config` (project defaults), `parjob-placement` (reads the effective
`span`), `job-status-detail-display`, `lsf-config-migration`.
## Existing Spec Touchpoints
- **Extends**: none.
- **Adjacent**: `resreq-maxtasks-wiring` (task counts), `scheduler-core`,
`scheduler-snapshot-job-allocation`, `scheduler-sim-decision-trace`.
## Constraints
- Additive Cap'n Proto only; old WAL records decode as "no profile, default
project", which is today's behaviour.
- Scheduler driver equivalence (`scheduler-driver-equivalence.md`): live and
simulator must compute identical effective values.
- Live and simulated re-evaluation on config change must not block the
delaymgr (`ha-async-io.md`).
- Touches persistence and scheduling: full 3-phase workflow.

View file

@ -60,6 +60,49 @@ At most about two plans plus the existing N. Configurable: the preference betwee
- **Decision trace.** The trace should name the heuristic that produced each plan, e.g. `plan 1 (min-hosts) [4,1]`, so pend explanations stay readable.
- **Pend reasons.** Still recorded once per job (`add_job_preason_once`, bug-1927).
## Task-range search and span constraints (discussion notes, 2026-10-09)
**Split (agreed).** bug-1931 (`e3b87d28`) already reuses the plan search for a
fixed task count (`sched_alloc_count_()`, `alloc_runner.c`) and bisects the
range (`sched_alloc_range_()`). Every span constraint (ptile, hosts=1,
block, host-count limit, capacity-driven shapes) lives in the planner **for a
fixed N**; the range search only chooses N. The search needs one contract
from the planner: over the job's *allowed* counts, "fits with N ⇒ fits with
every smaller allowed count". Each planned shape keeps it (balanced,
capacity-greedy, ptile, hosts=1, nhosts max), provided the allowed counts are
right. The range search asks the planner (optional allocator hook, default
"every N in [min, max]") for the ordered allowed counts and bisects over that
list.
**LSF 10.1 reference** (`../misc/lsf/IBM_Spectrum_LSF_10.1_Administering_LSF_Admin_Guide.pdf`,
PDF pages):
- `span[block=n]` (pp. 340-341): only multiples of n are allocated.
`-n 2,13` → 4, 8 or 12; `-n 5`, `-n 2`, `-n 3`, `-n 2,3`, `-n 9,10` are
rejected; the range minimum is silently rounded up (`-n 2,8` → `4,8`).
Packing: as many blocks as fit on a host, then the next; each host checked
once (`-n 9 block=3`, hosts with 6 free → 6 + 3). `block` cannot be combined
with `ptile` or `hosts`. Allowed counts: multiples of n.
- `span[ptile=n]` with a range (p. 339, written for resizable jobs, stated to
apply to the first allocation as well): either the maximum (n per host, one
host with fewer) or a multiple of n below it (`-n 1,14 ptile=4` → 14, or 4,
8, 12). Allowed counts: multiples of n plus max (requirements decide whether
to follow this or allow every N).
- `span[stripe]` / `span[stripe=max]` (p. 337): even spread over the free
candidate hosts, extra tasks to the first hosts; `stripe=max` caps per-host
tasks. **Candidate scope item** (close to today's balanced plans).
- Host-count limit: LSF has only `span[hosts=1]` per job; the host limit is the
queue's `HOSTLIMIT_PER_JOB` (p. 663), which pends a job that needs more
hosts and rejects a ptile job whose host count is known at submit (the
guide's `-n 1,4096 ptile=1` example instead says the job is limited to 20
hosts; pick one). A per-job min/max host count would be an XCE extension.
- Submission must reject a task range containing no allowed count.
**Where `span` comes from** (`/kiro-discovery` 2026-10-09): the job's
effective resreq produced by `job-config-merge`: the application profile's
resreq (`application-profile`) overridden section by section by the job's
`-R`; queues carry no resreq. A host-count limit from queue policy is a queue
hard limit in that merge.
Status: initialized only; requirements to be discussed with the user before `/kiro-spec-requirements`.
## Requirements

View file

@ -0,0 +1,85 @@
# Brief: project-config
(`/kiro-discovery`, 2026-10-09, Path E "configuration layering". Reference:
IBM Spectrum LSF 10.1 Administering LSF pp. 107 (default project), 446-453
(resource allocation limits, PROJECTS / PER_PROJECT); Using LSF on UNIX p21
(`bsub -P`).)
## Problem
Projects are meant to control **resource ownership**: who may charge work to
a project and how much of the cluster a project may use. Today a project is
only a free-text label the user types; anyone can name any project, project
limits cannot be configured end to end, and nothing shows usage per project.
## Current State
- Submit data carries `project_name` (`JobSubmitData`, `controller.capnp:582`
→ `job_submit_data_t`, `types.h:356-368`); `xcecli jobs submit -P`
(`xcecli/cmd/jobs/submit.go:60`); the apiserver passes it through
(`backend/convert.go:307-317`) without any check.
- The scheduler maps an empty name to `"default"` (`sched_get_project`,
`scheduler/src/core.c:617-623`). `SchedProject` (capnp:1260) has name and
priority but no config source.
- Limits already accept project scopes (`LimitConfig`, `limits.yaml`;
`CONSUMER_PROJECT`, `lib/xcelimit.c:23-26`, vres key `/p=<project>`), but
the scheduler's project-group manager `pgmgr` is created empty
(`core.c:185`) and never loaded (`scheduler.c:1144-1145` loads only
`ugmgr`/`hgmgr`), so project limits cannot be configured to work.
- No project fairshare, accounting view or access check exists.
## Desired Outcome (v1 = quota; user decision 2026-10-09)
- **Project definitions** in the cluster configuration: name, description,
members (users and user groups), priority; plus a cluster default project.
Schema, YAML, JSON schema, `xceadmin` mapping, live-upgrade replication.
- **Project selection** at submit: `-P` > the user's default project (if
defined) > the cluster default > `"default"` (LSF p107 order, without
environment variables).
- **Access check** at submit: the user must be a member of the project
(apiserver early check, controller authoritative), with a clear error;
`cluster_admin` keeps its existing bypass. Whether this is RBAC scope
(`project` scope type) or config membership is settled in requirements.
- **Quotas**: the scheduler loads project groups from config so
`PROJECTS` / `PER_PROJECT` limits in `limits.yaml` work end to end
(slots, jobs, mem, any resource), with pend reasons naming the project.
- **Visibility**: project shown in job details and listings; per-project
usage against its limits (REST, `xcecli`, REPL `info limit`).
- **Merge input**: the project is one level for `job-config-merge` (default
project; any project-level defaults decided in requirements).
## Approach
Follow the queue/usergroup config pattern (`config-design.md`): projects are
config entities pushed to controller and scheduler (`Conf4Sched`), the
scheduler fills `pgmgr` from them, and the existing limit machinery does the
rest. Projects already reach the scheduler on each job, so the WAL needs no
new job field.
## Scope
- **In**: project config model and pipeline; default project resolution;
submit access check; `pgmgr` loading; project limits end to end; display;
tests (unit, sim, live).
- **Out**: guaranteed shares, owned pools, borrowing and reclaim
(`project-guarantee`); fairshare; accounting history/billing.
## Boundary Candidates
- Config model and pipeline.
- Submit resolution and access check (apiserver, controller).
- Scheduler project groups and limits.
- Display surfaces.
## Out of Boundary
- Scheduling-order policy (queues), application settings (profiles).
## Upstream / Downstream
- **Upstream**: config pipeline (`admin-config-tool`, `config-upgrade-snapshot`),
limit plugin (`plugins/limit.c`, `lib/xcelimit.c`), RBAC (`rbac_check.go`).
- **Downstream**: `job-config-merge` (project level), `project-guarantee`,
`lsf-config-migration`, `rbac-persistence` (if access uses RBAC scopes).
## Existing Spec Touchpoints
- **Extends**: none.
- **Adjacent**: `scheduler-sim-diagnostics` (`info limit`),
`resource-distribution-mapping`, `rbac-persistence`.
## Constraints
- Middleware chain unchanged; additive Cap'n Proto; old jobs keep their
recorded project name even if the project is later removed from config
(requirements decide what a pending job with a removed project does).

View file

@ -0,0 +1,54 @@
# Brief: project-guarantee
(`/kiro-discovery`, 2026-10-09, Path E "configuration layering". Backlog:
not started until `project-config` and `scheduler-preemption` land.)
## Problem
With `project-config`, projects can only be capped. Owners of a project (a
group that funded hardware, a department with a committed share) need a
**guarantee**: a minimum share or a pool of resources they get when they ask
for it, while idle capacity is still lent to others.
## Current State
- No guarantee, share or pool concept exists; fairshare is in the queue YAML
schema only and dropped by `xceadmin` (`mapping.py:80-97`).
- Preemption is specified but not implemented (`scheduler-preemption`,
requirements generated).
## Desired Outcome
- A project can own a guarantee: a resource amount (slots, hosts or any
resource) or a pool of hosts.
- Idle guaranteed capacity may be borrowed by other projects' jobs.
- When an owner's pending job needs its guarantee back, borrowers are
reclaimed (preempted, requeued or allowed to finish, by policy).
- Guarantee state and borrowing are visible (REST, `xcecli`, REPL).
## Approach
To be chosen in discovery when the spec starts. Candidates: LSF-style
guarantee service classes (`lsb.resources` GuaranteedResourcePool), or
guarantees as reserved virtual resources built on the limit/vres machinery
plus preemption.
## Scope
- **In**: guarantee model, borrowing, reclaim policy, visibility.
- **Out**: quotas (`project-config`), fairshare ordering, accounting/billing.
## Boundary Candidates
- Guarantee model and config.
- Scheduler accounting of owned vs borrowed usage.
- Reclaim through `scheduler-preemption`.
## Out of Boundary
- Project definitions and membership (`project-config`).
## Upstream / Downstream
- **Upstream**: `project-config`, `scheduler-preemption`, `job-config-merge`.
- **Downstream**: `lsf-config-migration` (guarantee pools).
## Existing Spec Touchpoints
- **Extends**: none.
- **Adjacent**: `scheduler-preemption`, `resource-distribution-mapping`.
## Constraints
- Scheduler driver equivalence (live and simulator identical).
- Touches scheduling policy and HA state: full 3-phase workflow.

View file

@ -70,11 +70,19 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
- [ ] job-dependency-insight — NEW 2026-09-29. `bjdepinfo`-style query (parents/children/-p blocking/-r levels/-l verdicts, RBAC-filtered) via REST/xcecli/harness, dependency info in job details, optional orphan termination (per-job `-ti` + cluster grace period, default off; EXIT reason ORPHAN; failover-safe persisted grace start; minimal primary periodic work). Brief: `.kiro/specs/job-dependency-insight/brief.md`. Dependencies: job-dependency, job-time-window (exit-reason field).
- [ ] controller-job-cleanup — NEW 2026-09-29 (`/kiro-discovery`, Path C). Primary-only periodic sweep (configurable `job_cleanup_period`, default 300 s) removes DONE/EXIT jobs finished longer than configurable `job_retention` (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever `jobmgr` delete; refcount-safe `xkvmgr` delete), propagated to scheduler and apiserver (`OnJobDeleted`). Retires the node-local snapshot-only `job_should_retain` finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: `.kiro/specs/controller-job-cleanup/brief.md`. Dependencies: none (extends ha-entity-snapshots).
- [ ] job-status-detail-display — NEW 2026-09-28 (`/kiro-discovery`, Path E). End users (REST/`xcecli`) see NO pend reason for ANY job today: the apiserver cgo conversion (`backend/convert.go`) never maps `xce_job_t.pend_reasons`, so `pending_reason` is always empty. Maps it to a structured JSON (including `alternatives` from day one) using libxce's code→text mapping, and renders it in `xcecli` job detail. Open for requirements: also map `run_usage` → `JobRusage` (same gap; reserved vs. used for RUN jobs). Brief: `.kiro/specs/job-status-detail-display/brief.md`. Dependencies: none for flat reasons; scheduler-alter-pend-reasons for per-alternative rendering.
- [ ] job-config-merge — NEW 2026-10-09 (`/kiro-discovery`, Path E "configuration layering"; ref LSF 10.1 Admin Guide pp. 314-316, 435-437, 561-563, 643, 660). One merge function in `src/lib` turning the job's own values + application profile + queue policy + project into effective values (resreq incl. `span`, task range, limits, pre/post-exec, env). Simplified rules (user decision): resreq is two-level (profile, job), job overrides section by section, `select` ANDed; queue hard limits and task limits are ceilings that reject. **Live** merge (user decision): WAL keeps inputs + names; controller/apiserver validate at submit; scheduler computes effective values and re-evaluates pending jobs on config change. Queue is policy only: queue `res_req` deprecated, dropped queue limits (cpulimit/memlimit/runlimit/task limit) wired. Brief: `.kiro/specs/job-config-merge/brief.md`. Dependencies: none.
- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (members, priority, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config so `PROJECTS`/`PER_PROJECT` limits work end to end, per-project usage display. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none.
## Existing Spec Updates (2026-10-09 configuration layering discovery)
- [ ] application-profile — the profile carries the job's resource requirement (select/rusage/`span`/same) and task limits (min/default/max); "resolved and frozen at submit" replaced by the live merge of `job-config-merge` (brief updated). Dependencies: job-config-merge.
- [ ] parjob-placement — `span` read from the effective resreq; span constraints in the fixed-N planner; range search bisects over the planner's allowed counts (block: multiples; ptile: multiples + max); LSF 10.1 findings recorded in its requirements.md notes; `span[stripe]` candidate. Dependencies: job-config-merge (for `span` from profiles; job-level `-R` span can land first).
## Existing Spec Updates (2026-09-28 `info job` discovery)
- [x] scheduler-sim-diagnostics — `info job <jobid>` enhancement: a PEND job shows its pend reason set (per alternative once scheduler-alter-pend-reasons lands; shows the flat reason until then); a RUN job shows per-host/per-resource reserved (`job->allocs`) vs. real used (`job->rusages`), and when no rusage has been reported yet says "resource usage not reported yet" (no fabricated zeros; reserved still shown). Byte units per its Req 6. Read-only. Next: `/kiro-spec-requirements scheduler-sim-diagnostics`. Dependencies: none (alternatives display depends on scheduler-alter-pend-reasons).
## Future Specs / Backlog (dependency order)
- [ ] project-guarantee — NEW 2026-10-09 (configuration layering, backlog). Project-owned guarantees (resource amount or host pool), borrowing of idle guaranteed capacity, reclaim through preemption. Brief: `.kiro/specs/project-guarantee/brief.md`. Dependencies: project-config, scheduler-preemption.
- [ ] lsf-config-migration — NEW 2026-10-09 (user idea, backlog; no brief yet). A tool (likely an `xceadmin` subcommand) that converts LSF configuration (`lsb.queues`, `lsb.applications`, `lsb.resources` Limit sections, `lsb.users`/`lsb.hosts` groups, `lsb.params` defaults, project settings) into XCE `configs/*.yaml`, with a report of what does not map: queue `RES_REQ` (XCE queues are policy only; move into a profile or drop), LSF's full per-section merge rules vs XCE's simplified ones, compound/alternative resreqs, unsupported parameters. Dependencies: job-config-merge, application-profile, project-config, parjob-placement (span), admin-config-tool. Promote via `/kiro-discovery` when those land.
- [ ] **xce-harness-component** — **PROMOTED to spec 2026-09-13** (`/kiro-discovery`; brief at `.kiro/specs/xce-harness-component/brief.md`). Elevate `xce_test` (the REPL-driven wire-protocol test client used by HA soak / multi-host / resource-value verification) out of `src/controller/testsuite/` into its own first-class `src/harness/` component — own `meson.build`, `CLAUDE.md`, and unit-test binary (carved out of `controller_unit_test`, which today recompiles ~14 of xce_test's own files a second time paired with white-box tests) — fully decoupled from `/xce-build controller` (today's implicit "controller always builds xce_test" coupling, `build-deployment.md:24`). Verified finding: none of xce_test's 15 source files include any controller-private header — only libxce public headers + generated capnp schema — so this is a pure relocation + build-graph split, not a dependency untangling problem, and no xce_test behavior changes. Touches `/xce-test-ha`, `/xce-deploy-ha`, `/xce-test-multihost`, `/xce-test-resources` (each must build `harness` explicitly once decoupled) and is adjacent to (not overlapping) the three existing behavior specs `xce-test-cli-ux`/`xce-test-e2e-primitives`/`xce-test-rc-file`, whose file-path references go stale after the move. Next: `/kiro-spec-init xce-harness-component`.
- [x] config-upgrade-snapshot — **descoped from `ha-term-safety` R8 on 2026-06-21 (requirements generated; `.kiro/specs/config-upgrade-snapshot/`).** Closes **bug-857**: a configuration upgrade can leave members with DIVERGENT materialized state because today it does config-push + restart + per-node WAL replay under the new config (`config_repl.c` `do_upgrade_config_with_bytes` ~1001-1163), with no snapshot pinning an identical post-upgrade state (replay is config-dependent — `lost_and_found` re-homes orphaned entities). Approach: an upgrade materializes a **full entity-bearing state snapshot at the config-change marker's `(term,lsn)` boundary**, embeds the new config, and installs it on every member via the `snapshot_transfer_s` transport — no per-node re-derivation; supersedes `ha-term-safety` R7's interim config-push byte delivery for upgrades. **Former HARD dependency `entity-bearing-snapshots` is CLEARED** — it shipped as spec `ha-entity-snapshots` (**IMPLEMENTATION-COMPLETE, validate-impl GO 2026-07-18**; see Future Specs/Backlog below), and `ha-term-safety` (the config-change marker it anchors on) is also implemented. `config-upgrade-snapshot`'s own `spec.json` now has `blocked_on: []`. Verify by reproducing bug-857 (upgrade during chaotic multi-kill, diff materialized state). **Ready to promote to design.**