diff --git a/.kiro/specs/application-profile/brief.md b/.kiro/specs/application-profile/brief.md index 9a6dd3f4..e1cf663b 100644 --- a/.kiro/specs/application-profile/brief.md +++ b/.kiro/specs/application-profile/brief.md @@ -34,6 +34,15 @@ decisions):** still taken from the profile at dispatch time; requirements decide whether dispatch freezes them on the job record. +**Update 2026-10-10 (`/kiro-discovery`, long-running services):** a profile +may declare **kind `service`** with a service section: restart policy +(always / on-failure / never, backoff, give-up), liveness and readiness +probes (command / TCP / HTTP), named ports, default replica count, spread +rule and rollout settings. The section is defined with this spec and +consumed by `service-supervision` (worker probes and restarts), +`service-replica-set` (replicas, scaling, rolling update) and +`service-endpoint` (ports, endpoints). + ## Problem XCE users run many kinds of applications: trusted in-house codes, untrusted scripts, vendor binaries, MPI solvers. Each needs a different way of running: diff --git a/.kiro/specs/host-failure-job-recovery/brief.md b/.kiro/specs/host-failure-job-recovery/brief.md new file mode 100644 index 00000000..6025a34f --- /dev/null +++ b/.kiro/specs/host-failure-job-recovery/brief.md @@ -0,0 +1,86 @@ +# Brief: host-failure-job-recovery + +(`/kiro-discovery`, 2026-10-10, "Long-running services" phase 1. A +prerequisite for services, and a gap for batch jobs today.) + +## Problem +When a host dies, the jobs running on it are never recovered. Users see them +as RUN indefinitely, their resources stay charged, and nothing requeues or +fails them. The only exception is a host that comes back: when its worker +rejoins, the jobs it reports as "unrecognized" are reconciled. Every +long-running service needs its replicas recovered after a host failure, but +batch users are hurt by this already. + +## Current State +- `handle_worker_leave()` / `mark_worker_unavail()` + (`src/controller/worker.c:897,921`) mark the host UNAVAIL, record + `dead_since` and set its slots to 0. They do not touch the host's jobs. +- `controller/sched.c:631` says so: + `FIXME: need to handle job status in this worker`. +- Reconciliation exists only when a worker rejoins (`resync.c:692-710`): a + job it reports as "unrecognized" goes back to PEND. The rejoin check times + out after 15 s and then simply ends, leaving the jobs as they are + (`resync.c:259-290`). +- Job states `RUNKWN`/`UNKWN`/`TRANSIENT` are defined (`types.h:382-405`) but + no code sets them. +- `JOB_REQUEUE` exists only as an enum value; the apiserver's requeue control + is commented out (`jobs.go:2366-2382`). There is no rerunnable flag. + +## Desired Outcome +- After a configurable grace period with the host unreachable, each job that + ran on it is moved to an "unknown" state and then resolved by policy: + - requeued (PEND, new attempt) if the job is rerunnable (job option or + application profile default); + - otherwise EXIT with a clear reason ("host lost"). +- A multi-host job that loses one of its hosts is resolved as a whole (its + surviving tasks are stopped). A service replica is an ordinary job: it is + resolved like any other, and `service-replica-set` replaces it. +- If the host comes back with the job still running after the job was + resolved, the stale copy is killed (fencing by attempt number). +- Resources charged to the lost job are released; limits and quotas recover. +- Users and admins see the lost-host reason and the attempt count; + `xcecli`/REST can requeue a job manually (`JOB_REQUEUE`). +- Behaviour is identical after controller failover (facts in the WAL). + +## Approach +Controller-side, primary-only, trigger-only timer (delaymgr rule: +no blocking) that walks jobs of hosts unavailable longer than the grace +period, commits a "job lost" fact per job and applies the policy through the +existing commit-then-apply seam. The worker's rejoin resync gains a check +that kills jobs whose attempt number is stale. + +## Scope +- **In**: grace period config; unknown state; rerunnable flag (job and + profile default) and requeue-on-host-loss; EXIT with reason otherwise; + multi-host job handling; attempt numbers and stale-copy fencing on rejoin; + manual requeue (REST, `xcecli`); display; WAL/snapshot field completeness; + sim/live scheduler consistency for released resources. +- **Out**: requeue on job exit codes (`REQUEUE_EXIT_VALUES`, possible + follow-up); checkpoint/restart migration; replacing service replicas + (`service-replica-set`). + +## Boundary Candidates +- Host-loss detection and grace period (controller). +- Job-lost policy (requeue / exit) and attempt numbering. +- Stale-copy fencing on worker rejoin. +- Manual requeue control (apiserver, xcecli). + +## Out of Boundary +- Restarting a process on the same host (`service-supervision`). +- Keeping a service's replica count (`service-replica-set`). + +## Upstream / Downstream +- **Upstream**: worker liveness (`worker.c`), worker rejoin resync + (`worker-rejoin-resync`), commit/apply seam (`raft-apply-decoupling`). +- **Downstream**: `service-supervision`, `service-replica-set`, + `job-dependency` (lost jobs end as EXIT). + +## Existing Spec Touchpoints +- **Extends**: none. +- **Adjacent**: `worker-rejoin-resync`, `controller-job-cleanup`, + `scheduler-preemption` (requeue semantics). + +## Constraints +- HA/persistence: full 3-phase workflow; additive Cap'n Proto; the new facts + round-trip through WAL and snapshot (`/xce-audit-invariants`). +- The timer must never block the delaymgr (`ha-async-io.md`). diff --git a/.kiro/specs/job-resize/brief.md b/.kiro/specs/job-resize/brief.md new file mode 100644 index 00000000..31139901 --- /dev/null +++ b/.kiro/specs/job-resize/brief.md @@ -0,0 +1,66 @@ +# Brief: job-resize + +(`/kiro-discovery`, 2026-10-10, "Long-running services"; **backlog, not on +the services path**: services use independent replica jobs instead +(`service-replica-set`), so this stays an elastic-job feature. LSF +precedent: resizable jobs, `bresize`, autoresizable jobs; Admin Guide p. 339.) + +## Problem +A running parallel job's allocation is fixed at dispatch. Elastic +applications (e.g. elastic MPI or task-farm frameworks) cannot grow onto +free hosts or give hosts back without being killed and resubmitted. + +## Current State +- Allocation is decided once by the scheduler and committed at dispatch + (`scheduler-dispatch-commit`); job modify (`JOB_MODIFY`) is not + implemented. +- Job arrays have a fixed size; nothing adds or removes tasks of a running + job. + +## Desired Outcome +- A running parallel job can **grow** (request K more tasks; the scheduler + places them under the job's constraints, e.g. one per host, and the + controller adds them to the allocation and starts them) and **shrink** + (release named tasks or hosts; their processes are stopped first). +- A grow request pends until it can be placed and can be cancelled; the + job's min/max task range bounds every resize. +- Quotas, limits and accounting follow the current allocation. +- Requests come from the job's owner (REST/xcecli) and from the job itself + (an in-job API for elastic frameworks). +- Resizes survive controller failover (facts in the WAL) and the scheduler + simulator reproduces them. + +## Approach +A resize request is a pending allocation request attached to a running job; +the scheduler treats a grow like a parallel-job placement of K tasks with +the job's span constraints (`parjob-placement`) and existing hosts excluded +or preferred per policy; the controller commits the grown/shrunk allocation +and notifies the job's workers. + +## Scope +- **In**: grow/shrink requests (REST, xcecli, in-job API), scheduling of + grow requests, allocation commit and worker launch/stop, + accounting, persistence, simulator support. +- **Out**: changing per-task resource sizes (vertical resize); resizing + serial jobs; resize-driven autoscaling policies. + +## Boundary Candidates +- Resize request model and API. +- Scheduler placement of grow requests. +- Controller allocation change and worker task start/stop. + +## Out of Boundary +- Deciding when to resize (the user or the application). + +## Upstream / Downstream +- **Upstream**: `parjob-placement` (span constraints), scheduler allocation, + `host-failure-job-recovery` (lost tasks). +- **Downstream**: elastic-job frameworks; none of the service specs. + +## Existing Spec Touchpoints +- **Extends**: none. +- **Adjacent**: `parjob-placement`, `scheduler-preemption`, + `job-sandbox-multinode` (remote task launch). + +## Constraints +- HA/persistence and scheduler equivalence: full 3-phase workflow. diff --git a/.kiro/specs/service-endpoint/brief.md b/.kiro/specs/service-endpoint/brief.md new file mode 100644 index 00000000..797958d2 --- /dev/null +++ b/.kiro/specs/service-endpoint/brief.md @@ -0,0 +1,59 @@ +# Brief: service-endpoint + +(`/kiro-discovery`, 2026-10-10, "Long-running services" phase 2.) + +## Problem +Clients of a service need a stable way to reach its ready replicas. Today +users only see exec host names; there are no ports, no host addresses and no +registry, so every service invents its own discovery. + +## Current State +- No port or network resource, endpoint, DNS or load-balancing concept; + `xce_worker_t` has no IP field and the host API marks `ip_addr` as + unsupported (`models/hosts.go:43`). +- Readiness will be reported by workers (`service-supervision`). + +## Desired Outcome +- A service profile declares named ports (fixed or "any free"); each replica + is allocated its ports on its host like a resource, so two replicas never + collide; the job sees its ports in environment variables. +- Workers report host addresses; a service has an **endpoint**: its name + resolves to the list of `host:port` of ready replicas (readiness from + `service-supervision`), updated as replicas become ready or not. +- Users and clients read endpoints through REST and `xcecli` (and a simple + query interface usable from scripts); access follows RBAC (job view + scope). +- Out of v1 but kept possible: DNS records, a built-in proxy or load + balancer. + +## Approach +Ports become a per-host consumable resource the scheduler allocates; +readiness changes already reach the controller; the controller (or the +apiserver from pushed state) maintains the endpoint table per service. + +## Scope +- **In**: port declarations and allocation, host address reporting, + endpoint table from readiness, REST/xcecli endpoint queries, RBAC. +- **Out**: DNS, proxy/load balancing, TLS, ingress from outside the + cluster. + +## Boundary Candidates +- Port resource and allocation (scheduler, worker). +- Host address reporting. +- Endpoint table and query surfaces. + +## Out of Boundary +- Health probing itself (`service-supervision`). + +## Upstream / Downstream +- **Upstream**: `service-supervision` (readiness), `application-profile` + (port declarations), resource model (`resource-accounting.md`). +- **Downstream**: `service-replica-set` (a service's endpoint covers all its + replica jobs; rollouts keep it populated). + +## Existing Spec Touchpoints +- **Extends**: `application-profile` (ports in the service section). +- **Adjacent**: `resource-distribution-mapping`. + +## Constraints +- Endpoint updates are change-driven; no per-probe traffic. diff --git a/.kiro/specs/service-replica-set/brief.md b/.kiro/specs/service-replica-set/brief.md new file mode 100644 index 00000000..1ea160ff --- /dev/null +++ b/.kiro/specs/service-replica-set/brief.md @@ -0,0 +1,100 @@ +# Brief: service-replica-set + +(`/kiro-discovery`, 2026-10-10, "Long-running services" phase 2. Replaces +the earlier `service-head-agent` idea by user decision: no special replica, +no head, no promotion. Kubernetes analogue: Deployment/ReplicaSet with +independent pods.) + +## Problem +A service with N replicas needs N healthy replicas kept running: a replica +that gave up or was lost with its host must be replaced elsewhere, the +count must be changeable, and a new version must roll out without +downtime. Per-replica supervision (`service-supervision`) cannot do this +alone because no worker sees the other replicas. + +## Current State +- Phase 1 gives per-replica probes and in-place restart on each worker + (`service-supervision`) and host-loss recovery that ends or requeues a + job (`host-failure-job-recovery`). +- There is no service object, no replica grouping and no rollout concept; + job arrays are fixed-size. + +## Desired Outcome +- A **service** is a controller-held object: name, owner, project, queue, + application profile (kind `service`), desired replica count, version + (the profile/command revision it runs) and the list of its **replica + jobs**. +- **Replicas are independent, ordinary jobs.** Each is scheduled, limited, + charged, recovered and RBAC-checked like any job; the scheduler spreads a + service's replicas across hosts (anti-affinity rule from the profile, + e.g. at most one per host, best effort or strict). +- **No replica is special**: each replica's worker checks it + (`service-supervision`); there is no head, no promotion, no epoch. +- **Event-driven rules in the controller** (no reconcile loop, no new + timers on the delaymgr), applied when a fact commits: + - a replica job ends (gave up, lost with its host, killed outside the + service) while the service wants more → submit a replacement replica + job, with backoff when replacements keep failing (a crash-looping + service pends with a reason instead of churning); + - the desired count changes (scale through REST/xcecli) → submit or kill + replica jobs (scale-down picks not-ready replicas first, then newest); + - **rolling update** (new version) → replace replicas in batches limited + by max-unavailable / max-surge, each step advancing when the new + replicas report ready; pause, resume and rollback; a progress deadline + marks a stuck rollout. +- Deleting a service kills its replicas; killing one replica directly just + triggers its replacement. +- Users see service state: desired / running / ready counts, version, + rollout progress, last events; `xcecli services` commands and REST. +- Everything survives controller failover (service object and rollout state + are WAL facts); the scheduler simulator handles replica jobs as jobs. + +## Approach +A new controller entity with its own WAL facts and snapshot frames; rules +attached to the existing commit-then-apply seam (replica end, readiness +change, desired-count change, version change). Replica jobs carry their +service name and replica index. Considered and set aside: a head agent per +service (needs promotion and fencing) and a separate cluster-wide service +manager (one more HA component). + +## Scope +- **In**: service object and API (create, scale, update, pause/resume, + rollback, delete, list/show) in REST, xcecli and libxce; replica jobs + linked to the service; replacement with backoff; scaling; rolling update; + spread rule (with the scheduler); display; persistence (WAL, snapshot, + field completeness); RBAC on services (follows job permissions and + project scopes). +- **Out**: metric-based autoscaling; canary/blue-green beyond rolling + update; endpoints (`service-endpoint`); per-replica probes + (`service-supervision`); elastic resize of one job (`job-resize`). + +## Boundary Candidates +- Service entity, facts and API. +- Replacement and scaling rules. +- Rollout state machine. +- Spread placement rule (scheduler). +- Display surfaces. + +## Out of Boundary +- How a replica is probed or restarted on its host. +- Endpoint table mechanics. + +## Upstream / Downstream +- **Upstream**: `service-supervision` (readiness, gave-up events), + `host-failure-job-recovery` (lost replicas), `application-profile`, + `job-config-merge`, `project-config` (quotas cap replicas). +- **Downstream**: `service-endpoint` (the endpoint covers all replica + jobs), future autoscaling. + +## Existing Spec Touchpoints +- **Extends**: `application-profile` (default replicas, spread rule, + rollout settings in the service section). +- **Adjacent**: `job-array-single-wal-record` (grouped submission + precedent), `controller-job-cleanup` (finished replica jobs). + +## Constraints +- Rules run inside the commit/apply path and must stay bounded and + non-blocking (`ha-async-io.md`); replacement backoff without new delaymgr + work beyond trigger-only timers. +- HA/persistence: full 3-phase workflow; `/xce-audit-invariants` for the new + entity. diff --git a/.kiro/specs/service-supervision/brief.md b/.kiro/specs/service-supervision/brief.md new file mode 100644 index 00000000..029a888a --- /dev/null +++ b/.kiro/specs/service-supervision/brief.md @@ -0,0 +1,90 @@ +# Brief: service-supervision + +(`/kiro-discovery`, 2026-10-10, "Long-running services" phase 1.) + +## Problem +Users and admins want to run long-lived services on XCE next to batch work: +model servers, notebooks, databases and web apps for users; license servers, +workflow engines and monitoring agents for admins. Today a job that exits or +hangs simply ends or keeps running unnoticed: nothing restarts it and nothing +checks that it still works. + +## Current State +- A job runs in a systemd transient unit `xcecd-worker-` with no + `Restart=` (`src/worker/src/xcecd_cgroups.c`); the `xcecd_worker` process + forks the job and reports its exit (`xcecd_worker.c:467-523`); the + controller ends it DONE or EXIT (`job.c:2937-2951`). +- No health, liveness or readiness checks exist. +- Nothing kills an indefinitely running job unless the user sets `-W` + (`RuntimeMaxUSec` only when > 0), so running forever already works. +- A running job writes a usage WAL record every 10 s; checkpoints keep the + WAL bounded (no growth with job age). + +## Desired Outcome +- An application profile can declare **kind `service`** with: + - a restart policy: `always` / `on-failure` / `never`, backoff (initial, + maximum), and a give-up rule (max restarts in a window); + - a liveness probe: command, TCP connect or HTTP GET, with interval, + timeout, failure threshold and initial delay; + - a readiness probe (same forms), reported but not yet used for routing + (`service-endpoint`); + - whether the service is rerunnable after host loss (default yes, + `host-failure-job-recovery`). +- The worker supervises each service task on its host: runs the probes + locally, restarts the process in place on exit or failed liveness (keeping + its allocation), applies backoff, and gives up per policy (the job then + ends EXIT with the reason, or is requeued if the policy says so). +- The worker reports only changes to the controller: ready / not ready, + restarted (count, last reason), gave up. Users see state, restart count + and last failure in job details and `xcecli`. +- Stopping a service is the normal job kill; a service never ends DONE on + its own unless its policy is `never` and it exits 0. +- Service jobs are excluded from no-op expectations of batch jobs: no run + limit by default, not cleaned up while running. + +## Approach +Supervision lives in `xcecd_worker` (already the job's parent): a restart +loop around the launch with backoff, probe timers on the worker's event +loop, and change-only status reports over the existing worker→controller +channel. The profile's service section flows to the worker through the job's +effective values (`job-config-merge`). Using systemd `Restart=` was +considered and rejected for v1: it would rerun `xcecd_worker` itself +(including profile pre-exec) and hide restarts from XCE. + +## Scope +- **In**: profile service section (kind, restart policy, probes, rerunnable); + worker restart loop and probes; change-only health reports and their + persistence; display (state, restarts, last failure); xcecli/REST + submission of service jobs through profiles; tests incl. `xcecd_virtual` + simulation of probes and restarts. +- **Out**: replicas, scaling, replacement on other hosts, rolling update + (`service-replica-set`); endpoints and ports + (`service-endpoint`); host failure (`host-failure-job-recovery`). + +## Boundary Candidates +- Profile schema for services (with `application-profile`). +- Worker supervision loop and probe runner. +- Health/restart reporting and persistence (controller). +- Display surfaces. + +## Out of Boundary +- Any decision that needs a cluster-wide view (more replicas, new hosts). +- No replica is special: every replica's own worker checks it (user + decision 2026-10-10), so there is no head and no promotion. + +## Upstream / Downstream +- **Upstream**: `application-profile`, `job-config-merge` (effective + values), `host-failure-job-recovery` (host loss), worker launch path. +- **Downstream**: `service-replica-set` (replaces replicas that gave up, + gates rollouts on readiness), `service-endpoint` (readiness). + +## Existing Spec Touchpoints +- **Extends**: `application-profile` (service section of the profile). +- **Adjacent**: `job-sandbox-execution` (execution backends must support the + restart loop), `controller-job-cleanup`. + +## Constraints +- Health reports must be change-only (no per-probe WAL traffic). +- Real `xcecd` needs systemd/dbus to verify; `xcecd_virtual` must simulate + probe outcomes and restarts for CI. +- Persistence of health state: field completeness (`persistence.md`). diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index 367ec63e..bc923e07 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -73,6 +73,15 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han - [ ] job-config-merge — NEW 2026-10-09 (`/kiro-discovery`, Path E "configuration layering"; ref LSF 10.1 Admin Guide pp. 314-316, 435-437, 561-563, 643, 660). One merge function in `src/lib` turning the job's own values + application profile + queue policy + project into effective values (resreq incl. `span`, task range, limits, pre/post-exec, env). One owner per attribute (user decision; `.kiro/steering/job-config-layering.md`): job overrides a profile attribute only if it is *default* (for `select` only by adding ANDed terms); queue ceilings and project quotas only check (reject if never satisfiable, pend if full now). **Live** merge (user decision): WAL keeps inputs + names; controller/apiserver validate at submit; scheduler computes effective values and re-evaluates pending jobs on config change. Queue is policy only: queue `res_req` and `pre_exec`/`post_exec` deprecated (pre/post-exec moves to the profile), dropped queue limits (cpulimit/memlimit/runlimit/task limit) wired. Brief: `.kiro/specs/job-config-merge/brief.md`. Dependencies: none. - [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (members, priority, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config so `PROJECTS`/`PER_PROJECT` limits work end to end, per-project usage display. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none. +- [ ] host-failure-job-recovery — NEW 2026-10-10 (`/kiro-discovery`, "Long-running services" phase 1; also a batch gap: RUN jobs on a dead host stay RUN forever, `controller/sched.c:631` FIXME). After a grace period, jobs on a lost host go to an unknown state, then are requeued (rerunnable, job option or profile default) or end EXIT "host lost"; multi-host jobs resolved as a whole; attempt numbers fence a stale copy when the host returns; manual requeue (REST/xcecli). Brief: `.kiro/specs/host-failure-job-recovery/brief.md`. Dependencies: none. +- [ ] service-supervision — NEW 2026-10-10 (Long-running services phase 1). Application profile kind `service`: restart policy (always/on-failure/never, backoff, give-up), liveness and readiness probes (command/TCP/HTTP) run by each replica's own worker (no special replica), in-place restart, change-only health reports, display. Brief: `.kiro/specs/service-supervision/brief.md`. Dependencies: application-profile, host-failure-job-recovery. +- [ ] service-replica-set — NEW 2026-10-10 (Long-running services phase 2; user decision: no head agent, no promotion). Controller-held service object owning independent replica jobs; event-driven rules on commit (replace ended replicas with backoff, scale, rolling update with max-unavailable/surge, pause/resume/rollback); spread rule; REST/xcecli `services`. Brief: `.kiro/specs/service-replica-set/brief.md`. Dependencies: service-supervision, host-failure-job-recovery, application-profile. +- [ ] service-endpoint — NEW 2026-10-10 (Long-running services phase 2). Named ports allocated per replica as host resources, host addresses reported by workers, endpoint = `host:port` of ready replicas from readiness changes, REST/xcecli queries; DNS/proxy later. Brief: `.kiro/specs/service-endpoint/brief.md`. Dependencies: service-supervision, service-replica-set. + +## Existing Spec Updates (2026-10-10 long-running services discovery) +- [ ] application-profile — profile kind `service` section: restart policy, probes, ports, default replicas, spread rule, rollout settings (consumed by service-supervision, service-replica-set, service-endpoint). +- [ ] job-config-merge — service attributes follow the ownership rule (profile owns them; the job may override only `default` ones, e.g. replica count). + ## Existing Spec Updates (2026-10-09 configuration layering discovery) - [ ] application-profile — the profile carries the job's resource requirement (select/rusage/`span`/same) and task limits (min/default/max); "resolved and frozen at submit" replaced by the live merge of `job-config-merge` (brief updated). Dependencies: job-config-merge. - [ ] parjob-placement — `span` read from the effective resreq; span constraints in the fixed-N planner; range search bisects over the planner's allowed counts (block: multiples; ptile: multiples + max); LSF 10.1 findings recorded in its requirements.md notes; `span[stripe]` candidate. Dependencies: job-config-merge (for `span` from profiles; job-level `-R` span can land first). @@ -82,6 +91,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han ## Future Specs / Backlog (dependency order) - [ ] project-guarantee — NEW 2026-10-09 (configuration layering, backlog). Project-owned guarantees (resource amount or host pool), borrowing of idle guaranteed capacity, reclaim through preemption. Brief: `.kiro/specs/project-guarantee/brief.md`. Dependencies: project-config, scheduler-preemption. +- [ ] job-resize — NEW 2026-10-10 (from long-running services discovery; not on the services path). Grow/shrink a running parallel job's allocation (LSF resizable jobs, `bresize`) for elastic applications. Brief: `.kiro/specs/job-resize/brief.md`. Dependencies: parjob-placement, host-failure-job-recovery. - [ ] project-delegated-membership — NEW 2026-10-10 (from project-config requirements, user decision: future). Project managers edit their project's (and subtree's) members at runtime without an admin config push; needs a durable store that coexists with `xceadmin` config pushes (e.g. a separate delegated member list) and audit. Dependencies: project-config, rbac-persistence. - [ ] lsf-config-migration — NEW 2026-10-09 (user idea, backlog; no brief yet). A tool (likely an `xceadmin` subcommand) that converts LSF configuration (`lsb.queues`, `lsb.applications`, `lsb.resources` Limit sections, `lsb.users`/`lsb.hosts` groups, `lsb.params` defaults, project settings) into XCE `configs/*.yaml`, with a report of what does not map: queue `RES_REQ` (XCE queues are policy only; move into a profile or drop), LSF's full per-section merge rules vs XCE's simplified ones, compound/alternative resreqs, unsupported parameters. Dependencies: job-config-merge, application-profile, project-config, parjob-placement (span), admin-config-tool. Promote via `/kiro-discovery` when those land.