From 94472ee3e77a64f3bd5dcae8a76a61145fcb9352 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sat, 10 Oct 2026 08:11:08 +0000 Subject: [PATCH] docs(review): project and spec review: conflicts, code gaps, priorities 27 cross-spec conflicts among the 30 pending specs (5 high), code gaps against the five product goals and the no-shared-storage use case (silent failures such as unenforced queue/host close and REST submit dropping job limits, cwd, env and file transfers; inter-component and apiserver security; file staging; monitoring and accounting; scheduling policy; operations), and a tiered priority list (spec hygiene, P0 defects, P1 foundation specs, P2 features, P3 initiatives). Co-Authored-By: Claude Opus 5.5 (1M context) --- .../reviews/project-gap-review-2026-10-10.md | 363 ++++++++++++++++++ 1 file changed, 363 insertions(+) create mode 100644 .kiro/reviews/project-gap-review-2026-10-10.md diff --git a/.kiro/reviews/project-gap-review-2026-10-10.md b/.kiro/reviews/project-gap-review-2026-10-10.md new file mode 100644 index 00000000..f99545ce --- /dev/null +++ b/.kiro/reviews/project-gap-review-2026-10-10.md @@ -0,0 +1,363 @@ +# XCE Project and Spec Review: Conflicts, Gaps and Priorities (2026-10-10) + +Scope: every pending spec under `.kiro/specs/` (30 specs that are not +implemented), the steering rules, and the code base against the product goals +in `.kiro/steering/product.md`. The findings come from four parallel +investigations (spec conflicts; job lifecycle and hosts; scheduling; +security, HA and operations) plus direct checks of the most consequential +claims (marked *verified*). Paths are relative to `src/` unless they start +with `.kiro/`. + +## 1. Summary + +- **Specs:** 62 specs are implemented or complete, 2 superseded, 30 pending + (19 brief only, 5 initialized, 3 requirements generated, 1 design + generated, 1 brief, 1 decomposed umbrella). The pending set has **27 + cross-spec problems: 5 high, 15 medium, 7 low** (section 2). +- **Code:** HA and persistence (product goal 5) are mature: no open bugs + (all 564 bug records are closed), and phase P2.5 is done for workers, + scheduler and apiserver. The biggest gaps are elsewhere: + - several **shipped features silently do nothing**: queue close, host + close, job limits, cwd, env and file transfer submitted through REST, the + user's signal choice; + - **security** between components and in the apiserver; + - the **no-shared-storage** use case (file staging, output, checkpoint); + - **job monitoring** (exit status, pend reasons and history never reach + users); + - **scheduling policy** depth (no reservation, backfill, fairshare, + preemption, priority, windows); + - **operations** (metrics, health, docs, backup). +- **Priorities** (section 4): fix the silent defects and the security + basics first (P0), then the foundation specs many others depend on (P1), + then feature specs (P2), then the new initiatives (P3). Before any + further spec generation, resolve the 5 high spec conflicts (section 4, + "Spec hygiene"). + +## 2. Spec conflicts + +Severity: H = must be resolved before the affected specs go further; +M = resolve in the affected spec's next phase; L = cleanup. + +### High +1. **Scheduler reconnect: xce-node-manager vs component-failover-adoption + (implemented) and component-rejoin-resync.** The node-manager design + (Req 6.3; design.md:400, :12) forbids the scheduler from rejoining any + controller other than the one that started it and removes the reconnect + paths. component-failover-adoption (implementation-complete, Req 2/7) + makes the scheduler reconnect across the candidate list. + component-rejoin-resync still has scheduler reconnect cost in scope. + *Resolve:* in the node-manager revision, either retire the scheduler half + of failover-adoption (recorded in that spec) or keep reconnect only to + the same controller; drop the scheduler from component-rejoin-resync. +2. **Who replaces a lost or failed service replica.** service-supervision + defaults "rerunnable after host loss" to yes; host-failure-job-recovery + requeues rerunnable jobs; service-replica-set submits a replacement when + a replica ends. Together: N+1 replicas, or a replica that never ends so + the replica set never reacts. *Resolve:* replicas of a replica-set + service are never rerunnable and never self-requeue; the replica set + always replaces them; requeue applies to standalone service jobs only. + Record it in application-profile's service section and job-config-merge. +3. **application-profile brief still contradicts its own update and + job-config-merge.** "Resolution at submit time" and "Approach" still say + effective values are resolved and persisted at submit; the 2026-10-09 + update and job-config-merge say live merge, derived values never + persisted; roadmap.md:64 says the old thing. *Resolve:* rewrite those + sections and the roadmap line before `/kiro-spec-requirements + application-profile`; settle whether dispatch freezes execution values on + the job (then field-completeness applies). +4. **Pend-reason codes have no owner.** New codes come from + job-time-window, job-dependency, project-config, job-config-merge, + provisioning-demand, parjob-placement and service-replica-set; several + carry parameters (time, project, limit) that `JobPendReason` + (mainReason/hostReasons/resReasons) cannot hold; job-time-window and + job-dependency both claim the skipped-job publish path; + job-status-detail-display lists only the three existing codes and puts + "deciding what reasons are" out of scope. *Resolve:* one owner (the first + spec to land, or a small shared spec) for the code registry, the + parameter field and the skipped-job publish path; the display table is + extensible and each producing spec adds its rows; every new reason must + be stable across sessions or it defeats pend-reason-push-dedup. +5. **resource-distribution-mapping is blocked, but four specs build on it** + (elastic-shared-resources, provisioning-demand for licenses, + service-endpoint ports, project-guarantee). research.md says "do not + generate requirements until a scope is chosen". *Resolve:* record the + option-2 (shared pool) decision, generate requirements, and confirm it + also covers per-replica port numbers. + +### Medium +6. job-config-merge treats queue cpu/mem/run limits as hard limits that are + "wired through the merge", but job-config-layering.md says the queue never + owns limits enforced on the job. Also undefined: a job with no runtime in + a queue that has a maximum, and services with no run limit. Define "no + value vs ceiling" and the service case. +7. parjob-placement's host-count limit is a placement bound decided by the + planner, not a submit-time value check; the layering rule has no such + case. Add a third case (queue placement bounds applied by the planner) or + require a declared host count. +8. job-time-window uses the job's own run limit for its unmeetable-window + rule; after job-config-merge the effective limit can come from the + profile and change. Read the effective limit; add an "estimate exceeds + effective limit" reason. +9. Deadline, run-limit and user kills vs the service restart loop: neither + spec says whether they are terminal. Make them terminal and define how + the restart loop and deadline timer share the worker helper. +10. scheduler-preemption needs a revision before design: released memory of + SIGSTOP'd jobs vs resource-accounting.md, suspended service replicas + failing liveness, a stale "dispatch-commit not designed" note, state + naming (SSUSP/USUSP), and project-guarantee needing kill/requeue reclaim + that v1 excludes. +11. Requeue semantics across host-failure-job-recovery, job-dependency, + job-dependency-insight and controller-job-cleanup: which states are + requeueable; dependency verdicts for a requeued or unknown parent; + cleanup must not remove a job about to be requeued. +12. controller-job-cleanup vs job-dependency race: a dependent submitted + between the sweep's selection and the removal fact references a removed + parent; re-check at commit on the primary. +13. service-replica-set rules run in the apply seam on every node, but only + the primary may append facts; backoff and rollout deadlines are timers. + Fire rules only on the primary after commit; use trigger-only one-shots + handing off to the slow lane. +14. Provisioning overlap: host templates (elastic-hosts, provisioning-demand, + resource-provisioner), expected/orphan instances (elastic-hosts and + resource-provisioner), idle times (three specs). Assign: templates and + join matching to elastic-hosts; request/instance lifecycle, orphans and + reclaim policy to resource-provisioner; idle detection only to + provisioning-demand. +15. Preemption and provisioning-demand both react to allocation failure; + order undefined (preempt first, count demand only when no victim set). +16. RBAC scope types spread over application-profile (`app`), project-config + (`project`), service-replica-set; rbac-persistence's model lists only the + old scopes; security.md lists another set; validation of unknown names + differs; project-config's roadmap entry says no dependencies but relies + on rbac-persistence. +17. The scheduler-framework umbrella plans a project-aware picker and + project priority; project-config excludes project influence on ordering; + the project-config brief and roadmap still mention project priority; + `queue->project_name` exists in code (`scheduler/include/sched/types.h:115`). +18. scheduler-allocation Req 4 AC1 ("plan shape is a pure function of + maxTasks") is not yet amended; parjob-placement, provisioning-demand and + job-resize depend on the change. +19. xce-node-manager's new cross-spec items 4 and 5 need a channel to the + controller it does not have (local socket only). Route through the local + worker (it reports "restarting"; it exits with a do-not-restart code when + decommissioned). +20. Liveness probes for sandboxed jobs would run as root outside the unit; + job-sandbox-execution must offer "probe inside the unit" and "restart + without stopping the unit". + +### Low +21. Nobody owns a general `jobs modify` (job-time-window, job-dependency, + project-config, application-profile all assume one). +22. service-replica-set's cross-job spread rule has no scheduler owner. +23. Missing or unlisted dependencies: peer-authentication-hardening and + lsf-config-migration have no spec; edges service-supervision → + job-config-merge, host-failure → application-profile, job-resize → + job-config-merge are not in the roadmap. +24. Stale statements: roadmap.md:9 ("next: design scheduler-allocation"), + component-rejoin-resync ("failover-adoption in progress"), + scheduler-alter-pend-reasons ("preason-equality 0/8"), commit-batch-window + premise predates scheduler-dispatch-commit; pend-reason-push-dedup, + commit-batch-window and xce-node-manager lack roadmap entries. +25. job-config-merge: recordings must carry profile/queue/project config + changes so the simulator reproduces live re-evaluation. +26. elastic-hosts templates include dispatch windows, which the layering rule + gives to queues. +27. Service objects and dependency walks may reference jobs already removed + by cleanup; define the display. + +Checked without conflict: pre/post-exec ownership; queue `res_req` +deprecation; project membership vs RBAC; delaymgr non-blocking in pending +specs (except #13); field completeness statements; the pend-reason display +chain; job-dependency/cleanup `job_has_dependents()` interface; no live +references to the removed service-head-agent; sandbox single vs multinode +boundaries; job-time-window vs preemption. + +## 3. Code gaps against the product goals + +### Goal 1: Job lifecycle management +| Sev | Gap | Evidence | +|---|---|---| +| H | **REST submit drops most job settings** *(verified)*: `modelJobToSubmit` never sets `exdata`, `exechome` or `execcwd`, so `--cwd`, `--env`, file transfers, interactive mode, every `*_limit` (incl. `-W` run limit), `-i/-o/-e`, asked hosts and begin/end time never reach the worker | `frontend/apiserver/backend/convert.go:323-375` | +| H | File-transfer key mismatch *(verified)*: apiserver writes `transfer_files`, worker parses `transfer_file` | `controllers/jobs.go:1923` vs `worker/src/xcecd_exdata.c:115` | +| H | User signal ignored (JSON string parsed as a signal name, falls back to TERM/TSTP/CONT); ForceKill and checkpoint kill/method dropped | `controllers/jobs.go:2260-2277`; `backend/clib.go:697-705,746-762` | +| H | Kill/stop/resume refused unless the host status is exactly 0: no control of jobs on a closed or dead host | `controller/job.c:2145-2156` | +| H | Only kill, stop, resume, checkpoint, restart exist; REQUEUE, MIG, MODIFY, SWITCH, TOP/BOT are enum values only | `include/xce/types.h:614-627`; `controllers/jobs.go:2367-2384`; `controller/job.c:2171-2222` | +| H | Restart from checkpoint can never work (`ChkpntInfo` never filled); checkpoint hard-wired to `/shared/ckpt` | `controllers/jobs.go:934-970`; `worker/src/xcecd_worker.c:24` | +| H | Monitoring fields never filled: pend reason, exit status, pids, usage per host, reserve time, exec uid, checkpoint info; exit code and fail reason are FIXMEs in the controller | `models/jobs.go:528-571`; `backend/convert_pure.go:284-395`; `controller/apiserver.c:248,283,318` | +| H | No job history/accounting: finished jobs vanish from REST after 3600 s; no accounting file | `include/config.h:85`; `controller/persist.c:8736`; `backend/handler.go:83` | +| H | Jobs on a dead host stay RUN; TRANSIENT/UNKWN never set | `controller/sched.c:632`; `controller/apiserver.c:509`; `persist.c:8725` | +| M | Job control on a multi-host job goes only to the first host | `controller/job.c:2126-2131` | +| M | SSUSP never set; no load-based suspend | `controller/worker.c:409-411` | +| M | xcecli lacks requeue, modify, signal, switch, top/bottom | `xcecli/cmd/jobs/jobs.go:25-35` | +| M | `xcecd_virtual` restart loses its jobs (no recovery), one concurrent job only | `worker/src/xcecd.c:511-555` | + +### Use case: no-shared-storage HPC +| Sev | Gap | Evidence | +|---|---|---| +| H | Stage-in/stage-out not implemented in the worker (`transfer_info` parsed, never used) | `worker/src/xcecd_exdata.c:114-175`; `xcecd_worker.c:505,774-810` | +| H | Remote-to-local copy is a placeholder (empty MD5, nothing copied; shared package globals) | `controllers/jobs.go:34-42,1455-1493,2979-2995` | +| H | Uploads land on the apiserver host (`/home//apiserver_uploads`); workers cannot fetch them; file DELETE disabled | `controllers/files.go:40,144-155`; `router/routers.go:150-153` | +| M | No result download/list in xcecli; output in `/tmp` on the exec host, lost with the host; user `-o/-e` never reach the worker | `xcecli/common/request/request.go:260`; `worker/include/job_output_path.h:24-41` | + +### Goal 2: Queue and scheduling +| Sev | Gap | Evidence | +|---|---|---| +| H | **Queue close/inactive not enforced** *(verified)*: `sched_queue_is_ready()` and `sched_job_is_ready()` are `return 1` stubs | `scheduler/src/core.c:646-649` | +| H | **Host close not enforced** *(verified)*: the scheduler never checks CLOSED / CLOSED_BY_ADMIN; parjob bad-host check empty | `scheduler/src` (no reference); `plugins/parjob_allocator.c:40-48` | +| H | Queue user/host lists not enforced (not in the scheduler's queue struct) | `include/sched/types.h:101-121` | +| H | Dispatch and run windows not enforced | `xceadmin/.../mapping.py:397-398`; no scheduler use | +| H | No reservation/backfill: big parallel jobs starve (picker skips a blocked job and lets smaller ones pass) | `plugins/builtin_fcfs.c:964-979`; `include/sched/types.h:166-169` | +| H | No preemption (decision type unreachable), no fairshare, no job priority (hard-coded 0), no begin time or dependencies | `driver.c:923-936`; `schedobj.c:214`; `core_events.c:353` | +| H | No `order[]` host sorting (not in the grammar) | `lib/section.peg:36-65`; `host_iter.c:72,126` | +| M | span/same/affinity parsed then ignored; rusage duration/decay ignored | `include/resreq.h:233-237`; `alloc_runner.c:429` | +| M | Consumable flag ignored (possible double count of `mem`; inferred, not run) | `resource.c:163,178`; `configs/resources.yaml:11-16` | +| M | Shared/global resources cannot be declared from YAML; no GPU support; `sched_stop` thresholds never checked | `mapping.py:372-377,91-95,396` | +| M | Project and queue groups never loaded | `scheduler.c:1144-1145`; `core.c:184-185` | +| M | **Simulator does not load user/host groups**: limits that name groups can behave differently in replay than live (driver-equivalence violation) | `core_events.c:846-850` vs `scheduler.c:1144` | +| M | A job moved to another queue takes the queue's priority, new jobs get 0, so moved jobs jump ahead | `core_events.c:1360`; `schedobj.c:684` | +| L | Many queue YAML fields dropped or ignored (description, nice, fairshare, limits, runtimeest, reservation, preemption, job_starter, attrs, accept_interval, sched_delay) | `mapping.py:85-97`; `core_events.c:380-381` | +| L | Per-dispatch linear walk of every queue; large parallel plans cost about maxTasks × hosts | `builtin_fcfs.c:375-400`; `parjob_allocator.c:165-200` | + +Works well: queue-priority FCFS ordering, select filtering with a per-session +host cache, compound and alternative resreqs, min/max task ranges +(bug-1931), user/queue/project/host limits, the unsatisfiable-request cache, +and recorder/simulator/REPL on one core. + +### Goal 3: Cluster and host management +| Sev | Gap | Evidence | +|---|---|---| +| H | LOCKEDU, LOCKEDW, BUSY never set; CLOSED only by an admin (never when full) | `controller/worker.c:345,1024` | +| H | Host fields never filled: host group, job counts, topology, sched_stop, cores/threads, dispatch window; `ip_addr`/`real_load` commented out | `models/hosts.go:41-90`; `backend/convert_pure.go:297-311,492` | +| M | Host control: open/close only (TAG/UNTAG return -1); no drain, remove or decommission; HIDDEN never set | `controller/worker.c:1013-1029`; `controllers/hosts.go:411-418` | +| M | No host-group/rack topology (`cluster_build_groups` is a stub) | `controller/cluster.c:1513-1517` | + +### Goal 4: Multi-tenant RBAC and pluggable auth +| Sev | Gap | Evidence | +|---|---|---| +| H | LDAP and OAuth2 are stubs ("not yet implemented"), selectable without warning; PAM real (Linux + CGO) | `backend/auth/ldap.go:9-11`, `oauth2.go:9-11`, `init.go:33-36` | +| H | Config-file passwords plaintext, compared with `!=`; shipped `admin/admin`, `user1/user1` | `config_file.go:60`; `config/auth.yaml:12-21` | +| H | Logout does not revoke tokens (deletes a file never read); no refresh | `controllers/auth.go:69-95`; `middleware/validatehandler.go:101-116` | +| H | Default JWT secret `xce-api-key` accepted when `XCE_JWT_SECRET` is unset | `config/app.yaml:17`; `utils/config.go:315-320` | +| H | Shipped IP whitelist allows everyone; `X-Forwarded-For` trusted (no `SetTrustedProxies`), so client IP is spoofable | `config/app.yaml:50-51`; `router/routers.go:19` | +| H | RBAC runtime changes in memory only (rbac-persistence is a brief) | `store/stores.go:64-93` | +| M | `RequirePermission` and IP whitelist write an error but do not abort; `ReturnError` never aborts | `rbac_check.go:119-122`; `whitelist.go:130-132`; `response.go:30-37` | +| M | Scope checks spread over 13 handlers; `users.go` discards errors and compares only `!= all`; a narrower `rbac:control` grant may allow self-assigning `cluster_admin` (latent, needs verification) | `controllers/users.go:41,183,404-412,455-462` | +| M | Rate limit per endpoint only (no per-user/IP, no login brute-force limit); no audit log; CORS after JWT | `middleware/rate_limit.go:16,65-85` | +| M | JWT algorithm not pinned in validation; nil-pointer on `auth.yaml` without `config_file` | `validatehandler.go:102-104`; `config_file.go:39` | +| Fixed | `self_ug` acted as `all` (bug-1934), project filter rejected (bug-1935), limit scope swap (bug-1933) | commits 6ecd418d, 4a38bb3c | + +### Goal 5 and HA phases +| Sev | Gap | Evidence | +|---|---|---| +| H | **No certificate verification anywhere**: any TLS client without a certificate can send any opcode; `validate_host()` returns 1; `upgradeConfig`/`schedAdmin` check only "is primary"; `OP_replConfig` gives the full config to any peer | `lib/netio.c:1235-1269`; `myapp.c:116`; `controller/join.c:13-16`; `cluster.c:948,1004` | +| H | Remote tasks of multi-host jobs run as root (limits never set for task contexts) | `worker/src/controller_connector.c:1028` | +| M | P2.5 done for worker/scheduler/apiserver (libxce-failover, component-failover-adoption, worker-rejoin-resync); remaining: component-rejoin-resync (full state on rejoin), stale reads (no read leases; apiserver cache refresh 300 s), no step-down on quorum loss | roadmap.md:113,119,120 | +| M | xce-node-manager design only (no tasks/code); needs requirements revision | roadmap.md | + +### Operations (P3–P4) +| Sev | Gap | Evidence | +|---|---|---| +| H | No metrics or health endpoints (only `/ping`) | `router/routers.go:53` | +| H | No shipped process supervision (no systemd units/containers); real worker needs systemd/dbus, so multi-host tests rely on `xcecd_virtual` | `docs/deployment.md:287` | +| M | Thin docs (`docs/xce.md` 138 lines, `deployment.md` 302 lines; no admin guide for auth/RBAC/LDAP/PAM/backup/upgrade; a stale contradiction at deployment.md:291 vs :105) | `src/docs/` | +| M | No backup/restore tooling (`snapdump` todo, no config diff/rollback/history); no daemon log rotation | roadmap.md:140; deployment.md:108 | +| L | Packaging: Nix flake + AppImage, all-or-nothing build | `flake.nix:70-104` | + +### Bugs +All 564 bug records are closed (550 fixed, 8 cannot-reproduce, 5 wontfix, +1 duplicate). The defects above marked as silent failures are **not yet +logged** as bug records; the security items are tracked only as roadmap +items (peer-authentication-hardening, rbac-persistence). + +## 4. Priorities for the remaining work + +Order inside each tier is the suggested order. + +### Spec hygiene (before any further spec generation; small, documentation only) +- Resolve conflicts H1–H5: node-manager vs failover-adoption; replica + replacement owner; rewrite the application-profile brief's stale + sections; assign the pend-reason registry owner; record the + resource-distribution-mapping option-2 decision. +- Fix the stale roadmap statements (L24) and add missing dependency edges + (L23); drop "project priority" from project-config (M17). + +### P0: defects in shipped features and security basics (direct fixes as bugs, no spec) +1. REST submit drops exdata/cwd/env/limits/files, and the `transfer_file` + key mismatch (goal 1; silent data loss of user intent). +2. Queue close/inactive and host close not enforced (goal 2/3; admin + controls silently ignored). +3. Job control: user signal ignored, ForceKill dropped, control refused on + non-OK hosts, multi-host control only to the first host. +4. Simulator does not load user/host groups (driver-equivalence rule). +5. apiserver security basics: refuse the default JWT secret, token + revocation on logout, hashed passwords and no shipped defaults, + `SetTrustedProxies`, abort on middleware failures, pin the JWT + algorithm, verify the `rbac:control` escalation path, the `auth.yaml` + nil-pointer. +6. Moved jobs taking the queue's priority (ordering bug). + +### P1: foundation specs (unblock many others or close core product gaps) +1. **peer-authentication-hardening** (promote from roadmap to a spec): TLS + certificate verification, `validate_host`, role-based opcode allowlists, + remote tasks not as root. Highest security gap. +2. **host-failure-job-recovery**: dead-host jobs (batch gap; prerequisite + for services and elastic hosts). +3. **job-status-detail-display** (requirements generated), extended with + exit status/reason, pids and usage per host, plus the pend-reason + registry (H4). +4. **New: job-file-staging** for the no-shared-storage use case: stage-in + and stage-out on the worker, upload/download/list, output retrieval + after the host is gone, checkpoint without shared storage. +5. **New: job-accounting-history**: durable finished-job records and usage + history beyond the 3600 s retention (feeds bacct-like views and project + accounting). +6. **New: queue-policy-enforcement**: queue user/host lists, dispatch/run + windows, `sched_stop` thresholds, auto CLOSED/BUSY host states, and the + queue fields dropped by xceadmin that remain queue-owned (per + job-config-layering). +7. **job-config-merge** (initialized) and **resource-distribution-mapping** + (decision then requirements): both are dependencies of most P2/P3 specs. +8. **rbac-persistence**: runtime RBAC changes survive restart (project and + app scopes build on it). +9. **New: observability**: health/readiness endpoints, metrics for + controller, scheduler, apiserver and workers. +10. **xce-node-manager**: requirements revision (H1, M19, cross-spec + items), then tasks; ships process supervision. + +### P2: scheduling and feature depth +1. **New: slot-reservation-backfill**: reserve slots for blocked big jobs + and backfill around them (starvation today). +2. **scheduler-preemption** (revise requirements first, M10). +3. **parjob-placement** (span[ptile/hosts/block], allowed task counts), + plus host `order[]` sorting and rusage duration (new small items or part + of it). +4. **application-profile**, **project-config** (requirements generated), + **job-time-window**, **job-dependency** and **job-dependency-insight**, + **controller-job-cleanup**, **scheduler-alter-pend-reasons**, + **pend-reason-push-dedup**, **commit-batch-window** (re-check premise). +5. **New: job-priority-fairshare**: user job priority, top/bottom, fairshare + by user/project. +6. Host management: drain/remove/decommission, topology, host fields + (fold into elastic-hosts or a small spec). +7. Docs and backup/restore tooling (snapdump, walcompare, config history). + +### P3: new initiatives +- Long-running services: service-supervision, service-replica-set, + service-endpoint. +- Auto provisioning: provisioning-demand, resource-provisioner, + elastic-hosts, elastic-shared-resources. +- Sandbox execution: job-sandbox-execution, job-sandbox-multinode. +- project-guarantee, project-delegated-membership, job-resize, + lsf-config-migration, component-rejoin-resync (only if measured). + +## 5. Suggested next steps +1. Log the P0 defects as bug records (`buglog`) and fix them one by one + with tests. +2. Do the spec-hygiene edits (H1–H5, L23–L24, M17) in one documentation + commit. +3. Promote peer-authentication-hardening to a spec (`/kiro-spec-init`), then + host-failure-job-recovery. +4. Run `/kiro-discovery` for the new P1/P2 specs named above (job-file-staging, + job-accounting-history, queue-policy-enforcement, observability, + slot-reservation-backfill, job-priority-fairshare) to add them to the + roadmap with briefs.