docs(steering): N24 decided: gang-groups demoted to backlog as a co-start group

LLM training is one parallel job with topology requirements (PR-10).
gang-groups moves from the spec list to the roadmap backlog, re-scoped as
a co-start group of separately managed jobs and services (RL
post-training, disaggregated serving, streaming pipelines). resource-model
phases and the brief updated; job-resize no longer resizes gang groups.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.kiro/specs/gang-groups/brief.md

@@ -1,54 +1,93 @@
 # Brief: gang-groups
 
-(Resource-model discovery, 2026-10-11. Phase 5 of `.kiro/steering/resource-model.md`, driven by the LLM training/serving platform direction.)
+(Resource-model discovery, 2026-10-11. **Demoted to the backlog 2026-10-12**
+by user decision N24 in `.kiro/steering/resource-questions.md`. LLM training
+is one parallel job with topology requirements (PR-10 there), not a gang.
+This spec is re-scoped as a **co-start group** of separately managed jobs
+and services, to revisit when RL post-training or disaggregated serving
+reaches the roadmap.)
 
 ## Problem
-Distributed training is often several cooperating jobs that must start together or not at all, and with their storage ready:
+Some workloads are several cooperating parts that must start together, yet
-- a launcher plus worker groups;
+each part needs its own identity: application profile or container, owner,
-- a parameter server plus trainers;
+lifecycle, or independent scaling. One parallel job cannot express that.
-- heterogeneous roles with different resource requests (a Kubernetes JobSet, or a Volcano/Kueue gang).
+Starting only some parts wastes accelerators, and can deadlock the cluster
+when parts wait on each other.
 
-Starting part of a gang wastes accelerators and can deadlock the cluster. XCE places one parallel job all-or-nothing, but has no all-or-nothing group of jobs.
+**Use cases** (review 2026-10-11, "Gang use cases" in `resource-questions.md`):
+- **RL post-training** (RLHF, GRPO): a training job, rollout engines (e.g.
+  vLLM), and reward and reference models. Each is a different app;
+  rollouts restart and scale on their own.
+- **Disaggregated LLM serving:** prefill pool, decode pool and router. They
+  come up together and are placed close for KV-cache transfer, then scale
+  independently as long-running services.
+- **Streaming pipelines:** a live producer feeding a consumer (preprocessing
+  into training, online evaluation) with different profiles or owners.
+
+**Not this spec:**
+- data-, tensor- or pipeline-parallel training; MPMD codes; coordinator plus
+  workers: one parallel job with a compound resreq and topology
+  requirements (PR-10, `resource-instances`, `parjob-placement`);
+- a multi-host replica of a served model: one parallel job per replica in
+  `service-replica-set`;
+- jobs that only need to be placed near each other: placement groups
+  (PR-11), which give co-placement without a co-start.
 
 ## Current State
-- **One parallel job is already a gang:** all its tasks are placed together under `span[]` (`parjob-placement`).
+- One parallel job is already a gang: all its tasks are placed together
-- **Heterogeneous single jobs:** compound resreqs (`N*{...}+M*{...}`, `lib/xce_resreq.peg:60-83`) express different task shapes within one job.
+  (`parjob-placement`), and compound resreqs (`lib/xce_resreq.peg:60-83`)
-- **Missing:** a group of separate jobs scheduled atomically, topology packing across a gang (`resource-instances`), and readiness of the gang's claims (`resource-claims`).
+  mix task shapes within it.
+- There is no all-or-nothing start for a group of separate jobs or services.
 
 ## Desired Outcome
-- **Gang group:** the jobs of a group are scheduled all-or-nothing in one session decision (or by a reservation that accumulates until complete, with a timeout and release).
+- **Co-start group:** a named group whose members are batch jobs and/or
-- **Placement:** packed by topology (switch block, NVLink domain) when requested.
+  service replica sets.
-- **Start:** no member starts until every member is placed and all their claims are READY.
+- **Placement:** all members are placed all-or-nothing, in one session
-- **Group policy:** if a member fails, the group policy decides: fail all, restart all, or tolerate.
+  decision or through a reservation that accumulates until complete, with a
-- **Interactions:** with `job-resize`, preemption and backfill.
+  timeout and release. Placement is packed by topology when requested,
-- **Surfaces:** visible in xcecli/REST (group state, member states, pend reason).
+  reusing PR-11 placement groups.
+- **Start barrier:** no member starts until every member is placed and all
+  their claims are READY.
+- **Failure policy:** after the start, members keep their own lifecycles
+  (restart, scale). When a member fails, the group policy decides: fail the
+  group, restart the member, or tolerate.
+- **Display:** in xcecli/REST, with the group state, member states and the
+  pend reason.
 
 ## Approach
-- A group object, or a job-array-like construct, in the controller.
+A group object in the controller; a scheduler co-placement step on top of
-- A scheduler gang step on top of the allocator framework, possibly reusing the parallel-job planner.
+the allocator framework and placement groups; a group start barrier in the
-- A group-level start barrier in the dispatch path.
+dispatch path.
 
 ## Scope
-- **In:** group definition and submission, atomic placement, start barrier, failure policy, display, simulator support.
+- **In:** group definition, mixing jobs and service replica sets; atomic
-- **Out:** elastic training frameworks themselves; claim and instance mechanics (other specs).
+  placement; start barrier; failure policy; display; simulator support.
+- **Out:** RL and serving frameworks themselves; single-job training
+  (PR-10); claim and instance mechanics.
 
 ## Boundary Candidates
 - Group object.
-- Scheduler gang step.
+- Scheduler co-placement step.
 - Dispatch barrier.
 - Failure policy.
 
 ## Out of Boundary
-- Application-level rendezvous (torchrun etc.).
+- Application-level rendezvous (torchrun, Ray).
 
 ## Upstream / Downstream
-- **Upstream:** `resource-instances`, `resource-claims`, `parjob-placement`, `scheduler-framework`.
+- **Upstream:** `resource-instances`, `resource-claims`,
-- **Downstream:** LLM training workflows, `job-resize` (group resize).
+  `service-replica-set`, placement groups (PR-11), `scheduler-framework`.
+- **Downstream:** RL post-training and disaggregated-serving workflows.
 
 ## Existing Spec Touchpoints
-- **Adjacent:** `job-dependency` (ordering, not co-scheduling), `scheduler-preemption`, `service-replica-set` (replicas are independent, not a gang).
+- **Adjacent:**
+  - `job-dependency`: ordering, not co-start;
+  - `scheduler-preemption`;
+  - `service-replica-set`: replicas are independent; a co-start group can
+    contain replica sets;
+  - `job-resize`.
 
 ## Constraints
-- **Scheduler:** deterministic for sim/replay.
+- Deterministic for sim/replay.
-- **Persistence:** group state is durable and field-complete in snapshots.
+- Group state durable and field-complete in snapshots.
-- **HA/persistence:** full 3-phase spec workflow.
+- Full 3-phase spec workflow (HA/persistence).

.kiro/steering/resource-model.md

@@ -113,8 +113,7 @@ rules of the resource-provisioner contract
   resize or replica replacement. Examples: checkpoint volumes (one writer
   gang), model weights (read-many by every replica), dataset caches.
 - **States:** `PENDING → PREPARING → READY → RELEASING → RELEASED/FAILED`.
-  Dispatch waits until every claim of the job (or of the whole gang) is
+  Dispatch waits until every claim of the job is READY.
-  READY.
 
 ## Rules every resource design must keep
 
@@ -173,10 +172,12 @@ device-plugin **wire protocols** is open decision 4.
   `same[]`/`span[]` already pack against.
 - **MIG/MPS:** partitions are reported as instances. Changing a layout is a
   provider action that drains the GPU, never a scheduler decision.
-- **Gangs:** one parallel job is already placed all-or-nothing
+- **Gangs:** training is **one parallel job** (decided 2026-10-12, N24 in
-  (`parjob-placement`). A gang of several jobs (launcher plus workers) needs
+  `resource-questions.md`). One parallel job is already placed
-  a gang group; the whole gang and its claims are READY before any part
+  all-or-nothing (`parjob-placement`), and compound resreqs mix roles
-  starts.
+  inside it. A gang of several jobs (`gang-groups`, backlog) is only for
+  roles that need separate lifecycles, profiles or owners: RL
+  post-training, disaggregated serving, streaming pipelines.
 - **Elastic resize and restart:** standalone claims are re-bound to the new
   host set.
 - **Serving:** replicas share one read-many weights claim and have
@@ -198,8 +199,12 @@ device-plugin **wire protocols** is open decision 4.
    the provider contract, worker prepare/publish, and a stage-in/stage-out
    gate. First users: sandbox scratch, `transfer_files` staging,
    checkpoint volumes.
-5. **`gang-groups`:** gangs of several jobs. Claim re-attach in `job-resize`
+5. **Claim users in existing specs:** claim re-attach in `job-resize`, and
-   and shared weights claims in `service-replica-set` rollouts.
+   shared weights claims in `service-replica-set` rollouts.
+
+`gang-groups` is not a phase. It is in the roadmap backlog as a co-start
+group for RL post-training, disaggregated serving and streaming pipelines
+(N24).
 
 **Config-only now:** document the static custom resources that already
 work.

.kiro/steering/resource-questions.md

@@ -280,7 +280,7 @@ Source: the CSI specification, from general knowledge; to be checked against the
 | N21 | PR-10: cross-host topology syntax: an LSF-style `cu[type=...:maxcus=...:pref=...]` clause, or an extension of `same[]`/`span[]`? | open | Leaning `cu[]`, because XCE's resreq follows LSF. |
 | N22 | PR-10: relax policy: wait up to a limit for a better topology, then relax (Slurm `--switches=N@maxwait`)? Per job, or per queue or profile? | open | |
 | N23 | PR-10: rank ordering: does the scheduler return hosts in topology order, and is that order part of the persisted allocation and shown to the launcher (hostfile, env)? | open | |
-| N24 | PR-10: demote `gang-groups` to optional/backlog, kept only for roles with separate lifecycles (e.g. RL trainer plus inference services)? | leaning | Leaning yes. Use-case review (2026-10-11), below "Gang use cases": a multi-job gang is needed mainly for RL post-training, disaggregated serving and streaming pipelines. All three mix batch and service components, so re-scope it as a "co-start group" across `service-replica-set` and jobs when RL or disaggregated serving comes onto the roadmap. |
+| N24 | PR-10: demote `gang-groups` to optional/backlog, kept only for roles with separate lifecycles (e.g. RL trainer plus inference services)? | decided | User 2026-10-12: confirmed. Training is one parallel job (PR-10); `gang-groups` moves to the backlog, re-scoped as a co-start group. Roadmap, `resource-model.md` and the brief updated. Use-case review (2026-10-11), below "Gang use cases": a multi-job gang is needed mainly for RL post-training, disaggregated serving and streaming pipelines. All three mix batch and service components, so re-scope it as a "co-start group" across `service-replica-set` and jobs when RL or disaggregated serving comes onto the roadmap. |
 | N25 | PR-10: reservation or plan-ahead for large topology-constrained jobs, to prevent starvation and fragmentation: a new spec, or part of an existing one? | open | |
 | N26 | PR-11: placement groups only, or also direct job-to-job affinity (`affinity[job(...)]`)? | open | Leaning groups first (cheaper, simpler to explain). |
 | N27 | PR-11: which modes and strictness in v1: pack and spread, hard (`required`) and soft (`preferred`)? | open | |
@@ -290,6 +290,7 @@ Source: the CSI specification, from general knowledge; to be checked against the
 | C3 | CSI compatibility: nice-to-have or a must? If a must, which side: the controller plugin through the proxy, the node plugin on the worker, or both? | discussing | Background and recommendation in "CSI notes" below (2026-10-11). |
 
 ## History
+- 2026-10-12: N24 decided (user): `gang-groups` demoted to the backlog and re-scoped as a co-start group for RL post-training, disaggregated serving and streaming pipelines.
 - 2026-10-11: user asked about data cache, data and communication topology between jobs → PR-11 (one topology model; data locality with delay scheduling; placement groups with pack or spread); N26–N30 added.
 - 2026-10-11: user asked for the use cases of gangs → "Gang use cases" section; N24 leaning (demote `gang-groups`, re-scope later as a co-start group for RL, disaggregated serving and streaming).
 - 2026-10-11: user asked whether LLM training can be a single job with complex topology requirements → PR-10 (single parallel job as the default; topology at three levels; `gang-groups` optional); N21–N25 added.

.kiro/steering/roadmap.md

@@ -87,7 +87,6 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
 - [ ] custom-resource-reporting — NEW 2026-10-11 (`/kiro-discovery`, "Unified resource model" phase 2; `resource-distribution-mapping` option 3 split out, pending user decision 1 in `resource-model.md`). Runtime custom resource values: an elim-like script reporter on workers (`posix_spawn`) and/or an external reporter role (decision 2), staleness and `interval` enforced, a string-set kind for dataset locality, instance-inventory publishing (DRA ResourceSlice-like). Brief: `.kiro/specs/custom-resource-reporting/brief.md`. Dependencies: resource-distribution-mapping, peer-authentication-hardening.
 - [ ] resource-instances — NEW 2026-10-11 (Unified resource model phase 3). Identified instances (GPU, MIG slice, NIC, ports) with attributes; topology-aware allocator plugin driving the unused `affinity[]` (NVLink domain, GPU-NIC affinity; switch block via `same[]`/`span[]`); durable per-job instance assignment (WAL/snapshot field-complete); worker binding (device cgroup, `CUDA_VISIBLE_DEVICES`, port env). Brief: `.kiro/specs/resource-instances/brief.md`. Dependencies: custom-resource-reporting, scheduler-framework.
 - [ ] resource-claims — NEW 2026-10-11 (Unified resource model phase 4). Data and volumes as lifecycle resources: template claims (per job/replica: scratch, staging, KV cache) and standalone claims (K8s PVC/ResourceClaim-like: checkpoint volumes, model weights, dataset caches; access modes RWO/ROX/RWX/RWOP; retain/delete), claim WAL entity, dispatch readiness gate (stage-in/out), storage provider contract (reused from resource-provisioner), worker publish into the sandbox, orphan handling. Absorbs the sandbox workspace and `transfer_files` staging (bug-1937 deferral). Brief: `.kiro/specs/resource-claims/brief.md`. Dependencies: resource-provisioner, resource-instances, job-sandbox-execution.
-- [ ] gang-groups — NEW 2026-10-11 (Unified resource model phase 5, LLM training). All-or-nothing scheduling of a group of jobs (launcher + workers, JobSet-like), topology packing across the gang, start barrier until every member is placed and its claims are READY, group failure policy. Brief: `.kiro/specs/gang-groups/brief.md`. Dependencies: resource-instances, resource-claims, parjob-placement.
 
 ## Existing Spec Updates (2026-10-11 unified resource model discovery)
 Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generic supply map incl. provisioned supply, PR-2 resource provider proxy, PR-3 datasets in the map with staging as provisioning) and open questions: `.kiro/steering/resource-questions.md`. Duration/decay: must-have, future spec (R8).
@@ -95,7 +94,7 @@ Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generi
 - [ ] service-endpoint — ports as identified instances of `resource-instances` instead of a separate port model.
 - [ ] job-sandbox-execution — the per-job workspace becomes the first template claim of `resource-claims`; publish (mounts) through the claim's worker step. Where data staging lives is decision 9.
 - [ ] application-profile — default claims (model weights, checkpoint) and GPU instance requests in profiles; `container` execution mode decides volume publishing (decision 6).
-- [ ] job-resize — re-bind a job's standalone claims to the new host set; resize a gang group.
+- [ ] job-resize — re-bind a job's standalone claims to the new host set.
 - [ ] service-replica-set — replicas share one read-many weights claim; rollout attaches version N+1 while N stays attached; per-replica template claims.
 - [ ] resource-provisioner — its provider contract, lease, epoch and orphan machinery are shared with storage/claim providers.
 - [ ] provisioning-demand — claim shortfall as provisionable demand.
@@ -119,6 +118,7 @@ Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generi
 - [x] scheduler-sim-diagnostics — `info job <jobid>` enhancement: a PEND job shows its pend reason set (per alternative once scheduler-alter-pend-reasons lands; shows the flat reason until then); a RUN job shows per-host/per-resource reserved (`job->allocs`) vs. real used (`job->rusages`), and when no rusage has been reported yet says "resource usage not reported yet" (no fabricated zeros; reserved still shown). Byte units per its Req 6. Read-only. Next: `/kiro-spec-requirements scheduler-sim-diagnostics`. Dependencies: none (alternatives display depends on scheduler-alter-pend-reasons).
 
 ## Future Specs / Backlog (dependency order)
+- [ ] gang-groups — NEW 2026-10-11; **demoted to backlog 2026-10-12** (user decision N24 in `resource-questions.md`). LLM training is one parallel job with topology requirements (PR-10), so it needs no gang. Re-scoped as a **co-start group** of separately managed jobs and services: RL post-training (trainer, rollout engines, reward and reference models), disaggregated serving (prefill and decode pools, router), streaming pipelines. All-or-nothing start, close placement, group failure policy. Revisit when RL or disaggregated serving reaches the roadmap. Brief: `.kiro/specs/gang-groups/brief.md`. Dependencies: resource-instances, resource-claims, service-replica-set.
 - [ ] project-guarantee — NEW 2026-10-09 (configuration layering, backlog). Project-owned guarantees (resource amount or host pool), borrowing of idle guaranteed capacity, reclaim through preemption. Brief: `.kiro/specs/project-guarantee/brief.md`. Dependencies: project-config, scheduler-preemption.
 - [ ] job-resize — NEW 2026-10-10 (from long-running services discovery; not on the services path). Grow/shrink a running parallel job's allocation (LSF resizable jobs, `bresize`) for elastic applications. Brief: `.kiro/specs/job-resize/brief.md`. Dependencies: parjob-placement, host-failure-job-recovery.
 - [ ] project-delegated-membership — NEW 2026-10-10 (from project-config requirements, user decision: future). Project managers edit their project's (and subtree's) members at runtime without an admin config push; needs a durable store that coexists with `xceadmin` config pushes (e.g. a separate delegated member list) and audit. Dependencies: project-config, rbac-persistence.
@@ -181,7 +181,7 @@ Smaller items that likely don't warrant a full spec; pick up opportunistically o
   - Claims come in two forms, template and standalone (Kubernetes PVC/ResourceClaim-like), as WAL entities with the resource-provisioner discipline.
   - Adopt Kubernetes DRA/CSI concepts, not (yet) their APIs.
 
-  **Phases:** `resource-distribution-mapping` (shared pools) → `custom-resource-reporting` → `resource-instances` → `resource-claims` → `gang-groups`. Nine user decisions are pending, listed in `resource-model.md`.
+  **Phases:** `resource-distribution-mapping` (shared pools) → `custom-resource-reporting` → `resource-instances` → `resource-claims`; `gang-groups` demoted to the backlog (N24 decided 2026-10-12). Nine user decisions are pending, listed in `resource-model.md`.
 - **2026-07-06 admin-config-tool discovery** (`/kiro-discovery`, Path E mixed decomposition):
 
   | # | Workstream | Type | Language | Dependencies |