docs(steering): resource log: gang use cases; N24 leaning (demote gang-groups)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.kiro/steering/resource-questions.md

@@ -151,6 +151,22 @@ silently.
 - **Affects:** `resource-instances` (level 1), `parjob-placement` (levels 2 and 3, locality), the resreq syntax (`cu[]` or an extended `same[]`), `gang-groups` (demote), the scheduler reservation and backfill work.
 - **Open:** see N21–N25.
 
+## Gang use cases (2026-10-11, for N24)
+- **Two kinds of gang:**
+  - **Inside one job:** all tasks of a parallel job start together. XCE already does this, and compound resreqs mix role shapes in one job (like Slurm heterogeneous jobs).
+  - **A gang of several jobs** (`gang-groups`): needed only when roles must co-start **and** each needs its own identity: application profile or container, owner, lifecycle, or independent scaling.
+- **Covered by a single job:**
+  - data-, tensor- or pipeline-parallel training (PR-10);
+  - MPMD coupled simulation (compound resreq);
+  - coordinator or parameter server plus workers;
+  - a multi-host replica of a huge served model: each replica is a parallel job inside `service-replica-set` (like Kubernetes LeaderWorkerSet).
+- **Need a multi-job gang:**
+  - **RL post-training** (RLHF, GRPO): trainer plus rollout engines plus reward and reference models; different apps, independent restart and scaling.
+  - **Disaggregated LLM serving:** prefill pool, decode pool and router; co-start and close placement for KV-cache transfer, then independent scaling.
+  - **Streaming pipelines:** a live producer feeding a consumer (preprocessing to training, online evaluation) with different profiles or owners.
+- **Not gangs:** hyperparameter sweeps and independent trials (job arrays). Synchronous population-based training is rare; a co-start barrier on an array would do.
+- **Prior art:** Volcano PodGroup (`minMember`), Kueue with JobSet, Kubernetes LeaderWorkerSet, Slurm heterogeneous jobs. LSF has no multi-job gang.
+
 ## CSI notes (2026-10-11, for C3 and R4)
 Source: the CSI specification, from general knowledge; to be checked against the current spec version during design.
 
@@ -244,11 +260,12 @@ Source: the CSI specification, from general knowledge; to be checked against the
 | N21 | PR-10: cross-host topology syntax: an LSF-style `cu[type=...:maxcus=...:pref=...]` clause, or an extension of `same[]`/`span[]`? | open | Leaning `cu[]`, because XCE's resreq follows LSF. |
 | N22 | PR-10: relax policy: wait up to a limit for a better topology, then relax (Slurm `--switches=N@maxwait`)? Per job, or per queue or profile? | open | |
 | N23 | PR-10: rank ordering: does the scheduler return hosts in topology order, and is that order part of the persisted allocation and shown to the launcher (hostfile, env)? | open | |
-| N24 | PR-10: demote `gang-groups` to optional/backlog, kept only for roles with separate lifecycles (e.g. RL trainer plus inference services)? | open | Leaning yes. |
+| N24 | PR-10: demote `gang-groups` to optional/backlog, kept only for roles with separate lifecycles (e.g. RL trainer plus inference services)? | leaning | Leaning yes. Use-case review (2026-10-11), below "Gang use cases": a multi-job gang is needed mainly for RL post-training, disaggregated serving and streaming pipelines. All three mix batch and service components, so re-scope it as a "co-start group" across `service-replica-set` and jobs when RL or disaggregated serving comes onto the roadmap. |
 | N25 | PR-10: reservation or plan-ahead for large topology-constrained jobs, to prevent starvation and fragmentation: a new spec, or part of an existing one? | open | |
 | C3 | CSI compatibility: nice-to-have or a must? If a must, which side: the controller plugin through the proxy, the node plugin on the worker, or both? | discussing | Background and recommendation in "CSI notes" below (2026-10-11). |
 
 ## History
+- 2026-10-11: user asked for the use cases of gangs → "Gang use cases" section; N24 leaning (demote `gang-groups`, re-scope later as a co-start group for RL, disaggregated serving and streaming).
 - 2026-10-11: user asked whether LLM training can be a single job with complex topology requirements → PR-10 (single parallel job as the default; topology at three levels; `gang-groups` optional); N21–N25 added.
 - 2026-10-11: user input: keep submit options simple → PR-7 (N6 leaning); data producer/consumer dependencies → PR-8 (N16–N19); isolation between users, sharing low priority → PR-9 (N20); requeue/restart reuse the previous directory, details later → N11/N13 leaning.
 - 2026-10-11: user proposed a shared volume with per-user/per-job subdirectories (`shared_volume/<user>/<jobid>/{in,out,err}`) instead of a volume per job → PR-6 refinement (leaning). N9 and N10 leaning; N8 reframed; N12–N15 added.