diff --git a/.kiro/steering/resource-questions.md b/.kiro/steering/resource-questions.md index e4d29e18..44bc7ccb 100644 --- a/.kiro/steering/resource-questions.md +++ b/.kiro/steering/resource-questions.md @@ -151,6 +151,22 @@ silently. - **Affects:** `resource-instances` (level 1), `parjob-placement` (levels 2 and 3, locality), the resreq syntax (`cu[]` or an extended `same[]`), `gang-groups` (demote), the scheduler reservation and backfill work. - **Open:** see N21–N25. +## Gang use cases (2026-10-11, for N24) +- **Two kinds of gang:** + - **Inside one job:** all tasks of a parallel job start together. XCE already does this, and compound resreqs mix role shapes in one job (like Slurm heterogeneous jobs). + - **A gang of several jobs** (`gang-groups`): needed only when roles must co-start **and** each needs its own identity: application profile or container, owner, lifecycle, or independent scaling. +- **Covered by a single job:** + - data-, tensor- or pipeline-parallel training (PR-10); + - MPMD coupled simulation (compound resreq); + - coordinator or parameter server plus workers; + - a multi-host replica of a huge served model: each replica is a parallel job inside `service-replica-set` (like Kubernetes LeaderWorkerSet). +- **Need a multi-job gang:** + - **RL post-training** (RLHF, GRPO): trainer plus rollout engines plus reward and reference models; different apps, independent restart and scaling. + - **Disaggregated LLM serving:** prefill pool, decode pool and router; co-start and close placement for KV-cache transfer, then independent scaling. + - **Streaming pipelines:** a live producer feeding a consumer (preprocessing to training, online evaluation) with different profiles or owners. +- **Not gangs:** hyperparameter sweeps and independent trials (job arrays). Synchronous population-based training is rare; a co-start barrier on an array would do. +- **Prior art:** Volcano PodGroup (`minMember`), Kueue with JobSet, Kubernetes LeaderWorkerSet, Slurm heterogeneous jobs. LSF has no multi-job gang. + ## CSI notes (2026-10-11, for C3 and R4) Source: the CSI specification, from general knowledge; to be checked against the current spec version during design. @@ -244,11 +260,12 @@ Source: the CSI specification, from general knowledge; to be checked against the | N21 | PR-10: cross-host topology syntax: an LSF-style `cu[type=...:maxcus=...:pref=...]` clause, or an extension of `same[]`/`span[]`? | open | Leaning `cu[]`, because XCE's resreq follows LSF. | | N22 | PR-10: relax policy: wait up to a limit for a better topology, then relax (Slurm `--switches=N@maxwait`)? Per job, or per queue or profile? | open | | | N23 | PR-10: rank ordering: does the scheduler return hosts in topology order, and is that order part of the persisted allocation and shown to the launcher (hostfile, env)? | open | | -| N24 | PR-10: demote `gang-groups` to optional/backlog, kept only for roles with separate lifecycles (e.g. RL trainer plus inference services)? | open | Leaning yes. | +| N24 | PR-10: demote `gang-groups` to optional/backlog, kept only for roles with separate lifecycles (e.g. RL trainer plus inference services)? | leaning | Leaning yes. Use-case review (2026-10-11), below "Gang use cases": a multi-job gang is needed mainly for RL post-training, disaggregated serving and streaming pipelines. All three mix batch and service components, so re-scope it as a "co-start group" across `service-replica-set` and jobs when RL or disaggregated serving comes onto the roadmap. | | N25 | PR-10: reservation or plan-ahead for large topology-constrained jobs, to prevent starvation and fragmentation: a new spec, or part of an existing one? | open | | | C3 | CSI compatibility: nice-to-have or a must? If a must, which side: the controller plugin through the proxy, the node plugin on the worker, or both? | discussing | Background and recommendation in "CSI notes" below (2026-10-11). | ## History +- 2026-10-11: user asked for the use cases of gangs → "Gang use cases" section; N24 leaning (demote `gang-groups`, re-scope later as a co-start group for RL, disaggregated serving and streaming). - 2026-10-11: user asked whether LLM training can be a single job with complex topology requirements → PR-10 (single parallel job as the default; topology at three levels; `gang-groups` optional); N21–N25 added. - 2026-10-11: user input: keep submit options simple → PR-7 (N6 leaning); data producer/consumer dependencies → PR-8 (N16–N19); isolation between users, sharing low priority → PR-9 (N20); requeue/restart reuse the previous directory, details later → N11/N13 leaning. - 2026-10-11: user proposed a shared volume with per-user/per-job subdirectories (`shared_volume///{in,out,err}`) instead of a volume per job → PR-6 refinement (leaning). N9 and N10 leaning; N8 reframed; N12–N15 added.