diff --git a/.kiro/specs/gang-groups/brief.md b/.kiro/specs/gang-groups/brief.md index b4fa999c..08b70bf9 100644 --- a/.kiro/specs/gang-groups/brief.md +++ b/.kiro/specs/gang-groups/brief.md @@ -1,54 +1,93 @@ # Brief: gang-groups -(Resource-model discovery, 2026-10-11. Phase 5 of `.kiro/steering/resource-model.md`, driven by the LLM training/serving platform direction.) +(Resource-model discovery, 2026-10-11. **Demoted to the backlog 2026-10-12** +by user decision N24 in `.kiro/steering/resource-questions.md`. LLM training +is one parallel job with topology requirements (PR-10 there), not a gang. +This spec is re-scoped as a **co-start group** of separately managed jobs +and services, to revisit when RL post-training or disaggregated serving +reaches the roadmap.) ## Problem -Distributed training is often several cooperating jobs that must start together or not at all, and with their storage ready: -- a launcher plus worker groups; -- a parameter server plus trainers; -- heterogeneous roles with different resource requests (a Kubernetes JobSet, or a Volcano/Kueue gang). +Some workloads are several cooperating parts that must start together, yet +each part needs its own identity: application profile or container, owner, +lifecycle, or independent scaling. One parallel job cannot express that. +Starting only some parts wastes accelerators, and can deadlock the cluster +when parts wait on each other. -Starting part of a gang wastes accelerators and can deadlock the cluster. XCE places one parallel job all-or-nothing, but has no all-or-nothing group of jobs. +**Use cases** (review 2026-10-11, "Gang use cases" in `resource-questions.md`): +- **RL post-training** (RLHF, GRPO): a training job, rollout engines (e.g. + vLLM), and reward and reference models. Each is a different app; + rollouts restart and scale on their own. +- **Disaggregated LLM serving:** prefill pool, decode pool and router. They + come up together and are placed close for KV-cache transfer, then scale + independently as long-running services. +- **Streaming pipelines:** a live producer feeding a consumer (preprocessing + into training, online evaluation) with different profiles or owners. + +**Not this spec:** +- data-, tensor- or pipeline-parallel training; MPMD codes; coordinator plus + workers: one parallel job with a compound resreq and topology + requirements (PR-10, `resource-instances`, `parjob-placement`); +- a multi-host replica of a served model: one parallel job per replica in + `service-replica-set`; +- jobs that only need to be placed near each other: placement groups + (PR-11), which give co-placement without a co-start. ## Current State -- **One parallel job is already a gang:** all its tasks are placed together under `span[]` (`parjob-placement`). -- **Heterogeneous single jobs:** compound resreqs (`N*{...}+M*{...}`, `lib/xce_resreq.peg:60-83`) express different task shapes within one job. -- **Missing:** a group of separate jobs scheduled atomically, topology packing across a gang (`resource-instances`), and readiness of the gang's claims (`resource-claims`). +- One parallel job is already a gang: all its tasks are placed together + (`parjob-placement`), and compound resreqs (`lib/xce_resreq.peg:60-83`) + mix task shapes within it. +- There is no all-or-nothing start for a group of separate jobs or services. ## Desired Outcome -- **Gang group:** the jobs of a group are scheduled all-or-nothing in one session decision (or by a reservation that accumulates until complete, with a timeout and release). -- **Placement:** packed by topology (switch block, NVLink domain) when requested. -- **Start:** no member starts until every member is placed and all their claims are READY. -- **Group policy:** if a member fails, the group policy decides: fail all, restart all, or tolerate. -- **Interactions:** with `job-resize`, preemption and backfill. -- **Surfaces:** visible in xcecli/REST (group state, member states, pend reason). +- **Co-start group:** a named group whose members are batch jobs and/or + service replica sets. +- **Placement:** all members are placed all-or-nothing, in one session + decision or through a reservation that accumulates until complete, with a + timeout and release. Placement is packed by topology when requested, + reusing PR-11 placement groups. +- **Start barrier:** no member starts until every member is placed and all + their claims are READY. +- **Failure policy:** after the start, members keep their own lifecycles + (restart, scale). When a member fails, the group policy decides: fail the + group, restart the member, or tolerate. +- **Display:** in xcecli/REST, with the group state, member states and the + pend reason. ## Approach -- A group object, or a job-array-like construct, in the controller. -- A scheduler gang step on top of the allocator framework, possibly reusing the parallel-job planner. -- A group-level start barrier in the dispatch path. +A group object in the controller; a scheduler co-placement step on top of +the allocator framework and placement groups; a group start barrier in the +dispatch path. ## Scope -- **In:** group definition and submission, atomic placement, start barrier, failure policy, display, simulator support. -- **Out:** elastic training frameworks themselves; claim and instance mechanics (other specs). +- **In:** group definition, mixing jobs and service replica sets; atomic + placement; start barrier; failure policy; display; simulator support. +- **Out:** RL and serving frameworks themselves; single-job training + (PR-10); claim and instance mechanics. ## Boundary Candidates - Group object. -- Scheduler gang step. +- Scheduler co-placement step. - Dispatch barrier. - Failure policy. ## Out of Boundary -- Application-level rendezvous (torchrun etc.). +- Application-level rendezvous (torchrun, Ray). ## Upstream / Downstream -- **Upstream:** `resource-instances`, `resource-claims`, `parjob-placement`, `scheduler-framework`. -- **Downstream:** LLM training workflows, `job-resize` (group resize). +- **Upstream:** `resource-instances`, `resource-claims`, + `service-replica-set`, placement groups (PR-11), `scheduler-framework`. +- **Downstream:** RL post-training and disaggregated-serving workflows. ## Existing Spec Touchpoints -- **Adjacent:** `job-dependency` (ordering, not co-scheduling), `scheduler-preemption`, `service-replica-set` (replicas are independent, not a gang). +- **Adjacent:** + - `job-dependency`: ordering, not co-start; + - `scheduler-preemption`; + - `service-replica-set`: replicas are independent; a co-start group can + contain replica sets; + - `job-resize`. ## Constraints -- **Scheduler:** deterministic for sim/replay. -- **Persistence:** group state is durable and field-complete in snapshots. -- **HA/persistence:** full 3-phase spec workflow. +- Deterministic for sim/replay. +- Group state durable and field-complete in snapshots. +- Full 3-phase spec workflow (HA/persistence). diff --git a/.kiro/steering/resource-model.md b/.kiro/steering/resource-model.md index a3cf156a..cfb0dd8c 100644 --- a/.kiro/steering/resource-model.md +++ b/.kiro/steering/resource-model.md @@ -113,8 +113,7 @@ rules of the resource-provisioner contract resize or replica replacement. Examples: checkpoint volumes (one writer gang), model weights (read-many by every replica), dataset caches. - **States:** `PENDING → PREPARING → READY → RELEASING → RELEASED/FAILED`. - Dispatch waits until every claim of the job (or of the whole gang) is - READY. + Dispatch waits until every claim of the job is READY. ## Rules every resource design must keep @@ -173,10 +172,12 @@ device-plugin **wire protocols** is open decision 4. `same[]`/`span[]` already pack against. - **MIG/MPS:** partitions are reported as instances. Changing a layout is a provider action that drains the GPU, never a scheduler decision. -- **Gangs:** one parallel job is already placed all-or-nothing - (`parjob-placement`). A gang of several jobs (launcher plus workers) needs - a gang group; the whole gang and its claims are READY before any part - starts. +- **Gangs:** training is **one parallel job** (decided 2026-10-12, N24 in + `resource-questions.md`). One parallel job is already placed + all-or-nothing (`parjob-placement`), and compound resreqs mix roles + inside it. A gang of several jobs (`gang-groups`, backlog) is only for + roles that need separate lifecycles, profiles or owners: RL + post-training, disaggregated serving, streaming pipelines. - **Elastic resize and restart:** standalone claims are re-bound to the new host set. - **Serving:** replicas share one read-many weights claim and have @@ -198,8 +199,12 @@ device-plugin **wire protocols** is open decision 4. the provider contract, worker prepare/publish, and a stage-in/stage-out gate. First users: sandbox scratch, `transfer_files` staging, checkpoint volumes. -5. **`gang-groups`:** gangs of several jobs. Claim re-attach in `job-resize` - and shared weights claims in `service-replica-set` rollouts. +5. **Claim users in existing specs:** claim re-attach in `job-resize`, and + shared weights claims in `service-replica-set` rollouts. + +`gang-groups` is not a phase. It is in the roadmap backlog as a co-start +group for RL post-training, disaggregated serving and streaming pipelines +(N24). **Config-only now:** document the static custom resources that already work. diff --git a/.kiro/steering/resource-questions.md b/.kiro/steering/resource-questions.md index 530a4a3f..9c9970fc 100644 --- a/.kiro/steering/resource-questions.md +++ b/.kiro/steering/resource-questions.md @@ -280,7 +280,7 @@ Source: the CSI specification, from general knowledge; to be checked against the | N21 | PR-10: cross-host topology syntax: an LSF-style `cu[type=...:maxcus=...:pref=...]` clause, or an extension of `same[]`/`span[]`? | open | Leaning `cu[]`, because XCE's resreq follows LSF. | | N22 | PR-10: relax policy: wait up to a limit for a better topology, then relax (Slurm `--switches=N@maxwait`)? Per job, or per queue or profile? | open | | | N23 | PR-10: rank ordering: does the scheduler return hosts in topology order, and is that order part of the persisted allocation and shown to the launcher (hostfile, env)? | open | | -| N24 | PR-10: demote `gang-groups` to optional/backlog, kept only for roles with separate lifecycles (e.g. RL trainer plus inference services)? | leaning | Leaning yes. Use-case review (2026-10-11), below "Gang use cases": a multi-job gang is needed mainly for RL post-training, disaggregated serving and streaming pipelines. All three mix batch and service components, so re-scope it as a "co-start group" across `service-replica-set` and jobs when RL or disaggregated serving comes onto the roadmap. | +| N24 | PR-10: demote `gang-groups` to optional/backlog, kept only for roles with separate lifecycles (e.g. RL trainer plus inference services)? | decided | User 2026-10-12: confirmed. Training is one parallel job (PR-10); `gang-groups` moves to the backlog, re-scoped as a co-start group. Roadmap, `resource-model.md` and the brief updated. Use-case review (2026-10-11), below "Gang use cases": a multi-job gang is needed mainly for RL post-training, disaggregated serving and streaming pipelines. All three mix batch and service components, so re-scope it as a "co-start group" across `service-replica-set` and jobs when RL or disaggregated serving comes onto the roadmap. | | N25 | PR-10: reservation or plan-ahead for large topology-constrained jobs, to prevent starvation and fragmentation: a new spec, or part of an existing one? | open | | | N26 | PR-11: placement groups only, or also direct job-to-job affinity (`affinity[job(...)]`)? | open | Leaning groups first (cheaper, simpler to explain). | | N27 | PR-11: which modes and strictness in v1: pack and spread, hard (`required`) and soft (`preferred`)? | open | | @@ -290,6 +290,7 @@ Source: the CSI specification, from general knowledge; to be checked against the | C3 | CSI compatibility: nice-to-have or a must? If a must, which side: the controller plugin through the proxy, the node plugin on the worker, or both? | discussing | Background and recommendation in "CSI notes" below (2026-10-11). | ## History +- 2026-10-12: N24 decided (user): `gang-groups` demoted to the backlog and re-scoped as a co-start group for RL post-training, disaggregated serving and streaming pipelines. - 2026-10-11: user asked about data cache, data and communication topology between jobs → PR-11 (one topology model; data locality with delay scheduling; placement groups with pack or spread); N26–N30 added. - 2026-10-11: user asked for the use cases of gangs → "Gang use cases" section; N24 leaning (demote `gang-groups`, re-scope later as a co-start group for RL, disaggregated serving and streaming). - 2026-10-11: user asked whether LLM training can be a single job with complex topology requirements → PR-10 (single parallel job as the default; topology at three levels; `gang-groups` optional); N21–N25 added. diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index 5352e4ef..c98344af 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -87,7 +87,6 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han - [ ] custom-resource-reporting — NEW 2026-10-11 (`/kiro-discovery`, "Unified resource model" phase 2; `resource-distribution-mapping` option 3 split out, pending user decision 1 in `resource-model.md`). Runtime custom resource values: an elim-like script reporter on workers (`posix_spawn`) and/or an external reporter role (decision 2), staleness and `interval` enforced, a string-set kind for dataset locality, instance-inventory publishing (DRA ResourceSlice-like). Brief: `.kiro/specs/custom-resource-reporting/brief.md`. Dependencies: resource-distribution-mapping, peer-authentication-hardening. - [ ] resource-instances — NEW 2026-10-11 (Unified resource model phase 3). Identified instances (GPU, MIG slice, NIC, ports) with attributes; topology-aware allocator plugin driving the unused `affinity[]` (NVLink domain, GPU-NIC affinity; switch block via `same[]`/`span[]`); durable per-job instance assignment (WAL/snapshot field-complete); worker binding (device cgroup, `CUDA_VISIBLE_DEVICES`, port env). Brief: `.kiro/specs/resource-instances/brief.md`. Dependencies: custom-resource-reporting, scheduler-framework. - [ ] resource-claims — NEW 2026-10-11 (Unified resource model phase 4). Data and volumes as lifecycle resources: template claims (per job/replica: scratch, staging, KV cache) and standalone claims (K8s PVC/ResourceClaim-like: checkpoint volumes, model weights, dataset caches; access modes RWO/ROX/RWX/RWOP; retain/delete), claim WAL entity, dispatch readiness gate (stage-in/out), storage provider contract (reused from resource-provisioner), worker publish into the sandbox, orphan handling. Absorbs the sandbox workspace and `transfer_files` staging (bug-1937 deferral). Brief: `.kiro/specs/resource-claims/brief.md`. Dependencies: resource-provisioner, resource-instances, job-sandbox-execution. -- [ ] gang-groups — NEW 2026-10-11 (Unified resource model phase 5, LLM training). All-or-nothing scheduling of a group of jobs (launcher + workers, JobSet-like), topology packing across the gang, start barrier until every member is placed and its claims are READY, group failure policy. Brief: `.kiro/specs/gang-groups/brief.md`. Dependencies: resource-instances, resource-claims, parjob-placement. ## Existing Spec Updates (2026-10-11 unified resource model discovery) Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generic supply map incl. provisioned supply, PR-2 resource provider proxy, PR-3 datasets in the map with staging as provisioning) and open questions: `.kiro/steering/resource-questions.md`. Duration/decay: must-have, future spec (R8). @@ -95,7 +94,7 @@ Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generi - [ ] service-endpoint — ports as identified instances of `resource-instances` instead of a separate port model. - [ ] job-sandbox-execution — the per-job workspace becomes the first template claim of `resource-claims`; publish (mounts) through the claim's worker step. Where data staging lives is decision 9. - [ ] application-profile — default claims (model weights, checkpoint) and GPU instance requests in profiles; `container` execution mode decides volume publishing (decision 6). -- [ ] job-resize — re-bind a job's standalone claims to the new host set; resize a gang group. +- [ ] job-resize — re-bind a job's standalone claims to the new host set. - [ ] service-replica-set — replicas share one read-many weights claim; rollout attaches version N+1 while N stays attached; per-replica template claims. - [ ] resource-provisioner — its provider contract, lease, epoch and orphan machinery are shared with storage/claim providers. - [ ] provisioning-demand — claim shortfall as provisionable demand. @@ -119,6 +118,7 @@ Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generi - [x] scheduler-sim-diagnostics — `info job ` enhancement: a PEND job shows its pend reason set (per alternative once scheduler-alter-pend-reasons lands; shows the flat reason until then); a RUN job shows per-host/per-resource reserved (`job->allocs`) vs. real used (`job->rusages`), and when no rusage has been reported yet says "resource usage not reported yet" (no fabricated zeros; reserved still shown). Byte units per its Req 6. Read-only. Next: `/kiro-spec-requirements scheduler-sim-diagnostics`. Dependencies: none (alternatives display depends on scheduler-alter-pend-reasons). ## Future Specs / Backlog (dependency order) +- [ ] gang-groups — NEW 2026-10-11; **demoted to backlog 2026-10-12** (user decision N24 in `resource-questions.md`). LLM training is one parallel job with topology requirements (PR-10), so it needs no gang. Re-scoped as a **co-start group** of separately managed jobs and services: RL post-training (trainer, rollout engines, reward and reference models), disaggregated serving (prefill and decode pools, router), streaming pipelines. All-or-nothing start, close placement, group failure policy. Revisit when RL or disaggregated serving reaches the roadmap. Brief: `.kiro/specs/gang-groups/brief.md`. Dependencies: resource-instances, resource-claims, service-replica-set. - [ ] project-guarantee — NEW 2026-10-09 (configuration layering, backlog). Project-owned guarantees (resource amount or host pool), borrowing of idle guaranteed capacity, reclaim through preemption. Brief: `.kiro/specs/project-guarantee/brief.md`. Dependencies: project-config, scheduler-preemption. - [ ] job-resize — NEW 2026-10-10 (from long-running services discovery; not on the services path). Grow/shrink a running parallel job's allocation (LSF resizable jobs, `bresize`) for elastic applications. Brief: `.kiro/specs/job-resize/brief.md`. Dependencies: parjob-placement, host-failure-job-recovery. - [ ] project-delegated-membership — NEW 2026-10-10 (from project-config requirements, user decision: future). Project managers edit their project's (and subtree's) members at runtime without an admin config push; needs a durable store that coexists with `xceadmin` config pushes (e.g. a separate delegated member list) and audit. Dependencies: project-config, rbac-persistence. @@ -181,7 +181,7 @@ Smaller items that likely don't warrant a full spec; pick up opportunistically o - Claims come in two forms, template and standalone (Kubernetes PVC/ResourceClaim-like), as WAL entities with the resource-provisioner discipline. - Adopt Kubernetes DRA/CSI concepts, not (yet) their APIs. - **Phases:** `resource-distribution-mapping` (shared pools) → `custom-resource-reporting` → `resource-instances` → `resource-claims` → `gang-groups`. Nine user decisions are pending, listed in `resource-model.md`. + **Phases:** `resource-distribution-mapping` (shared pools) → `custom-resource-reporting` → `resource-instances` → `resource-claims`; `gang-groups` demoted to the backlog (N24 decided 2026-10-12). Nine user decisions are pending, listed in `resource-model.md`. - **2026-07-06 admin-config-tool discovery** (`/kiro-discovery`, Path E mixed decomposition): | # | Workstream | Type | Language | Dependencies |