Rongsong Shen
c14fe3dab5
resource-model.md: one resource type descriptor and provider contract over three layers (L1 attribute, L2 counter, L3 instance/claim); the scheduler only selects from its cached snapshot, providers run after dispatch; batch stays LSF-style; template and standalone (PVC/ResourceClaim-like) claims as WAL entities with the resource-provisioner discipline; LSF/K8s mapping; LLM training/serving needs; phases; nine pending user decisions. New briefs: custom-resource-reporting, resource-instances, resource-claims, gang-groups. Roadmap: long-term LLM training + serving platform direction (features from LSF and Kubernetes), the four new specs, updates to ten existing specs, discovery note. resource-distribution-mapping becomes phase 1 (shared pools + descriptor fields). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
54 lines
2.8 KiB
Markdown
54 lines
2.8 KiB
Markdown
# Brief: gang-groups
|
|
|
|
(Resource-model discovery, 2026-10-11. Phase 5 of `.kiro/steering/resource-model.md`, driven by the LLM training/serving platform direction.)
|
|
|
|
## Problem
|
|
Distributed training is often several cooperating jobs that must start together or not at all, and with their storage ready:
|
|
- a launcher plus worker groups;
|
|
- a parameter server plus trainers;
|
|
- heterogeneous roles with different resource requests (a Kubernetes JobSet, or a Volcano/Kueue gang).
|
|
|
|
Starting part of a gang wastes accelerators and can deadlock the cluster. XCE places one parallel job all-or-nothing, but has no all-or-nothing group of jobs.
|
|
|
|
## Current State
|
|
- **One parallel job is already a gang:** all its tasks are placed together under `span[]` (`parjob-placement`).
|
|
- **Heterogeneous single jobs:** compound resreqs (`N*{...}+M*{...}`, `lib/xce_resreq.peg:60-83`) express different task shapes within one job.
|
|
- **Missing:** a group of separate jobs scheduled atomically, topology packing across a gang (`resource-instances`), and readiness of the gang's claims (`resource-claims`).
|
|
|
|
## Desired Outcome
|
|
- **Gang group:** the jobs of a group are scheduled all-or-nothing in one session decision (or by a reservation that accumulates until complete, with a timeout and release).
|
|
- **Placement:** packed by topology (switch block, NVLink domain) when requested.
|
|
- **Start:** no member starts until every member is placed and all their claims are READY.
|
|
- **Group policy:** if a member fails, the group policy decides: fail all, restart all, or tolerate.
|
|
- **Interactions:** with `job-resize`, preemption and backfill.
|
|
- **Surfaces:** visible in xcecli/REST (group state, member states, pend reason).
|
|
|
|
## Approach
|
|
- A group object, or a job-array-like construct, in the controller.
|
|
- A scheduler gang step on top of the allocator framework, possibly reusing the parallel-job planner.
|
|
- A group-level start barrier in the dispatch path.
|
|
|
|
## Scope
|
|
- **In:** group definition and submission, atomic placement, start barrier, failure policy, display, simulator support.
|
|
- **Out:** elastic training frameworks themselves; claim and instance mechanics (other specs).
|
|
|
|
## Boundary Candidates
|
|
- Group object.
|
|
- Scheduler gang step.
|
|
- Dispatch barrier.
|
|
- Failure policy.
|
|
|
|
## Out of Boundary
|
|
- Application-level rendezvous (torchrun etc.).
|
|
|
|
## Upstream / Downstream
|
|
- **Upstream:** `resource-instances`, `resource-claims`, `parjob-placement`, `scheduler-framework`.
|
|
- **Downstream:** LLM training workflows, `job-resize` (group resize).
|
|
|
|
## Existing Spec Touchpoints
|
|
- **Adjacent:** `job-dependency` (ordering, not co-scheduling), `scheduler-preemption`, `service-replica-set` (replicas are independent, not a gang).
|
|
|
|
## Constraints
|
|
- **Scheduler:** deterministic for sim/replay.
|
|
- **Persistence:** group state is durable and field-complete in snapshots.
|
|
- **HA/persistence:** full 3-phase spec workflow.
|