docs(steering): unified resource model; briefs for reporting, instances, claims, gangs

resource-model.md: one resource type descriptor and provider contract over
three layers (L1 attribute, L2 counter, L3 instance/claim); the scheduler
only selects from its cached snapshot, providers run after dispatch; batch
stays LSF-style; template and standalone (PVC/ResourceClaim-like) claims
as WAL entities with the resource-provisioner discipline; LSF/K8s mapping;
LLM training/serving needs; phases; nine pending user decisions.

New briefs: custom-resource-reporting, resource-instances,
resource-claims, gang-groups. Roadmap: long-term LLM training + serving
platform direction (features from LSF and Kubernetes), the four new specs,
updates to ten existing specs, discovery note. resource-distribution-mapping
becomes phase 1 (shared pools + descriptor fields).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-10-12 03:13:44 +00:00

2.8 KiB

Raw Blame History

Brief: gang-groups

(Resource-model discovery, 2026-10-11. Phase 5 of .kiro/steering/resource-model.md, driven by the LLM training/serving platform direction.)

Problem

Distributed training is often several cooperating jobs that must start together or not at all, and with their storage ready:

a launcher plus worker groups;
a parameter server plus trainers;
heterogeneous roles with different resource requests (a Kubernetes JobSet, or a Volcano/Kueue gang).

Starting part of a gang wastes accelerators and can deadlock the cluster. XCE places one parallel job all-or-nothing, but has no all-or-nothing group of jobs.

Current State

One parallel job is already a gang: all its tasks are placed together under span[] (parjob-placement).
Heterogeneous single jobs: compound resreqs (N*{...}+M*{...}, lib/xce_resreq.peg:60-83) express different task shapes within one job.
Missing: a group of separate jobs scheduled atomically, topology packing across a gang (resource-instances), and readiness of the gang's claims (resource-claims).

Desired Outcome

Gang group: the jobs of a group are scheduled all-or-nothing in one session decision (or by a reservation that accumulates until complete, with a timeout and release).
Placement: packed by topology (switch block, NVLink domain) when requested.
Start: no member starts until every member is placed and all their claims are READY.
Group policy: if a member fails, the group policy decides: fail all, restart all, or tolerate.
Interactions: with job-resize, preemption and backfill.
Surfaces: visible in xcecli/REST (group state, member states, pend reason).

Approach

A group object, or a job-array-like construct, in the controller.
A scheduler gang step on top of the allocator framework, possibly reusing the parallel-job planner.
A group-level start barrier in the dispatch path.

Scope

In: group definition and submission, atomic placement, start barrier, failure policy, display, simulator support.
Out: elastic training frameworks themselves; claim and instance mechanics (other specs).

Boundary Candidates

Group object.
Scheduler gang step.
Dispatch barrier.
Failure policy.

Out of Boundary

Application-level rendezvous (torchrun etc.).

Upstream / Downstream

Upstream: resource-instances, resource-claims, parjob-placement, scheduler-framework.
Downstream: LLM training workflows, job-resize (group resize).

Existing Spec Touchpoints

Adjacent: job-dependency (ordering, not co-scheduling), scheduler-preemption, service-replica-set (replicas are independent, not a gang).

Constraints

Scheduler: deterministic for sim/replay.
Persistence: group state is durable and field-complete in snapshots.
HA/persistence: full 3-phase spec workflow.

2.8 KiB Raw Blame History