docs(steering): unified resource model; briefs for reporting, instances, claims, gangs

resource-model.md: one resource type descriptor and provider contract over
three layers (L1 attribute, L2 counter, L3 instance/claim); the scheduler
only selects from its cached snapshot, providers run after dispatch; batch
stays LSF-style; template and standalone (PVC/ResourceClaim-like) claims
as WAL entities with the resource-provisioner discipline; LSF/K8s mapping;
LLM training/serving needs; phases; nine pending user decisions.

New briefs: custom-resource-reporting, resource-instances,
resource-claims, gang-groups. Roadmap: long-term LLM training + serving
platform direction (features from LSF and Kubernetes), the four new specs,
updates to ten existing specs, discovery note. resource-distribution-mapping
becomes phase 1 (shared pools + descriptor fields).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-10-12 03:13:44 +00:00

4.2 KiB

Raw Blame History

Brief: resource-instances

(Resource-model discovery, 2026-10-11. Phase 3 of .kiro/steering/resource-model.md. Open decisions 5 (topology depth) and 7 (who changes MIG layouts) in that document are pending.)

Problem

Some resources are not interchangeable units. A job needs which GPUs it got (to set CUDA_VISIBLE_DEVICES and cgroup device rules), which ports (service endpoints), and units that sit close together: 8 GPUs in one NVLink domain, each with an affine NIC on the same PCIe switch, on hosts in one switch block. LLM training and serving depend on this. XCE allocations are counts only.

Current State

No identities: allocations are counts; xce_res_account_t.instid is a per-process counter, not a device identity (scheduler/src/sim_verify.c:73, bug-1844).
affinity[] is parsed and stored (lib/resreq.c:127) but never used.
GPUs: not modelled; only an unmapped gpu_req queue string (configs/queues.yaml:30). The scheduler-framework brief deferred GPU and affinity allocators as plugins (scheduler-framework/brief.md:364).
Ports: service-endpoint plans per-replica named ports as host resources whose allocated values matter.
Topology: whole parallel jobs are placed all-or-nothing with span[] (parjob-placement); there is no topology model below the host.

Desired Outcome

Identified instances: a resource type may declare identified instances (GPU, MIG slice, NIC, port range) with attributes (model, memory, NUMA node, NVLink domain, PCIe switch, rail). Inventory comes from built-in discovery or a reporter (custom-resource-reporting).
Placement:
- the scheduler counts instances like today's consumables, and also picks specific instances that meet attribute predicates and topology constraints (same domain, NIC affine to GPU, pack or spread);
- an allocator plugin consumes affinity[];
- host-level topology (switch block) is an attribute used by same[]/span[].
Durable identity: the instances allocated to a job are part of its allocation. They are durable (WAL/snapshot, field-complete), shown by info job/xcecli, and survive failover and replay.
Worker prepare step: binds the assigned instances to the job (device cgroup rules, CUDA_VISIBLE_DEVICES/equivalents, port env vars) and frees them at job end.
MIG/MPS: partitions are reported as instances. Changing the layout is a provider or admin action that drains the GPU, never a scheduler decision.

Approach

An allocator plugin in the scheduler framework works on the cached inventory snapshot (the DRA structured-parameters model: the scheduler selects, drivers prepare).
Allocations gain an instance list.
The worker gets a prepare/release hook.
The simulator replays instance inventory like resvals.

Scope

In:
- the instance inventory model and attributes;
- attribute predicates in requests;
- the topology-aware allocator (affinity[]);
- durable instance assignment;
- worker binding for GPUs and ports;
- display;
- a simulated GPU inventory.
Out: lifecycle resources (resource-claims); MIG layout changes (a provider action); multi-job gangs (gang-groups).

Boundary Candidates

Inventory and attribute model.
Allocator plugin.
Durable assignment (wire, WAL, snapshot).
Worker binding.
Request syntax for instance predicates.

Out of Boundary

Accounting policy for real usage (resource-accounting.md unchanged).

Upstream / Downstream

Upstream: custom-resource-reporting (inventory), scheduler-framework (allocator plugin interface), parjob-placement.
Downstream: service-endpoint (ports as identified instances), resource-claims (claims bound to instances), gang-groups, LLM training placement.

Existing Spec Touchpoints

Extends: scheduler-framework's deferred GPU/affinity allocator.
Adjacent: service-endpoint (should reuse this model for ports instead of its own), application-profile (default GPU requests), job-resize.

Constraints

Scheduler state: session cost stays bounded; instance choice is deterministic for sim/replay state-equivalence.
Persistence: assignment is field-complete in every save/restore seam (persistence.md; run /xce-audit-invariants).
Worker: binding follows the no-fork rule.

4.2 KiB Raw Blame History