docs(steering): unified resource model; briefs for reporting, instances, claims, gangs
resource-model.md: one resource type descriptor and provider contract over three layers (L1 attribute, L2 counter, L3 instance/claim); the scheduler only selects from its cached snapshot, providers run after dispatch; batch stays LSF-style; template and standalone (PVC/ResourceClaim-like) claims as WAL entities with the resource-provisioner discipline; LSF/K8s mapping; LLM training/serving needs; phases; nine pending user decisions. New briefs: custom-resource-reporting, resource-instances, resource-claims, gang-groups. Roadmap: long-term LLM training + serving platform direction (features from LSF and Kubernetes), the four new specs, updates to ten existing specs, discovery note. resource-distribution-mapping becomes phase 1 (shared pools + descriptor fields). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
b01b341869
commit
c14fe3dab5
GPG key ID:
80533536B0EB651C
8 changed files with 523 additions and 1 deletions
66
.kiro/specs/custom-resource-reporting/brief.md
Normal file
66
.kiro/specs/custom-resource-reporting/brief.md
Normal file
|
|
@ -0,0 +1,66 @@
|
|||
# Brief: custom-resource-reporting
|
||||
|
||||
(Resource-model discovery, 2026-10-11. Phase 2 of `.kiro/steering/resource-model.md`, which is `resource-distribution-mapping` option 3, split out into its own spec. Pending open decisions 1 and 2 in that document: whether this stays a separate spec, and who reports values.)
|
||||
|
||||
## Problem
|
||||
Admins can declare custom resources, but only with static per-host values. A value that changes at runtime has no way in: free space in a scratch filesystem, a dataset present in a node's cache, the number of free licenses on a vendor server, a site-specific health flag. LSF solves this with elim, PBS with server and mom hooks, Kubernetes with device plugins and DRA ResourceSlices.
|
||||
|
||||
## Current State
|
||||
- **Built-in values only:** the worker reports 11 compiled-in load indices (`src/worker/src/resource_sync.c:84-104`). `resource_sync_register_load_index()` (`:116`) has no external callers, and there is no `dlopen` or script path.
|
||||
- **Worker-only updates:** `do_update_resvals()` accepts updates only from a `HostRole_worker` connection (`src/controller/opupd.c:11-26`).
|
||||
- **Unused descriptor fields:** `resources.yaml` `interval` is never enforced, and `last_report` is never used (`resource-distribution-mapping/research.md:17`).
|
||||
- **No set type:** "has datasets {a,b,c}" needs one boolean resource per dataset.
|
||||
|
||||
## Desired Outcome
|
||||
- **Reporters:** a custom resource names a reporter in its type descriptor. The reporter is either a script run on each worker (elim-like, started with `posix_spawn`, emitting JSON values each `interval`) or an external reporter role connected to the controller, for cluster-scope values such as a license server's free count.
|
||||
- **Value lifecycle:**
|
||||
- values are checked against the declared kind;
|
||||
- a value not refreshed within its interval turns stale and is handled per resource: unknown, or excluded from `select[]`;
|
||||
- a reporter's failure is visible to admins.
|
||||
- **New kind:** string-set, so `select[datasets has "imagenet"]` can express locality.
|
||||
- **Inventory:** a reporter can also publish instance inventory with attributes. That is the input `resource-instances` consumes, shaped like a DRA ResourceSlice.
|
||||
- **Scheduler:** the scheduler sees reported values exactly like built-ins, from its cached snapshot, with no new work in the session hot path.
|
||||
|
||||
## Approach
|
||||
- Extend the worker's `resource_sync` with a script reporter, under the no-fork rule (`tech.md` "Process Creation").
|
||||
- Add a reporter role (or reuse the provisioner role) with an allowlisted update opcode next to `do_update_resvals`.
|
||||
- The wire format is additive, and values round-trip through the scheduler snapshot.
|
||||
|
||||
## Scope
|
||||
- **In:**
|
||||
- script reporters on workers and an external reporter role;
|
||||
- staleness and interval;
|
||||
- the string-set kind and its `select[]` operators;
|
||||
- inventory publishing (format only; its consumers are in `resource-instances`);
|
||||
- xcecli/REST display of reported values and reporter health;
|
||||
- a simulated reporter for the scheduler simulator.
|
||||
- **Out:** shared pools themselves (`resource-distribution-mapping` option 2); device identity and allocation (`resource-instances`); lifecycle (`resource-claims`).
|
||||
|
||||
## Boundary Candidates
|
||||
- Worker script reporter.
|
||||
- Controller reporter role and update path.
|
||||
- Value kinds and staleness.
|
||||
- Inventory format.
|
||||
|
||||
## Out of Boundary
|
||||
- Scheduling policy changes.
|
||||
- Provisioning.
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream:**
|
||||
- `resource-distribution-mapping`: the descriptor and shared pools;
|
||||
- `peer-authentication-hardening`: the reporter role on the allowlist;
|
||||
- the no-fork rule.
|
||||
- **Downstream:**
|
||||
- `resource-instances`, which consumes the inventory;
|
||||
- `elastic-shared-resources`, which may reuse the reporter role for license-server counts;
|
||||
- data-locality placement.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends:** `resource-distribution-mapping` (its option 3 scope).
|
||||
- **Adjacent:** `resource-value-verification` (end-to-end value propagation checks), `resource-provisioner` (role and lease patterns), `xce-node-manager` (supervision of reporter processes).
|
||||
|
||||
## Constraints
|
||||
- **Scheduler:** it never calls a reporter.
|
||||
- **Persistence:** reported values follow the persistence rules for their kind. Dynamic values are not WAL facts, like load indices, but must be in the scheduler snapshot.
|
||||
- **Credentials:** reporter credentials stay with the reporter.
|
||||
54
.kiro/specs/gang-groups/brief.md
Normal file
54
.kiro/specs/gang-groups/brief.md
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
# Brief: gang-groups
|
||||
|
||||
(Resource-model discovery, 2026-10-11. Phase 5 of `.kiro/steering/resource-model.md`, driven by the LLM training/serving platform direction.)
|
||||
|
||||
## Problem
|
||||
Distributed training is often several cooperating jobs that must start together or not at all, and with their storage ready:
|
||||
- a launcher plus worker groups;
|
||||
- a parameter server plus trainers;
|
||||
- heterogeneous roles with different resource requests (a Kubernetes JobSet, or a Volcano/Kueue gang).
|
||||
|
||||
Starting part of a gang wastes accelerators and can deadlock the cluster. XCE places one parallel job all-or-nothing, but has no all-or-nothing group of jobs.
|
||||
|
||||
## Current State
|
||||
- **One parallel job is already a gang:** all its tasks are placed together under `span[]` (`parjob-placement`).
|
||||
- **Heterogeneous single jobs:** compound resreqs (`N*{...}+M*{...}`, `lib/xce_resreq.peg:60-83`) express different task shapes within one job.
|
||||
- **Missing:** a group of separate jobs scheduled atomically, topology packing across a gang (`resource-instances`), and readiness of the gang's claims (`resource-claims`).
|
||||
|
||||
## Desired Outcome
|
||||
- **Gang group:** the jobs of a group are scheduled all-or-nothing in one session decision (or by a reservation that accumulates until complete, with a timeout and release).
|
||||
- **Placement:** packed by topology (switch block, NVLink domain) when requested.
|
||||
- **Start:** no member starts until every member is placed and all their claims are READY.
|
||||
- **Group policy:** if a member fails, the group policy decides: fail all, restart all, or tolerate.
|
||||
- **Interactions:** with `job-resize`, preemption and backfill.
|
||||
- **Surfaces:** visible in xcecli/REST (group state, member states, pend reason).
|
||||
|
||||
## Approach
|
||||
- A group object, or a job-array-like construct, in the controller.
|
||||
- A scheduler gang step on top of the allocator framework, possibly reusing the parallel-job planner.
|
||||
- A group-level start barrier in the dispatch path.
|
||||
|
||||
## Scope
|
||||
- **In:** group definition and submission, atomic placement, start barrier, failure policy, display, simulator support.
|
||||
- **Out:** elastic training frameworks themselves; claim and instance mechanics (other specs).
|
||||
|
||||
## Boundary Candidates
|
||||
- Group object.
|
||||
- Scheduler gang step.
|
||||
- Dispatch barrier.
|
||||
- Failure policy.
|
||||
|
||||
## Out of Boundary
|
||||
- Application-level rendezvous (torchrun etc.).
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream:** `resource-instances`, `resource-claims`, `parjob-placement`, `scheduler-framework`.
|
||||
- **Downstream:** LLM training workflows, `job-resize` (group resize).
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Adjacent:** `job-dependency` (ordering, not co-scheduling), `scheduler-preemption`, `service-replica-set` (replicas are independent, not a gang).
|
||||
|
||||
## Constraints
|
||||
- **Scheduler:** deterministic for sim/replay.
|
||||
- **Persistence:** group state is durable and field-complete in snapshots.
|
||||
- **HA/persistence:** full 3-phase spec workflow.
|
||||
80
.kiro/specs/resource-claims/brief.md
Normal file
80
.kiro/specs/resource-claims/brief.md
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
# Brief: resource-claims
|
||||
|
||||
(Resource-model discovery, 2026-10-11. Phase 4 of `.kiro/steering/resource-model.md`. Open decisions 3 (first-class claims and quotas), 4 (own contract vs CSI wire compatibility), 6 (containers) and 9 (where staging lives) in that document are pending.)
|
||||
|
||||
## Problem
|
||||
Data and volumes are resources with a lifecycle: something must be created, staged or attached before a job runs, and released or cleaned up after it, even when the job fails. Some must outlive any single job and be shared:
|
||||
- a checkpoint volume re-attached when training restarts or resizes;
|
||||
- model weights read by every serving replica;
|
||||
- a dataset cache used by many jobs.
|
||||
|
||||
XCE has no such object. File staging (`transfer_files`) is parsed but never performed, and sandbox workspaces are planned without a resource model.
|
||||
|
||||
## Current State
|
||||
- **Staging:** stage-in/out "remains unimplemented (future spec)" (`.wolf/bugs/bug-1937.md:48`).
|
||||
- **Workspaces:** `job-sandbox-execution` plans per-job workspaces with create, hand-over, mandatory cleanup or re-own (`brief.md:78-100`). This is a per-job claim in all but name.
|
||||
- **Missing everywhere:** volumes, burst buffers, datasets, checkpoints and access modes.
|
||||
- **Prior design:** `resource-provisioner` research (2026-10-10) already set the rules for externally created things that cost money or hold data: idempotent `op_id`, owner tags, intent committed before action, lease epoch, orphan classes, a JSON provider contract, `posix_spawn`.
|
||||
|
||||
## Desired Outcome
|
||||
- **Two claim forms:**
|
||||
- **Template claim:** declared in a job's resreq or application profile; created per job, task or replica; released with it. Examples: scratch, staging, KV cache.
|
||||
- **Standalone claim:** a controller entity with its own id, owner/project, access mode (RWO/ROX/RWX/RWOP), size and parameters, and reclaim policy (`retain`/`delete`). Many jobs or replicas reference it (e.g. `--claim ckpt-7:rw`), and it outlives them.
|
||||
- **Lifecycle:** claim states `PENDING → PREPARING → READY → RELEASING → RELEASED/FAILED`.
|
||||
- Dispatch waits until all claims of the job (or gang) are READY: a stage-in gate.
|
||||
- Stage-out and release run after the job; failures are visible and retried.
|
||||
- **Placement:**
|
||||
- the scheduler reserves capacity (L2 counter) and placement;
|
||||
- a standalone claim's reachability (volume reachable from hosts H, cache present on H) is a placement constraint read from the cached snapshot.
|
||||
- **Providers:** a provider (create/attach/release/health) runs in the provisioner daemon for external storage. The worker does node-local prepare/publish: mount into the sandbox, set ownership.
|
||||
- **Recovery:** claims are WAL entities, field-complete in snapshots, and survive failover. Orphan scans cover claims with no owner and provider objects with no claim.
|
||||
- **Surfaces:** xcecli/REST to create, list, show and delete standalone claims; claims shown on jobs; audit.
|
||||
|
||||
## Approach
|
||||
- Reuse the resource-provisioner provider contract, lease and orphan machinery.
|
||||
- Absorb `job-sandbox-execution`'s workspace as the first template claim, and `transfer_files` staging as the stage-in/out claim (decision 9).
|
||||
- Add the checkpoint volume as the first standalone claim.
|
||||
|
||||
## Scope
|
||||
- **In:**
|
||||
- both claim forms;
|
||||
- claim facts and snapshot rows;
|
||||
- the dispatch readiness gate;
|
||||
- the provider contract for storage;
|
||||
- worker publish into the sandbox;
|
||||
- access modes;
|
||||
- reclaim policy;
|
||||
- orphan handling;
|
||||
- REST/xcecli;
|
||||
- a simulated storage provider.
|
||||
- **Out:** device identity (`resource-instances`); multi-job gangs (`gang-groups`); data-locality-aware caching policy beyond reachability (later).
|
||||
|
||||
## Boundary Candidates
|
||||
- Claim entity and lifecycle (controller, WAL, snapshot).
|
||||
- Readiness gate and job states.
|
||||
- Provider contract for storage.
|
||||
- Worker publish.
|
||||
- REST/xcecli/RBAC.
|
||||
|
||||
## Out of Boundary
|
||||
- Building a storage system.
|
||||
- The scheduler calling providers.
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream:**
|
||||
- `resource-provisioner` (provider contract, lease, orphans);
|
||||
- `resource-instances` (claims bound to instances);
|
||||
- `job-sandbox-execution` (where publish mounts);
|
||||
- `peer-authentication-hardening`;
|
||||
- `application-profile` (template claims in profiles).
|
||||
- **Downstream:** `job-resize` and `service-replica-set` (re-bind and shared weights), `gang-groups`, `provisioning-demand` (claim shortfall).
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends:** `job-sandbox-execution` (workspace as a claim); closes bug-1937's deferred staging.
|
||||
- **Adjacent:** `elastic-hosts`, `project-guarantee` (quota on claims, decision 3).
|
||||
|
||||
## Constraints
|
||||
- **HA/persistence:** full 3-phase spec workflow; a claim that holds data or costs money is durable from v1.
|
||||
- **Scheduler:** never calls or waits on a provider.
|
||||
- **Process creation:** no fork in threaded processes.
|
||||
- **Credentials:** stay with the provider; users cannot choose arbitrary mount paths.
|
||||
|
|
@ -28,3 +28,9 @@
|
|||
## Decision
|
||||
|
||||
Not yet made — user chose to record the options and decide later rather than commit now. **Do not proceed to requirements.md generation until one of the three scopes (or a variant) is chosen.** When resumed, re-run `/kiro-spec-requirements resource-distribution-mapping` with the chosen scope stated explicitly, since the EARS requirements (and everything downstream) differ substantially by option.
|
||||
|
||||
## Note (2026-10-11): unified resource model direction
|
||||
`.kiro/steering/resource-model.md` (resource-model discovery) makes this spec **phase 1** of the unified resource model:
|
||||
- Shared pools (option 2), plus the resource type descriptor fields `scope`, `unit`, `reporter`, `layer`, `instances` and `provider`, which make today's flags explicit.
|
||||
- Option 3 (runtime custom values) moves to the new `custom-resource-reporting` spec, unless the user keeps it here (pending decision 1 there).
|
||||
- Licenses (`elastic-shared-resources`) depend on the shared pool.
|
||||
|
|
|
|||
63
.kiro/specs/resource-instances/brief.md
Normal file
63
.kiro/specs/resource-instances/brief.md
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
# Brief: resource-instances
|
||||
|
||||
(Resource-model discovery, 2026-10-11. Phase 3 of `.kiro/steering/resource-model.md`. Open decisions 5 (topology depth) and 7 (who changes MIG layouts) in that document are pending.)
|
||||
|
||||
## Problem
|
||||
Some resources are not interchangeable units. A job needs *which* GPUs it got (to set `CUDA_VISIBLE_DEVICES` and cgroup device rules), *which* ports (service endpoints), and units that sit close together: 8 GPUs in one NVLink domain, each with an affine NIC on the same PCIe switch, on hosts in one switch block. LLM training and serving depend on this. XCE allocations are counts only.
|
||||
|
||||
## Current State
|
||||
- **No identities:** allocations are counts; `xce_res_account_t.instid` is a per-process counter, not a device identity (`scheduler/src/sim_verify.c:73`, bug-1844).
|
||||
- **`affinity[]`** is parsed and stored (`lib/resreq.c:127`) but never used.
|
||||
- **GPUs:** not modelled; only an unmapped `gpu_req` queue string (`configs/queues.yaml:30`). The scheduler-framework brief deferred GPU and affinity allocators as plugins (`scheduler-framework/brief.md:364`).
|
||||
- **Ports:** `service-endpoint` plans per-replica named ports as host resources whose allocated values matter.
|
||||
- **Topology:** whole parallel jobs are placed all-or-nothing with `span[]` (`parjob-placement`); there is no topology model below the host.
|
||||
|
||||
## Desired Outcome
|
||||
- **Identified instances:** a resource type may declare identified instances (GPU, MIG slice, NIC, port range) with attributes (model, memory, NUMA node, NVLink domain, PCIe switch, rail). Inventory comes from built-in discovery or a reporter (`custom-resource-reporting`).
|
||||
- **Placement:**
|
||||
- the scheduler counts instances like today's consumables, and also picks specific instances that meet attribute predicates and topology constraints (same domain, NIC affine to GPU, pack or spread);
|
||||
- an allocator plugin consumes `affinity[]`;
|
||||
- host-level topology (switch block) is an attribute used by `same[]`/`span[]`.
|
||||
- **Durable identity:** the instances allocated to a job are part of its allocation. They are durable (WAL/snapshot, field-complete), shown by `info job`/xcecli, and survive failover and replay.
|
||||
- **Worker prepare step:** binds the assigned instances to the job (device cgroup rules, `CUDA_VISIBLE_DEVICES`/equivalents, port env vars) and frees them at job end.
|
||||
- **MIG/MPS:** partitions are reported as instances. Changing the layout is a provider or admin action that drains the GPU, never a scheduler decision.
|
||||
|
||||
## Approach
|
||||
- An allocator plugin in the scheduler framework works on the cached inventory snapshot (the DRA structured-parameters model: the scheduler selects, drivers prepare).
|
||||
- Allocations gain an instance list.
|
||||
- The worker gets a prepare/release hook.
|
||||
- The simulator replays instance inventory like `resvals`.
|
||||
|
||||
## Scope
|
||||
- **In:**
|
||||
- the instance inventory model and attributes;
|
||||
- attribute predicates in requests;
|
||||
- the topology-aware allocator (`affinity[]`);
|
||||
- durable instance assignment;
|
||||
- worker binding for GPUs and ports;
|
||||
- display;
|
||||
- a simulated GPU inventory.
|
||||
- **Out:** lifecycle resources (`resource-claims`); MIG layout changes (a provider action); multi-job gangs (`gang-groups`).
|
||||
|
||||
## Boundary Candidates
|
||||
- Inventory and attribute model.
|
||||
- Allocator plugin.
|
||||
- Durable assignment (wire, WAL, snapshot).
|
||||
- Worker binding.
|
||||
- Request syntax for instance predicates.
|
||||
|
||||
## Out of Boundary
|
||||
- Accounting policy for real usage (`resource-accounting.md` unchanged).
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream:** `custom-resource-reporting` (inventory), `scheduler-framework` (allocator plugin interface), `parjob-placement`.
|
||||
- **Downstream:** `service-endpoint` (ports as identified instances), `resource-claims` (claims bound to instances), `gang-groups`, LLM training placement.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends:** `scheduler-framework`'s deferred GPU/affinity allocator.
|
||||
- **Adjacent:** `service-endpoint` (should reuse this model for ports instead of its own), `application-profile` (default GPU requests), `job-resize`.
|
||||
|
||||
## Constraints
|
||||
- **Scheduler state:** session cost stays bounded; instance choice is deterministic for sim/replay state-equivalence.
|
||||
- **Persistence:** assignment is field-complete in every save/restore seam (`persistence.md`; run `/xce-audit-invariants`).
|
||||
- **Worker:** binding follows the no-fork rule.
|
||||
225
.kiro/steering/resource-model.md
Normal file
225
.kiro/steering/resource-model.md
Normal file
|
|
@ -0,0 +1,225 @@
|
|||
# Resource Model: one vocabulary, three layers
|
||||
|
||||
**Status:** direction adopted 2026-10-11 (resource-model discovery, after the
|
||||
2026-10-10 resource-provisioner research). The open decisions at the end are
|
||||
the user's and still pending; specs built on this document must not settle
|
||||
them silently.
|
||||
|
||||
## Why
|
||||
|
||||
XCE is heading toward a unified platform for applications such as LLM
|
||||
training **and** deployment (serving), not only classic HPC batch. So
|
||||
resource features come from IBM LSF (consumables, `rusage`, elim, licenses,
|
||||
`-app` profiles, data manager) **and** Kubernetes (extended resources, device
|
||||
plugins, CSI volumes, Dynamic Resource Allocation claims). Three asks drive
|
||||
this document:
|
||||
|
||||
1. Treat data and volumes as resources.
|
||||
2. Let admins define custom resources.
|
||||
3. Give all resources one interface.
|
||||
|
||||
The answer is yes to all three, as **one resource type vocabulary and one
|
||||
provider contract over three layers of behaviour**. It is not a single
|
||||
mechanism: none of LSF, Slurm, PBS or Kubernetes uses one mechanism for
|
||||
everything, and a per-job claim object for `slots`/`mem` would put lifecycle
|
||||
cost and WAL traffic on the scheduler's hottest path.
|
||||
|
||||
## Current state (2026-10-11)
|
||||
|
||||
- **Descriptor:** `xce_resource_t` (`src/include/xce/types.h:133-145`) has
|
||||
three value kinds (string, numeric, boolean) and the flags BUILTIN,
|
||||
DYNAMIC, GLOBAL, SHARED, EXTERNAL, RELEASE, DEFAULT_HOST, CONSUMABLE and
|
||||
LIMIT (`types.h:81-89`). Config (`resources.yaml` via `xceadmin`) reaches
|
||||
only CONSUMABLE and RELEASE. `resourcemap` is dropped at build
|
||||
(`xceadmin/build.py:38-43`), and its capnp fields are commented out
|
||||
(`controller.capnp:213-214`).
|
||||
- **Works with config only:** static per-host custom resources (numeric,
|
||||
boolean, string, consumable or not) declared in `resources.yaml`, with
|
||||
per-host values in `hosts.yaml`.
|
||||
- **Missing:**
|
||||
- runtime custom values: only a worker connection may update values
|
||||
(`controller/opupd.c:11-26`); the 11 load indices are compiled in
|
||||
(`worker/src/resource_sync.c:84-104`); there is no elim-like reporter;
|
||||
- cluster-wide pools such as licenses: the pool machinery exists for limit
|
||||
virtual resources (`sharedInst`, `scheduler/src/resource.c:292`), but
|
||||
`mark_host_res()` rejects SHARED (`controller/worker.c:259`). The scope
|
||||
choice is pending in `resource-distribution-mapping`;
|
||||
- device or volume identity: allocations are counts, and
|
||||
`xce_res_account_t.instid` is a per-process counter, not an identity;
|
||||
- lifecycle (create, stage, attach, release);
|
||||
- `rusage` `duration`/`decay`: parsed, ignored (`alloc_runner.c:429`);
|
||||
- `affinity[]`: parsed (`lib/resreq.c:127`), never used;
|
||||
- GPUs: only an unmapped `gpu_req` queue string.
|
||||
- **Data:** only scratch capacity (`tmp`/`maxtmp`). File staging is parsed
|
||||
but never performed (bug-1937). Sandbox workspaces
|
||||
(`job-sandbox-execution`) are a planned per-job storage lifecycle, but are
|
||||
not modelled as resources.
|
||||
|
||||
## The model
|
||||
|
||||
### Resource type (one descriptor for every resource)
|
||||
|
||||
`name`, `kind` (numeric, boolean, string, later string-set), `unit`,
|
||||
`order`, `consumable`, `scope` (host, host group, shared pool, external),
|
||||
`reporter` (static, builtin, provider), `interval`, `layer`, `instances`
|
||||
(counted, or identified with attributes), `provider` (reference; optional).
|
||||
This makes today's flags explicit, and it is the descriptor that
|
||||
`resource-distribution-mapping` extends.
|
||||
|
||||
### Three layers
|
||||
|
||||
| Layer | What it is | Examples | Scheduler sees |
|
||||
|---|---|---|---|
|
||||
| **L1 attribute** | not consumed; matched by `select[]` | host model, dataset X cached here, NVLink domain, switch block | cached values (as today) |
|
||||
| **L2 counter** | consumed amount, host or pool scope | `slots`, `mem`, scratch GB, licenses, filesystem quota | cached `resvals`/`sharedInst` (as today) |
|
||||
| **L3 instance / claim** | *which* unit, optionally with a lifecycle | GPU, MIG slice, NIC rail, port; volumes, burst buffer, checkpoint and model-weight stores | identified inventory with attributes; claim placement as a constraint |
|
||||
|
||||
A resource opts into L3 only if it needs identity or lifecycle. L1 and L2
|
||||
stay as fast and deterministic as now.
|
||||
|
||||
### Provider contract
|
||||
|
||||
A resource can have a provider with optional capabilities: `report`
|
||||
(values, or an instance inventory with attributes), `prepare`/`attach` before
|
||||
use, `release`/`cleanup` after, and `health`. It shares the transport and
|
||||
rules of the resource-provisioner contract
|
||||
(`.kiro/specs/resource-provisioner/research.md`):
|
||||
|
||||
- JSON documents; one-shot scripts first, a persistent JSON-lines mode later;
|
||||
- every mutating action idempotent, keyed by an `op_id`;
|
||||
- owner tags on every external object;
|
||||
- an intent fact committed before any external action;
|
||||
- a lease epoch on facts;
|
||||
- orphan scans;
|
||||
- started with `posix_spawn()` or Go `os/exec`, never `fork()` in a threaded
|
||||
process (`tech.md` "Process Creation").
|
||||
|
||||
**Where providers run:**
|
||||
- node-local (GPU, scratch, mounts): under the worker, which does the prepare
|
||||
step (devices, cgroup, env);
|
||||
- external (cloud, storage arrays, license servers): under a provisioner
|
||||
daemon with its own controller role.
|
||||
|
||||
### Claims (L3 with a lifecycle)
|
||||
|
||||
- **Template claim** (like a K8s ResourceClaimTemplate): written in a job's
|
||||
resreq or application profile, created per job, task or replica, and
|
||||
released with it. Examples: scratch, staging, KV cache. This is the LSF
|
||||
mental model.
|
||||
- **Standalone claim** (like a K8s PVC or ResourceClaim): a controller entity
|
||||
with its own id, owner/project, access mode (RWO, ROX, RWX, RWOP) and
|
||||
reclaim policy (`retain`/`delete`). Many jobs or replicas reference it
|
||||
(e.g. `--claim ckpt-7:rw`), and it outlives them. It is re-bound on restart,
|
||||
resize or replica replacement. Examples: checkpoint volumes (one writer
|
||||
gang), model weights (read-many by every replica), dataset caches.
|
||||
- **States:** `PENDING → PREPARING → READY → RELEASING → RELEASED/FAILED`.
|
||||
Dispatch waits until every claim of the job (or of the whole gang) is
|
||||
READY.
|
||||
|
||||
## Rules every resource design must keep
|
||||
|
||||
- **The scheduler only chooses.** It reads numbers, attributes and inventory
|
||||
from its cached session snapshot (the DRA "structured parameters" model).
|
||||
It never calls a provider and never waits for one. Prepare, attach,
|
||||
staging and release happen in the controller, worker or provisioner after
|
||||
dispatch. Claim readiness is an input event, not something the scheduler
|
||||
polls. This keeps sessions fast and sim/replay deterministic; the
|
||||
simulator models provider latency.
|
||||
- **Persistence:** instance assignments (which GPU or port), claim records,
|
||||
shared totals and borrowed amounts are WAL facts or snapshot fields. They
|
||||
must round-trip through every save/restore and apply seam (`persistence.md`
|
||||
field-completeness; run `/xce-audit-invariants`). A claim that costs money
|
||||
or holds data is durable from its first version.
|
||||
- **HA:** claims follow the resource-provisioner discipline: intent before
|
||||
action, `op_id`, epoch, owner tags, orphan classes.
|
||||
- **Batch stays LSF-style:** counted consumables, `rusage`, `max(reserved,
|
||||
actual)` accounting (`resource-accounting.md`), licenses as shared pools,
|
||||
limits as virtual resources. These need no lifecycle and stay constant-cost
|
||||
per session.
|
||||
- **Request syntax stays LSF-style:** `select[]`, `rusage[]`, `span[]`,
|
||||
`same[]`, `affinity[]`, extended with attribute predicates over instances.
|
||||
There is no CEL engine in the scheduler.
|
||||
- **Security:**
|
||||
- provider credentials stay in the provider process, never in job exdata,
|
||||
cluster config sent to workers, or the wire;
|
||||
- every reporter and provider role is on the per-role opcode allowlist
|
||||
(`peer-authentication-hardening`);
|
||||
- users never choose arbitrary mount paths (application-profile mount
|
||||
rules).
|
||||
|
||||
## Mapping to LSF and Kubernetes
|
||||
|
||||
| XCE | LSF | Kubernetes |
|
||||
|---|---|---|
|
||||
| resource type descriptor | `lsf.shared` + ResourceMap | DeviceClass / StorageClass / extended resource |
|
||||
| L1/L2 reporter (static, builtin, provider `report`) | elim | device plugin ListAndWatch / DRA ResourceSlice |
|
||||
| L2 shared pool | shared resource, License Scheduler | (extended resource at cluster level) |
|
||||
| L3 identified instance | `-gpu`, affinity | device plugin Allocate / DRA device |
|
||||
| template claim | `rusage` + data manager staging | ResourceClaimTemplate, ephemeral volume |
|
||||
| standalone claim | (none) | PVC, ResourceClaim |
|
||||
| worker prepare/publish | (job starter, cgroup) | NodePrepareResources, CSI NodeStage/Publish |
|
||||
| provider create/attach | resource connector | CSI controller |
|
||||
| access mode | (none) | RWO/ROX/RWX/RWOP |
|
||||
|
||||
Adopt the Kubernetes **concepts**. Whether XCE also speaks the CSI or
|
||||
device-plugin **wire protocols** is open decision 4.
|
||||
|
||||
## LLM training and serving
|
||||
|
||||
- **Accelerator topology:** NVLink/NVSwitch domains, GPU and NIC on one PCIe
|
||||
switch, IB rails. These are L3 instance attributes plus constraints
|
||||
between instances, served by an allocator that consumes `affinity[]`.
|
||||
Cluster-level topology (switch block) is an L1 host attribute that
|
||||
`same[]`/`span[]` already pack against.
|
||||
- **MIG/MPS:** partitions are reported as instances. Changing a layout is a
|
||||
provider action that drains the GPU, never a scheduler decision.
|
||||
- **Gangs:** one parallel job is already placed all-or-nothing
|
||||
(`parjob-placement`). A gang of several jobs (launcher plus workers) needs
|
||||
a gang group; the whole gang and its claims are READY before any part
|
||||
starts.
|
||||
- **Elastic resize and restart:** standalone claims are re-bound to the new
|
||||
host set.
|
||||
- **Serving:** replicas share one read-many weights claim and have
|
||||
per-replica template claims. A rolling update attaches version N+1 while
|
||||
version N stays attached.
|
||||
|
||||
## Phases
|
||||
|
||||
1. **`resource-distribution-mapping`:** shared pools (option 2) plus the
|
||||
descriptor fields above. Unblocks licenses (`elastic-shared-resources`)
|
||||
and shared quotas.
|
||||
2. **`custom-resource-reporting`:** runtime custom values and inventory
|
||||
(option 3): reporters other than the worker connection, a string-set kind
|
||||
for dataset locality, and `interval` enforced.
|
||||
3. **`resource-instances`:** identified GPU, MIG, NIC and port instances
|
||||
with attributes, durable assignment, and a topology-aware allocator using
|
||||
`affinity[]`. Shared with `service-endpoint` (ports).
|
||||
4. **`resource-claims`:** template and standalone claims as WAL entities,
|
||||
the provider contract, worker prepare/publish, and a stage-in/stage-out
|
||||
gate. First users: sandbox scratch, `transfer_files` staging,
|
||||
checkpoint volumes.
|
||||
5. **`gang-groups`:** gangs of several jobs. Claim re-attach in `job-resize`
|
||||
and shared weights claims in `service-replica-set` rollouts.
|
||||
|
||||
**Config-only now:** document the static custom resources that already
|
||||
work.
|
||||
|
||||
## Open decisions (user, pending)
|
||||
|
||||
1. Phase 1 scope: `resource-distribution-mapping` option 2 now with option 3
|
||||
as `custom-resource-reporting`, or option 3 in one spec?
|
||||
2. Who reports runtime custom values: worker-run scripts (`posix_spawn`), a
|
||||
separate reporter role, or both?
|
||||
3. Are claims first-class REST/RBAC objects owned by a project from v1, and
|
||||
do they count against project quotas?
|
||||
4. Should XCE keep its own JSON provider contract, or be wire-compatible with
|
||||
CSI and device plugins to reuse their drivers (gRPC sidecar, container
|
||||
runtime coupling)?
|
||||
5. How deep is the topology model: host plus switch block, or per-GPU/NIC
|
||||
graphs?
|
||||
6. Will OCI containers be an execution mode? It decides how dataset and
|
||||
weights volumes are published into a job.
|
||||
7. Who changes MIG layouts: a provider automatically, or an admin?
|
||||
8. Should `rusage` `duration`/`decay` be honoured now (licenses need it)?
|
||||
9. Is data staging part of `job-sandbox-execution`'s workspace work, or
|
||||
`resource-claims`?
|
||||
|
|
@ -4,6 +4,8 @@
|
|||
|
||||
XCE is in the persistence + HA transformation (branch `persistent_ha`). Governing model: Raft consensus + CP (see `product.md` key decisions). Active spec work is tracked per-spec in `.kiro/specs/`; this file is the **single consolidated future-work document** — the forward-looking queue, long-deferred items, and hardening backlog. It absorbs the former `src/docs/FUTURE_WORK.md` (2026-05-12, config-replication-design review), which now redirects here.
|
||||
|
||||
**Long-term direction (user, 2026-10-10):** XCE is heading toward a **unified platform for applications such as LLM training and deployment (serving)**, not only classic HPC batch. Features are therefore drawn from **IBM LSF and Kubernetes**: consumables, `rusage`, elim, licenses and application profiles from LSF; extended resources, device plugins, CSI volumes, Dynamic Resource Allocation claims and services with replicas from Kubernetes. Designs for resources, providers, job lifecycle and APIs should serve both batch jobs and long-running services. Resource vocabulary: `resource-model.md`.
|
||||
|
||||
## In Flight
|
||||
|
||||
- ~~**scheduler-framework** — NEW 2026-07-06 (`/kiro-discovery`). Iterator-driven allocation architecture: lazy plan iterator + host iterator, combinatorial parallel job allocation, configurable job picker, session-level unsatisfiable resreq cache. Brief: `.kiro/specs/scheduler-framework/brief.md`.~~ **DECOMPOSED 2026-07-27** into three child specs (`.kiro/specs/scheduler-framework/spec.json` `child_specs`): **scheduler-core** — core/driver separation, pluggable job picker interface, and simulation mode; **IMPLEMENTATION-COMPLETE**, GO/NO-GO validation 2026-08-07/08 (2 non-blocking follow-ups: (a) tlog logging-integration built but never activated, (b) `fcfs.c`/`builtin_fcfs.c` doc+traceability gap — design.md text was corrected same-day, but tasks 2.1-2.6 in `tasks.md` still carry no `_Verified:` lines, **still open**), 45/45 sub-tasks done as of that validation. **Updated 2026-08-19**: task 12.5 (REPL live-operator-testing fixes, Req 5.22 amendment) was added after that validation from live-operator testing gaps (bug-1593/1594/1595/1596) and is now also done, independently re-verified with fresh evidence (`/kiro-verify-completion`) — see `tasks.md`'s own `_Verified:` line on that task; this closes follow-up (a) too (the tlog gap is now activated — see `design.md`'s `SchedulerCore` component and `research.md`). A separate retrospective design review (2026-08-19, unrelated to the original 2026-08-07 follow-ups) additionally found and fixed an `OPCode_schedAdmin` blocking-wait admission gap and extended `core.c`'s isolation-boundary test coverage to its `core_events.c`/`core_session.c`/`core_clock.c` siblings — see `research.md`'s "SUPERSEDES" notes. **scheduler-allocation** — lazy iterator-driven allocation architecture, parallel job support, session-level unsatisfiable resreq cache; requirements generated, `blocked_on: [scheduler-core]` — now unblocked. **scheduler-preemption** — SIGSTP-based victim selection + controller→worker pipeline; requirements generated, `blocked_on: [scheduler-core]` — now unblocked. The parent `scheduler-framework` spec is retained only for discovery-context provenance and is no longer in flight itself. Next: `/kiro-spec-design scheduler-allocation` and `/kiro-spec-design scheduler-preemption`.
|
||||
|
|
@ -82,6 +84,23 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
|
|||
- [ ] resource-provisioner — NEW 2026-10-10 (Auto resource provisioning). New daemon (`src/autoscaler`, an LSF RC-style port, is prior art only, not a required basis): several instances, each owning providers via controller leases (cloud and license provisioners may differ); providers describe capacity, cost, priority, lead time, granularity, hold time; script contract v2 (hosts and counts); automatic policy (no approval), budgets, cooldowns, retries, idle reclaim; request/instance state as controller facts; orphan cleanup; REST/xcecli/audit; simulated provider. Brief: `.kiro/specs/resource-provisioner/brief.md`. Dependencies: provisioning-demand.
|
||||
- [ ] elastic-hosts — NEW 2026-10-10 (Auto resource provisioning). Host templates in config, expected-host records matched at join, provisioned/draining/decommissioned states, drain-then-release, decommission control, spot loss as host loss. Brief: `.kiro/specs/elastic-hosts/brief.md`. Dependencies: resource-provisioner, host-failure-job-recovery.
|
||||
- [ ] elastic-shared-resources — NEW 2026-10-10 (Auto resource provisioning). Elastic shared resources (e.g. EDA licenses): base + borrowed total updated by the provisioner role through the controller, unused borrowed amounts returned, display. Brief: `.kiro/specs/elastic-shared-resources/brief.md`. Dependencies: resource-provisioner, resource-distribution-mapping.
|
||||
- [ ] custom-resource-reporting — NEW 2026-10-11 (`/kiro-discovery`, "Unified resource model" phase 2; `resource-distribution-mapping` option 3 split out, pending user decision 1 in `resource-model.md`). Runtime custom resource values: an elim-like script reporter on workers (`posix_spawn`) and/or an external reporter role (decision 2), staleness and `interval` enforced, a string-set kind for dataset locality, instance-inventory publishing (DRA ResourceSlice-like). Brief: `.kiro/specs/custom-resource-reporting/brief.md`. Dependencies: resource-distribution-mapping, peer-authentication-hardening.
|
||||
- [ ] resource-instances — NEW 2026-10-11 (Unified resource model phase 3). Identified instances (GPU, MIG slice, NIC, ports) with attributes; topology-aware allocator plugin driving the unused `affinity[]` (NVLink domain, GPU-NIC affinity; switch block via `same[]`/`span[]`); durable per-job instance assignment (WAL/snapshot field-complete); worker binding (device cgroup, `CUDA_VISIBLE_DEVICES`, port env). Brief: `.kiro/specs/resource-instances/brief.md`. Dependencies: custom-resource-reporting, scheduler-framework.
|
||||
- [ ] resource-claims — NEW 2026-10-11 (Unified resource model phase 4). Data and volumes as lifecycle resources: template claims (per job/replica: scratch, staging, KV cache) and standalone claims (K8s PVC/ResourceClaim-like: checkpoint volumes, model weights, dataset caches; access modes RWO/ROX/RWX/RWOP; retain/delete), claim WAL entity, dispatch readiness gate (stage-in/out), storage provider contract (reused from resource-provisioner), worker publish into the sandbox, orphan handling. Absorbs the sandbox workspace and `transfer_files` staging (bug-1937 deferral). Brief: `.kiro/specs/resource-claims/brief.md`. Dependencies: resource-provisioner, resource-instances, job-sandbox-execution.
|
||||
- [ ] gang-groups — NEW 2026-10-11 (Unified resource model phase 5, LLM training). All-or-nothing scheduling of a group of jobs (launcher + workers, JobSet-like), topology packing across the gang, start barrier until every member is placed and its claims are READY, group failure policy. Brief: `.kiro/specs/gang-groups/brief.md`. Dependencies: resource-instances, resource-claims, parjob-placement.
|
||||
|
||||
## Existing Spec Updates (2026-10-11 unified resource model discovery)
|
||||
Direction and pending decisions: `.kiro/steering/resource-model.md`.
|
||||
- [ ] resource-distribution-mapping — phase 1: shared pools (option 2) plus the resource type descriptor fields (`scope`, `unit`, `reporter`, `layer`, `instances`, `provider`); option 3 moves to `custom-resource-reporting` unless the user keeps it here (decision 1).
|
||||
- [ ] service-endpoint — ports as identified instances of `resource-instances` instead of a separate port model.
|
||||
- [ ] job-sandbox-execution — the per-job workspace becomes the first template claim of `resource-claims`; publish (mounts) through the claim's worker step. Where data staging lives is decision 9.
|
||||
- [ ] application-profile — default claims (model weights, checkpoint) and GPU instance requests in profiles; `container` execution mode decides volume publishing (decision 6).
|
||||
- [ ] job-resize — re-bind a job's standalone claims to the new host set; resize a gang group.
|
||||
- [ ] service-replica-set — replicas share one read-many weights claim; rollout attaches version N+1 while N stays attached; per-replica template claims.
|
||||
- [ ] resource-provisioner — its provider contract, lease, epoch and orphan machinery are shared with storage/claim providers.
|
||||
- [ ] provisioning-demand — claim shortfall as provisionable demand.
|
||||
- [ ] peer-authentication-hardening — reporter and claim-provider roles on the per-role opcode allowlist.
|
||||
- [ ] scheduler-framework — the deferred GPU/affinity allocator plugin is delivered by `resource-instances`; honour `rusage` `duration`/`decay`, parsed but ignored today (`alloc_runner.c:429`; decision 8, licenses need it).
|
||||
|
||||
## Existing Spec Updates (2026-10-10 auto resource provisioning discovery)
|
||||
- [ ] resource-distribution-mapping — choose its shared-pool scope (option 2) so a cluster-wide pool (licenses) can be declared in `resources.yaml`; prerequisite of elastic-shared-resources.
|
||||
|
|
@ -155,6 +174,14 @@ Smaller items that likely don't warrant a full spec; pick up opportunistically o
|
|||
|
||||
- Backlog items here are seeds for `/kiro-discovery` / `/kiro-spec-init`; promote an item by creating its spec under `.kiro/specs/`.
|
||||
- Provenance: items marked _(FUTURE_WORK.md §N)_ originate from the 2026-05-12 config-replication-design.md review, merged here 2026-06-10.
|
||||
- **2026-10-11 unified resource model discovery** (data/volumes as resources, custom resources, one interface; context: the LLM training + serving platform direction in the Overview). **Key decisions:**
|
||||
- One resource type descriptor and one provider contract over three layers: L1 attribute, L2 counter, L3 instance/claim. Not one mechanism for everything.
|
||||
- The scheduler only selects from its cached snapshot; providers run after dispatch.
|
||||
- Batch stays LSF-style.
|
||||
- Claims come in two forms, template and standalone (Kubernetes PVC/ResourceClaim-like), as WAL entities with the resource-provisioner discipline.
|
||||
- Adopt Kubernetes DRA/CSI concepts, not (yet) their APIs.
|
||||
|
||||
**Phases:** `resource-distribution-mapping` (shared pools) → `custom-resource-reporting` → `resource-instances` → `resource-claims` → `gang-groups`. Nine user decisions are pending, listed in `resource-model.md`.
|
||||
- **2026-07-06 admin-config-tool discovery** (`/kiro-discovery`, Path E mixed decomposition):
|
||||
|
||||
| # | Workstream | Type | Language | Dependencies |
|
||||
|
|
|
|||
|
|
@ -27,6 +27,7 @@ For details on any topic below, read the corresponding steering file in `.kiro/s
|
|||
| RBAC, JWT, auth backends, input validation, secrets | `security.md` |
|
||||
| WAL durability, recovery, save/restore field-completeness | `persistence.md` |
|
||||
| Real vs. virtual resources, reservation-vs-usage accounting (mem) | `resource-accounting.md` |
|
||||
| Resource model: attributes / counters / instances+claims, providers, K8s-LSF mapping, LLM direction | `resource-model.md` |
|
||||
| Job/app-profile/queue/project ownership and merge rules | `job-config-layering.md` |
|
||||
|
||||
## Key Rules (from steering — summarized for quick reference)
|
||||
|
|
@ -107,5 +108,5 @@ Kiro-style Spec-Driven Development on an agentic SDLC
|
|||
## Steering Configuration
|
||||
- Load entire `.kiro/steering/` as project memory — all `.md` files are authoritative
|
||||
- Core: `product.md`, `tech.md`, `structure.md`
|
||||
- Custom: `testing.md`, `error-handling.md`, `security.md`, `ha-async-io.md`, `api-standards.md`, `build-deployment.md`, `config-design.md`, `persistence.md`, `resource-accounting.md`, `job-config-layering.md`, `roadmap.md`
|
||||
- Custom: `testing.md`, `error-handling.md`, `security.md`, `ha-async-io.md`, `api-standards.md`, `build-deployment.md`, `config-design.md`, `persistence.md`, `resource-accounting.md`, `resource-model.md`, `job-config-layering.md`, `roadmap.md`
|
||||
- Add new custom files via `/kiro-steering-custom`; sync existing via `/kiro-steering`
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue