Rongsong Shen
c14fe3dab5
resource-model.md: one resource type descriptor and provider contract over three layers (L1 attribute, L2 counter, L3 instance/claim); the scheduler only selects from its cached snapshot, providers run after dispatch; batch stays LSF-style; template and standalone (PVC/ResourceClaim-like) claims as WAL entities with the resource-provisioner discipline; LSF/K8s mapping; LLM training/serving needs; phases; nine pending user decisions. New briefs: custom-resource-reporting, resource-instances, resource-claims, gang-groups. Roadmap: long-term LLM training + serving platform direction (features from LSF and Kubernetes), the four new specs, updates to ten existing specs, discovery note. resource-distribution-mapping becomes phase 1 (shared pools + descriptor fields). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
80 lines
4.8 KiB
Markdown
80 lines
4.8 KiB
Markdown
# Brief: resource-claims
|
|
|
|
(Resource-model discovery, 2026-10-11. Phase 4 of `.kiro/steering/resource-model.md`. Open decisions 3 (first-class claims and quotas), 4 (own contract vs CSI wire compatibility), 6 (containers) and 9 (where staging lives) in that document are pending.)
|
|
|
|
## Problem
|
|
Data and volumes are resources with a lifecycle: something must be created, staged or attached before a job runs, and released or cleaned up after it, even when the job fails. Some must outlive any single job and be shared:
|
|
- a checkpoint volume re-attached when training restarts or resizes;
|
|
- model weights read by every serving replica;
|
|
- a dataset cache used by many jobs.
|
|
|
|
XCE has no such object. File staging (`transfer_files`) is parsed but never performed, and sandbox workspaces are planned without a resource model.
|
|
|
|
## Current State
|
|
- **Staging:** stage-in/out "remains unimplemented (future spec)" (`.wolf/bugs/bug-1937.md:48`).
|
|
- **Workspaces:** `job-sandbox-execution` plans per-job workspaces with create, hand-over, mandatory cleanup or re-own (`brief.md:78-100`). This is a per-job claim in all but name.
|
|
- **Missing everywhere:** volumes, burst buffers, datasets, checkpoints and access modes.
|
|
- **Prior design:** `resource-provisioner` research (2026-10-10) already set the rules for externally created things that cost money or hold data: idempotent `op_id`, owner tags, intent committed before action, lease epoch, orphan classes, a JSON provider contract, `posix_spawn`.
|
|
|
|
## Desired Outcome
|
|
- **Two claim forms:**
|
|
- **Template claim:** declared in a job's resreq or application profile; created per job, task or replica; released with it. Examples: scratch, staging, KV cache.
|
|
- **Standalone claim:** a controller entity with its own id, owner/project, access mode (RWO/ROX/RWX/RWOP), size and parameters, and reclaim policy (`retain`/`delete`). Many jobs or replicas reference it (e.g. `--claim ckpt-7:rw`), and it outlives them.
|
|
- **Lifecycle:** claim states `PENDING → PREPARING → READY → RELEASING → RELEASED/FAILED`.
|
|
- Dispatch waits until all claims of the job (or gang) are READY: a stage-in gate.
|
|
- Stage-out and release run after the job; failures are visible and retried.
|
|
- **Placement:**
|
|
- the scheduler reserves capacity (L2 counter) and placement;
|
|
- a standalone claim's reachability (volume reachable from hosts H, cache present on H) is a placement constraint read from the cached snapshot.
|
|
- **Providers:** a provider (create/attach/release/health) runs in the provisioner daemon for external storage. The worker does node-local prepare/publish: mount into the sandbox, set ownership.
|
|
- **Recovery:** claims are WAL entities, field-complete in snapshots, and survive failover. Orphan scans cover claims with no owner and provider objects with no claim.
|
|
- **Surfaces:** xcecli/REST to create, list, show and delete standalone claims; claims shown on jobs; audit.
|
|
|
|
## Approach
|
|
- Reuse the resource-provisioner provider contract, lease and orphan machinery.
|
|
- Absorb `job-sandbox-execution`'s workspace as the first template claim, and `transfer_files` staging as the stage-in/out claim (decision 9).
|
|
- Add the checkpoint volume as the first standalone claim.
|
|
|
|
## Scope
|
|
- **In:**
|
|
- both claim forms;
|
|
- claim facts and snapshot rows;
|
|
- the dispatch readiness gate;
|
|
- the provider contract for storage;
|
|
- worker publish into the sandbox;
|
|
- access modes;
|
|
- reclaim policy;
|
|
- orphan handling;
|
|
- REST/xcecli;
|
|
- a simulated storage provider.
|
|
- **Out:** device identity (`resource-instances`); multi-job gangs (`gang-groups`); data-locality-aware caching policy beyond reachability (later).
|
|
|
|
## Boundary Candidates
|
|
- Claim entity and lifecycle (controller, WAL, snapshot).
|
|
- Readiness gate and job states.
|
|
- Provider contract for storage.
|
|
- Worker publish.
|
|
- REST/xcecli/RBAC.
|
|
|
|
## Out of Boundary
|
|
- Building a storage system.
|
|
- The scheduler calling providers.
|
|
|
|
## Upstream / Downstream
|
|
- **Upstream:**
|
|
- `resource-provisioner` (provider contract, lease, orphans);
|
|
- `resource-instances` (claims bound to instances);
|
|
- `job-sandbox-execution` (where publish mounts);
|
|
- `peer-authentication-hardening`;
|
|
- `application-profile` (template claims in profiles).
|
|
- **Downstream:** `job-resize` and `service-replica-set` (re-bind and shared weights), `gang-groups`, `provisioning-demand` (claim shortfall).
|
|
|
|
## Existing Spec Touchpoints
|
|
- **Extends:** `job-sandbox-execution` (workspace as a claim); closes bug-1937's deferred staging.
|
|
- **Adjacent:** `elastic-hosts`, `project-guarantee` (quota on claims, decision 3).
|
|
|
|
## Constraints
|
|
- **HA/persistence:** full 3-phase spec workflow; a claim that holds data or costs money is durable from v1.
|
|
- **Scheduler:** never calls or waits on a provider.
|
|
- **Process creation:** no fork in threaded processes.
|
|
- **Credentials:** stay with the provider; users cannot choose arbitrary mount paths.
|