myxce/.kiro/steering/resource-model.md

# Resource Model: one vocabulary, three layers

**Status:** direction adopted 2026-10-11 (resource-model discovery, after the
2026-10-10 resource-provisioner research). The open decisions at the end are
the user's and still pending; specs built on this document must not settle
them silently.

## Why

XCE is heading toward a unified platform for applications such as LLM
training **and** deployment (serving), not only classic HPC batch. So
resource features come from IBM LSF (consumables, `rusage`, elim, licenses,
`-app` profiles, data manager) **and** Kubernetes (extended resources, device
plugins, CSI volumes, Dynamic Resource Allocation claims). Three asks drive
this document:

1. Treat data and volumes as resources.
2. Let admins define custom resources.
3. Give all resources one interface.

The answer is yes to all three, as **one resource type vocabulary and one
provider contract over three layers of behaviour**. It is not a single
mechanism: none of LSF, Slurm, PBS or Kubernetes uses one mechanism for
everything, and a per-job claim object for `slots`/`mem` would put lifecycle
cost and WAL traffic on the scheduler's hottest path.

## Current state (2026-10-11)

- **Descriptor:** `xce_resource_t` (`src/include/xce/types.h:133-145`) has
  three value kinds (string, numeric, boolean) and the flags BUILTIN,
  DYNAMIC, GLOBAL, SHARED, EXTERNAL, RELEASE, DEFAULT_HOST, CONSUMABLE and
  LIMIT (`types.h:81-89`). Config (`resources.yaml` via `xceadmin`) reaches
  only CONSUMABLE and RELEASE. `resourcemap` is dropped at build
  (`xceadmin/build.py:38-43`), and its capnp fields are commented out
  (`controller.capnp:213-214`).
- **Works with config only:** static per-host custom resources (numeric,
  boolean, string, consumable or not) declared in `resources.yaml`, with
  per-host values in `hosts.yaml`.
- **Missing:**
  - runtime custom values: only a worker connection may update values
    (`controller/opupd.c:11-26`); the 11 load indices are compiled in
    (`worker/src/resource_sync.c:84-104`); there is no elim-like reporter;
  - cluster-wide pools such as licenses: the pool machinery exists for limit
    virtual resources (`sharedInst`, `scheduler/src/resource.c:292`), but
    `mark_host_res()` rejects SHARED (`controller/worker.c:259`). The scope
    choice is pending in `resource-distribution-mapping`;
  - device or volume identity: allocations are counts, and
    `xce_res_account_t.instid` is a per-process counter, not an identity;
  - lifecycle (create, stage, attach, release);
  - `rusage` `duration`/`decay`: parsed, ignored (`alloc_runner.c:429`);
  - `affinity[]`: parsed (`lib/resreq.c:127`), never used;
  - GPUs: only an unmapped `gpu_req` queue string.
- **Data:** only scratch capacity (`tmp`/`maxtmp`). File staging is parsed
  but never performed (bug-1937). Sandbox workspaces
  (`job-sandbox-execution`) are a planned per-job storage lifecycle, but are
  not modelled as resources.

## The model

### Resource type (one descriptor for every resource)

`name`, `kind` (numeric, boolean, string, later string-set), `unit`,
`order`, `consumable`, `scope` (host, host group, shared pool, external),
`reporter` (static, builtin, provider), `interval`, `layer`, `instances`
(counted, or identified with attributes), `provider` (reference; optional).
This makes today's flags explicit, and it is the descriptor that
`resource-distribution-mapping` extends.

### Three layers

| Layer | What it is | Examples | Scheduler sees |
|---|---|---|---|
| **L1 attribute** | not consumed; matched by `select[]` | host model, dataset X cached here, NVLink domain, switch block | cached values (as today) |
| **L2 counter** | consumed amount, host or pool scope | `slots`, `mem`, scratch GB, licenses, filesystem quota | cached `resvals`/`sharedInst` (as today) |
| **L3 instance / claim** | *which* unit, optionally with a lifecycle | GPU, MIG slice, NIC rail, port; volumes, burst buffer, checkpoint and model-weight stores | identified inventory with attributes; claim placement as a constraint |

A resource opts into L3 only if it needs identity or lifecycle. L1 and L2
stay as fast and deterministic as now.

### Provider contract

A resource can have a provider with optional capabilities: `report`
(values, or an instance inventory with attributes), `prepare`/`attach` before
use, `release`/`cleanup` after, and `health`. It shares the transport and
rules of the resource-provisioner contract
(`.kiro/specs/resource-provisioner/research.md`):

- JSON documents; one-shot scripts first, a persistent JSON-lines mode later;
- every mutating action idempotent, keyed by an `op_id`;
- owner tags on every external object;
- an intent fact committed before any external action;
- a lease epoch on facts;
- orphan scans;
- started with `posix_spawn()` or Go `os/exec`, never `fork()` in a threaded
  process (`tech.md` "Process Creation").

**Where providers run:**
- node-local (GPU, scratch, mounts): under the worker, which does the prepare
  step (devices, cgroup, env);
- external (cloud, storage arrays, license servers): under a provisioner
  daemon with its own controller role.

### Claims (L3 with a lifecycle)

- **Template claim** (like a K8s ResourceClaimTemplate): written in a job's
  resreq or application profile, created per job, task or replica, and
  released with it. Examples: scratch, staging, KV cache. This is the LSF
  mental model.
- **Standalone claim** (like a K8s PVC or ResourceClaim): a controller entity
  with its own id, owner/project, access mode (RWO, ROX, RWX, RWOP) and
  reclaim policy (`retain`/`delete`). Many jobs or replicas reference it
  (e.g. `--claim ckpt-7:rw`), and it outlives them. It is re-bound on restart,
  resize or replica replacement. Examples: checkpoint volumes (one writer
  gang), model weights (read-many by every replica), dataset caches.
- **States:** `PENDING → PREPARING → READY → RELEASING → RELEASED/FAILED`.
  Dispatch waits until every claim of the job (or of the whole gang) is
  READY.

## Rules every resource design must keep

- **The scheduler only chooses.** It reads numbers, attributes and inventory
  from its cached session snapshot (the DRA "structured parameters" model).
  It never calls a provider and never waits for one. Prepare, attach,
  staging and release happen in the controller, worker or provisioner after
  dispatch. Claim readiness is an input event, not something the scheduler
  polls. This keeps sessions fast and sim/replay deterministic; the
  simulator models provider latency.
- **Persistence:** instance assignments (which GPU or port), claim records,
  shared totals and borrowed amounts are WAL facts or snapshot fields. They
  must round-trip through every save/restore and apply seam (`persistence.md`
  field-completeness; run `/xce-audit-invariants`). A claim that costs money
  or holds data is durable from its first version.
- **HA:** claims follow the resource-provisioner discipline: intent before
  action, `op_id`, epoch, owner tags, orphan classes.
- **Batch stays LSF-style:** counted consumables, `rusage`, `max(reserved,
  actual)` accounting (`resource-accounting.md`), licenses as shared pools,
  limits as virtual resources. These need no lifecycle and stay constant-cost
  per session.
- **Request syntax stays LSF-style:** `select[]`, `rusage[]`, `span[]`,
  `same[]`, `affinity[]`, extended with attribute predicates over instances.
  There is no CEL engine in the scheduler.
- **Security:**
  - provider credentials stay in the provider process, never in job exdata,
    cluster config sent to workers, or the wire;
  - every reporter and provider role is on the per-role opcode allowlist
    (`peer-authentication-hardening`);
  - users never choose arbitrary mount paths (application-profile mount
    rules).

## Mapping to LSF and Kubernetes

| XCE | LSF | Kubernetes |
|---|---|---|
| resource type descriptor | `lsf.shared` + ResourceMap | DeviceClass / StorageClass / extended resource |
| L1/L2 reporter (static, builtin, provider `report`) | elim | device plugin ListAndWatch / DRA ResourceSlice |
| L2 shared pool | shared resource, License Scheduler | (extended resource at cluster level) |
| L3 identified instance | `-gpu`, affinity | device plugin Allocate / DRA device |
| template claim | `rusage` + data manager staging | ResourceClaimTemplate, ephemeral volume |
| standalone claim | (none) | PVC, ResourceClaim |
| worker prepare/publish | (job starter, cgroup) | NodePrepareResources, CSI NodeStage/Publish |
| provider create/attach | resource connector | CSI controller |
| access mode | (none) | RWO/ROX/RWX/RWOP |

Adopt the Kubernetes **concepts**. Whether XCE also speaks the CSI or
device-plugin **wire protocols** is open decision 4.

## LLM training and serving

- **Accelerator topology:** NVLink/NVSwitch domains, GPU and NIC on one PCIe
  switch, IB rails. These are L3 instance attributes plus constraints
  between instances, served by an allocator that consumes `affinity[]`.
  Cluster-level topology (switch block) is an L1 host attribute that
  `same[]`/`span[]` already pack against.
- **MIG/MPS:** partitions are reported as instances. Changing a layout is a
  provider action that drains the GPU, never a scheduler decision.
- **Gangs:** one parallel job is already placed all-or-nothing
  (`parjob-placement`). A gang of several jobs (launcher plus workers) needs
  a gang group; the whole gang and its claims are READY before any part
  starts.
- **Elastic resize and restart:** standalone claims are re-bound to the new
  host set.
- **Serving:** replicas share one read-many weights claim and have
  per-replica template claims. A rolling update attaches version N+1 while
  version N stays attached.

## Phases

1. **`resource-distribution-mapping`:** shared pools (option 2) plus the
   descriptor fields above. Unblocks licenses (`elastic-shared-resources`)
   and shared quotas.
2. **`custom-resource-reporting`:** runtime custom values and inventory
   (option 3): reporters other than the worker connection, a string-set kind
   for dataset locality, and `interval` enforced.
3. **`resource-instances`:** identified GPU, MIG, NIC and port instances
   with attributes, durable assignment, and a topology-aware allocator using
   `affinity[]`. Shared with `service-endpoint` (ports).
4. **`resource-claims`:** template and standalone claims as WAL entities,
   the provider contract, worker prepare/publish, and a stage-in/stage-out
   gate. First users: sandbox scratch, `transfer_files` staging,
   checkpoint volumes.
5. **`gang-groups`:** gangs of several jobs. Claim re-attach in `job-resize`
   and shared weights claims in `service-replica-set` rollouts.

**Config-only now:** document the static custom resources that already
work.

## Open decisions
Kept in `resource-questions.md`, the persistent decision and question log (R1–R9 from this document, the provisioner's P1–P9, new questions, and proposals PR-1 to PR-3 that may reshape the phases above: one supply map that includes provisioned supply, a resource provider proxy, datasets in the map with staging as provisioning). Check it before writing requirements for any phase.