myxce/.kiro/steering/resource-model.md

Resource Model: one vocabulary, three layers

Status: direction adopted 2026-10-11 (resource-model discovery, after the 2026-10-10 resource-provisioner research). The open decisions at the end are the user's and still pending; specs built on this document must not settle them silently.

Why

XCE is heading toward a unified platform for applications such as LLM training and deployment (serving), not only classic HPC batch. So resource features come from IBM LSF (consumables, rusage, elim, licenses, -app profiles, data manager) and Kubernetes (extended resources, device plugins, CSI volumes, Dynamic Resource Allocation claims). Three asks drive this document:

1. Treat data and volumes as resources.
2. Let admins define custom resources.
3. Give all resources one interface.

The answer is yes to all three, as one resource type vocabulary and one provider contract over three layers of behaviour. It is not a single mechanism: none of LSF, Slurm, PBS or Kubernetes uses one mechanism for everything, and a per-job claim object for slots/mem would put lifecycle cost and WAL traffic on the scheduler's hottest path.

Current state (2026-10-11)

• Descriptor: xce_resource_t (src/include/xce/types.h:133-145) has three value kinds (string, numeric, boolean) and the flags BUILTIN, DYNAMIC, GLOBAL, SHARED, EXTERNAL, RELEASE, DEFAULT_HOST, CONSUMABLE and LIMIT (types.h:81-89). Config (resources.yaml via xceadmin) reaches only CONSUMABLE and RELEASE. resourcemap is dropped at build (xceadmin/build.py:38-43), and its capnp fields are commented out (controller.capnp:213-214).
• Works with config only: static per-host custom resources (numeric, boolean, string, consumable or not) declared in resources.yaml, with per-host values in hosts.yaml.
• Missing:
  • runtime custom values: only a worker connection may update values (controller/opupd.c:11-26); the 11 load indices are compiled in (worker/src/resource_sync.c:84-104); there is no elim-like reporter;
  • cluster-wide pools such as licenses: the pool machinery exists for limit virtual resources (sharedInst, scheduler/src/resource.c:292), but mark_host_res() rejects SHARED (controller/worker.c:259). The scope choice is pending in resource-distribution-mapping;
  • device or volume identity: allocations are counts, and xce_res_account_t.instid is a per-process counter, not an identity;
  • lifecycle (create, stage, attach, release);
  • rusage duration/decay: parsed, ignored (alloc_runner.c:429);
  • affinity[]: parsed (lib/resreq.c:127), never used;
  • GPUs: only an unmapped gpu_req queue string.
• Data: only scratch capacity (tmp/maxtmp). File staging is parsed but never performed (bug-1937). Sandbox workspaces (job-sandbox-execution) are a planned per-job storage lifecycle, but are not modelled as resources.

The model

Resource type (one descriptor for every resource)

name, kind (numeric, boolean, string, later string-set), unit, order, consumable, scope (host, host group, shared pool, external), reporter (static, builtin, provider), interval, layer, instances (counted, or identified with attributes), provider (reference; optional). This makes today's flags explicit, and it is the descriptor that resource-distribution-mapping extends.

Three layers

Layer	What it is	Examples	Scheduler sees
L1 attribute	not consumed; matched by select[]	host model, dataset X cached here, NVLink domain, switch block	cached values (as today)
L2 counter	consumed amount, host or pool scope	slots, mem, scratch GB, licenses, filesystem quota	cached resvals/sharedInst (as today)
L3 instance / claim	which unit, optionally with a lifecycle	GPU, MIG slice, NIC rail, port; volumes, burst buffer, checkpoint and model-weight stores	identified inventory with attributes; claim placement as a constraint

A resource opts into L3 only if it needs identity or lifecycle. L1 and L2 stay as fast and deterministic as now.

Provider contract

A resource can have a provider with optional capabilities: report (values, or an instance inventory with attributes), prepare/attach before use, release/cleanup after, and health. It shares the transport and rules of the resource-provisioner contract (.kiro/specs/resource-provisioner/research.md):

• JSON documents; one-shot scripts first, a persistent JSON-lines mode later;
• every mutating action idempotent, keyed by an op_id;
• owner tags on every external object;
• an intent fact committed before any external action;
• a lease epoch on facts;
• orphan scans;
• started with posix_spawn() or Go os/exec, never fork() in a threaded process (tech.md "Process Creation").

Where providers run:

• node-local (GPU, scratch, mounts): under the worker, which does the prepare step (devices, cgroup, env);
• external (cloud, storage arrays, license servers): under a provisioner daemon with its own controller role.

Claims (L3 with a lifecycle)

• Template claim (like a K8s ResourceClaimTemplate): written in a job's resreq or application profile, created per job, task or replica, and released with it. Examples: scratch, staging, KV cache. This is the LSF mental model.
• Standalone claim (like a K8s PVC or ResourceClaim): a controller entity with its own id, owner/project, access mode (RWO, ROX, RWX, RWOP) and reclaim policy (retain/delete). Many jobs or replicas reference it (e.g. --claim ckpt-7:rw), and it outlives them. It is re-bound on restart, resize or replica replacement. Examples: checkpoint volumes (one writer gang), model weights (read-many by every replica), dataset caches.
• States: PENDING → PREPARING → READY → RELEASING → RELEASED/FAILED. Dispatch waits until every claim of the job (or of the whole gang) is READY.

Rules every resource design must keep

• The scheduler only chooses. It reads numbers, attributes and inventory from its cached session snapshot (the DRA "structured parameters" model). It never calls a provider and never waits for one. Prepare, attach, staging and release happen in the controller, worker or provisioner after dispatch. Claim readiness is an input event, not something the scheduler polls. This keeps sessions fast and sim/replay deterministic; the simulator models provider latency.
• Persistence: instance assignments (which GPU or port), claim records, shared totals and borrowed amounts are WAL facts or snapshot fields. They must round-trip through every save/restore and apply seam (persistence.md field-completeness; run /xce-audit-invariants). A claim that costs money or holds data is durable from its first version.
• HA: claims follow the resource-provisioner discipline: intent before action, op_id, epoch, owner tags, orphan classes.
• Batch stays LSF-style: counted consumables, rusage, max(reserved, actual) accounting (resource-accounting.md), licenses as shared pools, limits as virtual resources. These need no lifecycle and stay constant-cost per session.
• Request syntax stays LSF-style: select[], rusage[], span[], same[], affinity[], extended with attribute predicates over instances. There is no CEL engine in the scheduler.
• Security:
  • provider credentials stay in the provider process, never in job exdata, cluster config sent to workers, or the wire;
  • every reporter and provider role is on the per-role opcode allowlist (peer-authentication-hardening);
  • users never choose arbitrary mount paths (application-profile mount rules).

Mapping to LSF and Kubernetes

XCE	LSF	Kubernetes
resource type descriptor	lsf.shared + ResourceMap	DeviceClass / StorageClass / extended resource
L1/L2 reporter (static, builtin, provider report)	elim	device plugin ListAndWatch / DRA ResourceSlice
L2 shared pool	shared resource, License Scheduler	(extended resource at cluster level)
L3 identified instance	-gpu, affinity	device plugin Allocate / DRA device
template claim	rusage + data manager staging	ResourceClaimTemplate, ephemeral volume
standalone claim	(none)	PVC, ResourceClaim
worker prepare/publish	(job starter, cgroup)	NodePrepareResources, CSI NodeStage/Publish
provider create/attach	resource connector	CSI controller
access mode	(none)	RWO/ROX/RWX/RWOP

Adopt the Kubernetes concepts. Whether XCE also speaks the CSI or device-plugin wire protocols is open decision 4.

LLM training and serving

• Accelerator topology: NVLink/NVSwitch domains, GPU and NIC on one PCIe switch, IB rails. These are L3 instance attributes plus constraints between instances, served by an allocator that consumes affinity[]. Cluster-level topology (switch block) is an L1 host attribute that same[]/span[] already pack against.
• MIG/MPS: partitions are reported as instances. Changing a layout is a provider action that drains the GPU, never a scheduler decision.
• Gangs: one parallel job is already placed all-or-nothing (parjob-placement). A gang of several jobs (launcher plus workers) needs a gang group; the whole gang and its claims are READY before any part starts.
• Elastic resize and restart: standalone claims are re-bound to the new host set.
• Serving: replicas share one read-many weights claim and have per-replica template claims. A rolling update attaches version N+1 while version N stays attached.

Phases

1. resource-distribution-mapping: shared pools (option 2) plus the descriptor fields above. Unblocks licenses (elastic-shared-resources) and shared quotas.
2. custom-resource-reporting: runtime custom values and inventory (option 3): reporters other than the worker connection, a string-set kind for dataset locality, and interval enforced.
3. resource-instances: identified GPU, MIG, NIC and port instances with attributes, durable assignment, and a topology-aware allocator using affinity[]. Shared with service-endpoint (ports).
4. resource-claims: template and standalone claims as WAL entities, the provider contract, worker prepare/publish, and a stage-in/stage-out gate. First users: sandbox scratch, transfer_files staging, checkpoint volumes.
5. gang-groups: gangs of several jobs. Claim re-attach in job-resize and shared weights claims in service-replica-set rollouts.

Config-only now: document the static custom resources that already work.

Open decisions

Kept in resource-questions.md, the persistent decision and question log (R1–R9 from this document, the provisioner's P1–P9, new questions, and proposals PR-1 to PR-3 that may reshape the phases above: one supply map that includes provisioned supply, a resource provider proxy, datasets in the map with staging as provisioning). Check it before writing requirements for any phase.