myxce/.kiro/steering/resource-questions.md

Resource Design: Decisions, Proposals and Open Questions

The persistent discussion log for the unified resource model (resource-model.md) and the provisioning work (resource-provisioner, provisioning-demand, elastic-*). Specs must not settle an item here silently.

How to use:

• Status values: open (not discussed), discussing (user input recorded, not final), leaning (direction agreed, details open), decided, deferred (later, with a reason).
• Changes: update an item in place and add a dated line under "History". When an item is decided, also update the documents it names.
• IDs: R = resource model, P = provisioner, N = new questions raised in discussion, PR = proposals.

Proposals under discussion (user, 2026-10-11)

PR-1: one resource map for all supply, including provisioned supply (status: leaning)

• User: should resource-distribution-mapping be made generic, so it also handles resource provisioning?
• Analysis (2026-10-11): yes, for the data model.
  • The resource map becomes the single record of supply: for each resource, which locations (host, host group, pool, storage) hold how much.
  • Each entry has a source (static config, reporter, provisioner) and a state (requested, available, draining, releasing).
  • Provisioning only changes map entries over time, e.g. a license pool's base plus borrowed amount, or a new dataset copy.
  • The provisioning mechanism stays in the provisioner: policy, provider calls, leases, epochs, orphan scans. The map is the state; the provisioner is the actuator. This is the Kubernetes split too (drivers publish ResourceSlices; autoscalers and provisioners act).
  • The scheduler keeps reading only the map's cached snapshot.
• Affects: resource-distribution-mapping, elastic-shared-resources, resource-provisioner, provisioning-demand.
• Open: see N1, N2.

PR-2: a resource provider proxy between the controller and upstream providers (status: leaning)

• User: introduce a resource provider that proxies requests and states between the XCE controller and upstream resource providers.
• Analysis (2026-10-11): yes. It generalizes the resource-provisioner daemon. The proxy:
  • connects to the controller as one role over libxce/Cap'n Proto;
  • speaks each upstream provider's protocol: the JSON script contract, a persistent JSON-lines mode, later CSI or vendor APIs;
  • holds provider sessions, credentials and rate limits, and translates provider states into map entries and claim states;
  • publishes inventory and values (it is the external reporter of R2);
  • carries out provisioning and claim operations with op_id, an epoch and owner tags.
• Why it helps: the controller never blocks on or depends on an outside system, and credentials never reach the controller.
• Where node-local work runs: GPU binding, mounts and scratch stay on the worker, possibly behind the same contract.
• Affects: resource-provisioner (rename or generalize), custom-resource-reporting (external reporters), resource-claims (storage providers), R4/C3 (the CSI adapter would live here).
• Open: see N2, N4.

PR-3: datasets in the resource map; staging as data provisioning (status: discussing)

• User: could data instances be handled by the resource distribution map, and data staging by provisioning the data to a new location?
• Analysis (2026-10-11): this fits PR-1 well.
  • Datasets: a dataset is a resource whose map entries are the locations holding a copy. Locality is an L1 attribute the scheduler can use.
  • Staging: staging is provisioning a copy at a new location, with a lead time like any provider.
  • Eviction: evicting a copy is a release.
  • Shortage: "dataset X needed at host group Y" can become provisioning demand.
  • Locality, caching and stage-in become one mechanism.
• Caveats:
  • Job-private data (stage-out results, scratch, a job's own outputs) is still a per-job claim (resource-claims), not shared supply.
  • Datasets need an identity and version, capacity accounting on the storage holding a copy, an eviction policy, and consistency rules (read-only vs updated).
• Affects: resource-distribution-mapping, custom-resource-reporting (string-set locality may become map entries instead), resource-claims (scope shrinks to job-private data), R9.
• Open: see N3.

PR-4: hosts as supply entries; the worker connection only confirms them (status: discussing)

• User: hosts get into the controller directly today; how could they be unified with the rest of the supply?
• Today: a worker connects to the controller and is accepted only if its name matches a host config entry (find_matched_hostconfig, src/controller/join.c:~76). The host record is created by that first join (init_worker). elastic-hosts plans "expected host" records matched at join.
• Proposal: separate who decides a host should exist (control plane) from the worker's connection (data plane):

  • Host supply entries: a host is an entry in the supply map with a source and a lifecycle state:

    • config: a static hosts.yaml entry or pattern;
    • provisioned: created by a proxy as an expected host, with instance id, template and deadline;
    • discovered: an unknown host that joins, admitted only if a policy allows it.

    Lifecycle: expected → joining → available → draining → released, plus lost.

  • Join confirms: the worker still connects directly. Job control, heartbeats and resync stay on that long-lived connection. A join confirms an existing entry (or is admitted by policy) instead of being the only way a host appears.

  • Host resources: slots, mem and so on stay L2 counters at the host's location, as today.

  • Bring-up: xce-node-manager can act as the node-side provider that starts the worker on a provisioned machine.

• Gains: one place for static, elastic (cloud) and discovered hosts. Orphan rules apply uniformly (expected but never joined; joined but unknown). provisioning-demand sees hosts and other resources the same way.
• Affects: elastic-hosts (its expected-host records become supply entries), resource-distribution-mapping (PR-1), xce-node-manager, the controller join path.

PR-5: simplified dataset model, v1 (status: discussing)

• User: give a simplified dataset model first.
• Proposal:
  • Identity: a dataset is name@version and is immutable; a change is a new version. No update consistency problem.
  • Copies: supply-map entries of the dataset at a location (host, host group or storage pool) with a state staging → available → evicting and a size. The size is charged as an L2 counter on the location's cache or scratch capacity.
  • Jobs: a job names datasets, e.g. data[imagenet@3]. The scheduler prefers hosts with an available copy, as a locality attribute from its cached snapshot. v1 does not make placement wait for a copy.
  • Staging: in v1 staging is explicit. A user or admin asks for a copy at a location (xcecli data stage); a proxy provider copies it and reports available. Eviction is explicit too.
  • Later: scheduler-driven staging (shortfall as provisioning demand), automatic LRU eviction, dataset quotas, and writable datasets.
• Open: whether v1 job placement should require a copy (a hard constraint) or only prefer one; and what the dataset registry is (config, REST object, or both).

PR-6: job input/output through provisioned volumes (status: discussing)

• User (2026-10-11): use volume provisioning for job input and output.
• Context:
  • bug-1936 rejects -i/-o/-e, out_dir and transfer_files at submit, because no field carries them to the job and the worker never stages files.
  • bug-1937 defers stage-in/out to a future spec.
  • Worker file fetch is locked to its own output path (bug-1508).
• Proposal: job I/O as claims (resource-claims):
  • Input claims (read-only): from a dataset copy (PR-5), for large or shared inputs; from files uploaded at submit, which a provider puts into a per-job input volume (the new home of transfer_files/-f); or from an existing standalone claim (model weights, a previous job's output).
  • Output claim (read-write): per job (template) or a named standalone claim.
    • -o/-e and out_dir resolve inside it.
    • Results are fetched through the API server from the provider, not the worker (this removes the bug-1508 restriction).
    • Retention policy: keep until fetched, or N days.
    • Optional export after the job (stage-out) to another volume, object storage or the user's home directory.
  • Placement in the job: the sandbox mounts the claims at fixed points (/xce/in, /xce/out) or at profile-allowed paths. Direct-mode jobs get the paths through environment variables.
  • v1 provider: a directory provider on a shared filesystem. Create is mkdir with an owner and a quota, publish is a bind mount or path, delete is removal. CSI, cloud block and burst-buffer volumes plug into the same contract later.
  • Default: jobs without I/O options keep today's behaviour.
• Refinement (user, 2026-10-11): a shared volume with per-job subdirectories is the default (status: leaning). Jobs do not each get a volume; several jobs share one: shared_volume/<user>/<jobid>/{in,out,err}, array elements <jobid>/<index>/....
  • The shared volume: a long-lived, admin-provisioned standalone claim (read-write-many, retain). Its capacity is a pool counter in the supply map; the hosts that mount it are an L1 attribute, so placement only uses hosts that reach it.
  • A job's directory: a lightweight template claim of kind "subdirectory". Create is mkdir plus ownership, with no storage call; release is retention or cleanup.
  • Per-job volumes (burst buffer, large scratch) stay possible when a profile asks for one.
  • Settles: N9 (read-write-many, so multi-host jobs work) and N10 (a subdirectory per element). Reframes N8 as per-user or per-project quota on the shared volume (filesystem quota, or usage charged to the pool counter after the fact, consistent with max(reserved, actual)).
  • Prior art: Kubernetes subPath and the nfs-subdir provisioner; the shared output trees common at LSF and Slurm sites.
• Affects:
  • resource-claims (adds job I/O as its first use, ahead of checkpoints);
  • job-sandbox-execution (mount points);
  • the apiserver submit/fetch path (lifts the bug-1936 rejections once claims exist);
  • application-profile (default I/O claims, allowed paths);
  • PR-5 (inputs from datasets).

CSI notes (2026-10-11, for C3 and R4)

Source: the CSI specification, from general knowledge; to be checked against the current spec version during design.

• What it is: the Container Storage Interface, a CNCF specification: a gRPC API between a container orchestrator and a storage driver. It is orchestrator-neutral (used by Kubernetes, Nomad, Mesos and Cloud Foundry), though most drivers are written, packaged and tested for Kubernetes.
• Three services:
  • Identity (every plugin): GetPluginInfo, GetPluginCapabilities, Probe.
  • Controller (runs once, central): CreateVolume/DeleteVolume, ControllerPublishVolume/Unpublish (attach to a node), ValidateVolumeCapabilities, ListVolumes, GetCapacity, CreateSnapshot/DeleteSnapshot/ListSnapshots, ControllerExpandVolume, ControllerGetVolume.
  • Node (one per host, privileged): NodeStageVolume/Unstage (mount once per node at a staging path), NodePublishVolume/Unpublish (bind-mount into the workload's path), NodeGetInfo (node id, max volumes, accessible topology), NodeGetCapabilities, NodeExpandVolume, NodeGetVolumeStats.
• Key concepts:
  • Volume capability: block or filesystem, plus fs type and mount flags.
  • Access modes: single-node writer or reader, multi-node reader-only, single-writer or multi-writer.
  • Topology: driver-reported segments such as zone=a or rack=r1; CreateVolume takes requisite and preferred topology.
  • Opaque parameters (a storage class); per-call secrets; volume context returned by create.
• How Kubernetes runs it:
  • The controller plugin runs as a Deployment, with sidecars (external-provisioner, attacher, snapshotter, resizer) that turn PVCs into CSI calls.
  • The node plugin runs as a privileged DaemonSet, with node-driver-registrar, over a unix socket.
  • The sidecars are Kubernetes-specific; the CSI calls themselves are not.
• Mapping to XCE:
  • Controller side → PR-2 proxy: CreateVolume when a claim or dataset copy is created; ControllerPublish when it is bound to hosts; Delete on release; GetCapacity → pool counters in the map; snapshots → checkpoint snapshots; secrets stay in the proxy.
  • Node side → worker/xcecd: NodeStage on first use on a host; NodePublish into the job's sandbox (or container) path; unpublish and unstage at job end. NodeGetInfo topology → host attributes (L1).
  • Timing: CSI topology plus "create after placement" matches the rule that the scheduler only selects (Kubernetes WaitForFirstConsumer).
  • Access modes map one-to-one onto claim access modes.
• Costs:
  • gRPC clients: natural in Go (the proxy; P2), awkward in C (the worker). A small Go node helper, started with posix_spawn or run under xce-node-manager, could make the node calls.
  • The node plugin is a privileged daemon on every host, needing mount propagation.
  • Some drivers assume Kubernetes APIs or container packaging, so each target driver needs testing outside Kubernetes.
• Relevant drivers:
  • HPC parallel filesystems: IBM Spectrum Scale/GPFS CSI, BeeGFS CSI, Lustre (e.g. AWS FSx, DDN), WEKA, VAST.
  • General storage: NFS, Ceph RBD/CephFS, NetApp Trident, Portworx, cloud disks (EBS, GCE PD, Azure Disk), local path.
  • Object storage: S3 via Mountpoint for S3 or JuiceFS.
  • Dataset caching: Fluid (Alluxio/JuiceFS), a Kubernetes dataset-cache project relevant to PR-5.
• Recommendation:
  • Make XCE's own claim and provider contract CSI-shaped: the same verbs (create, publish, stage, node publish, capacity, snapshot, expand), capabilities, access modes, topology and per-call secrets.
  • v1 ships simple native JSON providers (local scratch, NFS or parallel-filesystem directories).
  • Add a CSI adapter later as one provider type in the proxy, plus a node helper for the worker.
  • That keeps CSI a cheap "must later" rather than a v1 cost. If containers (R6) or cloud block storage become early targets, CSI moves up.

Resource model questions (from resource-model.md)

ID	Question	Status	User input / notes
R1	Phase 1 scope: shared pools only (option 2) with runtime values split out, or option 3 in one spec?	discussing	Reframed by PR-1: phase 1 may become the generic supply map. Decide after PR-1.
R2	Who reports runtime custom values: worker scripts, a reporter role, or both?	leaning	Leaning both: worker scripts (elim-like, posix_spawn) for host values; the PR-2 proxy for external and cluster values.
R3	Are claims first-class objects owned by a project from v1? Do they count against quotas?	deferred	User 2026-10-11: RBAC is handled in the API server; the backend only provides configuration and state persistence. Discuss later (with N4).
R4	Own JSON driver contract, or CSI and device-plugin wire compatibility?	leaning	User 2026-10-11: CSI compatibility is better. Still to decide whether it is nice-to-have or a must in future (C3). Either way, model the operations on CSI so an adapter in the PR-2 proxy is mechanical.
R5	Topology depth: host plus switch block, or per-GPU/NIC graphs?	leaning	User 2026-10-11: build GPU/topology into the scheduler first, with the capability to keep improving it. Means resource-instances delivers an extensible inventory model plus an allocator plugin interface, starting simple.
R6	OCI containers as an execution mode?	leaning	User 2026-10-11: possible through the application profile (it already reserves a container mode). Profile defines image, runtime, mounts and device passthrough.
R7	Who changes MIG layouts: a provider or an admin?	open
R8	Honour rusage duration/decay?	decided	User 2026-10-11: must-have features, handled in a future spec (not the current phase).
R9	Where does data staging live?	discussing	PR-3: shared datasets are provisioned copies in the map; job-private staging is a claim.

Provisioner questions (from .kiro/specs/resource-provisioner/research.md)

ID	Question	Status	Notes
P1	Specify the persistent-provider JSON-lines format now (not in v1)?	open	Leaning yes, so v1 schemas carry over.
P2	Is Go via CGO into libxce acceptable for the provisioner/proxy daemon?	open	Leaning yes (apiserver pattern; cloud SDKs native).
P3	Must peer authentication (per-role opcode allowlist) ship before the provisioner?	open	Related to N4: component authentication is separate from user RBAC.
P4	Fix the wire byte order and publish the protocol as stable?	open
P5	Are cloud accounts dedicated to the cluster, or shared?	open	If shared, tagged-but-unknown instances stay report-only.
P6	Automatic return of never-joined instances in v1?	open	Leaning yes, after a join deadline.
P7	Offer stop (quarantine) as well as terminate?	open
P8	Settle window after failover: fixed time, or until expected workers rejoin?	open
P9	License server vs XCE in-use counts: which wins?	open

New questions raised in discussion

ID	Question	Status	Notes
N1	Does the supply map (PR-1) also cover hosts provisioned by elastic-hosts, or only resources on hosts and pools?	discussing	User 2026-10-11: hosts join the controller directly today; how could they be unified? Proposal PR-4 below.
N2	One proxy (PR-2) for every provider kind, or one per kind (cloud, license, storage)? How does it map onto the provisioner's per-provider leases?	leaning	User 2026-10-11: there may be multiple proxies. Matches the provisioner brief: several instances, each provider owned by exactly one active proxy at a time through a controller lease, with standbys taking over. Still open: N2a, N2b.
N2a	How does a proxy declare which providers it serves: config pushed by the controller, or self-declared at join (and checked against config)?	open
N2b	Can one proxy serve several kinds (cloud and storage), or one kind each?	open	Leaning: any mix; the lease is per provider, not per kind.
N3	Dataset model (PR-3): identity and versions, read-only vs updatable, capacity accounting at each location, eviction policy, who triggers staging (scheduler demand, user, admin)?	discussing	User 2026-10-11: start with a simplified model. Proposal PR-5 below.
N4	User RBAC in the apiserver vs component authorization in the controller (which daemon roles may call which opcodes): confirm both layers exist and where each check lives.	leaning	User 2026-10-11: XCE delegates access control to the API server, and the controller trusts it. Implication: the trust is only sound if the controller knows a request really comes from the apiserver (or another legitimate component). Today validate_host() accepts every peer (src/controller/join.c:13-16), so anything that reaches the controller port bypasses RBAC. Component authentication with a per-role opcode allowlist (peer-authentication-hardening, P3) is what makes the delegation safe, especially once proxies can spend money.
N6	PR-6: with I/O claims, should -o/-e/-i and out_dir be accepted again even when no output claim is configured (e.g. a default per-job output directory claim), or stay rejected?	open
N7	PR-6: where do uploaded inputs live between submit and job start (provider volume, apiserver staging area), and is there a size limit?	open	Large inputs should be datasets (PR-5).
N8	PR-6: retention and quota defaults for output claims; who is charged (pool counter per project)?	discussing	With the shared volume: per-user or per-project quota on that volume (filesystem quota, or usage charged to the pool counter after the fact). Retention: see N14.
N9	PR-6: multi-host jobs: a read-write-many output (shared filesystem), or per-host outputs collected after the job?	leaning	Read-write-many shared volume (PR-6 refinement).
N10	PR-6: job arrays: one output claim with a subdirectory per element, or one claim per element?	leaning	Subdirectory per element: <jobid>/<index>/... (PR-6 refinement).
N11	PR-6: requeue and restart: reuse the same output claim or create a new one? How does it relate to checkpoint claims?	open
N12	PR-6 shared volume: permissions and creation. <user> 0700 and user-owned under a root-owned top? Is the job directory created by the worker (as root, before dropping privileges) or by a provider? Path-safe encoding of array IDs with brackets?	open
N13	PR-6 shared volume: a requeued job keeps its ID. Reuse <jobid>/ or add an attempt subdirectory (<jobid>/attempt-2)? (Refines N11.)	open
N14	PR-6 shared volume: retention per job directory (TTL, after fetch), the cleaner (a provider task), and orphan directories whose job no longer exists.	open
N15	PR-6 shared volume: with several shared volumes, which one a job uses: by application profile, queue or project?	open
C3	CSI compatibility: nice-to-have or a must? If a must, which side: the controller plugin through the proxy, the node plugin on the worker, or both?	discussing	Background and recommendation in "CSI notes" below (2026-10-11).

History

• 2026-10-11: user proposed a shared volume with per-user/per-job subdirectories (shared_volume/<user>/<jobid>/{in,out,err}) instead of a volume per job → PR-6 refinement (leaning). N9 and N10 leaning; N8 reframed; N12–N15 added.
• 2026-10-11: user asked whether volume provisioning can carry job input and output → PR-6; questions N6–N11 added.
• 2026-10-11: user input on N1 (hosts join directly; asked how to unify → PR-4), N2 (multiple proxies → leaning), N3 (a simplified model first → PR-5), N4 (access control delegated to the API server, the controller trusts it → leaning, with the component-authentication implication), C3 (asked for more on CSI → CSI notes). N2a and N2b added.
• 2026-10-11: created from the resource-model discussion. User input recorded for PR-1, PR-2 and PR-3 (proposals) and for R3, R4, R5, R6 and R8. Questions N1–N4 and C3 added.