myxce/.kiro/steering/resource-questions.md

# Resource Design: Decisions, Proposals and Open Questions

The persistent discussion log for the unified resource model
(`resource-model.md`) and the provisioning work (`resource-provisioner`,
`provisioning-demand`, `elastic-*`). Specs must not settle an item here
silently.

**How to use:**
- **Status values:** `open` (not discussed), `discussing` (user input recorded, not final), `leaning` (direction agreed, details open), `decided`, `deferred` (later, with a reason).
- **Changes:** update an item in place and add a dated line under "History". When an item is decided, also update the documents it names.
- **IDs:** R = resource model, P = provisioner, N = new questions raised in discussion, PR = proposals.

## Proposals under discussion (user, 2026-10-11)

### PR-1: one resource map for all supply, including provisioned supply (status: leaning)
- **User:** should `resource-distribution-mapping` be made generic, so it also handles resource provisioning?
- **Analysis (2026-10-11):** yes, for the **data model**.
  - The resource map becomes the single record of supply: for each resource, which locations (host, host group, pool, storage) hold how much.
  - Each entry has a **source** (static config, reporter, provisioner) and a **state** (requested, available, draining, releasing).
  - Provisioning only changes map entries over time, e.g. a license pool's base plus borrowed amount, or a new dataset copy.
  - The provisioning **mechanism** stays in the provisioner: policy, provider calls, leases, epochs, orphan scans. The map is the state; the provisioner is the actuator. This is the Kubernetes split too (drivers publish ResourceSlices; autoscalers and provisioners act).
  - The scheduler keeps reading only the map's cached snapshot.
- **Affects:** `resource-distribution-mapping`, `elastic-shared-resources`, `resource-provisioner`, `provisioning-demand`.
- **Open:** see N1, N2.

### PR-2: a resource provider proxy between the controller and upstream providers (status: leaning)
- **User:** introduce a resource provider that proxies requests and states between the XCE controller and upstream resource providers.
- **Analysis (2026-10-11):** yes. It generalizes the `resource-provisioner` daemon. The proxy:
  - connects to the controller as one role over libxce/Cap'n Proto;
  - speaks each upstream provider's protocol: the JSON script contract, a persistent JSON-lines mode, later CSI or vendor APIs;
  - holds provider sessions, credentials and rate limits, and translates provider states into map entries and claim states;
  - publishes inventory and values (it is the external reporter of R2);
  - carries out provisioning and claim operations with `op_id`, an epoch and owner tags.
- **Why it helps:** the controller never blocks on or depends on an outside system, and credentials never reach the controller.
- **Where node-local work runs:** GPU binding, mounts and scratch stay on the worker, possibly behind the same contract.
- **Affects:** `resource-provisioner` (rename or generalize), `custom-resource-reporting` (external reporters), `resource-claims` (storage providers), R4/C3 (the CSI adapter would live here).
- **Open:** see N2, N4.

### PR-3: datasets in the resource map; staging as data provisioning (status: discussing)
- **User:** could data instances be handled by the resource distribution map, and data staging by provisioning the data to a new location?
- **Analysis (2026-10-11):** this fits PR-1 well.
  - **Datasets:** a dataset is a resource whose map entries are the locations holding a copy. Locality is an L1 attribute the scheduler can use.
  - **Staging:** staging is provisioning a copy at a new location, with a lead time like any provider.
  - **Eviction:** evicting a copy is a release.
  - **Shortage:** "dataset X needed at host group Y" can become provisioning demand.
  - Locality, caching and stage-in become one mechanism.
- **Caveats:**
  - Job-private data (stage-out results, scratch, a job's own outputs) is still a per-job claim (`resource-claims`), not shared supply.
  - Datasets need an identity and version, capacity accounting on the storage holding a copy, an eviction policy, and consistency rules (read-only vs updated).
- **Affects:** `resource-distribution-mapping`, `custom-resource-reporting` (string-set locality may become map entries instead), `resource-claims` (scope shrinks to job-private data), R9.
- **Open:** see N3.

### PR-4: hosts as supply entries; the worker connection only confirms them (status: discussing)
- **User:** hosts get into the controller directly today; how could they be unified with the rest of the supply?
- **Today:** a worker connects to the controller and is accepted only if its name matches a host config entry (`find_matched_hostconfig`, `src/controller/join.c:~76`). The host record is created by that first join (`init_worker`). `elastic-hosts` plans "expected host" records matched at join.
- **Proposal:** separate *who decides a host should exist* (control plane) from *the worker's connection* (data plane):
  - **Host supply entries:** a host is an entry in the supply map with a source and a lifecycle state:
    - `config`: a static `hosts.yaml` entry or pattern;
    - `provisioned`: created by a proxy as an expected host, with instance id, template and deadline;
    - `discovered`: an unknown host that joins, admitted only if a policy allows it.

    Lifecycle: `expected → joining → available → draining → released`, plus `lost`.
  - **Join confirms:** the worker still connects directly. Job control, heartbeats and resync stay on that long-lived connection. A join confirms an existing entry (or is admitted by policy) instead of being the only way a host appears.
  - **Host resources:** `slots`, `mem` and so on stay L2 counters at the host's location, as today.
  - **Bring-up:** `xce-node-manager` can act as the node-side provider that starts the worker on a provisioned machine.
- **Gains:** one place for static, elastic (cloud) and discovered hosts. Orphan rules apply uniformly (expected but never joined; joined but unknown). `provisioning-demand` sees hosts and other resources the same way.
- **Affects:** `elastic-hosts` (its expected-host records become supply entries), `resource-distribution-mapping` (PR-1), `xce-node-manager`, the controller join path.

### PR-5: simplified dataset model, v1 (status: discussing)
- **User:** give a simplified dataset model first.
- **Proposal:**
  - **Identity:** a dataset is `name@version` and is **immutable**; a change is a new version. No update consistency problem.
  - **Copies:** supply-map entries of the dataset at a location (host, host group or storage pool) with a state `staging → available → evicting` and a size. The size is charged as an L2 counter on the location's cache or scratch capacity.
  - **Jobs:** a job names datasets, e.g. `data[imagenet@3]`. The scheduler **prefers** hosts with an available copy, as a locality attribute from its cached snapshot. v1 does not make placement wait for a copy.
  - **Staging:** in v1 staging is explicit. A user or admin asks for a copy at a location (`xcecli data stage`); a proxy provider copies it and reports `available`. Eviction is explicit too.
  - **Later:** scheduler-driven staging (shortfall as provisioning demand), automatic LRU eviction, dataset quotas, and writable datasets.
- **Open:** whether v1 job placement should *require* a copy (a hard constraint) or only prefer one; and what the dataset registry is (config, REST object, or both).

### PR-6: job input/output through provisioned volumes (status: discussing)
- **User (2026-10-11):** use volume provisioning for job input and output.
- **Context:**
  - bug-1936 rejects `-i/-o/-e`, `out_dir` and `transfer_files` at submit, because no field carries them to the job and the worker never stages files.
  - bug-1937 defers stage-in/out to a future spec.
  - Worker file fetch is locked to its own output path (bug-1508).
- **Proposal:** job I/O as claims (`resource-claims`):
  - **Input claims (read-only):** from a dataset copy (PR-5), for large or shared inputs; from files uploaded at submit, which a provider puts into a per-job input volume (the new home of `transfer_files`/`-f`); or from an existing standalone claim (model weights, a previous job's output).
  - **Output claim (read-write):** per job (template) or a named standalone claim.
    - `-o/-e` and `out_dir` resolve inside it.
    - Results are fetched through the API server from the provider, not the worker (this removes the bug-1508 restriction).
    - Retention policy: keep until fetched, or N days.
    - Optional export after the job (stage-out) to another volume, object storage or the user's home directory.
  - **Placement in the job:** the sandbox mounts the claims at fixed points (`/xce/in`, `/xce/out`) or at profile-allowed paths. Direct-mode jobs get the paths through environment variables.
  - **v1 provider: a directory provider on a shared filesystem.** Create is `mkdir` with an owner and a quota, publish is a bind mount or path, delete is removal. CSI, cloud block and burst-buffer volumes plug into the same contract later.
  - **Default:** jobs without I/O options keep today's behaviour.
- **Refinement (user, 2026-10-11): a shared volume with per-job subdirectories is the default (status: leaning).** Jobs do not each get a volume; several jobs share one: `shared_volume/<user>/<jobid>/{in,out,err}`, array elements `<jobid>/<index>/...`.
  - **The shared volume:** a long-lived, admin-provisioned standalone claim (read-write-many, retain). Its capacity is a pool counter in the supply map; the hosts that mount it are an L1 attribute, so placement only uses hosts that reach it.
  - **A job's directory:** a lightweight template claim of kind "subdirectory". Create is `mkdir` plus ownership, with no storage call; release is retention or cleanup.
  - **Per-job volumes** (burst buffer, large scratch) stay possible when a profile asks for one.
  - **Settles:** N9 (read-write-many, so multi-host jobs work) and N10 (a subdirectory per element). Reframes N8 as per-user or per-project quota on the shared volume (filesystem quota, or usage charged to the pool counter after the fact, consistent with `max(reserved, actual)`).
  - Prior art: Kubernetes `subPath` and the nfs-subdir provisioner; the shared output trees common at LSF and Slurm sites.
- **Affects:**
  - `resource-claims` (adds job I/O as its first use, ahead of checkpoints);
  - `job-sandbox-execution` (mount points);
  - the apiserver submit/fetch path (lifts the bug-1936 rejections once claims exist);
  - `application-profile` (default I/O claims, allowed paths);
  - PR-5 (inputs from datasets).

## CSI notes (2026-10-11, for C3 and R4)
Source: the CSI specification, from general knowledge; to be checked against the current spec version during design.

- **What it is:** the Container Storage Interface, a CNCF specification: a gRPC API between a *container orchestrator* and a *storage driver*. It is orchestrator-neutral (used by Kubernetes, Nomad, Mesos and Cloud Foundry), though most drivers are written, packaged and tested for Kubernetes.
- **Three services:**
  - **Identity** (every plugin): `GetPluginInfo`, `GetPluginCapabilities`, `Probe`.
  - **Controller** (runs once, central): `CreateVolume`/`DeleteVolume`, `ControllerPublishVolume`/`Unpublish` (attach to a node), `ValidateVolumeCapabilities`, `ListVolumes`, `GetCapacity`, `CreateSnapshot`/`DeleteSnapshot`/`ListSnapshots`, `ControllerExpandVolume`, `ControllerGetVolume`.
  - **Node** (one per host, privileged): `NodeStageVolume`/`Unstage` (mount once per node at a staging path), `NodePublishVolume`/`Unpublish` (bind-mount into the workload's path), `NodeGetInfo` (node id, max volumes, accessible topology), `NodeGetCapabilities`, `NodeExpandVolume`, `NodeGetVolumeStats`.
- **Key concepts:**
  - Volume capability: block or filesystem, plus fs type and mount flags.
  - Access modes: single-node writer or reader, multi-node reader-only, single-writer or multi-writer.
  - Topology: driver-reported segments such as `zone=a` or `rack=r1`; `CreateVolume` takes requisite and preferred topology.
  - Opaque parameters (a storage class); per-call secrets; volume context returned by create.
- **How Kubernetes runs it:**
  - The controller plugin runs as a Deployment, with sidecars (external-provisioner, attacher, snapshotter, resizer) that turn PVCs into CSI calls.
  - The node plugin runs as a privileged DaemonSet, with node-driver-registrar, over a unix socket.
  - The sidecars are Kubernetes-specific; the CSI calls themselves are not.
- **Mapping to XCE:**
  - **Controller side → PR-2 proxy:** `CreateVolume` when a claim or dataset copy is created; `ControllerPublish` when it is bound to hosts; `Delete` on release; `GetCapacity` → pool counters in the map; snapshots → checkpoint snapshots; secrets stay in the proxy.
  - **Node side → worker/xcecd:** `NodeStage` on first use on a host; `NodePublish` into the job's sandbox (or container) path; unpublish and unstage at job end. `NodeGetInfo` topology → host attributes (L1).
  - **Timing:** CSI topology plus "create after placement" matches the rule that the scheduler only selects (Kubernetes `WaitForFirstConsumer`).
  - **Access modes** map one-to-one onto claim access modes.
- **Costs:**
  - gRPC clients: natural in Go (the proxy; P2), awkward in C (the worker). A small Go node helper, started with `posix_spawn` or run under `xce-node-manager`, could make the node calls.
  - The node plugin is a privileged daemon on every host, needing mount propagation.
  - Some drivers assume Kubernetes APIs or container packaging, so each target driver needs testing outside Kubernetes.
- **Relevant drivers:**
  - HPC parallel filesystems: IBM Spectrum Scale/GPFS CSI, BeeGFS CSI, Lustre (e.g. AWS FSx, DDN), WEKA, VAST.
  - General storage: NFS, Ceph RBD/CephFS, NetApp Trident, Portworx, cloud disks (EBS, GCE PD, Azure Disk), local path.
  - Object storage: S3 via Mountpoint for S3 or JuiceFS.
  - Dataset caching: Fluid (Alluxio/JuiceFS), a Kubernetes dataset-cache project relevant to PR-5.
- **Recommendation:**
  - Make XCE's own claim and provider contract **CSI-shaped**: the same verbs (create, publish, stage, node publish, capacity, snapshot, expand), capabilities, access modes, topology and per-call secrets.
  - v1 ships simple native JSON providers (local scratch, NFS or parallel-filesystem directories).
  - Add a **CSI adapter** later as one provider type in the proxy, plus a node helper for the worker.
  - That keeps CSI a cheap "must later" rather than a v1 cost. If containers (R6) or cloud block storage become early targets, CSI moves up.

## Resource model questions (from `resource-model.md`)

| ID | Question | Status | User input / notes |
|---|---|---|---|
| R1 | Phase 1 scope: shared pools only (option 2) with runtime values split out, or option 3 in one spec? | discussing | Reframed by PR-1: phase 1 may become the generic supply map. Decide after PR-1. |
| R2 | Who reports runtime custom values: worker scripts, a reporter role, or both? | leaning | Leaning both: worker scripts (elim-like, `posix_spawn`) for host values; the PR-2 proxy for external and cluster values. |
| R3 | Are claims first-class objects owned by a project from v1? Do they count against quotas? | deferred | User 2026-10-11: RBAC is handled in the API server; the backend only provides configuration and state persistence. Discuss later (with N4). |
| R4 | Own JSON driver contract, or CSI and device-plugin wire compatibility? | leaning | User 2026-10-11: CSI compatibility is better. Still to decide whether it is nice-to-have or a must in future (C3). Either way, model the operations on CSI so an adapter in the PR-2 proxy is mechanical. |
| R5 | Topology depth: host plus switch block, or per-GPU/NIC graphs? | leaning | User 2026-10-11: build GPU/topology into the scheduler first, with the capability to keep improving it. Means `resource-instances` delivers an extensible inventory model plus an allocator plugin interface, starting simple. |
| R6 | OCI containers as an execution mode? | leaning | User 2026-10-11: possible through the application profile (it already reserves a `container` mode). Profile defines image, runtime, mounts and device passthrough. |
| R7 | Who changes MIG layouts: a provider or an admin? | open | |
| R8 | Honour `rusage` `duration`/`decay`? | decided | User 2026-10-11: must-have features, handled in a future spec (not the current phase). |
| R9 | Where does data staging live? | discussing | PR-3: shared datasets are provisioned copies in the map; job-private staging is a claim. |

## Provisioner questions (from `.kiro/specs/resource-provisioner/research.md`)

| ID | Question | Status | Notes |
|---|---|---|---|
| P1 | Specify the persistent-provider JSON-lines format now (not in v1)? | open | Leaning yes, so v1 schemas carry over. |
| P2 | Is Go via CGO into libxce acceptable for the provisioner/proxy daemon? | open | Leaning yes (apiserver pattern; cloud SDKs native). |
| P3 | Must peer authentication (per-role opcode allowlist) ship before the provisioner? | open | Related to N4: component authentication is separate from user RBAC. |
| P4 | Fix the wire byte order and publish the protocol as stable? | open | |
| P5 | Are cloud accounts dedicated to the cluster, or shared? | open | If shared, tagged-but-unknown instances stay report-only. |
| P6 | Automatic return of never-joined instances in v1? | open | Leaning yes, after a join deadline. |
| P7 | Offer `stop` (quarantine) as well as `terminate`? | open | |
| P8 | Settle window after failover: fixed time, or until expected workers rejoin? | open | |
| P9 | License server vs XCE in-use counts: which wins? | open | |

## New questions raised in discussion

| ID | Question | Status | Notes |
|---|---|---|---|
| N1 | Does the supply map (PR-1) also cover hosts provisioned by `elastic-hosts`, or only resources on hosts and pools? | discussing | User 2026-10-11: hosts join the controller directly today; how could they be unified? Proposal **PR-4** below. |
| N2 | One proxy (PR-2) for every provider kind, or one per kind (cloud, license, storage)? How does it map onto the provisioner's per-provider leases? | leaning | User 2026-10-11: there may be multiple proxies. Matches the provisioner brief: several instances, each provider owned by exactly one active proxy at a time through a controller lease, with standbys taking over. Still open: N2a, N2b. |
| N2a | How does a proxy declare which providers it serves: config pushed by the controller, or self-declared at join (and checked against config)? | open | |
| N2b | Can one proxy serve several kinds (cloud and storage), or one kind each? | open | Leaning: any mix; the lease is per provider, not per kind. |
| N3 | Dataset model (PR-3): identity and versions, read-only vs updatable, capacity accounting at each location, eviction policy, who triggers staging (scheduler demand, user, admin)? | discussing | User 2026-10-11: start with a simplified model. Proposal **PR-5** below. |
| N4 | User RBAC in the apiserver vs component authorization in the controller (which daemon roles may call which opcodes): confirm both layers exist and where each check lives. | leaning | User 2026-10-11: XCE delegates access control to the API server, and the controller trusts it. Implication: the trust is only sound if the controller knows a request really comes from the apiserver (or another legitimate component). Today `validate_host()` accepts every peer (`src/controller/join.c:13-16`), so anything that reaches the controller port bypasses RBAC. Component authentication with a per-role opcode allowlist (`peer-authentication-hardening`, P3) is what makes the delegation safe, especially once proxies can spend money. |
| N6 | PR-6: with I/O claims, should `-o/-e/-i` and `out_dir` be accepted again even when no output claim is configured (e.g. a default per-job output directory claim), or stay rejected? | open | |
| N7 | PR-6: where do uploaded inputs live between submit and job start (provider volume, apiserver staging area), and is there a size limit? | open | Large inputs should be datasets (PR-5). |
| N8 | PR-6: retention and quota defaults for output claims; who is charged (pool counter per project)? | discussing | With the shared volume: per-user or per-project quota on that volume (filesystem quota, or usage charged to the pool counter after the fact). Retention: see N14. |
| N9 | PR-6: multi-host jobs: a read-write-many output (shared filesystem), or per-host outputs collected after the job? | leaning | Read-write-many shared volume (PR-6 refinement). |
| N10 | PR-6: job arrays: one output claim with a subdirectory per element, or one claim per element? | leaning | Subdirectory per element: `<jobid>/<index>/...` (PR-6 refinement). |
| N11 | PR-6: requeue and restart: reuse the same output claim or create a new one? How does it relate to checkpoint claims? | open | |
| N12 | PR-6 shared volume: permissions and creation. `<user>` 0700 and user-owned under a root-owned top? Is the job directory created by the worker (as root, before dropping privileges) or by a provider? Path-safe encoding of array IDs with brackets? | open | |
| N13 | PR-6 shared volume: a requeued job keeps its ID. Reuse `<jobid>/` or add an attempt subdirectory (`<jobid>/attempt-2`)? (Refines N11.) | open | |
| N14 | PR-6 shared volume: retention per job directory (TTL, after fetch), the cleaner (a provider task), and orphan directories whose job no longer exists. | open | |
| N15 | PR-6 shared volume: with several shared volumes, which one a job uses: by application profile, queue or project? | open | |
| C3 | CSI compatibility: nice-to-have or a must? If a must, which side: the controller plugin through the proxy, the node plugin on the worker, or both? | discussing | Background and recommendation in "CSI notes" below (2026-10-11). |

## History
- 2026-10-11: user proposed a shared volume with per-user/per-job subdirectories (`shared_volume/<user>/<jobid>/{in,out,err}`) instead of a volume per job → PR-6 refinement (leaning). N9 and N10 leaning; N8 reframed; N12–N15 added.
- 2026-10-11: user asked whether volume provisioning can carry job input and output → PR-6; questions N6–N11 added.
- 2026-10-11: user input on N1 (hosts join directly; asked how to unify → PR-4), N2 (multiple proxies → leaning), N3 (a simplified model first → PR-5), N4 (access control delegated to the API server, the controller trusts it → leaning, with the component-authentication implication), C3 (asked for more on CSI → CSI notes). N2a and N2b added.
- 2026-10-11: created from the resource-model discussion. User input recorded for PR-1, PR-2 and PR-3 (proposals) and for R3, R4, R5, R6 and R8. Questions N1–N4 and C3 added.