diff --git a/.kiro/steering/resource-questions.md b/.kiro/steering/resource-questions.md index 4e62fb90..e78c97fa 100644 --- a/.kiro/steering/resource-questions.md +++ b/.kiro/steering/resource-questions.md @@ -50,6 +50,69 @@ silently. - **Affects:** `resource-distribution-mapping`, `custom-resource-reporting` (string-set locality may become map entries instead), `resource-claims` (scope shrinks to job-private data), R9. - **Open:** see N3. +### PR-4: hosts as supply entries; the worker connection only confirms them (status: discussing) +- **User:** hosts get into the controller directly today; how could they be unified with the rest of the supply? +- **Today:** a worker connects to the controller and is accepted only if its name matches a host config entry (`find_matched_hostconfig`, `src/controller/join.c:~76`). The host record is created by that first join (`init_worker`). `elastic-hosts` plans "expected host" records matched at join. +- **Proposal:** separate *who decides a host should exist* (control plane) from *the worker's connection* (data plane): + - **Host supply entries:** a host is an entry in the supply map with a source and a lifecycle state: + - `config`: a static `hosts.yaml` entry or pattern; + - `provisioned`: created by a proxy as an expected host, with instance id, template and deadline; + - `discovered`: an unknown host that joins, admitted only if a policy allows it. + + Lifecycle: `expected → joining → available → draining → released`, plus `lost`. + - **Join confirms:** the worker still connects directly. Job control, heartbeats and resync stay on that long-lived connection. A join confirms an existing entry (or is admitted by policy) instead of being the only way a host appears. + - **Host resources:** `slots`, `mem` and so on stay L2 counters at the host's location, as today. + - **Bring-up:** `xce-node-manager` can act as the node-side provider that starts the worker on a provisioned machine. +- **Gains:** one place for static, elastic (cloud) and discovered hosts. Orphan rules apply uniformly (expected but never joined; joined but unknown). `provisioning-demand` sees hosts and other resources the same way. +- **Affects:** `elastic-hosts` (its expected-host records become supply entries), `resource-distribution-mapping` (PR-1), `xce-node-manager`, the controller join path. + +### PR-5: simplified dataset model, v1 (status: discussing) +- **User:** give a simplified dataset model first. +- **Proposal:** + - **Identity:** a dataset is `name@version` and is **immutable**; a change is a new version. No update consistency problem. + - **Copies:** supply-map entries of the dataset at a location (host, host group or storage pool) with a state `staging → available → evicting` and a size. The size is charged as an L2 counter on the location's cache or scratch capacity. + - **Jobs:** a job names datasets, e.g. `data[imagenet@3]`. The scheduler **prefers** hosts with an available copy, as a locality attribute from its cached snapshot. v1 does not make placement wait for a copy. + - **Staging:** in v1 staging is explicit. A user or admin asks for a copy at a location (`xcecli data stage`); a proxy provider copies it and reports `available`. Eviction is explicit too. + - **Later:** scheduler-driven staging (shortfall as provisioning demand), automatic LRU eviction, dataset quotas, and writable datasets. +- **Open:** whether v1 job placement should *require* a copy (a hard constraint) or only prefer one; and what the dataset registry is (config, REST object, or both). + +## CSI notes (2026-10-11, for C3 and R4) +Source: the CSI specification, from general knowledge; to be checked against the current spec version during design. + +- **What it is:** the Container Storage Interface, a CNCF specification: a gRPC API between a *container orchestrator* and a *storage driver*. It is orchestrator-neutral (used by Kubernetes, Nomad, Mesos and Cloud Foundry), though most drivers are written, packaged and tested for Kubernetes. +- **Three services:** + - **Identity** (every plugin): `GetPluginInfo`, `GetPluginCapabilities`, `Probe`. + - **Controller** (runs once, central): `CreateVolume`/`DeleteVolume`, `ControllerPublishVolume`/`Unpublish` (attach to a node), `ValidateVolumeCapabilities`, `ListVolumes`, `GetCapacity`, `CreateSnapshot`/`DeleteSnapshot`/`ListSnapshots`, `ControllerExpandVolume`, `ControllerGetVolume`. + - **Node** (one per host, privileged): `NodeStageVolume`/`Unstage` (mount once per node at a staging path), `NodePublishVolume`/`Unpublish` (bind-mount into the workload's path), `NodeGetInfo` (node id, max volumes, accessible topology), `NodeGetCapabilities`, `NodeExpandVolume`, `NodeGetVolumeStats`. +- **Key concepts:** + - Volume capability: block or filesystem, plus fs type and mount flags. + - Access modes: single-node writer or reader, multi-node reader-only, single-writer or multi-writer. + - Topology: driver-reported segments such as `zone=a` or `rack=r1`; `CreateVolume` takes requisite and preferred topology. + - Opaque parameters (a storage class); per-call secrets; volume context returned by create. +- **How Kubernetes runs it:** + - The controller plugin runs as a Deployment, with sidecars (external-provisioner, attacher, snapshotter, resizer) that turn PVCs into CSI calls. + - The node plugin runs as a privileged DaemonSet, with node-driver-registrar, over a unix socket. + - The sidecars are Kubernetes-specific; the CSI calls themselves are not. +- **Mapping to XCE:** + - **Controller side → PR-2 proxy:** `CreateVolume` when a claim or dataset copy is created; `ControllerPublish` when it is bound to hosts; `Delete` on release; `GetCapacity` → pool counters in the map; snapshots → checkpoint snapshots; secrets stay in the proxy. + - **Node side → worker/xcecd:** `NodeStage` on first use on a host; `NodePublish` into the job's sandbox (or container) path; unpublish and unstage at job end. `NodeGetInfo` topology → host attributes (L1). + - **Timing:** CSI topology plus "create after placement" matches the rule that the scheduler only selects (Kubernetes `WaitForFirstConsumer`). + - **Access modes** map one-to-one onto claim access modes. +- **Costs:** + - gRPC clients: natural in Go (the proxy; P2), awkward in C (the worker). A small Go node helper, started with `posix_spawn` or run under `xce-node-manager`, could make the node calls. + - The node plugin is a privileged daemon on every host, needing mount propagation. + - Some drivers assume Kubernetes APIs or container packaging, so each target driver needs testing outside Kubernetes. +- **Relevant drivers:** + - HPC parallel filesystems: IBM Spectrum Scale/GPFS CSI, BeeGFS CSI, Lustre (e.g. AWS FSx, DDN), WEKA, VAST. + - General storage: NFS, Ceph RBD/CephFS, NetApp Trident, Portworx, cloud disks (EBS, GCE PD, Azure Disk), local path. + - Object storage: S3 via Mountpoint for S3 or JuiceFS. + - Dataset caching: Fluid (Alluxio/JuiceFS), a Kubernetes dataset-cache project relevant to PR-5. +- **Recommendation:** + - Make XCE's own claim and provider contract **CSI-shaped**: the same verbs (create, publish, stage, node publish, capacity, snapshot, expand), capabilities, access modes, topology and per-call secrets. + - v1 ships simple native JSON providers (local scratch, NFS or parallel-filesystem directories). + - Add a **CSI adapter** later as one provider type in the proxy, plus a node helper for the worker. + - That keeps CSI a cheap "must later" rather than a v1 cost. If containers (R6) or cloud block storage become early targets, CSI moves up. + ## Resource model questions (from `resource-model.md`) | ID | Question | Status | User input / notes | @@ -82,11 +145,14 @@ silently. | ID | Question | Status | Notes | |---|---|---|---| -| N1 | Does the supply map (PR-1) also cover hosts provisioned by `elastic-hosts`, or only resources on hosts and pools? | open | Hosts have their own lifecycle (join, drain, decommission); they may stay separate, with the map referencing them. | -| N2 | One proxy (PR-2) for every provider kind, or one per kind (cloud, license, storage)? How does it map onto the provisioner's per-provider leases? | open | The brief already allows several instances, each owning providers by lease. | -| N3 | Dataset model (PR-3): identity and versions, read-only vs updatable, capacity accounting at each location, eviction policy, who triggers staging (scheduler demand, user, admin)? | open | | -| N4 | User RBAC in the apiserver vs component authorization in the controller (which daemon roles may call which opcodes): confirm both layers exist and where each check lives. | open | Raised by the user's RBAC position (R3) and P3. | -| C3 | CSI compatibility: nice-to-have or a must? If a must, which side: the controller plugin through the proxy, the node plugin on the worker, or both? | open | The node plugin implies gRPC on the worker and mount semantics for the sandbox or container mode. | +| N1 | Does the supply map (PR-1) also cover hosts provisioned by `elastic-hosts`, or only resources on hosts and pools? | discussing | User 2026-10-11: hosts join the controller directly today; how could they be unified? Proposal **PR-4** below. | +| N2 | One proxy (PR-2) for every provider kind, or one per kind (cloud, license, storage)? How does it map onto the provisioner's per-provider leases? | leaning | User 2026-10-11: there may be multiple proxies. Matches the provisioner brief: several instances, each provider owned by exactly one active proxy at a time through a controller lease, with standbys taking over. Still open: N2a, N2b. | +| N2a | How does a proxy declare which providers it serves: config pushed by the controller, or self-declared at join (and checked against config)? | open | | +| N2b | Can one proxy serve several kinds (cloud and storage), or one kind each? | open | Leaning: any mix; the lease is per provider, not per kind. | +| N3 | Dataset model (PR-3): identity and versions, read-only vs updatable, capacity accounting at each location, eviction policy, who triggers staging (scheduler demand, user, admin)? | discussing | User 2026-10-11: start with a simplified model. Proposal **PR-5** below. | +| N4 | User RBAC in the apiserver vs component authorization in the controller (which daemon roles may call which opcodes): confirm both layers exist and where each check lives. | leaning | User 2026-10-11: XCE delegates access control to the API server, and the controller trusts it. Implication: the trust is only sound if the controller knows a request really comes from the apiserver (or another legitimate component). Today `validate_host()` accepts every peer (`src/controller/join.c:13-16`), so anything that reaches the controller port bypasses RBAC. Component authentication with a per-role opcode allowlist (`peer-authentication-hardening`, P3) is what makes the delegation safe, especially once proxies can spend money. | +| C3 | CSI compatibility: nice-to-have or a must? If a must, which side: the controller plugin through the proxy, the node plugin on the worker, or both? | discussing | Background and recommendation in "CSI notes" below (2026-10-11). | ## History +- 2026-10-11: user input on N1 (hosts join directly; asked how to unify → PR-4), N2 (multiple proxies → leaning), N3 (a simplified model first → PR-5), N4 (access control delegated to the API server, the controller trusts it → leaning, with the component-authentication implication), C3 (asked for more on CSI → CSI notes). N2a and N2b added. - 2026-10-11: created from the resource-model discussion. User input recorded for PR-1, PR-2 and PR-3 (proposals) and for R3, R4, R5, R6 and R8. Questions N1–N4 and C3 added.