myxce/.kiro/steering/resource-questions.md

Resource Design: Decisions, Proposals and Open Questions

The persistent discussion log for the unified resource model (resource-model.md) and the provisioning work (resource-provisioner, provisioning-demand, elastic-*). Specs must not settle an item here silently.

How to use:

• Status values: open (not discussed), discussing (user input recorded, not final), leaning (direction agreed, details open), decided, deferred (later, with a reason).
• Changes: update an item in place and add a dated line under "History". When an item is decided, also update the documents it names.
• IDs: R = resource model, P = provisioner, N = new questions raised in discussion, PR = proposals.
• Start here: "Delivery plan" (tracks, first slices, the decisions each needs, the deliberately deferred items). Then proposals, then question tables.

Delivery plan: tracks and slices (proposed 2026-10-12, status: discussing)

The proposals below are many (PR-1 to PR-15, about 50 questions). They do not have to be decided or built together. This plan keeps them tractable, so none is forgotten and none blocks unrelated work. Not yet copied into resource-model.md or the roadmap (pending user confirmation).

Why it is tractable:

• Optional and additive: a job that uses none of it behaves as today (LSF-compatible defaults), so each piece lands on its own.
• One vocabulary: roles plus typed relationships (PR-14) cover placement inside a job, between jobs and toward data. Supply (PR-1) covers hosts, pools, datasets and provisioned resources.
• Hard boundary: the scheduler sees only placement constraints and cached numbers. Lifecycle, plumbing, staging and providers live outside it.

Track	First slice (small, useful alone)	Decisions needed for the first slice	Later slices	Proposals
A. Roles	Named compound parts; the role (part index) on every allocation, WAL/snapshot field-complete; per-task role, rank and hostfile environment	N31, N45	—	PR-12, PR-15 (1, 8)
B. Relationships and placement	Jobspec skeleton with only colocate/spread inside a job (scope host or switch); relationship-aware search across parts; topology labels from config	N41, N42 (v1 kinds), N46, N47, N29 (config only)	exchange; lifecycle kinds (sidecar-of, init-of, fate); job-step runtime (N36); between-job groups (PR-11, N44); cu[] syntax and relax policy (N21, N22); rank order (N23); QoS	PR-10, PR-11, PR-13, PR-14, PR-15
C. Supply	resource-distribution-mapping shared pools (licenses), shaped as the PR-1 supply map	R1, PR-1	runtime reporters (R2); hosts as supply entries (PR-4, N1); provider proxies (PR-2, N2a/N2b); provisioning (P1–P9)	PR-1, PR-2, PR-4
D. Data and I/O	Directory provider on a shared volume, <user>/<jobid>/{in,out,err}; brings back -i/-o/-e/-outdir; user isolation	N6, N12, N14, N15	datasets (PR-5, N3); staging (PR-3); data dependencies (PR-8, N16–N19); job-I/O details (N7, N8); CSI (C3)	PR-3, PR-5, PR-6, PR-7, PR-8, PR-9
E. Devices	GPU instances with simple attributes (count plus NVLink domain)	R5	MIG (R7); NIC rails; topology-aware allocator via affinity[]	PR-10 (level 1)

Order: A first; B builds on A; C, D and E are independent of each other and of A/B. Questions not listed for a first slice stay open until their slice comes up.

Deliberately deferred (each needs telemetry or a mature base first). Revisit when the named prerequisite exists:

Item	Revisit when
Interference-aware placement (N37)	node telemetry for contention exists
Bandwidth as reserved counters (N38)	node isolation (track B later slice) is in place
QoS classes for co-located serving and batch (N39)	services (service-supervision) and node isolation exist
Multi-job gangs (gang-groups, backlog, N24)	RL post-training or independent serving pools reach the roadmap
CSI compatibility (C3, R4)	track D needs a storage backend without a native provider, or containers (R6) arrive
Data sharing between users (N20)	datasets (PR-5) exist
Per-role resize (N33)	job-resize starts
Per-task replacement in running jobs (N32)	service-supervision starts
Claims as RBAC objects and quotas (R3)	the RBAC discussion the user deferred
Multi-host actual-usage reconciliation (N50)	after track A, if accounting for multi-role jobs matters
Reservation and plan-ahead for large topology jobs (N25)	track B's first slice shows starvation of large jobs

Proposals under discussion (user, 2026-10-11)

PR-1: one resource map for all supply, including provisioned supply (status: leaning)

• User: should resource-distribution-mapping be made generic, so it also handles resource provisioning?
• Analysis (2026-10-11): yes, for the data model.
  • The resource map becomes the single record of supply: for each resource, which locations (host, host group, pool, storage) hold how much.
  • Each entry has a source (static config, reporter, provisioner) and a state (requested, available, draining, releasing).
  • Provisioning only changes map entries over time, e.g. a license pool's base plus borrowed amount, or a new dataset copy.
  • The provisioning mechanism stays in the provisioner: policy, provider calls, leases, epochs, orphan scans. The map is the state; the provisioner is the actuator. This is the Kubernetes split too (drivers publish ResourceSlices; autoscalers and provisioners act).
  • The scheduler keeps reading only the map's cached snapshot.
• Affects: resource-distribution-mapping, elastic-shared-resources, resource-provisioner, provisioning-demand.
• Open: see N1, N2.

PR-2: a resource provider proxy between the controller and upstream providers (status: leaning)

• User: introduce a resource provider that proxies requests and states between the XCE controller and upstream resource providers.
• Analysis (2026-10-11): yes. It generalizes the resource-provisioner daemon. The proxy:
  • connects to the controller as one role over libxce/Cap'n Proto;
  • speaks each upstream provider's protocol: the JSON script contract, a persistent JSON-lines mode, later CSI or vendor APIs;
  • holds provider sessions, credentials and rate limits, and translates provider states into map entries and claim states;
  • publishes inventory and values (it is the external reporter of R2);
  • carries out provisioning and claim operations with op_id, an epoch and owner tags.
• Why it helps: the controller never blocks on or depends on an outside system, and credentials never reach the controller.
• Where node-local work runs: GPU binding, mounts and scratch stay on the worker, possibly behind the same contract.
• Affects: resource-provisioner (rename or generalize), custom-resource-reporting (external reporters), resource-claims (storage providers), R4/C3 (the CSI adapter would live here).
• Open: see N2, N4.

PR-3: datasets in the resource map; staging as data provisioning (status: discussing)

• User: could data instances be handled by the resource distribution map, and data staging by provisioning the data to a new location?
• Analysis (2026-10-11): this fits PR-1 well.
  • Datasets: a dataset is a resource whose map entries are the locations holding a copy. Locality is an L1 attribute the scheduler can use.
  • Staging: staging is provisioning a copy at a new location, with a lead time like any provider.
  • Eviction: evicting a copy is a release.
  • Shortage: "dataset X needed at host group Y" can become provisioning demand.
  • Locality, caching and stage-in become one mechanism.
• Caveats:
  • Job-private data (stage-out results, scratch, a job's own outputs) is still a per-job claim (resource-claims), not shared supply.
  • Datasets need an identity and version, capacity accounting on the storage holding a copy, an eviction policy, and consistency rules (read-only vs updated).
• Affects: resource-distribution-mapping, custom-resource-reporting (string-set locality may become map entries instead), resource-claims (scope shrinks to job-private data), R9.
• Open: see N3.

PR-4: hosts as supply entries; the worker connection only confirms them (status: discussing)

• User: hosts get into the controller directly today; how could they be unified with the rest of the supply?
• Today: a worker connects to the controller and is accepted only if its name matches a host config entry (find_matched_hostconfig, src/controller/join.c:~76). The host record is created by that first join (init_worker). elastic-hosts plans "expected host" records matched at join.
• Proposal: separate who decides a host should exist (control plane) from the worker's connection (data plane):

  • Host supply entries: a host is an entry in the supply map with a source and a lifecycle state:

    • config: a static hosts.yaml entry or pattern;
    • provisioned: created by a proxy as an expected host, with instance id, template and deadline;
    • discovered: an unknown host that joins, admitted only if a policy allows it.

    Lifecycle: expected → joining → available → draining → released, plus lost.

  • Join confirms: the worker still connects directly. Job control, heartbeats and resync stay on that long-lived connection. A join confirms an existing entry (or is admitted by policy) instead of being the only way a host appears.

  • Host resources: slots, mem and so on stay L2 counters at the host's location, as today.

  • Bring-up: xce-node-manager can act as the node-side provider that starts the worker on a provisioned machine.

• Gains: one place for static, elastic (cloud) and discovered hosts. Orphan rules apply uniformly (expected but never joined; joined but unknown). provisioning-demand sees hosts and other resources the same way.
• Affects: elastic-hosts (its expected-host records become supply entries), resource-distribution-mapping (PR-1), xce-node-manager, the controller join path.

PR-5: simplified dataset model, v1 (status: discussing)

• User: give a simplified dataset model first.
• Proposal:
  • Identity: a dataset is name@version and is immutable; a change is a new version. No update consistency problem.
  • Copies: supply-map entries of the dataset at a location (host, host group or storage pool) with a state staging → available → evicting and a size. The size is charged as an L2 counter on the location's cache or scratch capacity.
  • Jobs: a job names datasets, e.g. data[imagenet@3]. The scheduler prefers hosts with an available copy, as a locality attribute from its cached snapshot. v1 does not make placement wait for a copy.
  • Staging: in v1 staging is explicit. A user or admin asks for a copy at a location (xcecli data stage); a proxy provider copies it and reports available. Eviction is explicit too.
  • Later: scheduler-driven staging (shortfall as provisioning demand), automatic LRU eviction, dataset quotas, and writable datasets.
• Open: whether v1 job placement should require a copy (a hard constraint) or only prefer one; and what the dataset registry is (config, REST object, or both).

PR-6: job input/output through provisioned volumes (status: discussing)

• User (2026-10-11): use volume provisioning for job input and output.
• Context:
  • bug-1936 rejects -i/-o/-e, out_dir and transfer_files at submit, because no field carries them to the job and the worker never stages files.
  • bug-1937 defers stage-in/out to a future spec.
  • Worker file fetch is locked to its own output path (bug-1508).
• Proposal: job I/O as claims (resource-claims):
  • Input claims (read-only): from a dataset copy (PR-5), for large or shared inputs; from files uploaded at submit, which a provider puts into a per-job input volume (the new home of transfer_files/-f); or from an existing standalone claim (model weights, a previous job's output).
  • Output claim (read-write): per job (template) or a named standalone claim.
    • -o/-e and out_dir resolve inside it.
    • Results are fetched through the API server from the provider, not the worker (this removes the bug-1508 restriction).
    • Retention policy: keep until fetched, or N days.
    • Optional export after the job (stage-out) to another volume, object storage or the user's home directory.
  • Placement in the job: the sandbox mounts the claims at fixed points (/xce/in, /xce/out) or at profile-allowed paths. Direct-mode jobs get the paths through environment variables.
  • v1 provider: a directory provider on a shared filesystem. Create is mkdir with an owner and a quota, publish is a bind mount or path, delete is removal. CSI, cloud block and burst-buffer volumes plug into the same contract later.
  • Default: jobs without I/O options keep today's behaviour.
• Refinement (user, 2026-10-11): a shared volume with per-job subdirectories is the default (status: leaning). Jobs do not each get a volume; several jobs share one: shared_volume/<user>/<jobid>/{in,out,err}, array elements <jobid>/<index>/....
  • The shared volume: a long-lived, admin-provisioned standalone claim (read-write-many, retain). Its capacity is a pool counter in the supply map; the hosts that mount it are an L1 attribute, so placement only uses hosts that reach it.
  • A job's directory: a lightweight template claim of kind "subdirectory". Create is mkdir plus ownership, with no storage call; release is retention or cleanup.
  • Per-job volumes (burst buffer, large scratch) stay possible when a profile asks for one.
  • Settles: N9 (read-write-many, so multi-host jobs work) and N10 (a subdirectory per element). Reframes N8 as per-user or per-project quota on the shared volume (filesystem quota, or usage charged to the pool counter after the fact, consistent with max(reserved, actual)).
  • Prior art: Kubernetes subPath and the nfs-subdir provisioner; the shared output trees common at LSF and Slurm sites.
• Affects:
  • resource-claims (adds job I/O as its first use, ahead of checkpoints);
  • job-sandbox-execution (mount points);
  • the apiserver submit/fetch path (lifts the bug-1936 rejections once claims exist);
  • application-profile (default I/O claims, allowed paths);
  • PR-5 (inputs from datasets).

PR-7: keep job submission options simple (status: leaning)

• User (2026-10-11): keep the submit options simple.
• Proposal:
  • No volume or claim names for ordinary jobs. The application profile or queue selects the shared volume and the directory layout (resolves N15).
  • LSF options come back, resolved against the job's directory (PR-6): -i, -o, -e (with %J/%I), -outdir, and -f (copy local files in, or results back out). Resolves N6: the bug-1936 rejections are lifted once the job directory exists.
  • At most two advanced options: --data name@ver (input dataset, PR-5) and --claim name:mode (standalone claim, e.g. checkpoints or model weights). Everything else lives in profiles.

PR-8: job dependencies by data produced and consumed (status: discussing)

• User (2026-10-11): consider job dependencies based on producing and consuming data.
• Proposal: an extension of job-dependency. Prior art: LSF Data Manager data tags, Nextflow, Snakemake, Argo.
  • Producer: declares an output dataset (--produce results@auto). On successful completion its output directory is sealed as an immutable dataset version (PR-5).
  • Consumer: declares --data results@latest or a specific version. A dependency condition data(results@v) keeps it pending until the version is available. Then the version is mounted read-only as an input, and placement prefers hosts near it.
• Open: see N16–N19.

PR-9: isolation between users; data sharing later (status: leaning)

• User (2026-10-11): consider isolation between users; a mechanism for sharing data across users may be low priority.
• Isolation (v1 must):
  • Each user's tree is 0700 and user-owned; jobs run as the user's uid.
  • Claims and datasets are owned by a user or project.
  • The API server lets users list and fetch only their own data; admins see everything (N4: access control lives in the API server).
  • Filesystem permissions (uid, gid, ACLs) must enforce isolation as well, because jobs read the shared filesystem directly on workers. API server RBAC alone is not enough.
• Sharing (low priority):
  • A sealed, immutable dataset (PR-5) gets a visibility of private, project or public, always read-only.
  • The API server checks the visibility; the worker mounts the dataset read-only, or a group ACL grants access on the filesystem.
  • Shared writable space, if ever needed, is a project-owned directory with group permissions.
  • Sharing is what enables cross-user PR-8.

PR-10: LLM training as a single parallel job with topology requirements (status: discussing)

• User (2026-10-11): treat LLM training as a single job with complex topology requirements.
• Analysis: yes, as the default model. Training is one parallel job, not a gang of jobs.
  • Already in XCE:
    • all-or-nothing placement of one parallel job;
    • span[ptile|hosts|block] (parjob-placement);
    • compound resreqs for roles of different shapes inside one job (lib/xce_resreq.peg:60-83), e.g. 1*{select[!gpu]} + 16*{rusage[gpu=8] span[ptile=1]} (coordinator plus GPU workers);
    • one lifecycle for kill, requeue, checkpoint and accounting.
  • Topology, at three levels:
    1. Inside a host: GPU and NIC choice (NVLink domain, rail affinity, NUMA). From resource-instances (identified instances with attributes) and affinity[], which is parsed but unused (lib/resreq.c:127).
    2. Across hosts: topology domains (leaf switch, rail group, IB block, NVL72 rack), stored as host attributes in the supply map. A request bounds and packs domains. Precedents: LSF compute units cu[type=switch:maxcus=2:pref=minavail] (fits XCE's LSF-style syntax), and Slurm --switches=N@maxwait (wait up to a limit for better topology, then relax). parjob-placement already notes locality scoring as later work.
    3. Rank order: the allocation returns hosts in topology order so adjacent ranks or pipeline stages are close; the launcher's hostfile follows it. The ordered allocation is persisted field-complete.
  • Launch: the launcher on the first host gets the hostfile and environment (MASTER_ADDR, WORLD_SIZE, ranks) and starts torchrun, DeepSpeed or MPI, or starts remote tasks through xcecd's remote-task path.
  • Failure: host loss fails or requeues the job; it restarts from its checkpoint claim. Elastic training comes later via job-resize.
  • Scheduler impact: topology-aware selection is combinatorial and needs heuristics (best-fit domain packing). Large GPU jobs need reservation or plan-ahead so small jobs do not fragment domains and starve them.
• Consequence: gang-groups becomes optional and later. It is only for roles that need separate lifecycles, profiles or owners, e.g. an RL trainer plus rollout or inference services.
• Affects: resource-instances (level 1), parjob-placement (levels 2 and 3, locality), the resreq syntax (cu[] or an extended same[]), gang-groups (demote), the scheduler reservation and backfill work.
• Open: see N21–N25.

PR-11: data, cache and communication topology between jobs (status: discussing)

• User (2026-10-11): how about data cache, data and communication topology between jobs?
• One topology model:
  • Hosts (PR-4) carry hierarchical labels: host ⊂ leaf switch ⊂ block/spine ⊂ rack ⊂ zone. Storage pools carry where they are reachable from, and their speed.
  • The scheduler derives one distance: same host < same switch < same block < anywhere. It serves within-job topology (PR-10 level 2), between-job placement and data locality alike.
  • Labels first, not a full graph (as in R5). Admin config first; discovery (ibnetdiscover, LLDP) later through a reporter.
• 1. Data and cache locality between jobs (producer to consumer through files):
  • Dataset copies and node-local caches are supply-map entries with locations (PR-5, PR-8). A consumer prefers the closest copy (Spark locality levels: node, switch, storage pool, any).
  • Delay scheduling (Spark locality.wait): wait briefly for a near host, then relax. The same policy as N22, timed in session time so simulation stays deterministic.
  • Move job or move data: place far, or stage a copy (PR-3), decided by size and lead time. Start with "prefer locality, stage explicitly".
• 2. Communication between jobs (RL trainer and rollouts, prefill and decode, streaming producer and consumer, services and heavy clients):
  • Placement groups (like AWS cluster placement groups): a named group that jobs join (--placement-group <name>, or set by a profile).
  • Modes: pack (same or nearest domain) or spread (fault tolerance; replica anti-affinity for service-replica-set).
  • Strictness: hard (required) or soft (preferred), like Kubernetes inter-pod affinity with topologyKey.
  • Co-placed, not co-started: covers most RL and prefill/decode needs without a multi-job gang (supports N24).
  • Caveat: later members are tied to the first member's domain and may wait. Optionally the group reserves domain capacity (N25).
• 3. Scheduler: soft preferences become host scoring in the scheduler framework's host iterator; hard ones become filters. Both read only the cached snapshot (other jobs' allocations, labels, copy locations), so the scheduler stays deterministic and never calls a provider. Groups are cheaper than arbitrary job-to-job affinity.
• Affects: scheduler-framework (scoring and filters), parjob-placement (shared topology labels), service-replica-set (spread), PR-4 (labels on host entries), PR-5/PR-8 (copy locations), custom-resource-reporting (topology discovery), N22/N25.
• Open: see N26–N30.

PR-12: disaggregated LLM serving as one job per unit, replicated (status: discussing)

• User (2026-10-12): can disaggregated LLM serving be put into a single job?
• Proposal: yes, one job per unit.
  • A unit is one compound job: a router, P prefill and D decode hosts, e.g. 1P3D: 1*{router: select[!gpu]} + 1*{prefill: rusage[gpu=8] span[ptile=1]} + 3*{decode: rusage[gpu=8] span[ptile=1]}.
  • What one job gives for free: all-or-nothing start with no gang; topology packing inside the job, so prefill and decode share a switch or NVLink domain for KV transfer (PR-10); one lifecycle and one accounting record.
  • Scaling: service-replica-set scales whole units (each replica is one unit job); service-endpoint balances across units' routers.
  • Prior art: fixed xPyD units, replicated as Kubernetes LeaderWorkerSet groups or NVIDIA Dynamo/llm-d graphs.
• Limits and remedies:
  • Prefill:decode ratio is fixed per unit: offer several unit shapes (1P3D, 2P2D) as profiles; later, per-part resize (N33).
  • Scaling one pool alone is impossible within a unit: scale units. Independent pools behind a global router remain a gang-groups co-start case (N34).
  • A host failure fails the unit: acceptable while other replicas serve; later, per-task replacement (N32).
  • Rolling updates happen per unit through service-replica-set.
• Repo facts (2026-10-12): compound parts are an unnamed list (xce_compound_resreq_t.primaries, src/include/resreq.h:255-257). Tasks see only XCE_JOBID, XCE_JOBINDEX and internal variables; there is no role, rank or hostfile (N31).
• Effect on N24: disaggregated serving mostly moves to single-job units. gang-groups remains for independent pools behind a global router, RL post-training and streaming pipelines.
• Affects: the resreq syntax (named parts), worker task environment and hostfiles (N31, also needed by PR-10 training), service-replica-set, service-supervision (N32), job-resize (N33), service-endpoint (N34), gang-groups (scope note).

PR-13: task relationship model, typed edges between roles (status: discussing)

• User (2026-10-12): consider how different work cooperates at the task level. Tasks race for resources (CPU, memory, network bandwidth, cache), exchange data by memory, disk or network, and depend on each other in piped processing, plus other relationships. We need to manage complex relationships between tasks.
• Relationship types:
  1. Contention: CPU cores, memory capacity and bandwidth, last-level cache, NUMA, PCIe/NVLink, NIC bandwidth, disk I/O, GPU SMs (MPS).
     • Isolate (worker): cgroups cpuset/memory/io; NUMA binding via affinity[] (parsed, unused, lib/resreq.c:127); Intel RDT cache and memory-bandwidth allocation; NIC traffic classes; MIG.
     • Place around interference (scheduler): keep noisy tasks apart, pair complementary ones.
     • Measurable bandwidth can become L2 counters.
  2. Data exchange:
     • shared memory or local disk → same host or NUMA scope;
     • network (RDMA, NCCL, TCP) → switch or rail proximity, plus bandwidth;
     • plumbing: shared IPC namespace or shm mount, a shared scratch claim, endpoint and rank discovery (N31).
  3. Pipelines and dependencies:
     • streaming (concurrent): co-start, a channel, lifecycle coupling;
     • staged (B after A): a DAG inside the job, i.e. PR-8 at task granularity;
     • flow control and backpressure stay in the application.
  4. Others:
     • lifecycle coupling: fate policy (fail together, restart one, tolerate); sidecar tasks (log shipper, data loader, proxy); init tasks (download weights, warm cache); readiness gating between roles;
     • co-location and spread: with a scope from NUMA to block, hard or soft;
     • communication pattern: ring, all-to-all (MoE), pipeline neighbours → rank-to-topology mapping (PR-10);
     • QoS between co-located tasks: latency-critical serving next to batch;
     • isolation between tenants sharing a node (PR-9; exclusive hosts);
     • shared budgets: power and thermal caps as pool counters.
• Making it manageable:
  • Edges between roles (N31 named parts), expanded per task by the scheduler.
  • One vocabulary for task-to-task, job-to-job (PR-11 placement groups) and job-to-data (PR-5 locality).
  • Layering: the scheduler turns edges into placement constraints and scores from its cached snapshot; the worker enforces contention and exchange (cgroups, RDT, namespaces, mounts) and runs local sidecar and init ordering; the controller runs cross-host lifecycle (fate sharing, restarts, readiness gates); the application keeps flow control.
  • Allocation plus steps (N36): the scheduler allocates once, topology-aware; a job-step runtime inside the allocation launches tasks and steps by the graph (init, then main plus sidecars, then staged pipeline steps), like Slurm job steps or Flux nested scheduling. Sequencing stays out of the scheduler's hot path.
• Prior art: Kubernetes pods (shared network, IPC and volumes), sidecar and init containers, pod affinity, Topology/CPU Manager, QoS classes; Volcano task-topology plugin (affinity between task roles); Ray placement-group bundles; Slurm job steps, heterogeneous jobs and CPU binding; LSF affinity[], blaunch, compound resreqs; Flux hierarchical scheduling; Intel RDT.
• Suggested phasing:
  1. N31 role identity.
  2. Placement edges between roles (PR-11 vocabulary).
  3. Node isolation: cpuset/NUMA via affinity[], memory/io cgroups; RDT, NIC QoS and bandwidth counters later.
  4. Lifecycle edges: init and sidecar tasks, fate policy, per-task restart (N32).
  5. Job-step runtime: staged and streaming pipelines inside an allocation, with data handoff.
• Affects: resreq syntax (roles and edges), scheduler-framework (constraints and scoring), worker (xcecd_cgroups.c, isolation, namespaces), service-supervision (lifecycle), job-dependency (task-level DAG), parjob-placement, resource-instances.

PR-14: one uniform way to describe task relationships, a canonical jobspec (status: discussing)

• User (2026-10-12): should we add a uniform approach to describe the relationships of tasks?
• Proposal: yes. Without one, each relationship kind grows its own syntax, as in Kubernetes (node and pod affinity, topology spread, init containers, sidecars, dependencies). Precedent: Flux's jobspec, a canonical description of resources and tasks.
  • One record shape: {kind, from, to, scope, strength, params}.
  • Subjects and objects: role:<name> (N31), job:<id|label>, group:<placement-group>, data:<name@ver>, claim:<name>.
  • A fixed set of kinds, each with its enforcing layer:
    • colocate/spread: scope numa…zone, strength required or preferred, weight. Scheduler.
    • exchange: channel shm, localdisk, sharedfs or network, plus bandwidth. Scheduler (the channel implies a scope) and worker (plumbing).
    • with/after/ready-after: co-start, after completion, after readiness. Controller or job-step runtime.
    • sidecar-of/init-of: worker and controller.
    • fate: fail-together, restart-member, tolerate. Controller.
    • isolate/share: exclusivity or QoS class, deliberate sharing (MPS, cache). Worker and scheduler.
    • consumes/produces: data dependency and locality (PR-8). Scheduler and controller.
  • One vocabulary at three levels: inside a job (role to role), between jobs (job:, group:; PR-11 placement groups become relations toward a group:), and to data (data:, claim:).
• Canonical form plus shorthands:
  • Canonical: a structured job spec (YAML/JSON over REST), persisted on the job (WAL submit fact, field-complete in snapshots) and shown by info job. The resreq stays the per-role resource request inside it.
  • Shorthands that compile into it: xcecli flags (--placement-group, --after, --data); application-profile relationship templates (e.g. "1P3D serving unit", "training with a data-loader sidecar"); and existing span[]/same[]/affinity[], translated into colocate, spread and isolate.
• Example (a PR-12 unit): roles router (1), prefill (1), decode (3), metrics (per host). Relations:
  • colocate prefill↔decode scope=switch required
  • exchange prefill→decode channel=network rdma
  • ready-after router→{prefill,decode}
  • sidecar-of metrics→{prefill,decode}
  • fate {prefill,decode,router} fail-together
• Rules:
  • The scheduler sees only placement relations, compiled into filters and scores; the rest passes through. The hot path stays cheap and deterministic.
  • Validation at submit: contradictions, ordering cycles, unknown roles, impossible scopes.
  • A pending job names the relation that blocks it (scheduler-alter-pend-reasons).
  • Configuration layering: profile relations are profile-owned and bind; a job may only add relations (additive rule, job-config-layering.md, 2026-10-12).
  • The schema is versioned, and new kinds are additive.
• Would become a spec, e.g. task-relationship-spec (the canonical jobspec), after N31.
• Open: see N41–N44.

PR-15: relationships and compound resreqs (status: discussing)

• User (2026-10-12): consider the feature interaction with compound resreqs.
• Repo facts (2026-10-12):
  • Grammar: alter := {compound} || ..., compound := N*{primary} + .... Each part has its own select/rusage/span/same/affinity and a fixed count (minTasks = maxTasks = N, src/lib/xce_resreq.peg:83-99). Parts are unnamed.
  • The scheduler allocates a compound part by part, greedily, on one context (sched_ctx_set_primary, src/scheduler/src/schedobj.c:547; bug-1768, bug-1931). An alternative resreq tries one alternative at a time (bug-1769). There is no backtracking between parts.
  • xce_alloc_t (instid, from, duration) has no part index, so allocations lose which part a host serves.
• Interactions and proposed rules:
  1. Roles are compound parts. Add named parts (N31); unnamed parts get implicit names part0, part1, … for LSF compatibility. The jobspec's roles: is the compound, and relationships refer to part names.
  2. Cross-part relationships vs greedy allocation: part 1 may take a domain where part 2 cannot fit, so the job pends although a placement exists. Use relationship-aware allocation instead: order parts by constraint tightness, enumerate candidate domains for a cross-part colocate scope=D, and place all related parts inside one domain (a bounded search, shared with PR-10).
  3. Unsatisfiable-resreq cache: the key must include a hash of the relationships. Never cache results that depend on another job's placement (relationships between jobs).
  4. Alternatives: relationships are declared once per job over role names. Every alternative must define all roles named by a required relationship (checked at submit); a preferred relationship naming a missing role is skipped. Pend reasons are per alternative.
  5. Per-part span/same/affinity compile into self-relationships (span[hosts=1] → colocate scope=host; same[...] → same-attribute constraint). Contradictions are rejected at submit.
  6. Sidecar roles: by default attached to the parent's tasks, sharing their allocation and hosts, with no separate part. Optionally a separate part with a derived count (N48).
  7. Task ranges (-n min,max, parjob-placement range search): ranges per role, or only on primary jobs (N49).
  8. Allocation record: each host allocation records its role (part index). WAL and snapshot stay field-complete. Dispatch gives each task its role, rank and hostfile (N31); info job shows allocations per role.
  9. Configuration layering: the profile owns the compound template and relationships; the job may only choose counts within the profile's ranges and add relations (additive rule, 2026-10-12).
  10. Resize (N33): per-role resize re-checks the role's relationships.
  11. Pend reasons and trace: per part and per relationship (e.g. "decode: no switch with 3 free 8-GPU hosts near prefill"); trace capture per part; deterministic for sim/replay.
  12. Accounting: actual-usage reconciliation (max(reserved, actual)) is single-host only (resource-accounting.md); multi-role multi-host jobs inherit that gap. Record it explicitly (N50).
• Recommendation: the compound resreq is the resource half of the canonical jobspec (PR-14), and relationships are the other half. The essential engineering is items 1, 2 and 8. Items 3 and 4 are correctness rules required in v1.
• Affects: resreq grammar and resreq.h, schedobj.c and the allocator plugins (parjob_allocator.c), alloc_cache.c, xce_alloc_t and the WAL/snapshot allocation encoding, worker dispatch and task environment, parjob-placement, job-resize, scheduler-alter-pend-reasons, the decision trace.

Gang use cases (2026-10-11, for N24)

• Two kinds of gang:
  • Inside one job: all tasks of a parallel job start together. XCE already does this, and compound resreqs mix role shapes in one job (like Slurm heterogeneous jobs).
  • A gang of several jobs (gang-groups): needed only when roles must co-start and each needs its own identity: application profile or container, owner, lifecycle, or independent scaling.
• Covered by a single job:
  • data-, tensor- or pipeline-parallel training (PR-10);
  • MPMD coupled simulation (compound resreq);
  • coordinator or parameter server plus workers;
  • a multi-host replica of a huge served model: each replica is a parallel job inside service-replica-set (like Kubernetes LeaderWorkerSet).
• Need a multi-job gang:
  • RL post-training (RLHF, GRPO): trainer plus rollout engines plus reward and reference models; different apps, independent restart and scaling.
  • Disaggregated LLM serving: prefill pool, decode pool and router; co-start and close placement for KV-cache transfer, then independent scaling.
  • Streaming pipelines: a live producer feeding a consumer (preprocessing to training, online evaluation) with different profiles or owners.
• Not gangs: hyperparameter sweeps and independent trials (job arrays). Synchronous population-based training is rare; a co-start barrier on an array would do.
• Prior art: Volcano PodGroup (minMember), Kueue with JobSet, Kubernetes LeaderWorkerSet, Slurm heterogeneous jobs. LSF has no multi-job gang.

CSI notes (2026-10-11, for C3 and R4)

Source: the CSI specification, from general knowledge; to be checked against the current spec version during design.

• What it is: the Container Storage Interface, a CNCF specification: a gRPC API between a container orchestrator and a storage driver. It is orchestrator-neutral (used by Kubernetes, Nomad, Mesos and Cloud Foundry), though most drivers are written, packaged and tested for Kubernetes.
• Three services:
  • Identity (every plugin): GetPluginInfo, GetPluginCapabilities, Probe.
  • Controller (runs once, central): CreateVolume/DeleteVolume, ControllerPublishVolume/Unpublish (attach to a node), ValidateVolumeCapabilities, ListVolumes, GetCapacity, CreateSnapshot/DeleteSnapshot/ListSnapshots, ControllerExpandVolume, ControllerGetVolume.
  • Node (one per host, privileged): NodeStageVolume/Unstage (mount once per node at a staging path), NodePublishVolume/Unpublish (bind-mount into the workload's path), NodeGetInfo (node id, max volumes, accessible topology), NodeGetCapabilities, NodeExpandVolume, NodeGetVolumeStats.
• Key concepts:
  • Volume capability: block or filesystem, plus fs type and mount flags.
  • Access modes: single-node writer or reader, multi-node reader-only, single-writer or multi-writer.
  • Topology: driver-reported segments such as zone=a or rack=r1; CreateVolume takes requisite and preferred topology.
  • Opaque parameters (a storage class); per-call secrets; volume context returned by create.
• How Kubernetes runs it:
  • The controller plugin runs as a Deployment, with sidecars (external-provisioner, attacher, snapshotter, resizer) that turn PVCs into CSI calls.
  • The node plugin runs as a privileged DaemonSet, with node-driver-registrar, over a unix socket.
  • The sidecars are Kubernetes-specific; the CSI calls themselves are not.
• Mapping to XCE:
  • Controller side → PR-2 proxy: CreateVolume when a claim or dataset copy is created; ControllerPublish when it is bound to hosts; Delete on release; GetCapacity → pool counters in the map; snapshots → checkpoint snapshots; secrets stay in the proxy.
  • Node side → worker/xcecd: NodeStage on first use on a host; NodePublish into the job's sandbox (or container) path; unpublish and unstage at job end. NodeGetInfo topology → host attributes (L1).
  • Timing: CSI topology plus "create after placement" matches the rule that the scheduler only selects (Kubernetes WaitForFirstConsumer).
  • Access modes map one-to-one onto claim access modes.
• Costs:
  • gRPC clients: natural in Go (the proxy; P2), awkward in C (the worker). A small Go node helper, started with posix_spawn or run under xce-node-manager, could make the node calls.
  • The node plugin is a privileged daemon on every host, needing mount propagation.
  • Some drivers assume Kubernetes APIs or container packaging, so each target driver needs testing outside Kubernetes.
• Relevant drivers:
  • HPC parallel filesystems: IBM Spectrum Scale/GPFS CSI, BeeGFS CSI, Lustre (e.g. AWS FSx, DDN), WEKA, VAST.
  • General storage: NFS, Ceph RBD/CephFS, NetApp Trident, Portworx, cloud disks (EBS, GCE PD, Azure Disk), local path.
  • Object storage: S3 via Mountpoint for S3 or JuiceFS.
  • Dataset caching: Fluid (Alluxio/JuiceFS), a Kubernetes dataset-cache project relevant to PR-5.
• Recommendation:
  • Make XCE's own claim and provider contract CSI-shaped: the same verbs (create, publish, stage, node publish, capacity, snapshot, expand), capabilities, access modes, topology and per-call secrets.
  • v1 ships simple native JSON providers (local scratch, NFS or parallel-filesystem directories).
  • Add a CSI adapter later as one provider type in the proxy, plus a node helper for the worker.
  • That keeps CSI a cheap "must later" rather than a v1 cost. If containers (R6) or cloud block storage become early targets, CSI moves up.

Resource model questions (from resource-model.md)

ID	Question	Status	User input / notes
R1	Phase 1 scope: shared pools only (option 2) with runtime values split out, or option 3 in one spec?	discussing	Reframed by PR-1: phase 1 may become the generic supply map. Decide after PR-1.
R2	Who reports runtime custom values: worker scripts, a reporter role, or both?	leaning	Leaning both: worker scripts (elim-like, posix_spawn) for host values; the PR-2 proxy for external and cluster values.
R3	Are claims first-class objects owned by a project from v1? Do they count against quotas?	deferred	User 2026-10-11: RBAC is handled in the API server; the backend only provides configuration and state persistence. Discuss later (with N4).
R4	Own JSON driver contract, or CSI and device-plugin wire compatibility?	leaning	User 2026-10-11: CSI compatibility is better. Still to decide whether it is nice-to-have or a must in future (C3). Either way, model the operations on CSI so an adapter in the PR-2 proxy is mechanical.
R5	Topology depth: host plus switch block, or per-GPU/NIC graphs?	leaning	User 2026-10-11: build GPU/topology into the scheduler first, with the capability to keep improving it. Means resource-instances delivers an extensible inventory model plus an allocator plugin interface, starting simple.
R6	OCI containers as an execution mode?	leaning	User 2026-10-11: possible through the application profile (it already reserves a container mode). Profile defines image, runtime, mounts and device passthrough.
R7	Who changes MIG layouts: a provider or an admin?	open
R8	Honour rusage duration/decay?	decided	User 2026-10-11: must-have features, handled in a future spec (not the current phase).
R9	Where does data staging live?	discussing	PR-3: shared datasets are provisioned copies in the map; job-private staging is a claim.

Provisioner questions (from .kiro/specs/resource-provisioner/research.md)

ID	Question	Status	Notes
P1	Specify the persistent-provider JSON-lines format now (not in v1)?	open	Leaning yes, so v1 schemas carry over.
P2	Is Go via CGO into libxce acceptable for the provisioner/proxy daemon?	open	Leaning yes (apiserver pattern; cloud SDKs native).
P3	Must peer authentication (per-role opcode allowlist) ship before the provisioner?	open	Related to N4: component authentication is separate from user RBAC.
P4	Fix the wire byte order and publish the protocol as stable?	open
P5	Are cloud accounts dedicated to the cluster, or shared?	open	If shared, tagged-but-unknown instances stay report-only.
P6	Automatic return of never-joined instances in v1?	open	Leaning yes, after a join deadline.
P7	Offer stop (quarantine) as well as terminate?	open
P8	Settle window after failover: fixed time, or until expected workers rejoin?	open
P9	License server vs XCE in-use counts: which wins?	open

New questions raised in discussion

ID	Question	Status	Notes
N1	Does the supply map (PR-1) also cover hosts provisioned by elastic-hosts, or only resources on hosts and pools?	discussing	User 2026-10-11: hosts join the controller directly today; how could they be unified? Proposal PR-4 below.
N2	One proxy (PR-2) for every provider kind, or one per kind (cloud, license, storage)? How does it map onto the provisioner's per-provider leases?	leaning	User 2026-10-11: there may be multiple proxies. Matches the provisioner brief: several instances, each provider owned by exactly one active proxy at a time through a controller lease, with standbys taking over. Still open: N2a, N2b.
N2a	How does a proxy declare which providers it serves: config pushed by the controller, or self-declared at join (and checked against config)?	open
N2b	Can one proxy serve several kinds (cloud and storage), or one kind each?	open	Leaning: any mix; the lease is per provider, not per kind.
N3	Dataset model (PR-3): identity and versions, read-only vs updatable, capacity accounting at each location, eviction policy, who triggers staging (scheduler demand, user, admin)?	discussing	User 2026-10-11: start with a simplified model. Proposal PR-5 below.
N4	User RBAC in the apiserver vs component authorization in the controller (which daemon roles may call which opcodes): confirm both layers exist and where each check lives.	leaning	User 2026-10-11: XCE delegates access control to the API server, and the controller trusts it. Implication: the trust is only sound if the controller knows a request really comes from the apiserver (or another legitimate component). Today validate_host() accepts every peer (src/controller/join.c:13-16), so anything that reaches the controller port bypasses RBAC. Component authentication with a per-role opcode allowlist (peer-authentication-hardening, P3) is what makes the delegation safe, especially once proxies can spend money.
N6	PR-6: with I/O claims, should -o/-e/-i and out_dir be accepted again even when no output claim is configured (e.g. a default per-job output directory claim), or stay rejected?	leaning	PR-7: accepted again, resolved against the job directory the profile or queue provides. Rejected only where no shared volume is configured.
N7	PR-6: where do uploaded inputs live between submit and job start (provider volume, apiserver staging area), and is there a size limit?	open	Large inputs should be datasets (PR-5).
N8	PR-6: retention and quota defaults for output claims; who is charged (pool counter per project)?	discussing	With the shared volume: per-user or per-project quota on that volume (filesystem quota, or usage charged to the pool counter after the fact). Retention: see N14.
N9	PR-6: multi-host jobs: a read-write-many output (shared filesystem), or per-host outputs collected after the job?	leaning	Read-write-many shared volume (PR-6 refinement).
N10	PR-6: job arrays: one output claim with a subdirectory per element, or one claim per element?	leaning	Subdirectory per element: <jobid>/<index>/... (PR-6 refinement).
N11	PR-6: requeue and restart: reuse the same output claim or create a new one? How does it relate to checkpoint claims?	leaning (details deferred)	User 2026-10-11: requeue and restart can use the previous one; details later.
N12	PR-6 shared volume: permissions and creation. <user> 0700 and user-owned under a root-owned top? Is the job directory created by the worker (as root, before dropping privileges) or by a provider? Path-safe encoding of array IDs with brackets?	open
N13	PR-6 shared volume: a requeued job keeps its ID. Reuse <jobid>/ or add an attempt subdirectory (<jobid>/attempt-2)? (Refines N11.)	leaning (details deferred)	Reuse the previous directory (user 2026-10-11); whether earlier attempts' files are kept, overwritten or rotated is to be discussed later.
N14	PR-6 shared volume: retention per job directory (TTL, after fetch), the cleaner (a provider task), and orphan directories whose job no longer exists.	open
N15	PR-6 shared volume: with several shared volumes, which one a job uses: by application profile, queue or project?	open
N16	PR-8: what counts as "complete" for sealing an output: job DONE with exit 0, or an explicit seal by the job?	open
N17	PR-8: partial outputs from a failed or killed producer: discard, keep unsealed, or seal as a failed version?	open
N18	PR-8: version numbering (@auto increment, timestamps, user tags) and @latest resolution: at submit or at dispatch?	open
N19	PR-8: a consumer whose producer fails or never seals: pend forever, or fail like an unsatisfiable job-dependency condition?	open	Follow job-dependency's rule for never-satisfied conditions.
N20	PR-9: sharing granularity: dataset visibility (private/project/public, read-only) is enough, or are explicit per-user grants and writable shared space needed? How is visibility enforced on the filesystem (group ACLs, read-only mounts)?	open	Low priority (user 2026-10-11).
N21	PR-10: cross-host topology syntax: an LSF-style cu[type=...:maxcus=...:pref=...] clause, or an extension of same[]/span[]?	open	Leaning cu[], because XCE's resreq follows LSF.
N22	PR-10: relax policy: wait up to a limit for a better topology, then relax (Slurm --switches=N@maxwait)? Per job, or per queue or profile?	open
N23	PR-10: rank ordering: does the scheduler return hosts in topology order, and is that order part of the persisted allocation and shown to the launcher (hostfile, env)?	open
N24	PR-10: demote gang-groups to optional/backlog, kept only for roles with separate lifecycles (e.g. RL trainer plus inference services)?	decided	User 2026-10-12: confirmed. Training is one parallel job (PR-10); gang-groups moves to the backlog, re-scoped as a co-start group. Roadmap, resource-model.md and the brief updated. Use-case review (2026-10-11), below "Gang use cases": a multi-job gang is needed mainly for RL post-training, disaggregated serving and streaming pipelines. All three mix batch and service components, so re-scope it as a "co-start group" across service-replica-set and jobs when RL or disaggregated serving comes onto the roadmap.
N25	PR-10: reservation or plan-ahead for large topology-constrained jobs, to prevent starvation and fragmentation: a new spec, or part of an existing one?	open
N26	PR-11: placement groups only, or also direct job-to-job affinity (affinity[job(...)])?	open	Leaning groups first (cheaper, simpler to explain).
N27	PR-11: which modes and strictness in v1: pack and spread, hard (required) and soft (preferred)?	open
N28	PR-11: delay scheduling for data locality: wait per job, queue or profile; default?	open	Same mechanism as N22.
N29	PR-11: where topology comes from: admin config only in v1, discovery (ibnetdiscover, LLDP) later via a reporter?	open
N30	PR-11: data distance: simple locality tiers, or a size-weighted cost deciding between staging a copy and placing far?	open	Leaning tiers in v1.
N31	PR-12/PR-10: role identity in compound jobs: named parts in the resreq (name=prefill), plus per-task part name, part index, rank and per-part hostfile or environment (XCE_ROLE, XCE_ROLE_INDEX, XCE_HOSTS_<role>)? None exists today.	open	Needed for disaggregated serving units and for training (coordinator plus workers).
N32	PR-12: long-running service jobs: restart policy and per-task replacement inside a running job, instead of failing the whole unit?	open	Extends service-supervision.
N33	PR-12: per-part resize of a compound job (job-resize), or scaling by whole units only?	open	Leaning units first.
N34	PR-12: a router inside each unit plus an outer load balancer, or one global router across units?	open	A global router pulls toward independent pools (gang-groups).
N35	PR-13: relationship edges between roles only, or per-task edges too?	open	Leaning roles only (compact for jobs with thousands of tasks).
N36	PR-13: who sequences tasks and steps: the scheduler, or a job-step runtime inside the allocation (Slurm steps, Flux)?	open	Leaning the job-step runtime.
N37	PR-13: contention in v1: isolation only, or also interference-aware placement (needs telemetry and a model)?	open	Leaning isolation only in v1.
N38	PR-13: network, memory and I/O bandwidth as reserved counters (L2), or best-effort QoS?	open
N39	PR-13: QoS classes for co-locating latency-critical serving with batch work on the same node?	open
N40	PR-13: defaults for sidecar and init tasks and fate sharing: MPI-style fail-together, or tolerate?	open
N41	PR-14: a canonical structured jobspec plus resreq and CLI shorthands, or extend the resreq string only?	open	Leaning canonical plus shorthands.
N42	PR-14: relationship kinds in v1?	open	Suggested v1: colocate/spread, exchange, with/after, sidecar-of/init-of, fate. Later: isolate/share QoS, bandwidth, consumes/produces.
N43	PR-14: where relationships live in config layering: profile templates, which are overridable, per-job additions?	open
N44	PR-14: does the same vocabulary cover relationships between jobs, replacing a separate placement-group mechanism (PR-11)?	open	Leaning yes.
N45	PR-15: implicit names (part0…) for unnamed compound parts; is the name syntax inside the brace (N*{name=prefill: ...}) or in the jobspec only?	open
N46	PR-15: allocation search for cross-part relationships: domain-first enumeration, bounded backtracking, or both? What bound keeps the session cost acceptable?	open
N47	PR-15: relationships vs alternatives: one relationship set per job over role names (validated per alternative), or separate relationship sets per alternative?	open	Leaning one set per job.
N48	PR-15: sidecar roles: attached to the parent's tasks (shared allocation), or a separate compound part with a derived count?	open	Leaning attached by default.
N49	PR-15: task ranges in compound jobs: per-role ranges searched by the planner, or ranges only for primary (single-part) jobs?	open
N50	PR-15: actual-usage reconciliation for multi-role, multi-host jobs (today single-host only): extend it, or record as an accepted gap?	open
C3	CSI compatibility: nice-to-have or a must? If a must, which side: the controller plugin through the proxy, the node plugin on the worker, or both?	discussing	Background and recommendation in "CSI notes" below (2026-10-11).

History

• 2026-10-12: user noted the requirements are complex → "Delivery plan" section: five tracks (roles, relationships and placement, supply, data and I/O, devices), first slices with the decisions each needs, and a deferred list with revisit triggers, so that nothing is forgotten.
• 2026-10-12: user asked about the interaction with compound resreqs → PR-15 (12 interactions; named parts, relationship-aware cross-part search, role on every allocation, cache-key and alternative rules); N45–N50 added.
• 2026-10-12: user asked for a uniform approach to describe task relationships → PR-14 (canonical jobspec with typed relationship records; shorthands compile into it); N41–N44 added.
• 2026-10-12: user raised task-level cooperation (contention, data exchange, piped dependencies, more) → PR-13 (task relationship model with typed edges between roles; allocation plus job-step runtime); N35–N40 added.
• 2026-10-12: user asked whether disaggregated serving can be one job → PR-12 (one compound job per unit, replicated by service-replica-set); N31–N34 added; gang-groups scope narrowed.
• 2026-10-12: N24 decided (user): gang-groups demoted to the backlog and re-scoped as a co-start group for RL post-training, disaggregated serving and streaming pipelines.
• 2026-10-11: user asked about data cache, data and communication topology between jobs → PR-11 (one topology model; data locality with delay scheduling; placement groups with pack or spread); N26–N30 added.
• 2026-10-11: user asked for the use cases of gangs → "Gang use cases" section; N24 leaning (demote gang-groups, re-scope later as a co-start group for RL, disaggregated serving and streaming).
• 2026-10-11: user asked whether LLM training can be a single job with complex topology requirements → PR-10 (single parallel job as the default; topology at three levels; gang-groups optional); N21–N25 added.
• 2026-10-11: user input: keep submit options simple → PR-7 (N6 leaning); data producer/consumer dependencies → PR-8 (N16–N19); isolation between users, sharing low priority → PR-9 (N20); requeue/restart reuse the previous directory, details later → N11/N13 leaning.
• 2026-10-11: user proposed a shared volume with per-user/per-job subdirectories (shared_volume/<user>/<jobid>/{in,out,err}) instead of a volume per job → PR-6 refinement (leaning). N9 and N10 leaning; N8 reframed; N12–N15 added.
• 2026-10-11: user asked whether volume provisioning can carry job input and output → PR-6; questions N6–N11 added.
• 2026-10-11: user input on N1 (hosts join directly; asked how to unify → PR-4), N2 (multiple proxies → leaning), N3 (a simplified model first → PR-5), N4 (access control delegated to the API server, the controller trusts it → leaning, with the component-authentication implication), C3 (asked for more on CSI → CSI notes). N2a and N2b added.
• 2026-10-11: created from the resource-model discussion. User input recorded for PR-1, PR-2 and PR-3 (proposals) and for R3, R4, R5, R6 and R8. Questions N1–N4 and C3 added.