From 5b05d51b7a4ef6453e0cf4446ec16bb53c2a45b2 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Mon, 12 Oct 2026 04:00:47 +0000 Subject: [PATCH] docs(steering): resource decision and question log (PR-1..3 proposals, R/P/N items) Persistent log of the resource-design discussion: the user's proposals (generic supply map incl. provisioned supply; a resource provider proxy between the controller and upstream providers; datasets in the map with staging as data provisioning) with analysis, the user's 2026-10-11 input (duration/decay must-have but future; RBAC in the apiserver, discuss later; OCI containers via application profiles; CSI compatibility preferred; GPU/topology scheduler-first and extensible), the open R1-R9/P1-P9 items and new questions N1-N4, C3. resource-model.md, the provisioner research, roadmap and CLAUDE.md point at it. Co-Authored-By: Claude Opus 5.5 (1M context) --- .kiro/specs/resource-provisioner/research.md | 2 + .kiro/steering/resource-model.md | 21 +---- .kiro/steering/resource-questions.md | 92 ++++++++++++++++++++ .kiro/steering/roadmap.md | 2 +- CLAUDE.md | 3 +- 5 files changed, 99 insertions(+), 21 deletions(-) create mode 100644 .kiro/steering/resource-questions.md diff --git a/.kiro/specs/resource-provisioner/research.md b/.kiro/specs/resource-provisioner/research.md index 8aff2d6a..19945b19 100644 --- a/.kiro/specs/resource-provisioner/research.md +++ b/.kiro/specs/resource-provisioner/research.md @@ -178,6 +178,8 @@ A partitioned old owner can finish a cloud call already in flight. Mitigations: ## Open questions for the user +(Tracked since 2026-10-11 as P1–P9 in `.kiro/steering/resource-questions.md`, which is authoritative for their status. PR-2 there proposes generalizing this daemon into a resource provider proxy.) + 1. Should the JSON-lines persistent-provider framing be specified now, though not implemented in v1 (decision 1)? 2. Is Go via CGO into libxce acceptable for the provisioner daemon, or is a CGO-free daemon required (re-implementing failover/rejoin)? 3. Is `peer-authentication-hardening` (per-role opcode allowlist) a hard dependency before the provisioner ships? diff --git a/.kiro/steering/resource-model.md b/.kiro/steering/resource-model.md index 119e6379..a3cf156a 100644 --- a/.kiro/steering/resource-model.md +++ b/.kiro/steering/resource-model.md @@ -204,22 +204,5 @@ device-plugin **wire protocols** is open decision 4. **Config-only now:** document the static custom resources that already work. -## Open decisions (user, pending) - -1. Phase 1 scope: `resource-distribution-mapping` option 2 now with option 3 - as `custom-resource-reporting`, or option 3 in one spec? -2. Who reports runtime custom values: worker-run scripts (`posix_spawn`), a - separate reporter role, or both? -3. Are claims first-class REST/RBAC objects owned by a project from v1, and - do they count against project quotas? -4. Should XCE keep its own JSON provider contract, or be wire-compatible with - CSI and device plugins to reuse their drivers (gRPC sidecar, container - runtime coupling)? -5. How deep is the topology model: host plus switch block, or per-GPU/NIC - graphs? -6. Will OCI containers be an execution mode? It decides how dataset and - weights volumes are published into a job. -7. Who changes MIG layouts: a provider automatically, or an admin? -8. Should `rusage` `duration`/`decay` be honoured now (licenses need it)? -9. Is data staging part of `job-sandbox-execution`'s workspace work, or - `resource-claims`? +## Open decisions +Kept in `resource-questions.md`, the persistent decision and question log (R1–R9 from this document, the provisioner's P1–P9, new questions, and proposals PR-1 to PR-3 that may reshape the phases above: one supply map that includes provisioned supply, a resource provider proxy, datasets in the map with staging as provisioning). Check it before writing requirements for any phase. diff --git a/.kiro/steering/resource-questions.md b/.kiro/steering/resource-questions.md new file mode 100644 index 00000000..4e62fb90 --- /dev/null +++ b/.kiro/steering/resource-questions.md @@ -0,0 +1,92 @@ +# Resource Design: Decisions, Proposals and Open Questions + +The persistent discussion log for the unified resource model +(`resource-model.md`) and the provisioning work (`resource-provisioner`, +`provisioning-demand`, `elastic-*`). Specs must not settle an item here +silently. + +**How to use:** +- **Status values:** `open` (not discussed), `discussing` (user input recorded, not final), `leaning` (direction agreed, details open), `decided`, `deferred` (later, with a reason). +- **Changes:** update an item in place and add a dated line under "History". When an item is decided, also update the documents it names. +- **IDs:** R = resource model, P = provisioner, N = new questions raised in discussion, PR = proposals. + +## Proposals under discussion (user, 2026-10-11) + +### PR-1: one resource map for all supply, including provisioned supply (status: leaning) +- **User:** should `resource-distribution-mapping` be made generic, so it also handles resource provisioning? +- **Analysis (2026-10-11):** yes, for the **data model**. + - The resource map becomes the single record of supply: for each resource, which locations (host, host group, pool, storage) hold how much. + - Each entry has a **source** (static config, reporter, provisioner) and a **state** (requested, available, draining, releasing). + - Provisioning only changes map entries over time, e.g. a license pool's base plus borrowed amount, or a new dataset copy. + - The provisioning **mechanism** stays in the provisioner: policy, provider calls, leases, epochs, orphan scans. The map is the state; the provisioner is the actuator. This is the Kubernetes split too (drivers publish ResourceSlices; autoscalers and provisioners act). + - The scheduler keeps reading only the map's cached snapshot. +- **Affects:** `resource-distribution-mapping`, `elastic-shared-resources`, `resource-provisioner`, `provisioning-demand`. +- **Open:** see N1, N2. + +### PR-2: a resource provider proxy between the controller and upstream providers (status: leaning) +- **User:** introduce a resource provider that proxies requests and states between the XCE controller and upstream resource providers. +- **Analysis (2026-10-11):** yes. It generalizes the `resource-provisioner` daemon. The proxy: + - connects to the controller as one role over libxce/Cap'n Proto; + - speaks each upstream provider's protocol: the JSON script contract, a persistent JSON-lines mode, later CSI or vendor APIs; + - holds provider sessions, credentials and rate limits, and translates provider states into map entries and claim states; + - publishes inventory and values (it is the external reporter of R2); + - carries out provisioning and claim operations with `op_id`, an epoch and owner tags. +- **Why it helps:** the controller never blocks on or depends on an outside system, and credentials never reach the controller. +- **Where node-local work runs:** GPU binding, mounts and scratch stay on the worker, possibly behind the same contract. +- **Affects:** `resource-provisioner` (rename or generalize), `custom-resource-reporting` (external reporters), `resource-claims` (storage providers), R4/C3 (the CSI adapter would live here). +- **Open:** see N2, N4. + +### PR-3: datasets in the resource map; staging as data provisioning (status: discussing) +- **User:** could data instances be handled by the resource distribution map, and data staging by provisioning the data to a new location? +- **Analysis (2026-10-11):** this fits PR-1 well. + - **Datasets:** a dataset is a resource whose map entries are the locations holding a copy. Locality is an L1 attribute the scheduler can use. + - **Staging:** staging is provisioning a copy at a new location, with a lead time like any provider. + - **Eviction:** evicting a copy is a release. + - **Shortage:** "dataset X needed at host group Y" can become provisioning demand. + - Locality, caching and stage-in become one mechanism. +- **Caveats:** + - Job-private data (stage-out results, scratch, a job's own outputs) is still a per-job claim (`resource-claims`), not shared supply. + - Datasets need an identity and version, capacity accounting on the storage holding a copy, an eviction policy, and consistency rules (read-only vs updated). +- **Affects:** `resource-distribution-mapping`, `custom-resource-reporting` (string-set locality may become map entries instead), `resource-claims` (scope shrinks to job-private data), R9. +- **Open:** see N3. + +## Resource model questions (from `resource-model.md`) + +| ID | Question | Status | User input / notes | +|---|---|---|---| +| R1 | Phase 1 scope: shared pools only (option 2) with runtime values split out, or option 3 in one spec? | discussing | Reframed by PR-1: phase 1 may become the generic supply map. Decide after PR-1. | +| R2 | Who reports runtime custom values: worker scripts, a reporter role, or both? | leaning | Leaning both: worker scripts (elim-like, `posix_spawn`) for host values; the PR-2 proxy for external and cluster values. | +| R3 | Are claims first-class objects owned by a project from v1? Do they count against quotas? | deferred | User 2026-10-11: RBAC is handled in the API server; the backend only provides configuration and state persistence. Discuss later (with N4). | +| R4 | Own JSON driver contract, or CSI and device-plugin wire compatibility? | leaning | User 2026-10-11: CSI compatibility is better. Still to decide whether it is nice-to-have or a must in future (C3). Either way, model the operations on CSI so an adapter in the PR-2 proxy is mechanical. | +| R5 | Topology depth: host plus switch block, or per-GPU/NIC graphs? | leaning | User 2026-10-11: build GPU/topology into the scheduler first, with the capability to keep improving it. Means `resource-instances` delivers an extensible inventory model plus an allocator plugin interface, starting simple. | +| R6 | OCI containers as an execution mode? | leaning | User 2026-10-11: possible through the application profile (it already reserves a `container` mode). Profile defines image, runtime, mounts and device passthrough. | +| R7 | Who changes MIG layouts: a provider or an admin? | open | | +| R8 | Honour `rusage` `duration`/`decay`? | decided | User 2026-10-11: must-have features, handled in a future spec (not the current phase). | +| R9 | Where does data staging live? | discussing | PR-3: shared datasets are provisioned copies in the map; job-private staging is a claim. | + +## Provisioner questions (from `.kiro/specs/resource-provisioner/research.md`) + +| ID | Question | Status | Notes | +|---|---|---|---| +| P1 | Specify the persistent-provider JSON-lines format now (not in v1)? | open | Leaning yes, so v1 schemas carry over. | +| P2 | Is Go via CGO into libxce acceptable for the provisioner/proxy daemon? | open | Leaning yes (apiserver pattern; cloud SDKs native). | +| P3 | Must peer authentication (per-role opcode allowlist) ship before the provisioner? | open | Related to N4: component authentication is separate from user RBAC. | +| P4 | Fix the wire byte order and publish the protocol as stable? | open | | +| P5 | Are cloud accounts dedicated to the cluster, or shared? | open | If shared, tagged-but-unknown instances stay report-only. | +| P6 | Automatic return of never-joined instances in v1? | open | Leaning yes, after a join deadline. | +| P7 | Offer `stop` (quarantine) as well as `terminate`? | open | | +| P8 | Settle window after failover: fixed time, or until expected workers rejoin? | open | | +| P9 | License server vs XCE in-use counts: which wins? | open | | + +## New questions raised in discussion + +| ID | Question | Status | Notes | +|---|---|---|---| +| N1 | Does the supply map (PR-1) also cover hosts provisioned by `elastic-hosts`, or only resources on hosts and pools? | open | Hosts have their own lifecycle (join, drain, decommission); they may stay separate, with the map referencing them. | +| N2 | One proxy (PR-2) for every provider kind, or one per kind (cloud, license, storage)? How does it map onto the provisioner's per-provider leases? | open | The brief already allows several instances, each owning providers by lease. | +| N3 | Dataset model (PR-3): identity and versions, read-only vs updatable, capacity accounting at each location, eviction policy, who triggers staging (scheduler demand, user, admin)? | open | | +| N4 | User RBAC in the apiserver vs component authorization in the controller (which daemon roles may call which opcodes): confirm both layers exist and where each check lives. | open | Raised by the user's RBAC position (R3) and P3. | +| C3 | CSI compatibility: nice-to-have or a must? If a must, which side: the controller plugin through the proxy, the node plugin on the worker, or both? | open | The node plugin implies gRPC on the worker and mount semantics for the sandbox or container mode. | + +## History +- 2026-10-11: created from the resource-model discussion. User input recorded for PR-1, PR-2 and PR-3 (proposals) and for R3, R4, R5, R6 and R8. Questions N1–N4 and C3 added. diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index 1bcc0922..5352e4ef 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -90,7 +90,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han - [ ] gang-groups — NEW 2026-10-11 (Unified resource model phase 5, LLM training). All-or-nothing scheduling of a group of jobs (launcher + workers, JobSet-like), topology packing across the gang, start barrier until every member is placed and its claims are READY, group failure policy. Brief: `.kiro/specs/gang-groups/brief.md`. Dependencies: resource-instances, resource-claims, parjob-placement. ## Existing Spec Updates (2026-10-11 unified resource model discovery) -Direction and pending decisions: `.kiro/steering/resource-model.md`. +Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generic supply map incl. provisioned supply, PR-2 resource provider proxy, PR-3 datasets in the map with staging as provisioning) and open questions: `.kiro/steering/resource-questions.md`. Duration/decay: must-have, future spec (R8). - [ ] resource-distribution-mapping — phase 1: shared pools (option 2) plus the resource type descriptor fields (`scope`, `unit`, `reporter`, `layer`, `instances`, `provider`); option 3 moves to `custom-resource-reporting` unless the user keeps it here (decision 1). - [ ] service-endpoint — ports as identified instances of `resource-instances` instead of a separate port model. - [ ] job-sandbox-execution — the per-job workspace becomes the first template claim of `resource-claims`; publish (mounts) through the claim's worker step. Where data staging lives is decision 9. diff --git a/CLAUDE.md b/CLAUDE.md index 41057cfe..bc538224 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -28,6 +28,7 @@ For details on any topic below, read the corresponding steering file in `.kiro/s | WAL durability, recovery, save/restore field-completeness | `persistence.md` | | Real vs. virtual resources, reservation-vs-usage accounting (mem) | `resource-accounting.md` | | Resource model: attributes / counters / instances+claims, providers, K8s-LSF mapping, LLM direction | `resource-model.md` | +| Resource design decisions, proposals and open questions (discussion log) | `resource-questions.md` | | Job/app-profile/queue/project ownership and merge rules | `job-config-layering.md` | ## Key Rules (from steering — summarized for quick reference) @@ -108,5 +109,5 @@ Kiro-style Spec-Driven Development on an agentic SDLC ## Steering Configuration - Load entire `.kiro/steering/` as project memory — all `.md` files are authoritative - Core: `product.md`, `tech.md`, `structure.md` -- Custom: `testing.md`, `error-handling.md`, `security.md`, `ha-async-io.md`, `api-standards.md`, `build-deployment.md`, `config-design.md`, `persistence.md`, `resource-accounting.md`, `resource-model.md`, `job-config-layering.md`, `roadmap.md` +- Custom: `testing.md`, `error-handling.md`, `security.md`, `ha-async-io.md`, `api-standards.md`, `build-deployment.md`, `config-design.md`, `persistence.md`, `resource-accounting.md`, `resource-model.md`, `resource-questions.md`, `job-config-layering.md`, `roadmap.md` - Add new custom files via `/kiro-steering-custom`; sync existing via `/kiro-steering`