docs(steering): resource decision and question log (PR-1..3 proposals, R/P/N items)
Persistent log of the resource-design discussion: the user's proposals (generic supply map incl. provisioned supply; a resource provider proxy between the controller and upstream providers; datasets in the map with staging as data provisioning) with analysis, the user's 2026-10-11 input (duration/decay must-have but future; RBAC in the apiserver, discuss later; OCI containers via application profiles; CSI compatibility preferred; GPU/topology scheduler-first and extensible), the open R1-R9/P1-P9 items and new questions N1-N4, C3. resource-model.md, the provisioner research, roadmap and CLAUDE.md point at it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
bab73826f0
commit
5b05d51b7a
GPG key ID:
80533536B0EB651C
5 changed files with 99 additions and 21 deletions
|
|
@ -178,6 +178,8 @@ A partitioned old owner can finish a cloud call already in flight. Mitigations:
|
|||
|
||||
|
||||
## Open questions for the user
|
||||
(Tracked since 2026-10-11 as P1–P9 in `.kiro/steering/resource-questions.md`, which is authoritative for their status. PR-2 there proposes generalizing this daemon into a resource provider proxy.)
|
||||
|
||||
1. Should the JSON-lines persistent-provider framing be specified now, though not implemented in v1 (decision 1)?
|
||||
2. Is Go via CGO into libxce acceptable for the provisioner daemon, or is a CGO-free daemon required (re-implementing failover/rejoin)?
|
||||
3. Is `peer-authentication-hardening` (per-role opcode allowlist) a hard dependency before the provisioner ships?
|
||||
|
|
|
|||
|
|
@ -204,22 +204,5 @@ device-plugin **wire protocols** is open decision 4.
|
|||
**Config-only now:** document the static custom resources that already
|
||||
work.
|
||||
|
||||
## Open decisions (user, pending)
|
||||
|
||||
1. Phase 1 scope: `resource-distribution-mapping` option 2 now with option 3
|
||||
as `custom-resource-reporting`, or option 3 in one spec?
|
||||
2. Who reports runtime custom values: worker-run scripts (`posix_spawn`), a
|
||||
separate reporter role, or both?
|
||||
3. Are claims first-class REST/RBAC objects owned by a project from v1, and
|
||||
do they count against project quotas?
|
||||
4. Should XCE keep its own JSON provider contract, or be wire-compatible with
|
||||
CSI and device plugins to reuse their drivers (gRPC sidecar, container
|
||||
runtime coupling)?
|
||||
5. How deep is the topology model: host plus switch block, or per-GPU/NIC
|
||||
graphs?
|
||||
6. Will OCI containers be an execution mode? It decides how dataset and
|
||||
weights volumes are published into a job.
|
||||
7. Who changes MIG layouts: a provider automatically, or an admin?
|
||||
8. Should `rusage` `duration`/`decay` be honoured now (licenses need it)?
|
||||
9. Is data staging part of `job-sandbox-execution`'s workspace work, or
|
||||
`resource-claims`?
|
||||
## Open decisions
|
||||
Kept in `resource-questions.md`, the persistent decision and question log (R1–R9 from this document, the provisioner's P1–P9, new questions, and proposals PR-1 to PR-3 that may reshape the phases above: one supply map that includes provisioned supply, a resource provider proxy, datasets in the map with staging as provisioning). Check it before writing requirements for any phase.
|
||||
|
|
|
|||
92
.kiro/steering/resource-questions.md
Normal file
92
.kiro/steering/resource-questions.md
Normal file
|
|
@ -0,0 +1,92 @@
|
|||
# Resource Design: Decisions, Proposals and Open Questions
|
||||
|
||||
The persistent discussion log for the unified resource model
|
||||
(`resource-model.md`) and the provisioning work (`resource-provisioner`,
|
||||
`provisioning-demand`, `elastic-*`). Specs must not settle an item here
|
||||
silently.
|
||||
|
||||
**How to use:**
|
||||
- **Status values:** `open` (not discussed), `discussing` (user input recorded, not final), `leaning` (direction agreed, details open), `decided`, `deferred` (later, with a reason).
|
||||
- **Changes:** update an item in place and add a dated line under "History". When an item is decided, also update the documents it names.
|
||||
- **IDs:** R = resource model, P = provisioner, N = new questions raised in discussion, PR = proposals.
|
||||
|
||||
## Proposals under discussion (user, 2026-10-11)
|
||||
|
||||
### PR-1: one resource map for all supply, including provisioned supply (status: leaning)
|
||||
- **User:** should `resource-distribution-mapping` be made generic, so it also handles resource provisioning?
|
||||
- **Analysis (2026-10-11):** yes, for the **data model**.
|
||||
- The resource map becomes the single record of supply: for each resource, which locations (host, host group, pool, storage) hold how much.
|
||||
- Each entry has a **source** (static config, reporter, provisioner) and a **state** (requested, available, draining, releasing).
|
||||
- Provisioning only changes map entries over time, e.g. a license pool's base plus borrowed amount, or a new dataset copy.
|
||||
- The provisioning **mechanism** stays in the provisioner: policy, provider calls, leases, epochs, orphan scans. The map is the state; the provisioner is the actuator. This is the Kubernetes split too (drivers publish ResourceSlices; autoscalers and provisioners act).
|
||||
- The scheduler keeps reading only the map's cached snapshot.
|
||||
- **Affects:** `resource-distribution-mapping`, `elastic-shared-resources`, `resource-provisioner`, `provisioning-demand`.
|
||||
- **Open:** see N1, N2.
|
||||
|
||||
### PR-2: a resource provider proxy between the controller and upstream providers (status: leaning)
|
||||
- **User:** introduce a resource provider that proxies requests and states between the XCE controller and upstream resource providers.
|
||||
- **Analysis (2026-10-11):** yes. It generalizes the `resource-provisioner` daemon. The proxy:
|
||||
- connects to the controller as one role over libxce/Cap'n Proto;
|
||||
- speaks each upstream provider's protocol: the JSON script contract, a persistent JSON-lines mode, later CSI or vendor APIs;
|
||||
- holds provider sessions, credentials and rate limits, and translates provider states into map entries and claim states;
|
||||
- publishes inventory and values (it is the external reporter of R2);
|
||||
- carries out provisioning and claim operations with `op_id`, an epoch and owner tags.
|
||||
- **Why it helps:** the controller never blocks on or depends on an outside system, and credentials never reach the controller.
|
||||
- **Where node-local work runs:** GPU binding, mounts and scratch stay on the worker, possibly behind the same contract.
|
||||
- **Affects:** `resource-provisioner` (rename or generalize), `custom-resource-reporting` (external reporters), `resource-claims` (storage providers), R4/C3 (the CSI adapter would live here).
|
||||
- **Open:** see N2, N4.
|
||||
|
||||
### PR-3: datasets in the resource map; staging as data provisioning (status: discussing)
|
||||
- **User:** could data instances be handled by the resource distribution map, and data staging by provisioning the data to a new location?
|
||||
- **Analysis (2026-10-11):** this fits PR-1 well.
|
||||
- **Datasets:** a dataset is a resource whose map entries are the locations holding a copy. Locality is an L1 attribute the scheduler can use.
|
||||
- **Staging:** staging is provisioning a copy at a new location, with a lead time like any provider.
|
||||
- **Eviction:** evicting a copy is a release.
|
||||
- **Shortage:** "dataset X needed at host group Y" can become provisioning demand.
|
||||
- Locality, caching and stage-in become one mechanism.
|
||||
- **Caveats:**
|
||||
- Job-private data (stage-out results, scratch, a job's own outputs) is still a per-job claim (`resource-claims`), not shared supply.
|
||||
- Datasets need an identity and version, capacity accounting on the storage holding a copy, an eviction policy, and consistency rules (read-only vs updated).
|
||||
- **Affects:** `resource-distribution-mapping`, `custom-resource-reporting` (string-set locality may become map entries instead), `resource-claims` (scope shrinks to job-private data), R9.
|
||||
- **Open:** see N3.
|
||||
|
||||
## Resource model questions (from `resource-model.md`)
|
||||
|
||||
| ID | Question | Status | User input / notes |
|
||||
|---|---|---|---|
|
||||
| R1 | Phase 1 scope: shared pools only (option 2) with runtime values split out, or option 3 in one spec? | discussing | Reframed by PR-1: phase 1 may become the generic supply map. Decide after PR-1. |
|
||||
| R2 | Who reports runtime custom values: worker scripts, a reporter role, or both? | leaning | Leaning both: worker scripts (elim-like, `posix_spawn`) for host values; the PR-2 proxy for external and cluster values. |
|
||||
| R3 | Are claims first-class objects owned by a project from v1? Do they count against quotas? | deferred | User 2026-10-11: RBAC is handled in the API server; the backend only provides configuration and state persistence. Discuss later (with N4). |
|
||||
| R4 | Own JSON driver contract, or CSI and device-plugin wire compatibility? | leaning | User 2026-10-11: CSI compatibility is better. Still to decide whether it is nice-to-have or a must in future (C3). Either way, model the operations on CSI so an adapter in the PR-2 proxy is mechanical. |
|
||||
| R5 | Topology depth: host plus switch block, or per-GPU/NIC graphs? | leaning | User 2026-10-11: build GPU/topology into the scheduler first, with the capability to keep improving it. Means `resource-instances` delivers an extensible inventory model plus an allocator plugin interface, starting simple. |
|
||||
| R6 | OCI containers as an execution mode? | leaning | User 2026-10-11: possible through the application profile (it already reserves a `container` mode). Profile defines image, runtime, mounts and device passthrough. |
|
||||
| R7 | Who changes MIG layouts: a provider or an admin? | open | |
|
||||
| R8 | Honour `rusage` `duration`/`decay`? | decided | User 2026-10-11: must-have features, handled in a future spec (not the current phase). |
|
||||
| R9 | Where does data staging live? | discussing | PR-3: shared datasets are provisioned copies in the map; job-private staging is a claim. |
|
||||
|
||||
## Provisioner questions (from `.kiro/specs/resource-provisioner/research.md`)
|
||||
|
||||
| ID | Question | Status | Notes |
|
||||
|---|---|---|---|
|
||||
| P1 | Specify the persistent-provider JSON-lines format now (not in v1)? | open | Leaning yes, so v1 schemas carry over. |
|
||||
| P2 | Is Go via CGO into libxce acceptable for the provisioner/proxy daemon? | open | Leaning yes (apiserver pattern; cloud SDKs native). |
|
||||
| P3 | Must peer authentication (per-role opcode allowlist) ship before the provisioner? | open | Related to N4: component authentication is separate from user RBAC. |
|
||||
| P4 | Fix the wire byte order and publish the protocol as stable? | open | |
|
||||
| P5 | Are cloud accounts dedicated to the cluster, or shared? | open | If shared, tagged-but-unknown instances stay report-only. |
|
||||
| P6 | Automatic return of never-joined instances in v1? | open | Leaning yes, after a join deadline. |
|
||||
| P7 | Offer `stop` (quarantine) as well as `terminate`? | open | |
|
||||
| P8 | Settle window after failover: fixed time, or until expected workers rejoin? | open | |
|
||||
| P9 | License server vs XCE in-use counts: which wins? | open | |
|
||||
|
||||
## New questions raised in discussion
|
||||
|
||||
| ID | Question | Status | Notes |
|
||||
|---|---|---|---|
|
||||
| N1 | Does the supply map (PR-1) also cover hosts provisioned by `elastic-hosts`, or only resources on hosts and pools? | open | Hosts have their own lifecycle (join, drain, decommission); they may stay separate, with the map referencing them. |
|
||||
| N2 | One proxy (PR-2) for every provider kind, or one per kind (cloud, license, storage)? How does it map onto the provisioner's per-provider leases? | open | The brief already allows several instances, each owning providers by lease. |
|
||||
| N3 | Dataset model (PR-3): identity and versions, read-only vs updatable, capacity accounting at each location, eviction policy, who triggers staging (scheduler demand, user, admin)? | open | |
|
||||
| N4 | User RBAC in the apiserver vs component authorization in the controller (which daemon roles may call which opcodes): confirm both layers exist and where each check lives. | open | Raised by the user's RBAC position (R3) and P3. |
|
||||
| C3 | CSI compatibility: nice-to-have or a must? If a must, which side: the controller plugin through the proxy, the node plugin on the worker, or both? | open | The node plugin implies gRPC on the worker and mount semantics for the sandbox or container mode. |
|
||||
|
||||
## History
|
||||
- 2026-10-11: created from the resource-model discussion. User input recorded for PR-1, PR-2 and PR-3 (proposals) and for R3, R4, R5, R6 and R8. Questions N1–N4 and C3 added.
|
||||
|
|
@ -90,7 +90,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
|
|||
- [ ] gang-groups — NEW 2026-10-11 (Unified resource model phase 5, LLM training). All-or-nothing scheduling of a group of jobs (launcher + workers, JobSet-like), topology packing across the gang, start barrier until every member is placed and its claims are READY, group failure policy. Brief: `.kiro/specs/gang-groups/brief.md`. Dependencies: resource-instances, resource-claims, parjob-placement.
|
||||
|
||||
## Existing Spec Updates (2026-10-11 unified resource model discovery)
|
||||
Direction and pending decisions: `.kiro/steering/resource-model.md`.
|
||||
Direction: `.kiro/steering/resource-model.md`. Decisions, proposals (PR-1 generic supply map incl. provisioned supply, PR-2 resource provider proxy, PR-3 datasets in the map with staging as provisioning) and open questions: `.kiro/steering/resource-questions.md`. Duration/decay: must-have, future spec (R8).
|
||||
- [ ] resource-distribution-mapping — phase 1: shared pools (option 2) plus the resource type descriptor fields (`scope`, `unit`, `reporter`, `layer`, `instances`, `provider`); option 3 moves to `custom-resource-reporting` unless the user keeps it here (decision 1).
|
||||
- [ ] service-endpoint — ports as identified instances of `resource-instances` instead of a separate port model.
|
||||
- [ ] job-sandbox-execution — the per-job workspace becomes the first template claim of `resource-claims`; publish (mounts) through the claim's worker step. Where data staging lives is decision 9.
|
||||
|
|
|
|||
|
|
@ -28,6 +28,7 @@ For details on any topic below, read the corresponding steering file in `.kiro/s
|
|||
| WAL durability, recovery, save/restore field-completeness | `persistence.md` |
|
||||
| Real vs. virtual resources, reservation-vs-usage accounting (mem) | `resource-accounting.md` |
|
||||
| Resource model: attributes / counters / instances+claims, providers, K8s-LSF mapping, LLM direction | `resource-model.md` |
|
||||
| Resource design decisions, proposals and open questions (discussion log) | `resource-questions.md` |
|
||||
| Job/app-profile/queue/project ownership and merge rules | `job-config-layering.md` |
|
||||
|
||||
## Key Rules (from steering — summarized for quick reference)
|
||||
|
|
@ -108,5 +109,5 @@ Kiro-style Spec-Driven Development on an agentic SDLC
|
|||
## Steering Configuration
|
||||
- Load entire `.kiro/steering/` as project memory — all `.md` files are authoritative
|
||||
- Core: `product.md`, `tech.md`, `structure.md`
|
||||
- Custom: `testing.md`, `error-handling.md`, `security.md`, `ha-async-io.md`, `api-standards.md`, `build-deployment.md`, `config-design.md`, `persistence.md`, `resource-accounting.md`, `resource-model.md`, `job-config-layering.md`, `roadmap.md`
|
||||
- Custom: `testing.md`, `error-handling.md`, `security.md`, `ha-async-io.md`, `api-standards.md`, `build-deployment.md`, `config-design.md`, `persistence.md`, `resource-accounting.md`, `resource-model.md`, `resource-questions.md`, `job-config-layering.md`, `roadmap.md`
|
||||
- Add new custom files via `/kiro-steering-custom`; sync existing via `/kiro-steering`
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue