diff --git a/.kiro/specs/resource-provisioner/brief.md b/.kiro/specs/resource-provisioner/brief.md index 3515a1ff..27724a50 100644 --- a/.kiro/specs/resource-provisioner/brief.md +++ b/.kiro/specs/resource-provisioner/brief.md @@ -102,3 +102,25 @@ controller facts. - Provider scripts never run in the controller or scheduler process. - Credentials stay with the provisioner instance and its scripts. - HA/persistence: full 3-phase workflow. +- No fork in multi-threaded processes (`.kiro/steering/tech.md`, "Process + Creation"): the provisioner starts providers with `posix_spawn()` (or + is itself started as a systemd unit); bug-1969 is the prior-art violation. + +## Decisions (user, 2026-10-10) +Answers to the design questions raised by the provider research +(stateful providers, start model, protocol): +1. **Stateful (long-running) providers:** the design must accommodate them + (protocol and supervision shaped for a persistent provider process + holding sessions, rate limits, in-flight operations), but v1 need not + ship them; v1 may support one-shot provider calls only. +2. **Implementation language:** prefer language independence; needs more + investigation (which parts must be language-neutral: provider contract, + provisioner daemon, controller link). +3. **Provider ownership lease:** investigate whether it can work with no + WAL entry (lease held in the primary's memory, rebuilt after failover). +4. **Demand delivery:** push from the controller to the provisioner, like + job dispatch. +5. **Orphan cleanup** (automatic vs admin-confirmed): needs more + investigation. +6. **LSF resource-connector script names:** keeping them is preferred, so + existing provider scripts can be reused; other names are acceptable. diff --git a/.kiro/specs/resource-provisioner/research.md b/.kiro/specs/resource-provisioner/research.md new file mode 100644 index 00000000..8aff2d6a --- /dev/null +++ b/.kiro/specs/resource-provisioner/research.md @@ -0,0 +1,189 @@ +# Research: resource-provisioner + +(2026-10-10. Follows the user's decisions in `brief.md`, "Decisions": items 2, 3 and 5 needed investigation. Read-only research; findings cite repo paths. Earlier context: the provider survey of the same day found the stateful-provider requirement in this brief and in `xce-node-manager` requirements.md "Cross-Spec Interactions" 3, and found no existing wire protocol or start model.) + +## Summary +| Question | Result | +|---|---| +| Q2 language independence | The **provider contract** is language-neutral: JSON documents, one-shot LSF RC scripts in v1, JSON-lines persistent mode later. The **daemon** is Go via CGO into libxce. The **controller link** is Cap'n Proto over libxce with a new `HostRole_provisioner`, and is a documented protocol that other languages can implement. | +| Q3 lease without WAL | **Feasible.** The lease is held in the primary's memory with epoch `(HA term, seq)`. Grants need a fresh quorum, a grace of TTL after promotion applies before a different holder is granted, the epoch is stamped on and checked for every request/instance fact, intent is committed before acting, and the holder stops itself on its TTL. A WAL lease entry would not remove the grace period. | +| Q4 demand delivery | Push, like job dispatch (user decision). Q2 adds: pushes are fire-and-forget, so demand goes as **level-triggered snapshots** with a resync after rejoin. | +| Q5 orphan cleanup | **Per class:** never-joined → automatic return; tagged-but-unknown → report (adopt if it matches a live host; stop/terminate opt-in); ghost → automatic fact retirement; license → report plus reconciliation. Strict safety preconditions, rate limits, dry-run. | + +## Design constraints these results add +- **Idempotent actions:** every provider action is idempotent and keyed by an op_id, which is also passed to the cloud as its idempotency token where the API allows. +- **Owner tags:** every provisioned cloud resource is tagged with cluster id, provider, request id/op_id and lease epoch. +- **Lease self-stop:** a persistent provider (decision 1) stops making changes when its lease lapses. +- **Commit before acting:** an intent fact is committed before any cloud mutation, including terminate. +- **Hard dependency:** the per-role opcode allowlist from `peer-authentication-hardening`. Today any TLS peer is accepted (`roadmap.md:133-134`). +- **No fork:** the provisioner spawns providers with `posix_spawn` or Go `os/exec`, never fork from threaded C. + +## Q2: Language independence (decision 2) + +**Finding:** language neutrality is needed at the provider contract, and only there. The controller wire protocol is already usable from any language: the Python `xceadmin` speaks it. + +### Found in repo +- **Wire format:** + - Each packet is an 8-byte host-endian length followed by the body (`src/include/netio.h:29-32`, `src/lib/netio.c:197-205`). + - The body is an unpacked Cap'n Proto message (`src/lib/request.c:35-54,103`). + - It travels over server-side TLS with no client certificates (`security.md:71-75`, `src/controller/myapp.c:116`). + - Replies come wrapped as `Request{opcode: reply}` (`src/xceadmin/src/xceadmin/discovery.py:16-28`). +- **Non-C client:** `xcecommon/wire.py` is 155 lines of Python (pycapnp + ssl) and follows redirects (`discovery.py:110,202`). It is one-shot only: it never joins as a role, sends no heartbeat and takes no pushes. +- **What a long-lived role client implements:** + - join with a `HostRole` (`controller.capnp:150-156`, `src/controller/join.c:307-326`), with per-role cleanup when the connection closes (`cluster.c:71-110`); + - failover rotation (`src/lib/client_failover.c`, about 435 lines); + - rejoin after reconnect (`apiserver backend/reconnect.go:20-60`); + - heartbeat (`backend/heartbeat.go`). +- **Job dispatch push:** + - The worker connects out and joins (`src/worker/src/controller_connector.c:402`). + - The controller sends `xce_request` down that connection (`src/lib/request.c:2143`). + - The worker's opcode switch (`controller_connector.c:1845-1873`) handles it. + - After a rejoin, state is reconciled with `jobStateQuery`/`jobStateReport`. + - The apiserver receives the same pushes through `go_push_handler` (`backend/clib.go:23-47,405`). +- **Apiserver:** has no SSE, WebSocket or gRPC streaming. +- **Authentication:** `validate_host()` accepts every peer (`roadmap.md:133-134`). `peer-authentication-hardening` is a roadmap entry only (`roadmap.md:88`). + +### Seams and recommendation +| Seam | Recommendation | Why | +|---|---|---| +| A. Provisioner ↔ provider | **Language-neutral JSON documents.** v1 transport: LSF RC one-shot (`