lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(spec): resource-provisioner decisions and research (language, in-memory lease, orphan cleanup)

Browse source 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-10 13:46:58 +00:00
parent 832e397351
commit ce3772868f
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
2 changed files with 211 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

22
.kiro/specs/resource-provisioner/brief.md
View file

@ -102,3 +102,25 @@ controller facts.
- Provider scripts never run in the controller or scheduler process.
- Credentials stay with the provisioner instance and its scripts.
- HA/persistence: full 3-phase workflow.
- No fork in multi-threaded processes (`.kiro/steering/tech.md`, "Process
Creation"): the provisioner starts providers with `posix_spawn()` (or
is itself started as a systemd unit); bug-1969 is the prior-art violation.
## Decisions (user, 2026-10-10)
Answers to the design questions raised by the provider research
(stateful providers, start model, protocol):
1. **Stateful (long-running) providers:** the design must accommodate them
(protocol and supervision shaped for a persistent provider process
holding sessions, rate limits, in-flight operations), but v1 need not
ship them; v1 may support one-shot provider calls only.
2. **Implementation language:** prefer language independence; needs more
investigation (which parts must be language-neutral: provider contract,
provisioner daemon, controller link).
3. **Provider ownership lease:** investigate whether it can work with no
WAL entry (lease held in the primary's memory, rebuilt after failover).
4. **Demand delivery:** push from the controller to the provisioner, like
job dispatch.
5. **Orphan cleanup** (automatic vs admin-confirmed): needs more
investigation.
6. **LSF resource-connector script names:** keeping them is preferred, so
existing provider scripts can be reused; other names are acceptable.

189
.kiro/specs/resource-provisioner/research.md Normal file
View file

@ -0,0 +1,189 @@
# Research: resource-provisioner
(2026-10-10. Follows the user's decisions in `brief.md`, "Decisions": items 2, 3 and 5 needed investigation. Read-only research; findings cite repo paths. Earlier context: the provider survey of the same day found the stateful-provider requirement in this brief and in `xce-node-manager` requirements.md "Cross-Spec Interactions" 3, and found no existing wire protocol or start model.)
## Summary
| Question | Result |
|---|---|
| Q2 language independence | The **provider contract** is language-neutral: JSON documents, one-shot LSF RC scripts in v1, JSON-lines persistent mode later. The **daemon** is Go via CGO into libxce. The **controller link** is Cap'n Proto over libxce with a new `HostRole_provisioner`, and is a documented protocol that other languages can implement. |
| Q3 lease without WAL | **Feasible.** The lease is held in the primary's memory with epoch `(HA term, seq)`. Grants need a fresh quorum, a grace of TTL after promotion applies before a different holder is granted, the epoch is stamped on and checked for every request/instance fact, intent is committed before acting, and the holder stops itself on its TTL. A WAL lease entry would not remove the grace period. |
| Q4 demand delivery | Push, like job dispatch (user decision). Q2 adds: pushes are fire-and-forget, so demand goes as **level-triggered snapshots** with a resync after rejoin. |
| Q5 orphan cleanup | **Per class:** never-joined → automatic return; tagged-but-unknown → report (adopt if it matches a live host; stop/terminate opt-in); ghost → automatic fact retirement; license → report plus reconciliation. Strict safety preconditions, rate limits, dry-run. |
## Design constraints these results add
- **Idempotent actions:** every provider action is idempotent and keyed by an op_id, which is also passed to the cloud as its idempotency token where the API allows.
- **Owner tags:** every provisioned cloud resource is tagged with cluster id, provider, request id/op_id and lease epoch.
- **Lease self-stop:** a persistent provider (decision 1) stops making changes when its lease lapses.
- **Commit before acting:** an intent fact is committed before any cloud mutation, including terminate.
- **Hard dependency:** the per-role opcode allowlist from `peer-authentication-hardening`. Today any TLS peer is accepted (`roadmap.md:133-134`).
- **No fork:** the provisioner spawns providers with `posix_spawn` or Go `os/exec`, never fork from threaded C.
## Q2: Language independence (decision 2)
**Finding:** language neutrality is needed at the provider contract, and only there. The controller wire protocol is already usable from any language: the Python `xceadmin` speaks it.
### Found in repo
- **Wire format:**
- Each packet is an 8-byte host-endian length followed by the body (`src/include/netio.h:29-32`, `src/lib/netio.c:197-205`).
- The body is an unpacked Cap'n Proto message (`src/lib/request.c:35-54,103`).
- It travels over server-side TLS with no client certificates (`security.md:71-75`, `src/controller/myapp.c:116`).
- Replies come wrapped as `Request{opcode: reply}` (`src/xceadmin/src/xceadmin/discovery.py:16-28`).
- **Non-C client:** `xcecommon/wire.py` is 155 lines of Python (pycapnp + ssl) and follows redirects (`discovery.py:110,202`). It is one-shot only: it never joins as a role, sends no heartbeat and takes no pushes.
- **What a long-lived role client implements:**
- join with a `HostRole` (`controller.capnp:150-156`, `src/controller/join.c:307-326`), with per-role cleanup when the connection closes (`cluster.c:71-110`);
- failover rotation (`src/lib/client_failover.c`, about 435 lines);
- rejoin after reconnect (`apiserver backend/reconnect.go:20-60`);
- heartbeat (`backend/heartbeat.go`).
- **Job dispatch push:**
- The worker connects out and joins (`src/worker/src/controller_connector.c:402`).
- The controller sends `xce_request` down that connection (`src/lib/request.c:2143`).
- The worker's opcode switch (`controller_connector.c:1845-1873`) handles it.
- After a rejoin, state is reconciled with `jobStateQuery`/`jobStateReport`.
- The apiserver receives the same pushes through `go_push_handler` (`backend/clib.go:23-47,405`).
- **Apiserver:** has no SSE, WebSocket or gRPC streaming.
- **Authentication:** `validate_host()` accepts every peer (`roadmap.md:133-134`). `peer-authentication-hardening` is a roadmap entry only (`roadmap.md:88`).
### Seams and recommendation
| Seam | Recommendation | Why |
|---|---|---|
| A. Provisioner ↔ provider | **Language-neutral JSON documents.** v1 transport: LSF RC one-shot (`<script> -f in.json`, JSON on stdout, exit code), keeping the script names (decision 6). Later transport: persistent JSON-lines `{id, action, input}` → `{id, result\|error}` with the same action names and schemas (decision 1). | Bash or any language can implement it; one schema serves both transports; a built-in simulated provider implements the same schema |
| B. Provisioner daemon | **Go via CGO into libxce**, like the apiserver | Reuses libxce failover and TLS; `os/exec` is allowed (Go is outside the no-fork rule); cloud SDKs are native |
| C. Provisioner ↔ controller | **Cap'n Proto over libxce with a new `HostRole_provisioner`**; neutrality comes from documenting the protocol | Push works natively, like worker dispatch (decision 4), and HA redirect, rejoin and resync are already built. A REST facade would first need streaming, and a dedicated protocol would repeat all of this |
### Risks
- **Lost pushes:** pushes are fire-and-forget, so demand sent during a failover is lost. Send demand as level-triggered snapshots and resync after a rejoin.
- **Authentication:** without peer authentication, any TLS peer can pose as a provisioner. The per-role opcode allowlist is effectively a blocker.
- **Byte order:** the frame header is host-endian. Document it or fix it before third-party clients exist.
### Open questions
1. Should the JSON-lines persistent framing be specified now, though not implemented in v1?
2. Is Go via CGO acceptable for the daemon?
3. Should demand be pushed as snapshots or as deltas?
4. Is `peer-authentication-hardening` (the per-role allowlist) a hard dependency?
5. Should the wire byte order be fixed and the protocol published as stable?
## Q3: Provider-ownership lease with no WAL entry (decision 3)
**Finding:** this is feasible. A WAL lease entry would not remove the hard part: a new primary must wait one lease TTL before granting a provider to a *different* instance either way, because it cannot know when the last renewal happened on its own clock (the Chubby result).
### Found in repo
- **In-memory sessions, rebuilt after failover, are an existing pattern:**
- The scheduler session is held only in memory: `c->sched.ep` is set on join with no WAL entry (`src/controller/join.c:166-177`) and cleared on leave (`sched.c:769-797`).
- Clients re-join from scratch after a failover (`component-rejoin-resync/brief.md`).
- **Workers are mixed:** the connection is in memory (`worker.c:951`), while the join is a WAL fact with a commit barrier (`join.c:126-150`).
- **Failover:** clients find the new primary through redirect then rotation (`src/lib/client_failover.c:102-138`).
- **The HA term is a durable fencing token:**
- It is commented as one (`src/controller/ha.h:169`).
- It is fsync'd to `ha-term.bin` before any vote or term increase (`ha.c:5545-5588`).
- It is monotonic (`ha-async-io.md:57`).
- A new primary appends a NOOP in its own term before committing anything (`ha.c:2334-2338`).
- **Gaps that matter here:**
- A partitioned old primary does not step down until it sees a higher term (`ha.h:871`, `ha-async-io.md:67`).
- The write-quorum gate counts *connected* standbys (`ha.c:4847-4875`).
- Some fact opcodes are admitted without quorum (`ha.c:4990-4998`).
- No lease code exists yet. `delay.c` uses `CLOCK_MONOTONIC`.
### Mechanisms required for an in-memory lease
1. **Epoch:** `(durable HA term, in-memory seq)`. Raft allows one leader per term, so the epoch is unique and ordered.
2. **Grant and renew only under a fresh quorum:** a majority acknowledged a heartbeat in the current term within `T_check`, not merely connected standbys. Lease and provisioner opcodes are never admitted without quorum.
3. **Grace after promotion:** wait `TTL + T_check` (plus a drift margin) before granting a provider to a different instance. The previous holder, whose epoch matches the latest committed fact for that provider, is re-granted at once.
4. **Every request and instance fact carries the epoch,** and the primary rejects a fact whose epoch is not the current lease epoch. This makes the lease safe without its own WAL entry, at the cost of a few bytes per fact.
5. **Commit before acting:** the intent is committed (commit barrier, on the slow lane or under `COMMIT_WAIT_RESERVED`) before any cloud call.
6. **The holder stops itself:** it treats the lease as valid until `renew_sent + TTL − margin` on its monotonic clock, then makes no new mutations.
### Comparison (10 providers, TTL 30s, renewal every 10s)
| Option | Lease WAL entries | Needs grace | Previous owner known after failover |
|---|---|---|---|
| (i) In-memory + term epoch + grace | 0 | yes | from the epoch on the latest committed fact |
| (ii) WAL fact per renewal | ~86,400/day (majority fsync) | yes | yes |
| (ii′/iii) WAL only on acquire/transfer | tens/day | yes | yes |
**Recommendation: (i), no WAL entry for the lease.**
- **Durable:** the request, instance and return facts, each stamped with the epoch, and the HA term.
- **Not durable:** the lease table.
### What the epoch cannot fence: the cloud
A partitioned old owner can finish a cloud call already in flight. Mitigations:
- **Idempotency keys:** use the request id as the cloud client token (AWS `ClientToken`, GCE `requestId`).
- **Owner tags:** tag each instance with cluster, provider, request id and epoch. The successor checks the tags before retrying an unknown outcome, and the orphan scan uses them too.
- **Self-stop:** the provider stops on TTL expiry.
- **Cloud backstop:** use the cloud's maximum lifetime or auto-terminate where it exists.
**Design constraints that follow (feed into decision 1):**
- every provider action is idempotent;
- a persistent provider must stop mutating when its lease lapses.
## Q5: Orphan cleanup, automatic or admin-confirmed (decision 5)
**Finding:** use a mix, set per class of orphan. Automate cleanup where XCE's own records prove that no work can be on the resource. Report, and let an admin confirm, where the only evidence that XCE owns the resource is a cloud tag.
### Found
- **Repo:**
- The brief requires that orphans which never joined are found and returned, with no approval step in v1.
- `elastic-hosts` says that a host which misses its join deadline "is reported for return".
- `elastic-shared-resources` says "amounts in use are never returned".
- `src/autoscaler` keeps its state in memory only and has no join deadline (`scaler.c:148`, `include/scaler/types.h:65,97`).
- The Baidu provider tags instances with `REQ_ID`, `RC_ACCOUNT` and `PROVIDER_NAME` (`providers/baidu/src/bcctool/client/bccClient.go:176`) and keeps its own `_db.json`, which is a second source of truth.
- **Other systems:**
- **LSF resource connector:** returns a host that hasn't joined 10 minutes after it is ready. Untracked instances are cleaned up by hand.
- **Slurm:** after `ResumeTimeout` the node goes DOWN and a site script handles the cloud side.
- **Kubernetes cluster-autoscaler:** removes nodes still unregistered after `--max-node-provision-time` (15 min). It halts if too many nodes are unready and caps bulk deletes.
- **Karpenter:** deletes tagged instances that have no NodeClaim after 30 s.
- **AWS ParallelCluster:** terminates tagged instances with no Slurm node after 300 s, in batches. It skips the whole cycle if either Slurm or EC2 state can't be read, and has a protected mode for repeated bootstrap failures.
- **Nomad:** drains a node before scale-in and honours scale-in protection.
- **Terraform:** only reports drift.
- **Common patterns:**
- Ownership is a cluster/manager tag, and untagged resources are never touched.
- The grace period is measured from the cloud's own launch time.
- Cleanup skips the cycle when either view can't be read.
- Batch limits and a circuit breaker.
- Protection tags exempt individual resources.
- Systems that own their account clean up automatically; systems that share it only report.
### Recommended policy
| Class | Default | Why |
|---|---|---|
| (a) requested, never joined | **automatic return** after `join_deadline` (lead time + margin, e.g. 15 min) | XCE has the request fact and knows no worker joined, so no job can be on the instance |
| (b) tagged in the cloud, no XCE fact | **report** (admin confirms). `stop`/`terminate` after `orphan_grace` (default 30 min) are opt-in. If the instance matches a live XCE host by hostname or IP, **adopt** it instead | It exists only because XCE's records failed somewhere; it could be a joined worker whose fact was lost |
| (c) ghost: XCE fact, no cloud instance | **automatic** retirement of the fact through the host-loss path, optionally after two "not found" reads in a row | Costs no money; jobs on it are already lost |
| (d) license grant not returned | **report** plus reconciliation. Automatic return only if XCE's in-use count is 0 **and** the license server agrees | Returning a license in use breaks running jobs |
- **Admin knobs, per provider:**
- `orphan_action: report|stop|terminate`
- `join_deadline`, `orphan_grace`
- `max_cleanup_per_interval`
- a circuit breaker on repeated failed joins
- `protect_tag`
- `dry_run`
- an admin adopt/return command
- **Preconditions for any automatic action:**
1. the resource has this cluster's ID tag and the provider tag;
2. the provisioner holds the lease, and its epoch is stamped on the action;
3. the controller is primary, its own-term NOOP is committed, and the settle window after failover has passed;
4. both the provider's list and the controller's view were read successfully this cycle;
5. the cloud launch time is older than the grace period;
6. tags carry the request id or op_id, so a create that crashed before its fact was written maps to class (a), not (b).
- **What to record:**
- **WAL facts:** instance RETURNING, RETURNED and LOST; orphan REPORTED and ADOPTED; an intent-to-terminate fact written *before* the provider call.
- **Audit log:** every detection with its evidence (tags, launch time, epoch), the action, who confirmed it, and the provider's response.
### Sources
- LSF resource connector: IBM docs ("LSF resource connector overview", pre/post-provisioning); SAS PDF "Configuring the LSF resource connector for AWS"
- Slurm: power_save.html
- Kubernetes cluster-autoscaler: FAQ.md
- Karpenter: aws `nodeclaim/garbagecollection/controller.go`
- AWS ParallelCluster: processes-v3, slurm-protected-mode-v3, `clustermgtd.py`
- Nomad: aws-asg target plugin
- Terraform: resource-drift tutorial
## Open questions for the user
1. Should the JSON-lines persistent-provider framing be specified now, though not implemented in v1 (decision 1)?
2. Is Go via CGO into libxce acceptable for the provisioner daemon, or is a CGO-free daemon required (re-implementing failover/rejoin)?
3. Is `peer-authentication-hardening` (per-role opcode allowlist) a hard dependency before the provisioner ships?
4. Should the wire frame header be fixed to a defined byte order and the protocol published as stable for third-party provisioners?
5. Are the cloud accounts dedicated to the cluster or shared? If shared, tagged-but-unknown orphans stay report-only for good.
6. Is automatic return of never-joined instances acceptable in v1 (the brief says no approval step)?
7. Should `stop` (quarantine) be offered where the provider supports it, or only `terminate`?
8. Should the settle window after failover be a fixed time, or last until expected workers have rejoined or timed out?
9. When the license server and XCE disagree on in-use counts, which is authoritative?