myxce/.kiro/specs/resource-provisioner/research.md

Research: resource-provisioner

(2026-10-10. Follows the user's decisions in brief.md, "Decisions": items 2, 3 and 5 needed investigation. Read-only research; findings cite repo paths. Earlier context: the provider survey of the same day found the stateful-provider requirement in this brief and in xce-node-manager requirements.md "Cross-Spec Interactions" 3, and found no existing wire protocol or start model.)

Summary

Question	Result
Q2 language independence	The provider contract is language-neutral: JSON documents, one-shot LSF RC scripts in v1, JSON-lines persistent mode later. The daemon is Go via CGO into libxce. The controller link is Cap'n Proto over libxce with a new HostRole_provisioner, and is a documented protocol that other languages can implement.
Q3 lease without WAL	Feasible. The lease is held in the primary's memory with epoch (HA term, seq). Grants need a fresh quorum, a grace of TTL after promotion applies before a different holder is granted, the epoch is stamped on and checked for every request/instance fact, intent is committed before acting, and the holder stops itself on its TTL. A WAL lease entry would not remove the grace period.
Q4 demand delivery	Push, like job dispatch (user decision). Q2 adds: pushes are fire-and-forget, so demand goes as level-triggered snapshots with a resync after rejoin.
Q5 orphan cleanup	Per class: never-joined → automatic return; tagged-but-unknown → report (adopt if it matches a live host; stop/terminate opt-in); ghost → automatic fact retirement; license → report plus reconciliation. Strict safety preconditions, rate limits, dry-run.

Design constraints these results add

• Idempotent actions: every provider action is idempotent and keyed by an op_id, which is also passed to the cloud as its idempotency token where the API allows.
• Owner tags: every provisioned cloud resource is tagged with cluster id, provider, request id/op_id and lease epoch.
• Lease self-stop: a persistent provider (decision 1) stops making changes when its lease lapses.
• Commit before acting: an intent fact is committed before any cloud mutation, including terminate.
• Hard dependency: the per-role opcode allowlist from peer-authentication-hardening. Today any TLS peer is accepted (roadmap.md:133-134).
• No fork: the provisioner spawns providers with posix_spawn or Go os/exec, never fork from threaded C.

Q2: Language independence (decision 2)

Finding: language neutrality is needed at the provider contract, and only there. The controller wire protocol is already usable from any language: the Python xceadmin speaks it.

Found in repo

• Wire format:
  • Each packet is an 8-byte host-endian length followed by the body (src/include/netio.h:29-32, src/lib/netio.c:197-205).
  • The body is an unpacked Cap'n Proto message (src/lib/request.c:35-54,103).
  • It travels over server-side TLS with no client certificates (security.md:71-75, src/controller/myapp.c:116).
  • Replies come wrapped as Request{opcode: reply} (src/xceadmin/src/xceadmin/discovery.py:16-28).
• Non-C client: xcecommon/wire.py is 155 lines of Python (pycapnp + ssl) and follows redirects (discovery.py:110,202). It is one-shot only: it never joins as a role, sends no heartbeat and takes no pushes.
• What a long-lived role client implements:
  • join with a HostRole (controller.capnp:150-156, src/controller/join.c:307-326), with per-role cleanup when the connection closes (cluster.c:71-110);
  • failover rotation (src/lib/client_failover.c, about 435 lines);
  • rejoin after reconnect (apiserver backend/reconnect.go:20-60);
  • heartbeat (backend/heartbeat.go).
• Job dispatch push:
  • The worker connects out and joins (src/worker/src/controller_connector.c:402).
  • The controller sends xce_request down that connection (src/lib/request.c:2143).
  • The worker's opcode switch (controller_connector.c:1845-1873) handles it.
  • After a rejoin, state is reconciled with jobStateQuery/jobStateReport.
  • The apiserver receives the same pushes through go_push_handler (backend/clib.go:23-47,405).
• Apiserver: has no SSE, WebSocket or gRPC streaming.
• Authentication: validate_host() accepts every peer (roadmap.md:133-134). peer-authentication-hardening is a roadmap entry only (roadmap.md:88).

Seams and recommendation

Seam	Recommendation	Why
A. Provisioner ↔ provider	Language-neutral JSON documents. v1 transport: LSF RC one-shot (<script> -f in.json, JSON on stdout, exit code), keeping the script names (decision 6). Later transport: persistent JSON-lines {id, action, input} → {id, result|error} with the same action names and schemas (decision 1).	Bash or any language can implement it; one schema serves both transports; a built-in simulated provider implements the same schema
B. Provisioner daemon	Go via CGO into libxce, like the apiserver	Reuses libxce failover and TLS; os/exec is allowed (Go is outside the no-fork rule); cloud SDKs are native
C. Provisioner ↔ controller	Cap'n Proto over libxce with a new HostRole_provisioner; neutrality comes from documenting the protocol	Push works natively, like worker dispatch (decision 4), and HA redirect, rejoin and resync are already built. A REST facade would first need streaming, and a dedicated protocol would repeat all of this

Risks

• Lost pushes: pushes are fire-and-forget, so demand sent during a failover is lost. Send demand as level-triggered snapshots and resync after a rejoin.
• Authentication: without peer authentication, any TLS peer can pose as a provisioner. The per-role opcode allowlist is effectively a blocker.
• Byte order: the frame header is host-endian. Document it or fix it before third-party clients exist.

Open questions

1. Should the JSON-lines persistent framing be specified now, though not implemented in v1?
2. Is Go via CGO acceptable for the daemon?
3. Should demand be pushed as snapshots or as deltas?
4. Is peer-authentication-hardening (the per-role allowlist) a hard dependency?
5. Should the wire byte order be fixed and the protocol published as stable?

Q3: Provider-ownership lease with no WAL entry (decision 3)

Finding: this is feasible. A WAL lease entry would not remove the hard part: a new primary must wait one lease TTL before granting a provider to a different instance either way, because it cannot know when the last renewal happened on its own clock (the Chubby result).

Found in repo

• In-memory sessions, rebuilt after failover, are an existing pattern:
  • The scheduler session is held only in memory: c->sched.ep is set on join with no WAL entry (src/controller/join.c:166-177) and cleared on leave (sched.c:769-797).
  • Clients re-join from scratch after a failover (component-rejoin-resync/brief.md).
• Workers are mixed: the connection is in memory (worker.c:951), while the join is a WAL fact with a commit barrier (join.c:126-150).
• Failover: clients find the new primary through redirect then rotation (src/lib/client_failover.c:102-138).
• The HA term is a durable fencing token:
  • It is commented as one (src/controller/ha.h:169).
  • It is fsync'd to ha-term.bin before any vote or term increase (ha.c:5545-5588).
  • It is monotonic (ha-async-io.md:57).
  • A new primary appends a NOOP in its own term before committing anything (ha.c:2334-2338).
• Gaps that matter here:
  • A partitioned old primary does not step down until it sees a higher term (ha.h:871, ha-async-io.md:67).
  • The write-quorum gate counts connected standbys (ha.c:4847-4875).
  • Some fact opcodes are admitted without quorum (ha.c:4990-4998).
  • No lease code exists yet. delay.c uses CLOCK_MONOTONIC.

Mechanisms required for an in-memory lease

1. Epoch: (durable HA term, in-memory seq). Raft allows one leader per term, so the epoch is unique and ordered.
2. Grant and renew only under a fresh quorum: a majority acknowledged a heartbeat in the current term within T_check, not merely connected standbys. Lease and provisioner opcodes are never admitted without quorum.
3. Grace after promotion: wait TTL + T_check (plus a drift margin) before granting a provider to a different instance. The previous holder, whose epoch matches the latest committed fact for that provider, is re-granted at once.
4. Every request and instance fact carries the epoch, and the primary rejects a fact whose epoch is not the current lease epoch. This makes the lease safe without its own WAL entry, at the cost of a few bytes per fact.
5. Commit before acting: the intent is committed (commit barrier, on the slow lane or under COMMIT_WAIT_RESERVED) before any cloud call.
6. The holder stops itself: it treats the lease as valid until renew_sent + TTL − margin on its monotonic clock, then makes no new mutations.

Comparison (10 providers, TTL 30s, renewal every 10s)

Option	Lease WAL entries	Needs grace	Previous owner known after failover
(i) In-memory + term epoch + grace	0	yes	from the epoch on the latest committed fact
(ii) WAL fact per renewal	~86,400/day (majority fsync)	yes	yes
(ii′/iii) WAL only on acquire/transfer	tens/day	yes	yes

Recommendation: (i), no WAL entry for the lease.

• Durable: the request, instance and return facts, each stamped with the epoch, and the HA term.
• Not durable: the lease table.

What the epoch cannot fence: the cloud

A partitioned old owner can finish a cloud call already in flight. Mitigations:

• Idempotency keys: use the request id as the cloud client token (AWS ClientToken, GCE requestId).
• Owner tags: tag each instance with cluster, provider, request id and epoch. The successor checks the tags before retrying an unknown outcome, and the orphan scan uses them too.
• Self-stop: the provider stops on TTL expiry.
• Cloud backstop: use the cloud's maximum lifetime or auto-terminate where it exists.

Design constraints that follow (feed into decision 1):

• every provider action is idempotent;
• a persistent provider must stop mutating when its lease lapses.

Q5: Orphan cleanup, automatic or admin-confirmed (decision 5)

Finding: use a mix, set per class of orphan. Automate cleanup where XCE's own records prove that no work can be on the resource. Report, and let an admin confirm, where the only evidence that XCE owns the resource is a cloud tag.

Found

• Repo:
  • The brief requires that orphans which never joined are found and returned, with no approval step in v1.
  • elastic-hosts says that a host which misses its join deadline "is reported for return".
  • elastic-shared-resources says "amounts in use are never returned".
  • src/autoscaler keeps its state in memory only and has no join deadline (scaler.c:148, include/scaler/types.h:65,97).
  • The Baidu provider tags instances with REQ_ID, RC_ACCOUNT and PROVIDER_NAME (providers/baidu/src/bcctool/client/bccClient.go:176) and keeps its own _db.json, which is a second source of truth.
• Other systems:
  • LSF resource connector: returns a host that hasn't joined 10 minutes after it is ready. Untracked instances are cleaned up by hand.
  • Slurm: after ResumeTimeout the node goes DOWN and a site script handles the cloud side.
  • Kubernetes cluster-autoscaler: removes nodes still unregistered after --max-node-provision-time (15 min). It halts if too many nodes are unready and caps bulk deletes.
  • Karpenter: deletes tagged instances that have no NodeClaim after 30 s.
  • AWS ParallelCluster: terminates tagged instances with no Slurm node after 300 s, in batches. It skips the whole cycle if either Slurm or EC2 state can't be read, and has a protected mode for repeated bootstrap failures.
  • Nomad: drains a node before scale-in and honours scale-in protection.
  • Terraform: only reports drift.
• Common patterns:
  • Ownership is a cluster/manager tag, and untagged resources are never touched.
  • The grace period is measured from the cloud's own launch time.
  • Cleanup skips the cycle when either view can't be read.
  • Batch limits and a circuit breaker.
  • Protection tags exempt individual resources.
  • Systems that own their account clean up automatically; systems that share it only report.

Recommended policy

Class	Default	Why
(a) requested, never joined	automatic return after join_deadline (lead time + margin, e.g. 15 min)	XCE has the request fact and knows no worker joined, so no job can be on the instance
(b) tagged in the cloud, no XCE fact	report (admin confirms). stop/terminate after orphan_grace (default 30 min) are opt-in. If the instance matches a live XCE host by hostname or IP, adopt it instead	It exists only because XCE's records failed somewhere; it could be a joined worker whose fact was lost
(c) ghost: XCE fact, no cloud instance	automatic retirement of the fact through the host-loss path, optionally after two "not found" reads in a row	Costs no money; jobs on it are already lost
(d) license grant not returned	report plus reconciliation. Automatic return only if XCE's in-use count is 0 and the license server agrees	Returning a license in use breaks running jobs

• Admin knobs, per provider:
  • orphan_action: report|stop|terminate
  • join_deadline, orphan_grace
  • max_cleanup_per_interval
  • a circuit breaker on repeated failed joins
  • protect_tag
  • dry_run
  • an admin adopt/return command
• Preconditions for any automatic action:
  1. the resource has this cluster's ID tag and the provider tag;
  2. the provisioner holds the lease, and its epoch is stamped on the action;
  3. the controller is primary, its own-term NOOP is committed, and the settle window after failover has passed;
  4. both the provider's list and the controller's view were read successfully this cycle;
  5. the cloud launch time is older than the grace period;
  6. tags carry the request id or op_id, so a create that crashed before its fact was written maps to class (a), not (b).
• What to record:
  • WAL facts: instance RETURNING, RETURNED and LOST; orphan REPORTED and ADOPTED; an intent-to-terminate fact written before the provider call.
  • Audit log: every detection with its evidence (tags, launch time, epoch), the action, who confirmed it, and the provider's response.

Sources

• LSF resource connector: IBM docs ("LSF resource connector overview", pre/post-provisioning); SAS PDF "Configuring the LSF resource connector for AWS"
• Slurm: power_save.html
• Kubernetes cluster-autoscaler: FAQ.md
• Karpenter: aws nodeclaim/garbagecollection/controller.go
• AWS ParallelCluster: processes-v3, slurm-protected-mode-v3, clustermgtd.py
• Nomad: aws-asg target plugin
• Terraform: resource-drift tutorial

Open questions for the user

(Tracked since 2026-10-11 as P1–P9 in .kiro/steering/resource-questions.md, which is authoritative for their status. PR-2 there proposes generalizing this daemon into a resource provider proxy.)

1. Should the JSON-lines persistent-provider framing be specified now, though not implemented in v1 (decision 1)?
2. Is Go via CGO into libxce acceptable for the provisioner daemon, or is a CGO-free daemon required (re-implementing failover/rejoin)?
3. Is peer-authentication-hardening (per-role opcode allowlist) a hard dependency before the provisioner ships?
4. Should the wire frame header be fixed to a defined byte order and the protocol published as stable for third-party provisioners?
5. Are the cloud accounts dedicated to the cluster or shared? If shared, tagged-but-unknown orphans stay report-only for good.
6. Is automatic return of never-joined instances acceptable in v1 (the brief says no approval step)?
7. Should stop (quarantine) be offered where the provider supports it, or only terminate?
8. Should the settle window after failover be a fixed time, or last until expected workers have rejoined or timed out?
9. When the license server and XCE disagree on in-use counts, which is authoritative?