myxce/.kiro/specs/resource-provisioner/brief.md

Brief: resource-provisioner

(/kiro-discovery, 2026-10-10, "Auto resource provisioning". Prior art: src/autoscaler, an unused port of the LSF resource connector's provider layer. It is a source of ideas only, not a required basis (user 2026-10-10): the design decides whether any of its code or its script contract is reused.)

Problem

Demand computed by the scheduler (provisioning-demand) must turn into real requests to resource providers (cloud APIs, license pool managers, others later), and borrowed resources must be given back, automatically, with as little admin involvement as possible while the cluster runs, and without ever losing track of something that costs money.

Current State

• Prior art, src/autoscaler (static library, linked by nothing): provider registry (scaler.yaml / hostProviders.yaml), a 5-script provider contract (getAvailableTemplates, requestMachines, requestReturnMachines, getRequestStatus, getReturnRequests; plug_mgr.c runs sh <script> -f <in.json> and reads JSON), templates (templateId, maxNumber, priority, attributes, price), request and instance state machines, a monitor loop and a demand queue; state in memory only. One provider: Baidu BCC (providers/baidu, Go bcctool).
• scaler.yaml already flows through xceadmin (hproviders) to the controller and scheduler (Conf4Sched), unused.

Desired Outcome

• Provisioner program, several instances: different resources may need different provisioners (user 2026-10-10): e.g. a cloud provisioner on a host that holds cloud credentials, a license provisioner next to the license server. Each instance declares the providers it serves; each provider is owned by exactly one active instance at a time (lease granted by the controller; a standby takes over when the lease lapses).
• Providers are generic (hosts and countable resources first, other kinds later) and describe themselves (user 2026-10-10) so policy can run without manual steps: capacity (maximum hosts or units), cost (per host-hour or unit), priority, lead time (how long a request takes), granularity (minimum request size, step), lease or minimum hold time, region/zone, and extensible attributes.
• Script contract: JSON in/out, actions for both kinds (templates or quote, request, request status, return, return status), timeouts, exit codes, environment. The LSF-style 5-script contract of src/autoscaler is one input to this design, not a constraint.
• Automatic policy (no approval step in v1): from finalized demand and provider descriptions, choose providers (priority, then cost), request amounts (respecting capacity, granularity, per-provider and global budgets, cooldowns), and reclaim idle supply after its idle time, respecting minimum hold times. Requests that fail or time out are retried or moved to the next provider.
• Durable state: every request and every provisioned instance or grant is a controller fact (WAL), so a provisioner restart or failover never forgets a cloud instance; orphaned instances (requested, never joined) are found and returned.
• Visibility: REST and xcecli show providers, their descriptions and usage, requests, instances, costs so far, and recent decisions; audit log.
• Testing: a simulated provider (scripts or built-in) for CI and for the scheduler simulator's offline evaluation.

Approach

A new daemon (implementation language and code reuse decided in design; src/autoscaler may be reused, adapted or replaced), connected to the controller through libxce with its own role; reads finalized demand and idle supply, runs provider scripts off any HA path, and records state through controller facts.

Scope

• In: daemon, multi-instance ownership via leases, provider registry and self-description, script contract v2, policy engine (selection, sizing, budgets, cooldowns, retries, reclaim), durable request/instance state, orphan cleanup, REST/xcecli/audit, simulated provider; at least one real host provider and one license provider as reference implementations (whether the existing Baidu provider is one of them is a design choice).
• Out: computing demand (provisioning-demand); host join/drain/ decommission mechanics (elastic-hosts); license pool totals in the scheduler (elastic-shared-resources); approval workflow (later, if ever).

Boundary Candidates

• Daemon and leases.
• Provider registry, description and script contract.
• Policy engine.
• Durable request/instance facts in the controller.
• Status surfaces.

Out of Boundary

• Scheduling decisions.

Upstream / Downstream

• Upstream: provisioning-demand, controller (facts, leases), libxce, peer-authentication-hardening (provisioner role).
• Downstream: elastic-hosts, elastic-shared-resources.

Existing Spec Touchpoints

• Extends: none (src/autoscaler is prior art only).
• Adjacent: component-failover-adoption, admin-config-tool (provider config), xce-node-manager: a provisioner node role keeps the daemon running; which instance is active is decided by the Controller-granted lease, not by the Node Manager (xce-node-manager requirements.md, "Cross-Spec Interactions" 3).

Constraints

• Provider scripts never run in the controller or scheduler process.
• Credentials stay with the provisioner instance and its scripts.
• HA/persistence: full 3-phase workflow.
• No fork in multi-threaded processes (.kiro/steering/tech.md, "Process Creation"): the provisioner starts providers with posix_spawn() (or is itself started as a systemd unit); bug-1969 is the prior-art violation.

Decisions (user, 2026-10-10)

Answers to the design questions raised by the provider research (stateful providers, start model, protocol):

1. Stateful (long-running) providers: the design must accommodate them (protocol and supervision shaped for a persistent provider process holding sessions, rate limits, in-flight operations), but v1 need not ship them; v1 may support one-shot provider calls only.
2. Implementation language: prefer language independence; needs more investigation (which parts must be language-neutral: provider contract, provisioner daemon, controller link).
3. Provider ownership lease: investigate whether it can work with no WAL entry (lease held in the primary's memory, rebuilt after failover).
4. Demand delivery: push from the controller to the provisioner, like job dispatch.
5. Orphan cleanup (automatic vs admin-confirmed): needs more investigation.
6. LSF resource-connector script names: keeping them is preferred, so existing provider scripts can be reused; other names are acceptable.