126 lines
6.6 KiB
Markdown
126 lines
6.6 KiB
Markdown
# Brief: resource-provisioner
|
|
|
|
(`/kiro-discovery`, 2026-10-10, "Auto resource provisioning". Prior art:
|
|
`src/autoscaler`, an unused port of the LSF resource connector's provider
|
|
layer. It is a source of ideas only, not a required basis (user
|
|
2026-10-10): the design decides whether any of its code or its script
|
|
contract is reused.)
|
|
|
|
## Problem
|
|
Demand computed by the scheduler (`provisioning-demand`) must turn into real
|
|
requests to resource providers (cloud APIs, license pool managers, others
|
|
later), and borrowed resources must be given back, automatically, with as
|
|
little admin involvement as possible while the cluster runs, and without
|
|
ever losing track of something that costs money.
|
|
|
|
## Current State
|
|
- Prior art, `src/autoscaler` (static library, linked by nothing): provider registry
|
|
(`scaler.yaml` / `hostProviders.yaml`), a 5-script provider contract
|
|
(getAvailableTemplates, requestMachines, requestReturnMachines,
|
|
getRequestStatus, getReturnRequests; `plug_mgr.c` runs
|
|
`sh <script> -f <in.json>` and reads JSON), templates (`templateId`,
|
|
`maxNumber`, `priority`, attributes, price), request and instance state
|
|
machines, a monitor loop and a demand queue; state in memory only. One
|
|
provider: Baidu BCC (`providers/baidu`, Go `bcctool`).
|
|
- `scaler.yaml` already flows through `xceadmin` (`hproviders`) to the
|
|
controller and scheduler (`Conf4Sched`), unused.
|
|
|
|
## Desired Outcome
|
|
- **Provisioner program, several instances:** different resources may need
|
|
different provisioners (user 2026-10-10): e.g. a cloud provisioner on a
|
|
host that holds cloud credentials, a license provisioner next to the
|
|
license server. Each instance declares the providers it serves; each
|
|
provider is owned by exactly one active instance at a time (lease granted
|
|
by the controller; a standby takes over when the lease lapses).
|
|
- **Providers are generic** (hosts and countable resources first, other
|
|
kinds later) and **describe themselves** (user 2026-10-10) so policy can
|
|
run without manual steps: capacity (maximum hosts or units), cost (per
|
|
host-hour or unit), priority, lead time (how long a request takes),
|
|
granularity (minimum request size, step), lease or minimum hold time,
|
|
region/zone, and extensible attributes.
|
|
- **Script contract:** JSON in/out, actions for both kinds (templates or
|
|
quote, request, request status, return, return status), timeouts, exit
|
|
codes, environment. The LSF-style 5-script contract of `src/autoscaler`
|
|
is one input to this design, not a constraint.
|
|
- **Automatic policy** (no approval step in v1): from finalized demand and
|
|
provider descriptions, choose providers (priority, then cost), request
|
|
amounts (respecting capacity, granularity, per-provider and global
|
|
budgets, cooldowns), and reclaim idle supply after its idle time,
|
|
respecting minimum hold times. Requests that fail or time out are retried
|
|
or moved to the next provider.
|
|
- **Durable state:** every request and every provisioned instance or grant
|
|
is a controller fact (WAL), so a provisioner restart or failover never
|
|
forgets a cloud instance; orphaned instances (requested, never joined)
|
|
are found and returned.
|
|
- **Visibility:** REST and `xcecli` show providers, their descriptions and
|
|
usage, requests, instances, costs so far, and recent decisions; audit log.
|
|
- **Testing:** a simulated provider (scripts or built-in) for CI and for the
|
|
scheduler simulator's offline evaluation.
|
|
|
|
## Approach
|
|
A new daemon (implementation language and code reuse decided in design;
|
|
`src/autoscaler` may be reused, adapted or replaced), connected to the
|
|
controller through libxce with its own role; reads finalized demand and idle
|
|
supply, runs provider scripts off any HA path, and records state through
|
|
controller facts.
|
|
|
|
## Scope
|
|
- **In**: daemon, multi-instance ownership via leases, provider registry and
|
|
self-description, script contract v2, policy engine (selection, sizing,
|
|
budgets, cooldowns, retries, reclaim), durable request/instance state,
|
|
orphan cleanup, REST/xcecli/audit, simulated provider; at least one real
|
|
host provider and one license provider as reference implementations
|
|
(whether the existing Baidu provider is one of them is a design choice).
|
|
- **Out**: computing demand (`provisioning-demand`); host join/drain/
|
|
decommission mechanics (`elastic-hosts`); license pool totals in the
|
|
scheduler (`elastic-shared-resources`); approval workflow (later, if ever).
|
|
|
|
## Boundary Candidates
|
|
- Daemon and leases.
|
|
- Provider registry, description and script contract.
|
|
- Policy engine.
|
|
- Durable request/instance facts in the controller.
|
|
- Status surfaces.
|
|
|
|
## Out of Boundary
|
|
- Scheduling decisions.
|
|
|
|
## Upstream / Downstream
|
|
- **Upstream**: `provisioning-demand`, controller (facts, leases),
|
|
libxce, `peer-authentication-hardening` (provisioner role).
|
|
- **Downstream**: `elastic-hosts`, `elastic-shared-resources`.
|
|
|
|
## Existing Spec Touchpoints
|
|
- **Extends**: none (`src/autoscaler` is prior art only).
|
|
- **Adjacent**: `component-failover-adoption`, `admin-config-tool`
|
|
(provider config), `xce-node-manager`: a `provisioner` node role keeps the
|
|
daemon running; which instance is active is decided by the
|
|
Controller-granted lease, not by the Node Manager (`xce-node-manager`
|
|
requirements.md, "Cross-Spec Interactions" 3).
|
|
|
|
## Constraints
|
|
- Provider scripts never run in the controller or scheduler process.
|
|
- Credentials stay with the provisioner instance and its scripts.
|
|
- HA/persistence: full 3-phase workflow.
|
|
- No fork in multi-threaded processes (`.kiro/steering/tech.md`, "Process
|
|
Creation"): the provisioner starts providers with `posix_spawn()` (or
|
|
is itself started as a systemd unit); bug-1969 is the prior-art violation.
|
|
|
|
## Decisions (user, 2026-10-10)
|
|
Answers to the design questions raised by the provider research
|
|
(stateful providers, start model, protocol):
|
|
1. **Stateful (long-running) providers:** the design must accommodate them
|
|
(protocol and supervision shaped for a persistent provider process
|
|
holding sessions, rate limits, in-flight operations), but v1 need not
|
|
ship them; v1 may support one-shot provider calls only.
|
|
2. **Implementation language:** prefer language independence; needs more
|
|
investigation (which parts must be language-neutral: provider contract,
|
|
provisioner daemon, controller link).
|
|
3. **Provider ownership lease:** investigate whether it can work with no
|
|
WAL entry (lease held in the primary's memory, rebuilt after failover).
|
|
4. **Demand delivery:** push from the controller to the provisioner, like
|
|
job dispatch.
|
|
5. **Orphan cleanup** (automatic vs admin-confirmed): needs more
|
|
investigation.
|
|
6. **LSF resource-connector script names:** keeping them is preferred, so
|
|
existing provider scripts can be reused; other names are acceptable.
|