Rongsong Shen
a68aac1a2b
xce-node-manager: a "Cross-Spec Interactions" section (pending a requirements revision; approved criteria unchanged) with six points: a supervision/probe module shared with service-supervision, node-manager components vs service jobs, a lease-based provisioner role, role declaration and worker stop on decommission for provisioned hosts, a host-loss grace period longer than worker restart plus rejoin, and a future service proxy under the reverse-proxy role. design.md Open Questions points to it. The service-supervision, host-failure-job-recovery, elastic-hosts, resource-provisioner and service-endpoint briefs cross-reference the matching points; the roadmap lists the xce-node-manager revision. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
104 lines
5.3 KiB
Markdown
104 lines
5.3 KiB
Markdown
# Brief: resource-provisioner
|
|
|
|
(`/kiro-discovery`, 2026-10-10, "Auto resource provisioning". Prior art:
|
|
`src/autoscaler`, an unused port of the LSF resource connector's provider
|
|
layer. It is a source of ideas only, not a required basis (user
|
|
2026-10-10): the design decides whether any of its code or its script
|
|
contract is reused.)
|
|
|
|
## Problem
|
|
Demand computed by the scheduler (`provisioning-demand`) must turn into real
|
|
requests to resource providers (cloud APIs, license pool managers, others
|
|
later), and borrowed resources must be given back, automatically, with as
|
|
little admin involvement as possible while the cluster runs, and without
|
|
ever losing track of something that costs money.
|
|
|
|
## Current State
|
|
- Prior art, `src/autoscaler` (static library, linked by nothing): provider registry
|
|
(`scaler.yaml` / `hostProviders.yaml`), a 5-script provider contract
|
|
(getAvailableTemplates, requestMachines, requestReturnMachines,
|
|
getRequestStatus, getReturnRequests; `plug_mgr.c` runs
|
|
`sh <script> -f <in.json>` and reads JSON), templates (`templateId`,
|
|
`maxNumber`, `priority`, attributes, price), request and instance state
|
|
machines, a monitor loop and a demand queue; state in memory only. One
|
|
provider: Baidu BCC (`providers/baidu`, Go `bcctool`).
|
|
- `scaler.yaml` already flows through `xceadmin` (`hproviders`) to the
|
|
controller and scheduler (`Conf4Sched`), unused.
|
|
|
|
## Desired Outcome
|
|
- **Provisioner program, several instances:** different resources may need
|
|
different provisioners (user 2026-10-10): e.g. a cloud provisioner on a
|
|
host that holds cloud credentials, a license provisioner next to the
|
|
license server. Each instance declares the providers it serves; each
|
|
provider is owned by exactly one active instance at a time (lease granted
|
|
by the controller; a standby takes over when the lease lapses).
|
|
- **Providers are generic** (hosts and countable resources first, other
|
|
kinds later) and **describe themselves** (user 2026-10-10) so policy can
|
|
run without manual steps: capacity (maximum hosts or units), cost (per
|
|
host-hour or unit), priority, lead time (how long a request takes),
|
|
granularity (minimum request size, step), lease or minimum hold time,
|
|
region/zone, and extensible attributes.
|
|
- **Script contract:** JSON in/out, actions for both kinds (templates or
|
|
quote, request, request status, return, return status), timeouts, exit
|
|
codes, environment. The LSF-style 5-script contract of `src/autoscaler`
|
|
is one input to this design, not a constraint.
|
|
- **Automatic policy** (no approval step in v1): from finalized demand and
|
|
provider descriptions, choose providers (priority, then cost), request
|
|
amounts (respecting capacity, granularity, per-provider and global
|
|
budgets, cooldowns), and reclaim idle supply after its idle time,
|
|
respecting minimum hold times. Requests that fail or time out are retried
|
|
or moved to the next provider.
|
|
- **Durable state:** every request and every provisioned instance or grant
|
|
is a controller fact (WAL), so a provisioner restart or failover never
|
|
forgets a cloud instance; orphaned instances (requested, never joined)
|
|
are found and returned.
|
|
- **Visibility:** REST and `xcecli` show providers, their descriptions and
|
|
usage, requests, instances, costs so far, and recent decisions; audit log.
|
|
- **Testing:** a simulated provider (scripts or built-in) for CI and for the
|
|
scheduler simulator's offline evaluation.
|
|
|
|
## Approach
|
|
A new daemon (implementation language and code reuse decided in design;
|
|
`src/autoscaler` may be reused, adapted or replaced), connected to the
|
|
controller through libxce with its own role; reads finalized demand and idle
|
|
supply, runs provider scripts off any HA path, and records state through
|
|
controller facts.
|
|
|
|
## Scope
|
|
- **In**: daemon, multi-instance ownership via leases, provider registry and
|
|
self-description, script contract v2, policy engine (selection, sizing,
|
|
budgets, cooldowns, retries, reclaim), durable request/instance state,
|
|
orphan cleanup, REST/xcecli/audit, simulated provider; at least one real
|
|
host provider and one license provider as reference implementations
|
|
(whether the existing Baidu provider is one of them is a design choice).
|
|
- **Out**: computing demand (`provisioning-demand`); host join/drain/
|
|
decommission mechanics (`elastic-hosts`); license pool totals in the
|
|
scheduler (`elastic-shared-resources`); approval workflow (later, if ever).
|
|
|
|
## Boundary Candidates
|
|
- Daemon and leases.
|
|
- Provider registry, description and script contract.
|
|
- Policy engine.
|
|
- Durable request/instance facts in the controller.
|
|
- Status surfaces.
|
|
|
|
## Out of Boundary
|
|
- Scheduling decisions.
|
|
|
|
## Upstream / Downstream
|
|
- **Upstream**: `provisioning-demand`, controller (facts, leases),
|
|
libxce, `peer-authentication-hardening` (provisioner role).
|
|
- **Downstream**: `elastic-hosts`, `elastic-shared-resources`.
|
|
|
|
## Existing Spec Touchpoints
|
|
- **Extends**: none (`src/autoscaler` is prior art only).
|
|
- **Adjacent**: `component-failover-adoption`, `admin-config-tool`
|
|
(provider config), `xce-node-manager`: a `provisioner` node role keeps the
|
|
daemon running; which instance is active is decided by the
|
|
Controller-granted lease, not by the Node Manager (`xce-node-manager`
|
|
requirements.md, "Cross-Spec Interactions" 3).
|
|
|
|
## Constraints
|
|
- Provider scripts never run in the controller or scheduler process.
|
|
- Credentials stay with the provisioner instance and its scripts.
|
|
- HA/persistence: full 3-phase workflow.
|