Rongsong Shen
a68aac1a2b
xce-node-manager: a "Cross-Spec Interactions" section (pending a requirements revision; approved criteria unchanged) with six points: a supervision/probe module shared with service-supervision, node-manager components vs service jobs, a lease-based provisioner role, role declaration and worker stop on decommission for provisioned hosts, a host-loss grace period longer than worker restart plus rejoin, and a future service proxy under the reverse-proxy role. design.md Open Questions points to it. The service-supervision, host-failure-job-recovery, elastic-hosts, resource-provisioner and service-endpoint briefs cross-reference the matching points; the roadmap lists the xce-node-manager revision. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
75 lines
3.4 KiB
Markdown
75 lines
3.4 KiB
Markdown
# Brief: elastic-hosts
|
|
|
|
(`/kiro-discovery`, 2026-10-10, "Auto resource provisioning".)
|
|
|
|
## Problem
|
|
Hosts created by a provider (e.g. cloud burst) must join XCE with the right
|
|
configuration, be recognized as provisioned, be drained before they are
|
|
given back, and disappear cleanly afterwards. Today XCE has no notion of a
|
|
host that is expected, provisioned or being released.
|
|
|
|
## Current State
|
|
- Any host can join: `find_matched_hostconfig()` (`controller/worker.c:26`)
|
|
falls back to the `hostname: default` entry of `hosts.yaml`;
|
|
`validate_host()` (`join.c:13`) accepts everyone.
|
|
- No named host templates beyond `default`; no provisioning or releasing
|
|
status; host control has only OPEN/CLOSE/TAG/UNTAG;
|
|
`XCE_WORKER_HIDDEN` is defined but never set; UNAVAIL hosts are
|
|
garbage-collected after `worker_retain_s`.
|
|
|
|
## Desired Outcome
|
|
- **Host templates** in cluster config (type, model, slots, resources,
|
|
host groups, dispatch windows) matched by provider templates, so a
|
|
provisioned host joins with its template's configuration.
|
|
- **Expected hosts:** when the provisioner gets an instance, the controller
|
|
records it (instance id, template, expected host name or address,
|
|
deadline); a joining host is matched to it and marked provisioned; an
|
|
instance that does not join by its deadline is reported for return.
|
|
- **Release:** for reclaim, the host is closed to new work, running jobs
|
|
finish (or, past a drain timeout, are handled by policy), then the host is
|
|
removed (decommissioned, no longer listed) and the provisioner returns the
|
|
instance.
|
|
- **Provider-initiated loss** (spot reclaim, instance failure) is handled as
|
|
host loss (`host-failure-job-recovery`).
|
|
- Admins can decommission any host (not only provisioned ones).
|
|
- Host states and provisioning details are visible (REST, `xcecli`).
|
|
|
|
## Approach
|
|
Controller-side host lifecycle extension with WAL facts for expected and
|
|
provisioned hosts; template-based host config; a decommission control.
|
|
|
|
## Scope
|
|
- **In**: host templates; expected-host records and matching; provisioned,
|
|
draining and decommissioned states; drain timeout policy; decommission
|
|
control; display; persistence.
|
|
- **Out**: choosing when and how many (`resource-provisioner`); the cloud
|
|
calls themselves; join authentication beyond matching (see
|
|
`peer-authentication-hardening`).
|
|
|
|
## Boundary Candidates
|
|
- Host template config.
|
|
- Expected-host records and join matching.
|
|
- Drain and decommission lifecycle.
|
|
|
|
## Out of Boundary
|
|
- Provider scripts and policy.
|
|
|
|
## Upstream / Downstream
|
|
- **Upstream**: `resource-provisioner`, `host-failure-job-recovery`,
|
|
worker join path, config pipeline.
|
|
- **Downstream**: future autoscaling of on-premise power management (power
|
|
hosts off/on) can reuse the same lifecycle.
|
|
|
|
## Existing Spec Touchpoints
|
|
- **Extends**: none.
|
|
- **Adjacent**: `worker-join-refactor`, `peer-authentication-hardening`,
|
|
`xce-node-manager`: a provisioned host boots with the Node Manager and the
|
|
`worker` role, declared by its host template and written into the Node
|
|
Manager's configuration by the instance's boot script; on decommission
|
|
the Node Manager stops the worker (or drops its role) instead of
|
|
restarting it, and the Controller refuses a rejoin from a decommissioned
|
|
host (`xce-node-manager` requirements.md, "Cross-Spec Interactions" 4).
|
|
|
|
## Constraints
|
|
- HA/persistence: full 3-phase workflow; the scheduler must see the same
|
|
host lifecycle in live and simulation.
|