From 922188b5d798cd9eb74714a33316c5ccbe549328 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sat, 10 Oct 2026 07:37:25 +0000 Subject: [PATCH] docs(specs): discovery for auto resource provisioning New briefs: provisioning-demand (the scheduler knows a provisionable catalog, tries host templates as virtual hosts and license headroom after real resources fail, finalizes demand and idle supply at session end; simulator support is a hard requirement), resource-provisioner (new daemon, several instances owning providers via controller leases, self-describing providers, automatic policy, durable request/instance state; src/autoscaler is prior art only), elastic-hosts (host templates, expected hosts, drain and decommission), elastic-shared-resources (borrowed license tokens change the shared total). Roadmap: the new specs, plus updates to resource-distribution-mapping (shared pools) and peer-authentication-hardening (provisioner role). Co-Authored-By: Claude Opus 5.5 (1M context) --- .kiro/specs/elastic-hosts/brief.md | 70 ++++++++++++ .kiro/specs/elastic-shared-resources/brief.md | 65 +++++++++++ .kiro/specs/provisioning-demand/brief.md | 103 ++++++++++++++++++ .kiro/specs/resource-provisioner/brief.md | 101 +++++++++++++++++ .kiro/steering/roadmap.md | 9 ++ 5 files changed, 348 insertions(+) create mode 100644 .kiro/specs/elastic-hosts/brief.md create mode 100644 .kiro/specs/elastic-shared-resources/brief.md create mode 100644 .kiro/specs/provisioning-demand/brief.md create mode 100644 .kiro/specs/resource-provisioner/brief.md diff --git a/.kiro/specs/elastic-hosts/brief.md b/.kiro/specs/elastic-hosts/brief.md new file mode 100644 index 00000000..774c46db --- /dev/null +++ b/.kiro/specs/elastic-hosts/brief.md @@ -0,0 +1,70 @@ +# Brief: elastic-hosts + +(`/kiro-discovery`, 2026-10-10, "Auto resource provisioning".) + +## Problem +Hosts created by a provider (e.g. cloud burst) must join XCE with the right +configuration, be recognized as provisioned, be drained before they are +given back, and disappear cleanly afterwards. Today XCE has no notion of a +host that is expected, provisioned or being released. + +## Current State +- Any host can join: `find_matched_hostconfig()` (`controller/worker.c:26`) + falls back to the `hostname: default` entry of `hosts.yaml`; + `validate_host()` (`join.c:13`) accepts everyone. +- No named host templates beyond `default`; no provisioning or releasing + status; host control has only OPEN/CLOSE/TAG/UNTAG; + `XCE_WORKER_HIDDEN` is defined but never set; UNAVAIL hosts are + garbage-collected after `worker_retain_s`. + +## Desired Outcome +- **Host templates** in cluster config (type, model, slots, resources, + host groups, dispatch windows) matched by provider templates, so a + provisioned host joins with its template's configuration. +- **Expected hosts:** when the provisioner gets an instance, the controller + records it (instance id, template, expected host name or address, + deadline); a joining host is matched to it and marked provisioned; an + instance that does not join by its deadline is reported for return. +- **Release:** for reclaim, the host is closed to new work, running jobs + finish (or, past a drain timeout, are handled by policy), then the host is + removed (decommissioned, no longer listed) and the provisioner returns the + instance. +- **Provider-initiated loss** (spot reclaim, instance failure) is handled as + host loss (`host-failure-job-recovery`). +- Admins can decommission any host (not only provisioned ones). +- Host states and provisioning details are visible (REST, `xcecli`). + +## Approach +Controller-side host lifecycle extension with WAL facts for expected and +provisioned hosts; template-based host config; a decommission control. + +## Scope +- **In**: host templates; expected-host records and matching; provisioned, + draining and decommissioned states; drain timeout policy; decommission + control; display; persistence. +- **Out**: choosing when and how many (`resource-provisioner`); the cloud + calls themselves; join authentication beyond matching (see + `peer-authentication-hardening`). + +## Boundary Candidates +- Host template config. +- Expected-host records and join matching. +- Drain and decommission lifecycle. + +## Out of Boundary +- Provider scripts and policy. + +## Upstream / Downstream +- **Upstream**: `resource-provisioner`, `host-failure-job-recovery`, + worker join path, config pipeline. +- **Downstream**: future autoscaling of on-premise power management (power + hosts off/on) can reuse the same lifecycle. + +## Existing Spec Touchpoints +- **Extends**: none. +- **Adjacent**: `worker-join-refactor`, `peer-authentication-hardening`, + `xce-node-manager`. + +## Constraints +- HA/persistence: full 3-phase workflow; the scheduler must see the same + host lifecycle in live and simulation. diff --git a/.kiro/specs/elastic-shared-resources/brief.md b/.kiro/specs/elastic-shared-resources/brief.md new file mode 100644 index 00000000..4c5f64a2 --- /dev/null +++ b/.kiro/specs/elastic-shared-resources/brief.md @@ -0,0 +1,65 @@ +# Brief: elastic-shared-resources + +(`/kiro-discovery`, 2026-10-10, "Auto resource provisioning". Example: EDA +license tokens borrowed from a shared license pool.) + +## Problem +Cluster-wide countable resources such as EDA licenses are often shared with +other sites or clusters. When jobs wait for licenses, XCE should borrow more +tokens from the shared pool and return unused tokens when load is light, so +the cluster's total follows its need. + +## Current State +- Shared pooled resources work in the scheduler (`sharedInst` totals vs + `sharedInUse`, `resource.c:292`), and a total can change at runtime + (RESVALS_FULL events), but only resources flagged SHARED, and today only + limit virtual resources are: `resources.yaml` cannot declare a shared + pool (`xceadmin` maps only `consumable`/`release`), which is the open + `resource-distribution-mapping` decision. +- Only a worker may update a shared value (`do_update_resvals`, `opupd.c:11`, + `HostRole_worker`). + +## Desired Outcome +- A shared resource can be marked **elastic**, with its owned base amount + and the provider that can lend more. +- The provisioner's grants and returns change the cluster's total for that + resource (base + borrowed), through a controller-authorized update by the + provisioner role; the change is durable and visible. +- Borrowed amounts unused for a configured time (and past the provider's + minimum hold) are returned; amounts in use are never returned. +- Optionally the observed in-use count from the real license server is + reported, so XCE's view does not drift from reality. +- Users see the total, owned and borrowed amounts and current use. + +## Approach +Build on the shared-pool scope of `resource-distribution-mapping`; add an +elastic flag, a borrowed component of the total, and a provisioner update +path through the controller. + +## Scope +- **In**: elastic flag and config; borrowed component; provisioner update + path and its authorization; return rules; display; persistence. +- **Out**: the license server protocol itself (provider scripts); demand + computation (`provisioning-demand`); provider policy + (`resource-provisioner`). + +## Boundary Candidates +- Elastic shared resource model and config. +- Total update path (controller, scheduler). +- Return rules. + +## Out of Boundary +- Talking to license servers. + +## Upstream / Downstream +- **Upstream**: `resource-distribution-mapping` (shared pool declaration), + `resource-provisioner`. +- **Downstream**: other countable provisioned resources. + +## Existing Spec Touchpoints +- **Extends**: `resource-distribution-mapping` (needs its shared-pool scope). +- **Adjacent**: `resource-value-verification`. + +## Constraints +- Totals must be identical for live and simulated scheduling. +- HA/persistence: full 3-phase workflow. diff --git a/.kiro/specs/provisioning-demand/brief.md b/.kiro/specs/provisioning-demand/brief.md new file mode 100644 index 00000000..88c30514 --- /dev/null +++ b/.kiro/specs/provisioning-demand/brief.md @@ -0,0 +1,103 @@ +# Brief: provisioning-demand + +(`/kiro-discovery`, 2026-10-10, "Auto resource provisioning". LSF precedent: +the resource connector, where mbschd evaluates pending jobs against provider +templates. `src/autoscaler` is prior art only, not a required basis.) + +## Problem +Admins want XCE to obtain more resources when work is waiting for them (burst +hosts to a public cloud, borrow EDA license tokens from a shared pool) and to +give them back when load is light. Deciding *how much* to ask for needs an +exact view of which pending jobs could run if more of a resource existed, +respecting queue order, limits and quotas. Only the scheduler has that view. + +## Current State +- The scheduler records why each job pends (`PREASON_NO_RESOURCE` with host + and resource, `alloc_runner.c:106`, `resource.c:606`; deduplicated by + `add_job_preason_once`) and keeps an unsatisfiable-request cache per + session (`sched/alloc_cache.h`, bug-1932). Pend reasons are pushed to the + controller per job; nothing adds them up into demand. +- Shared (pooled) resources are checked in `sched_check_shared_satisfy_()` + (`resource.c:292`) against `sharedInst` totals that can change at runtime. +- Prior art: `src/autoscaler` has a `calculate()` (`scaler.c:391`: new = + demand − allocated − requesting, capped by template max) but no demand + source. +- The controller forwards `hproviders` (scaler.yaml) to the scheduler in + `Conf4Sched`; the scheduler ignores it. + +## Desired Outcome +- **Provisionable catalog** known to the scheduler (never the providers + themselves): + - host templates: attributes and resources of hosts a provider can create, + and how many more are allowed; + - countable resources (licenses): resource name and current headroom (how + much more could be obtained); + - supply in flight: hosts requested but not yet joined, counts requested + but not yet granted, so demand is not requested twice. + The static part comes from cluster config; runtime parts (headroom, in + flight) from the provisioner through the controller. +- **Demand computation:** after real resources fail for a job, the scheduler + tries the catalog: templates as virtual hosts (would the job fit on N new + hosts of template T, under its resreq, span and limits?) and headroom for + counts (is the job blocked only by a shortfall the pool could cover?). + Placement on virtual supply reserves it for the rest of the session, so + demand respects job order and is not double counted. +- **Finalized once per session (preferred design, user 2026-10-10):** demand + is aggregated and finalized in the scheduler's end-of-session handler: per + template, count of hosts; per countable resource, amount; with the jobs + that need it and their priority order. Then published to the controller + (in memory, replaced each session, not a WAL fact). +- **Idle supply:** the same handler reports reclaim candidates: provisioned + hosts with no running jobs for a configured idle time, and borrowed counts + unused for a configured time. +- **Visibility:** decision trace shows "would fit on template T" / "short + K of R, provisionable"; pend reasons say that a job waits for provisioning. +- **Simulator (hard requirement, user 2026-10-10):** the scheduler simulator + computes the same demand and idle supply as the live scheduler for the + same input (driver equivalence), shows them in the REPL (e.g. `info + demand`), records them in the decision trace and recordings, and can + replay with simulated provider responses (hosts that join after a lead + time, counts granted or refused) to evaluate policies offline. + +## Approach +Extend the allocation runner with a second, catalog-backed pass for jobs +that fail on real resources, using virtual host objects built from templates +and virtual headroom on shared resources; aggregate in the session-end +handler; publish through the existing scheduler→controller channel. + +## Scope +- **In**: catalog model and its config/runtime inputs; virtual-host and + headroom passes; reservation of virtual supply within a session; demand + and idle-supply aggregation at session end; publication to the + controller; trace, pend-reason wording; simulator support (computation, + REPL display, recording, simulated provider responses). +- **Out**: talking to providers, policy (which provider, when, how much to + actually request), request lifecycle (`resource-provisioner`); joining + and releasing hosts (`elastic-hosts`); license pool totals + (`elastic-shared-resources`). + +## Boundary Candidates +- Catalog model and inputs. +- Catalog pass in the allocation runner. +- Session-end aggregation and publication. +- Simulator and REPL support. + +## Out of Boundary +- Any provider call or policy decision. + +## Upstream / Downstream +- **Upstream**: scheduler allocation runner, session lifecycle, + `parjob-placement` (span rules on virtual hosts), `resource-distribution- + mapping` (shared pools), driver equivalence. +- **Downstream**: `resource-provisioner`, `elastic-hosts`, + `elastic-shared-resources`. + +## Existing Spec Touchpoints +- **Extends**: none. +- **Adjacent**: `scheduler-sim-decision-trace`, `scheduler-sim-diagnostics`, + `pend-reason-push-dedup`, `scheduler-preemption`. + +## Constraints +- Live/simulator equivalence (`scheduler-driver-equivalence.md`). +- Bounded cost per session: the catalog pass runs only for jobs that failed + on real resources, with a cap. diff --git a/.kiro/specs/resource-provisioner/brief.md b/.kiro/specs/resource-provisioner/brief.md new file mode 100644 index 00000000..7117251d --- /dev/null +++ b/.kiro/specs/resource-provisioner/brief.md @@ -0,0 +1,101 @@ +# Brief: resource-provisioner + +(`/kiro-discovery`, 2026-10-10, "Auto resource provisioning". Prior art: +`src/autoscaler`, an unused port of the LSF resource connector's provider +layer. It is a source of ideas only, not a required basis (user +2026-10-10): the design decides whether any of its code or its script +contract is reused.) + +## Problem +Demand computed by the scheduler (`provisioning-demand`) must turn into real +requests to resource providers (cloud APIs, license pool managers, others +later), and borrowed resources must be given back, automatically, with as +little admin involvement as possible while the cluster runs, and without +ever losing track of something that costs money. + +## Current State +- Prior art, `src/autoscaler` (static library, linked by nothing): provider registry + (`scaler.yaml` / `hostProviders.yaml`), a 5-script provider contract + (getAvailableTemplates, requestMachines, requestReturnMachines, + getRequestStatus, getReturnRequests; `plug_mgr.c` runs + `sh