docs(specs): discovery for auto resource provisioning

New briefs: provisioning-demand (the scheduler knows a provisionable
catalog, tries host templates as virtual hosts and license headroom after
real resources fail, finalizes demand and idle supply at session end;
simulator support is a hard requirement), resource-provisioner (new
daemon, several instances owning providers via controller leases,
self-describing providers, automatic policy, durable request/instance
state; src/autoscaler is prior art only), elastic-hosts (host templates,
expected hosts, drain and decommission), elastic-shared-resources
(borrowed license tokens change the shared total).

Roadmap: the new specs, plus updates to resource-distribution-mapping
(shared pools) and peer-authentication-hardening (provisioner role).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.kiro/specs/elastic-hosts/brief.md

@@ -0,0 +1,70 @@
+# Brief: elastic-hosts
+
+(`/kiro-discovery`, 2026-10-10, "Auto resource provisioning".)
+
+## Problem
+Hosts created by a provider (e.g. cloud burst) must join XCE with the right
+configuration, be recognized as provisioned, be drained before they are
+given back, and disappear cleanly afterwards. Today XCE has no notion of a
+host that is expected, provisioned or being released.
+
+## Current State
+- Any host can join: `find_matched_hostconfig()` (`controller/worker.c:26`)
+  falls back to the `hostname: default` entry of `hosts.yaml`;
+  `validate_host()` (`join.c:13`) accepts everyone.
+- No named host templates beyond `default`; no provisioning or releasing
+  status; host control has only OPEN/CLOSE/TAG/UNTAG;
+  `XCE_WORKER_HIDDEN` is defined but never set; UNAVAIL hosts are
+  garbage-collected after `worker_retain_s`.
+
+## Desired Outcome
+- **Host templates** in cluster config (type, model, slots, resources,
+  host groups, dispatch windows) matched by provider templates, so a
+  provisioned host joins with its template's configuration.
+- **Expected hosts:** when the provisioner gets an instance, the controller
+  records it (instance id, template, expected host name or address,
+  deadline); a joining host is matched to it and marked provisioned; an
+  instance that does not join by its deadline is reported for return.
+- **Release:** for reclaim, the host is closed to new work, running jobs
+  finish (or, past a drain timeout, are handled by policy), then the host is
+  removed (decommissioned, no longer listed) and the provisioner returns the
+  instance.
+- **Provider-initiated loss** (spot reclaim, instance failure) is handled as
+  host loss (`host-failure-job-recovery`).
+- Admins can decommission any host (not only provisioned ones).
+- Host states and provisioning details are visible (REST, `xcecli`).
+
+## Approach
+Controller-side host lifecycle extension with WAL facts for expected and
+provisioned hosts; template-based host config; a decommission control.
+
+## Scope
+- **In**: host templates; expected-host records and matching; provisioned,
+  draining and decommissioned states; drain timeout policy; decommission
+  control; display; persistence.
+- **Out**: choosing when and how many (`resource-provisioner`); the cloud
+  calls themselves; join authentication beyond matching (see
+  `peer-authentication-hardening`).
+
+## Boundary Candidates
+- Host template config.
+- Expected-host records and join matching.
+- Drain and decommission lifecycle.
+
+## Out of Boundary
+- Provider scripts and policy.
+
+## Upstream / Downstream
+- **Upstream**: `resource-provisioner`, `host-failure-job-recovery`,
+  worker join path, config pipeline.
+- **Downstream**: future autoscaling of on-premise power management (power
+  hosts off/on) can reuse the same lifecycle.
+
+## Existing Spec Touchpoints
+- **Extends**: none.
+- **Adjacent**: `worker-join-refactor`, `peer-authentication-hardening`,
+  `xce-node-manager`.
+
+## Constraints
+- HA/persistence: full 3-phase workflow; the scheduler must see the same
+  host lifecycle in live and simulation.

.kiro/specs/elastic-shared-resources/brief.md

@@ -0,0 +1,65 @@
+# Brief: elastic-shared-resources
+
+(`/kiro-discovery`, 2026-10-10, "Auto resource provisioning". Example: EDA
+license tokens borrowed from a shared license pool.)
+
+## Problem
+Cluster-wide countable resources such as EDA licenses are often shared with
+other sites or clusters. When jobs wait for licenses, XCE should borrow more
+tokens from the shared pool and return unused tokens when load is light, so
+the cluster's total follows its need.
+
+## Current State
+- Shared pooled resources work in the scheduler (`sharedInst` totals vs
+  `sharedInUse`, `resource.c:292`), and a total can change at runtime
+  (RESVALS_FULL events), but only resources flagged SHARED, and today only
+  limit virtual resources are: `resources.yaml` cannot declare a shared
+  pool (`xceadmin` maps only `consumable`/`release`), which is the open
+  `resource-distribution-mapping` decision.
+- Only a worker may update a shared value (`do_update_resvals`, `opupd.c:11`,
+  `HostRole_worker`).
+
+## Desired Outcome
+- A shared resource can be marked **elastic**, with its owned base amount
+  and the provider that can lend more.
+- The provisioner's grants and returns change the cluster's total for that
+  resource (base + borrowed), through a controller-authorized update by the
+  provisioner role; the change is durable and visible.
+- Borrowed amounts unused for a configured time (and past the provider's
+  minimum hold) are returned; amounts in use are never returned.
+- Optionally the observed in-use count from the real license server is
+  reported, so XCE's view does not drift from reality.
+- Users see the total, owned and borrowed amounts and current use.
+
+## Approach
+Build on the shared-pool scope of `resource-distribution-mapping`; add an
+elastic flag, a borrowed component of the total, and a provisioner update
+path through the controller.
+
+## Scope
+- **In**: elastic flag and config; borrowed component; provisioner update
+  path and its authorization; return rules; display; persistence.
+- **Out**: the license server protocol itself (provider scripts); demand
+  computation (`provisioning-demand`); provider policy
+  (`resource-provisioner`).
+
+## Boundary Candidates
+- Elastic shared resource model and config.
+- Total update path (controller, scheduler).
+- Return rules.
+
+## Out of Boundary
+- Talking to license servers.
+
+## Upstream / Downstream
+- **Upstream**: `resource-distribution-mapping` (shared pool declaration),
+  `resource-provisioner`.
+- **Downstream**: other countable provisioned resources.
+
+## Existing Spec Touchpoints
+- **Extends**: `resource-distribution-mapping` (needs its shared-pool scope).
+- **Adjacent**: `resource-value-verification`.
+
+## Constraints
+- Totals must be identical for live and simulated scheduling.
+- HA/persistence: full 3-phase workflow.

.kiro/specs/provisioning-demand/brief.md

@@ -0,0 +1,103 @@
+# Brief: provisioning-demand
+
+(`/kiro-discovery`, 2026-10-10, "Auto resource provisioning". LSF precedent:
+the resource connector, where mbschd evaluates pending jobs against provider
+templates. `src/autoscaler` is prior art only, not a required basis.)
+
+## Problem
+Admins want XCE to obtain more resources when work is waiting for them (burst
+hosts to a public cloud, borrow EDA license tokens from a shared pool) and to
+give them back when load is light. Deciding *how much* to ask for needs an
+exact view of which pending jobs could run if more of a resource existed,
+respecting queue order, limits and quotas. Only the scheduler has that view.
+
+## Current State
+- The scheduler records why each job pends (`PREASON_NO_RESOURCE` with host
+  and resource, `alloc_runner.c:106`, `resource.c:606`; deduplicated by
+  `add_job_preason_once`) and keeps an unsatisfiable-request cache per
+  session (`sched/alloc_cache.h`, bug-1932). Pend reasons are pushed to the
+  controller per job; nothing adds them up into demand.
+- Shared (pooled) resources are checked in `sched_check_shared_satisfy_()`
+  (`resource.c:292`) against `sharedInst` totals that can change at runtime.
+- Prior art: `src/autoscaler` has a `calculate()` (`scaler.c:391`: new =
+  demand − allocated − requesting, capped by template max) but no demand
+  source.
+- The controller forwards `hproviders` (scaler.yaml) to the scheduler in
+  `Conf4Sched`; the scheduler ignores it.
+
+## Desired Outcome
+- **Provisionable catalog** known to the scheduler (never the providers
+  themselves):
+  - host templates: attributes and resources of hosts a provider can create,
+    and how many more are allowed;
+  - countable resources (licenses): resource name and current headroom (how
+    much more could be obtained);
+  - supply in flight: hosts requested but not yet joined, counts requested
+    but not yet granted, so demand is not requested twice.
+  The static part comes from cluster config; runtime parts (headroom, in
+  flight) from the provisioner through the controller.
+- **Demand computation:** after real resources fail for a job, the scheduler
+  tries the catalog: templates as virtual hosts (would the job fit on N new
+  hosts of template T, under its resreq, span and limits?) and headroom for
+  counts (is the job blocked only by a shortfall the pool could cover?).
+  Placement on virtual supply reserves it for the rest of the session, so
+  demand respects job order and is not double counted.
+- **Finalized once per session (preferred design, user 2026-10-10):** demand
+  is aggregated and finalized in the scheduler's end-of-session handler: per
+  template, count of hosts; per countable resource, amount; with the jobs
+  that need it and their priority order. Then published to the controller
+  (in memory, replaced each session, not a WAL fact).
+- **Idle supply:** the same handler reports reclaim candidates: provisioned
+  hosts with no running jobs for a configured idle time, and borrowed counts
+  unused for a configured time.
+- **Visibility:** decision trace shows "would fit on template T" / "short
+  K of R, provisionable"; pend reasons say that a job waits for provisioning.
+- **Simulator (hard requirement, user 2026-10-10):** the scheduler simulator
+  computes the same demand and idle supply as the live scheduler for the
+  same input (driver equivalence), shows them in the REPL (e.g. `info
+  demand`), records them in the decision trace and recordings, and can
+  replay with simulated provider responses (hosts that join after a lead
+  time, counts granted or refused) to evaluate policies offline.
+
+## Approach
+Extend the allocation runner with a second, catalog-backed pass for jobs
+that fail on real resources, using virtual host objects built from templates
+and virtual headroom on shared resources; aggregate in the session-end
+handler; publish through the existing scheduler→controller channel.
+
+## Scope
+- **In**: catalog model and its config/runtime inputs; virtual-host and
+  headroom passes; reservation of virtual supply within a session; demand
+  and idle-supply aggregation at session end; publication to the
+  controller; trace, pend-reason wording; simulator support (computation,
+  REPL display, recording, simulated provider responses).
+- **Out**: talking to providers, policy (which provider, when, how much to
+  actually request), request lifecycle (`resource-provisioner`); joining
+  and releasing hosts (`elastic-hosts`); license pool totals
+  (`elastic-shared-resources`).
+
+## Boundary Candidates
+- Catalog model and inputs.
+- Catalog pass in the allocation runner.
+- Session-end aggregation and publication.
+- Simulator and REPL support.
+
+## Out of Boundary
+- Any provider call or policy decision.
+
+## Upstream / Downstream
+- **Upstream**: scheduler allocation runner, session lifecycle,
+  `parjob-placement` (span rules on virtual hosts), `resource-distribution-
+  mapping` (shared pools), driver equivalence.
+- **Downstream**: `resource-provisioner`, `elastic-hosts`,
+  `elastic-shared-resources`.
+
+## Existing Spec Touchpoints
+- **Extends**: none.
+- **Adjacent**: `scheduler-sim-decision-trace`, `scheduler-sim-diagnostics`,
+  `pend-reason-push-dedup`, `scheduler-preemption`.
+
+## Constraints
+- Live/simulator equivalence (`scheduler-driver-equivalence.md`).
+- Bounded cost per session: the catalog pass runs only for jobs that failed
+  on real resources, with a cap.

.kiro/specs/resource-provisioner/brief.md

@@ -0,0 +1,101 @@
+# Brief: resource-provisioner
+
+(`/kiro-discovery`, 2026-10-10, "Auto resource provisioning". Prior art:
+`src/autoscaler`, an unused port of the LSF resource connector's provider
+layer. It is a source of ideas only, not a required basis (user
+2026-10-10): the design decides whether any of its code or its script
+contract is reused.)
+
+## Problem
+Demand computed by the scheduler (`provisioning-demand`) must turn into real
+requests to resource providers (cloud APIs, license pool managers, others
+later), and borrowed resources must be given back, automatically, with as
+little admin involvement as possible while the cluster runs, and without
+ever losing track of something that costs money.
+
+## Current State
+- Prior art, `src/autoscaler` (static library, linked by nothing): provider registry
+  (`scaler.yaml` / `hostProviders.yaml`), a 5-script provider contract
+  (getAvailableTemplates, requestMachines, requestReturnMachines,
+  getRequestStatus, getReturnRequests; `plug_mgr.c` runs
+  `sh <script> -f <in.json>` and reads JSON), templates (`templateId`,
+  `maxNumber`, `priority`, attributes, price), request and instance state
+  machines, a monitor loop and a demand queue; state in memory only. One
+  provider: Baidu BCC (`providers/baidu`, Go `bcctool`).
+- `scaler.yaml` already flows through `xceadmin` (`hproviders`) to the
+  controller and scheduler (`Conf4Sched`), unused.
+
+## Desired Outcome
+- **Provisioner program, several instances:** different resources may need
+  different provisioners (user 2026-10-10): e.g. a cloud provisioner on a
+  host that holds cloud credentials, a license provisioner next to the
+  license server. Each instance declares the providers it serves; each
+  provider is owned by exactly one active instance at a time (lease granted
+  by the controller; a standby takes over when the lease lapses).
+- **Providers are generic** (hosts and countable resources first, other
+  kinds later) and **describe themselves** (user 2026-10-10) so policy can
+  run without manual steps: capacity (maximum hosts or units), cost (per
+  host-hour or unit), priority, lead time (how long a request takes),
+  granularity (minimum request size, step), lease or minimum hold time,
+  region/zone, and extensible attributes.
+- **Script contract:** JSON in/out, actions for both kinds (templates or
+  quote, request, request status, return, return status), timeouts, exit
+  codes, environment. The LSF-style 5-script contract of `src/autoscaler`
+  is one input to this design, not a constraint.
+- **Automatic policy** (no approval step in v1): from finalized demand and
+  provider descriptions, choose providers (priority, then cost), request
+  amounts (respecting capacity, granularity, per-provider and global
+  budgets, cooldowns), and reclaim idle supply after its idle time,
+  respecting minimum hold times. Requests that fail or time out are retried
+  or moved to the next provider.
+- **Durable state:** every request and every provisioned instance or grant
+  is a controller fact (WAL), so a provisioner restart or failover never
+  forgets a cloud instance; orphaned instances (requested, never joined)
+  are found and returned.
+- **Visibility:** REST and `xcecli` show providers, their descriptions and
+  usage, requests, instances, costs so far, and recent decisions; audit log.
+- **Testing:** a simulated provider (scripts or built-in) for CI and for the
+  scheduler simulator's offline evaluation.
+
+## Approach
+A new daemon (implementation language and code reuse decided in design;
+`src/autoscaler` may be reused, adapted or replaced), connected to the
+controller through libxce with its own role; reads finalized demand and idle
+supply, runs provider scripts off any HA path, and records state through
+controller facts.
+
+## Scope
+- **In**: daemon, multi-instance ownership via leases, provider registry and
+  self-description, script contract v2, policy engine (selection, sizing,
+  budgets, cooldowns, retries, reclaim), durable request/instance state,
+  orphan cleanup, REST/xcecli/audit, simulated provider; at least one real
+  host provider and one license provider as reference implementations
+  (whether the existing Baidu provider is one of them is a design choice).
+- **Out**: computing demand (`provisioning-demand`); host join/drain/
+  decommission mechanics (`elastic-hosts`); license pool totals in the
+  scheduler (`elastic-shared-resources`); approval workflow (later, if ever).
+
+## Boundary Candidates
+- Daemon and leases.
+- Provider registry, description and script contract.
+- Policy engine.
+- Durable request/instance facts in the controller.
+- Status surfaces.
+
+## Out of Boundary
+- Scheduling decisions.
+
+## Upstream / Downstream
+- **Upstream**: `provisioning-demand`, controller (facts, leases),
+  libxce, `peer-authentication-hardening` (provisioner role).
+- **Downstream**: `elastic-hosts`, `elastic-shared-resources`.
+
+## Existing Spec Touchpoints
+- **Extends**: none (`src/autoscaler` is prior art only).
+- **Adjacent**: `component-failover-adoption`, `admin-config-tool`
+  (provider config).
+
+## Constraints
+- Provider scripts never run in the controller or scheduler process.
+- Credentials stay with the provisioner instance and its scripts.
+- HA/persistence: full 3-phase workflow.

.kiro/steering/roadmap.md

@@ -78,6 +78,15 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
 - [ ] service-replica-set — NEW 2026-10-10 (Long-running services phase 2; user decision: no head agent, no promotion). Controller-held service object owning independent replica jobs; event-driven rules on commit (replace ended replicas with backoff, scale, rolling update with max-unavailable/surge, pause/resume/rollback); spread rule; REST/xcecli `services`. Brief: `.kiro/specs/service-replica-set/brief.md`. Dependencies: service-supervision, host-failure-job-recovery, application-profile.
 - [ ] service-endpoint — NEW 2026-10-10 (Long-running services phase 2). Named ports allocated per replica as host resources, host addresses reported by workers, endpoint = `host:port` of ready replicas from readiness changes, REST/xcecli queries; DNS/proxy later. Brief: `.kiro/specs/service-endpoint/brief.md`. Dependencies: service-supervision, service-replica-set.
 
+- [ ] provisioning-demand — NEW 2026-10-10 (`/kiro-discovery`, "Auto resource provisioning"; LSF resource-connector precedent). Scheduler knows a provisionable catalog (host templates, license headroom, supply in flight), tries it after real resources fail (templates as virtual hosts, headroom shortfall), finalizes demand and idle supply in the end-of-session handler (preferred design) and publishes it to the controller. **Simulator support is a hard requirement** (same demand as live, REPL display, recordings, simulated provider responses). Brief: `.kiro/specs/provisioning-demand/brief.md`. Dependencies: none (licenses need resource-distribution-mapping's shared pools).
+- [ ] resource-provisioner — NEW 2026-10-10 (Auto resource provisioning). New daemon (`src/autoscaler`, an LSF RC-style port, is prior art only, not a required basis): several instances, each owning providers via controller leases (cloud and license provisioners may differ); providers describe capacity, cost, priority, lead time, granularity, hold time; script contract v2 (hosts and counts); automatic policy (no approval), budgets, cooldowns, retries, idle reclaim; request/instance state as controller facts; orphan cleanup; REST/xcecli/audit; simulated provider. Brief: `.kiro/specs/resource-provisioner/brief.md`. Dependencies: provisioning-demand.
+- [ ] elastic-hosts — NEW 2026-10-10 (Auto resource provisioning). Host templates in config, expected-host records matched at join, provisioned/draining/decommissioned states, drain-then-release, decommission control, spot loss as host loss. Brief: `.kiro/specs/elastic-hosts/brief.md`. Dependencies: resource-provisioner, host-failure-job-recovery.
+- [ ] elastic-shared-resources — NEW 2026-10-10 (Auto resource provisioning). Elastic shared resources (e.g. EDA licenses): base + borrowed total updated by the provisioner role through the controller, unused borrowed amounts returned, display. Brief: `.kiro/specs/elastic-shared-resources/brief.md`. Dependencies: resource-provisioner, resource-distribution-mapping.
+
+## Existing Spec Updates (2026-10-10 auto resource provisioning discovery)
+- [ ] resource-distribution-mapping — choose its shared-pool scope (option 2) so a cluster-wide pool (licenses) can be declared in `resources.yaml`; prerequisite of elastic-shared-resources.
+- [ ] peer-authentication-hardening — a provisioner role with its allowed operations (provisioning facts, shared-total updates, expected hosts).
+
 ## Existing Spec Updates (2026-10-10 long-running services discovery)
 - [ ] application-profile — profile kind `service` section: restart policy, probes, ports, default replicas, spread rule, rollout settings (consumed by service-supervision, service-replica-set, service-endpoint).
 - [ ] job-config-merge — service attributes follow the ownership rule (profile owns them; the job may override only `default` ones, e.g. replica count).