myxce/.kiro/specs/provisioning-demand/brief.md

Brief: provisioning-demand

(/kiro-discovery, 2026-10-10, "Auto resource provisioning". LSF precedent: the resource connector, where mbschd evaluates pending jobs against provider templates. src/autoscaler is prior art only, not a required basis.)

Problem

Admins want XCE to obtain more resources when work is waiting for them (burst hosts to a public cloud, borrow EDA license tokens from a shared pool) and to give them back when load is light. Deciding how much to ask for needs an exact view of which pending jobs could run if more of a resource existed, respecting queue order, limits and quotas. Only the scheduler has that view.

Current State

• The scheduler records why each job pends (PREASON_NO_RESOURCE with host and resource, alloc_runner.c:106, resource.c:606; deduplicated by add_job_preason_once) and keeps an unsatisfiable-request cache per session (sched/alloc_cache.h, bug-1932). Pend reasons are pushed to the controller per job; nothing adds them up into demand.
• Shared (pooled) resources are checked in sched_check_shared_satisfy_() (resource.c:292) against sharedInst totals that can change at runtime.
• Prior art: src/autoscaler has a calculate() (scaler.c:391: new = demand − allocated − requesting, capped by template max) but no demand source.
• The controller forwards hproviders (scaler.yaml) to the scheduler in Conf4Sched; the scheduler ignores it.

Desired Outcome

• Provisionable catalog known to the scheduler (never the providers themselves):
  • host templates: attributes and resources of hosts a provider can create, and how many more are allowed;
  • countable resources (licenses): resource name and current headroom (how much more could be obtained);
  • supply in flight: hosts requested but not yet joined, counts requested but not yet granted, so demand is not requested twice. The static part comes from cluster config; runtime parts (headroom, in flight) from the provisioner through the controller.
• Demand computation: after real resources fail for a job, the scheduler tries the catalog: templates as virtual hosts (would the job fit on N new hosts of template T, under its resreq, span and limits?) and headroom for counts (is the job blocked only by a shortfall the pool could cover?). Placement on virtual supply reserves it for the rest of the session, so demand respects job order and is not double counted.
• Finalized once per session (preferred design, user 2026-10-10): demand is aggregated and finalized in the scheduler's end-of-session handler: per template, count of hosts; per countable resource, amount; with the jobs that need it and their priority order. Then published to the controller (in memory, replaced each session, not a WAL fact).
• Idle supply: the same handler reports reclaim candidates: provisioned hosts with no running jobs for a configured idle time, and borrowed counts unused for a configured time.
• Visibility: decision trace shows "would fit on template T" / "short K of R, provisionable"; pend reasons say that a job waits for provisioning.
• Simulator (hard requirement, user 2026-10-10): the scheduler simulator computes the same demand and idle supply as the live scheduler for the same input (driver equivalence), shows them in the REPL (e.g. info demand), records them in the decision trace and recordings, and can replay with simulated provider responses (hosts that join after a lead time, counts granted or refused) to evaluate policies offline.

Approach

Extend the allocation runner with a second, catalog-backed pass for jobs that fail on real resources, using virtual host objects built from templates and virtual headroom on shared resources; aggregate in the session-end handler; publish through the existing scheduler→controller channel.

Scope

• In: catalog model and its config/runtime inputs; virtual-host and headroom passes; reservation of virtual supply within a session; demand and idle-supply aggregation at session end; publication to the controller; trace, pend-reason wording; simulator support (computation, REPL display, recording, simulated provider responses).
• Out: talking to providers, policy (which provider, when, how much to actually request), request lifecycle (resource-provisioner); joining and releasing hosts (elastic-hosts); license pool totals (elastic-shared-resources).

Boundary Candidates

• Catalog model and inputs.
• Catalog pass in the allocation runner.
• Session-end aggregation and publication.
• Simulator and REPL support.

Out of Boundary

• Any provider call or policy decision.

Upstream / Downstream

• Upstream: scheduler allocation runner, session lifecycle, parjob-placement (span rules on virtual hosts), resource-distribution- mapping (shared pools), driver equivalence.
• Downstream: resource-provisioner, elastic-hosts, elastic-shared-resources.

Existing Spec Touchpoints

• Extends: none.
• Adjacent: scheduler-sim-decision-trace, scheduler-sim-diagnostics, pend-reason-push-dedup, scheduler-preemption.

Constraints

• Live/simulator equivalence (scheduler-driver-equivalence.md).
• Bounded cost per session: the catalog pass runs only for jobs that failed on real resources, with a cap.