# Brief: provisioning-demand

(`/kiro-discovery`, 2026-10-10, "Auto resource provisioning". LSF precedent:
the resource connector, where mbschd evaluates pending jobs against provider
templates. `src/autoscaler` is prior art only, not a required basis.)

## Problem
Admins want XCE to obtain more resources when work is waiting for them (burst
hosts to a public cloud, borrow EDA license tokens from a shared pool) and to
give them back when load is light. Deciding *how much* to ask for needs an
exact view of which pending jobs could run if more of a resource existed,
respecting queue order, limits and quotas. Only the scheduler has that view.

## Current State
- The scheduler records why each job pends (`PREASON_NO_RESOURCE` with host
  and resource, `alloc_runner.c:106`, `resource.c:606`; deduplicated by
  `add_job_preason_once`) and keeps an unsatisfiable-request cache per
  session (`sched/alloc_cache.h`, bug-1932). Pend reasons are pushed to the
  controller per job; nothing adds them up into demand.
- Shared (pooled) resources are checked in `sched_check_shared_satisfy_()`
  (`resource.c:292`) against `sharedInst` totals that can change at runtime.
- Prior art: `src/autoscaler` has a `calculate()` (`scaler.c:391`: new =
  demand − allocated − requesting, capped by template max) but no demand
  source.
- The controller forwards `hproviders` (scaler.yaml) to the scheduler in
  `Conf4Sched`; the scheduler ignores it.

## Desired Outcome
- **Provisionable catalog** known to the scheduler (never the providers
  themselves):
  - host templates: attributes and resources of hosts a provider can create,
    and how many more are allowed;
  - countable resources (licenses): resource name and current headroom (how
    much more could be obtained);
  - supply in flight: hosts requested but not yet joined, counts requested
    but not yet granted, so demand is not requested twice.
  The static part comes from cluster config; runtime parts (headroom, in
  flight) from the provisioner through the controller.
- **Demand computation:** after real resources fail for a job, the scheduler
  tries the catalog: templates as virtual hosts (would the job fit on N new
  hosts of template T, under its resreq, span and limits?) and headroom for
  counts (is the job blocked only by a shortfall the pool could cover?).
  Placement on virtual supply reserves it for the rest of the session, so
  demand respects job order and is not double counted.
- **Finalized once per session (preferred design, user 2026-10-10):** demand
  is aggregated and finalized in the scheduler's end-of-session handler: per
  template, count of hosts; per countable resource, amount; with the jobs
  that need it and their priority order. Then published to the controller
  (in memory, replaced each session, not a WAL fact).
- **Idle supply:** the same handler reports reclaim candidates: provisioned
  hosts with no running jobs for a configured idle time, and borrowed counts
  unused for a configured time.
- **Visibility:** decision trace shows "would fit on template T" / "short
  K of R, provisionable"; pend reasons say that a job waits for provisioning.
- **Simulator (hard requirement, user 2026-10-10):** the scheduler simulator
  computes the same demand and idle supply as the live scheduler for the
  same input (driver equivalence), shows them in the REPL (e.g. `info
  demand`), records them in the decision trace and recordings, and can
  replay with simulated provider responses (hosts that join after a lead
  time, counts granted or refused) to evaluate policies offline.

## Approach
Extend the allocation runner with a second, catalog-backed pass for jobs
that fail on real resources, using virtual host objects built from templates
and virtual headroom on shared resources; aggregate in the session-end
handler; publish through the existing scheduler→controller channel.

## Scope
- **In**: catalog model and its config/runtime inputs; virtual-host and
  headroom passes; reservation of virtual supply within a session; demand
  and idle-supply aggregation at session end; publication to the
  controller; trace, pend-reason wording; simulator support (computation,
  REPL display, recording, simulated provider responses).
- **Out**: talking to providers, policy (which provider, when, how much to
  actually request), request lifecycle (`resource-provisioner`); joining
  and releasing hosts (`elastic-hosts`); license pool totals
  (`elastic-shared-resources`).

## Boundary Candidates
- Catalog model and inputs.
- Catalog pass in the allocation runner.
- Session-end aggregation and publication.
- Simulator and REPL support.

## Out of Boundary
- Any provider call or policy decision.

## Upstream / Downstream
- **Upstream**: scheduler allocation runner, session lifecycle,
  `parjob-placement` (span rules on virtual hosts), `resource-distribution-
  mapping` (shared pools), driver equivalence.
- **Downstream**: `resource-provisioner`, `elastic-hosts`,
  `elastic-shared-resources`.

## Existing Spec Touchpoints
- **Extends**: none.
- **Adjacent**: `scheduler-sim-decision-trace`, `scheduler-sim-diagnostics`,
  `pend-reason-push-dedup`, `scheduler-preemption`.

## Constraints
- Live/simulator equivalence (`scheduler-driver-equivalence.md`).
- Bounded cost per session: the catalog pass runs only for jobs that failed
  on real resources, with a cap.