Rongsong Shen
922188b5d7
New briefs: provisioning-demand (the scheduler knows a provisionable catalog, tries host templates as virtual hosts and license headroom after real resources fail, finalizes demand and idle supply at session end; simulator support is a hard requirement), resource-provisioner (new daemon, several instances owning providers via controller leases, self-describing providers, automatic policy, durable request/instance state; src/autoscaler is prior art only), elastic-hosts (host templates, expected hosts, drain and decommission), elastic-shared-resources (borrowed license tokens change the shared total). Roadmap: the new specs, plus updates to resource-distribution-mapping (shared pools) and peer-authentication-hardening (provisioner role). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
103 lines
5.3 KiB
Markdown
103 lines
5.3 KiB
Markdown
# Brief: provisioning-demand
|
||
|
||
(`/kiro-discovery`, 2026-10-10, "Auto resource provisioning". LSF precedent:
|
||
the resource connector, where mbschd evaluates pending jobs against provider
|
||
templates. `src/autoscaler` is prior art only, not a required basis.)
|
||
|
||
## Problem
|
||
Admins want XCE to obtain more resources when work is waiting for them (burst
|
||
hosts to a public cloud, borrow EDA license tokens from a shared pool) and to
|
||
give them back when load is light. Deciding *how much* to ask for needs an
|
||
exact view of which pending jobs could run if more of a resource existed,
|
||
respecting queue order, limits and quotas. Only the scheduler has that view.
|
||
|
||
## Current State
|
||
- The scheduler records why each job pends (`PREASON_NO_RESOURCE` with host
|
||
and resource, `alloc_runner.c:106`, `resource.c:606`; deduplicated by
|
||
`add_job_preason_once`) and keeps an unsatisfiable-request cache per
|
||
session (`sched/alloc_cache.h`, bug-1932). Pend reasons are pushed to the
|
||
controller per job; nothing adds them up into demand.
|
||
- Shared (pooled) resources are checked in `sched_check_shared_satisfy_()`
|
||
(`resource.c:292`) against `sharedInst` totals that can change at runtime.
|
||
- Prior art: `src/autoscaler` has a `calculate()` (`scaler.c:391`: new =
|
||
demand − allocated − requesting, capped by template max) but no demand
|
||
source.
|
||
- The controller forwards `hproviders` (scaler.yaml) to the scheduler in
|
||
`Conf4Sched`; the scheduler ignores it.
|
||
|
||
## Desired Outcome
|
||
- **Provisionable catalog** known to the scheduler (never the providers
|
||
themselves):
|
||
- host templates: attributes and resources of hosts a provider can create,
|
||
and how many more are allowed;
|
||
- countable resources (licenses): resource name and current headroom (how
|
||
much more could be obtained);
|
||
- supply in flight: hosts requested but not yet joined, counts requested
|
||
but not yet granted, so demand is not requested twice.
|
||
The static part comes from cluster config; runtime parts (headroom, in
|
||
flight) from the provisioner through the controller.
|
||
- **Demand computation:** after real resources fail for a job, the scheduler
|
||
tries the catalog: templates as virtual hosts (would the job fit on N new
|
||
hosts of template T, under its resreq, span and limits?) and headroom for
|
||
counts (is the job blocked only by a shortfall the pool could cover?).
|
||
Placement on virtual supply reserves it for the rest of the session, so
|
||
demand respects job order and is not double counted.
|
||
- **Finalized once per session (preferred design, user 2026-10-10):** demand
|
||
is aggregated and finalized in the scheduler's end-of-session handler: per
|
||
template, count of hosts; per countable resource, amount; with the jobs
|
||
that need it and their priority order. Then published to the controller
|
||
(in memory, replaced each session, not a WAL fact).
|
||
- **Idle supply:** the same handler reports reclaim candidates: provisioned
|
||
hosts with no running jobs for a configured idle time, and borrowed counts
|
||
unused for a configured time.
|
||
- **Visibility:** decision trace shows "would fit on template T" / "short
|
||
K of R, provisionable"; pend reasons say that a job waits for provisioning.
|
||
- **Simulator (hard requirement, user 2026-10-10):** the scheduler simulator
|
||
computes the same demand and idle supply as the live scheduler for the
|
||
same input (driver equivalence), shows them in the REPL (e.g. `info
|
||
demand`), records them in the decision trace and recordings, and can
|
||
replay with simulated provider responses (hosts that join after a lead
|
||
time, counts granted or refused) to evaluate policies offline.
|
||
|
||
## Approach
|
||
Extend the allocation runner with a second, catalog-backed pass for jobs
|
||
that fail on real resources, using virtual host objects built from templates
|
||
and virtual headroom on shared resources; aggregate in the session-end
|
||
handler; publish through the existing scheduler→controller channel.
|
||
|
||
## Scope
|
||
- **In**: catalog model and its config/runtime inputs; virtual-host and
|
||
headroom passes; reservation of virtual supply within a session; demand
|
||
and idle-supply aggregation at session end; publication to the
|
||
controller; trace, pend-reason wording; simulator support (computation,
|
||
REPL display, recording, simulated provider responses).
|
||
- **Out**: talking to providers, policy (which provider, when, how much to
|
||
actually request), request lifecycle (`resource-provisioner`); joining
|
||
and releasing hosts (`elastic-hosts`); license pool totals
|
||
(`elastic-shared-resources`).
|
||
|
||
## Boundary Candidates
|
||
- Catalog model and inputs.
|
||
- Catalog pass in the allocation runner.
|
||
- Session-end aggregation and publication.
|
||
- Simulator and REPL support.
|
||
|
||
## Out of Boundary
|
||
- Any provider call or policy decision.
|
||
|
||
## Upstream / Downstream
|
||
- **Upstream**: scheduler allocation runner, session lifecycle,
|
||
`parjob-placement` (span rules on virtual hosts), `resource-distribution-
|
||
mapping` (shared pools), driver equivalence.
|
||
- **Downstream**: `resource-provisioner`, `elastic-hosts`,
|
||
`elastic-shared-resources`.
|
||
|
||
## Existing Spec Touchpoints
|
||
- **Extends**: none.
|
||
- **Adjacent**: `scheduler-sim-decision-trace`, `scheduler-sim-diagnostics`,
|
||
`pend-reason-push-dedup`, `scheduler-preemption`.
|
||
|
||
## Constraints
|
||
- Live/simulator equivalence (`scheduler-driver-equivalence.md`).
|
||
- Bounded cost per session: the catalog pass runs only for jobs that failed
|
||
on real resources, with a cap.
|