Rongsong Shen
c14fe3dab5
resource-model.md: one resource type descriptor and provider contract over three layers (L1 attribute, L2 counter, L3 instance/claim); the scheduler only selects from its cached snapshot, providers run after dispatch; batch stays LSF-style; template and standalone (PVC/ResourceClaim-like) claims as WAL entities with the resource-provisioner discipline; LSF/K8s mapping; LLM training/serving needs; phases; nine pending user decisions. New briefs: custom-resource-reporting, resource-instances, resource-claims, gang-groups. Roadmap: long-term LLM training + serving platform direction (features from LSF and Kubernetes), the four new specs, updates to ten existing specs, discovery note. resource-distribution-mapping becomes phase 1 (shared pools + descriptor fields). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
66 lines
4.3 KiB
Markdown
66 lines
4.3 KiB
Markdown
# Brief: custom-resource-reporting
|
|
|
|
(Resource-model discovery, 2026-10-11. Phase 2 of `.kiro/steering/resource-model.md`, which is `resource-distribution-mapping` option 3, split out into its own spec. Pending open decisions 1 and 2 in that document: whether this stays a separate spec, and who reports values.)
|
|
|
|
## Problem
|
|
Admins can declare custom resources, but only with static per-host values. A value that changes at runtime has no way in: free space in a scratch filesystem, a dataset present in a node's cache, the number of free licenses on a vendor server, a site-specific health flag. LSF solves this with elim, PBS with server and mom hooks, Kubernetes with device plugins and DRA ResourceSlices.
|
|
|
|
## Current State
|
|
- **Built-in values only:** the worker reports 11 compiled-in load indices (`src/worker/src/resource_sync.c:84-104`). `resource_sync_register_load_index()` (`:116`) has no external callers, and there is no `dlopen` or script path.
|
|
- **Worker-only updates:** `do_update_resvals()` accepts updates only from a `HostRole_worker` connection (`src/controller/opupd.c:11-26`).
|
|
- **Unused descriptor fields:** `resources.yaml` `interval` is never enforced, and `last_report` is never used (`resource-distribution-mapping/research.md:17`).
|
|
- **No set type:** "has datasets {a,b,c}" needs one boolean resource per dataset.
|
|
|
|
## Desired Outcome
|
|
- **Reporters:** a custom resource names a reporter in its type descriptor. The reporter is either a script run on each worker (elim-like, started with `posix_spawn`, emitting JSON values each `interval`) or an external reporter role connected to the controller, for cluster-scope values such as a license server's free count.
|
|
- **Value lifecycle:**
|
|
- values are checked against the declared kind;
|
|
- a value not refreshed within its interval turns stale and is handled per resource: unknown, or excluded from `select[]`;
|
|
- a reporter's failure is visible to admins.
|
|
- **New kind:** string-set, so `select[datasets has "imagenet"]` can express locality.
|
|
- **Inventory:** a reporter can also publish instance inventory with attributes. That is the input `resource-instances` consumes, shaped like a DRA ResourceSlice.
|
|
- **Scheduler:** the scheduler sees reported values exactly like built-ins, from its cached snapshot, with no new work in the session hot path.
|
|
|
|
## Approach
|
|
- Extend the worker's `resource_sync` with a script reporter, under the no-fork rule (`tech.md` "Process Creation").
|
|
- Add a reporter role (or reuse the provisioner role) with an allowlisted update opcode next to `do_update_resvals`.
|
|
- The wire format is additive, and values round-trip through the scheduler snapshot.
|
|
|
|
## Scope
|
|
- **In:**
|
|
- script reporters on workers and an external reporter role;
|
|
- staleness and interval;
|
|
- the string-set kind and its `select[]` operators;
|
|
- inventory publishing (format only; its consumers are in `resource-instances`);
|
|
- xcecli/REST display of reported values and reporter health;
|
|
- a simulated reporter for the scheduler simulator.
|
|
- **Out:** shared pools themselves (`resource-distribution-mapping` option 2); device identity and allocation (`resource-instances`); lifecycle (`resource-claims`).
|
|
|
|
## Boundary Candidates
|
|
- Worker script reporter.
|
|
- Controller reporter role and update path.
|
|
- Value kinds and staleness.
|
|
- Inventory format.
|
|
|
|
## Out of Boundary
|
|
- Scheduling policy changes.
|
|
- Provisioning.
|
|
|
|
## Upstream / Downstream
|
|
- **Upstream:**
|
|
- `resource-distribution-mapping`: the descriptor and shared pools;
|
|
- `peer-authentication-hardening`: the reporter role on the allowlist;
|
|
- the no-fork rule.
|
|
- **Downstream:**
|
|
- `resource-instances`, which consumes the inventory;
|
|
- `elastic-shared-resources`, which may reuse the reporter role for license-server counts;
|
|
- data-locality placement.
|
|
|
|
## Existing Spec Touchpoints
|
|
- **Extends:** `resource-distribution-mapping` (its option 3 scope).
|
|
- **Adjacent:** `resource-value-verification` (end-to-end value propagation checks), `resource-provisioner` (role and lease patterns), `xce-node-manager` (supervision of reporter processes).
|
|
|
|
## Constraints
|
|
- **Scheduler:** it never calls a reporter.
|
|
- **Persistence:** reported values follow the persistence rules for their kind. Dynamic values are not WAL facts, like load indices, but must be in the scheduler snapshot.
|
|
- **Credentials:** reporter credentials stay with the reporter.
|