save wolf state and document
This commit is contained in:
parent
c68935127c
commit
ea0ac81c92
9 changed files with 30215 additions and 429 deletions
200
.kiro/specs/scheduler-framework/brief.md
Normal file
200
.kiro/specs/scheduler-framework/brief.md
Normal file
|
|
@ -0,0 +1,200 @@
|
|||
# Brief: scheduler-framework
|
||||
|
||||
## Problem
|
||||
The XCE scheduler has a plugin architecture but three critical gaps prevent it from being a real framework:
|
||||
1. **Job selection policy is hardcoded** — `sched_job_picker()` in `scheduler.c:48` uses the literal string `"xce_sched_fcfs"`. No way to swap policies without recompilation.
|
||||
2. **Parallel job allocation is a stub** — `parjob_allocator.c:my_alloc()` returns 0 with no logic. Jobs requiring multiple slots (`maxTasks > 1`) produce no allocation.
|
||||
3. **Allocation is one-shot** — `sched_make_allocation()` tries the first matching host and stops. No mechanism to try alternative allocation plans (different host×slot combinations) when the first fails.
|
||||
|
||||
Operators need configurable scheduling policies and parallel job support. The scheduler must generate multiple candidate allocation plans and try them in preference order.
|
||||
|
||||
## Current State
|
||||
|
||||
### What exists
|
||||
- **Plugin framework** (`plugin.h`, `plugin.c`): `xce_sched_plugin_t` with 9 hooks + `job_elect_fn` / `job_alloc_fn`. Plugins registered in key-value map by name.
|
||||
- **FCFS job picker** (`fcfs.c`): dedicated `jobpicker` thread. Iterates queues by priority, jobs within queue by (priority desc, queue_time asc). Communicates picked jobid via `jbuf` cbuf.
|
||||
- **Seqjob allocator** (`seqjob_allocator.c`): handles `maxTasks == 1`. Checks shared resources → collects workers → tries one host → returns first match.
|
||||
- **Parjob allocator stub** (`parjob_allocator.c`): registered but `my_alloc()` is empty.
|
||||
- **Resource request types** (`resreq.h`): PRIMARY (minTasks/maxTasks/select/rusages/span/same/affinity), COMPOUND (list of primaries), ALTER (list of compounds).
|
||||
- **Select expression AST** (`resreq.h`): full parser/evaluator with per-session memoization (`select_eval_ctx_t`) and inter-session GC.
|
||||
- **Shared resources** (`xce_scheduler_t.sharedInst`/`sharedInUse`): zarray of resource values tracked per scheduling session.
|
||||
- **Scheduling loop** (`scheduler.c:sched_allocator()`): pick job → `sched_make_allocation()` → pend or dispatch → next.
|
||||
|
||||
### What's missing
|
||||
- Configurable job picker: the plugin name is a hardcoded string literal
|
||||
- Parallel job allocation: no combinatorial plan generation, no multi-slot host matching
|
||||
- Iterator/lazy generation: allocation is eager one-shot, no retry with alternative plans
|
||||
- Cross-job resreq optimization: no mechanism to skip already-failed resreq patterns within a session
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
1. **Configurable job selection**: operator can set `job_pick_plugin` in scheduler config to swap policies (FCFS, priority, custom) without recompilation. Plugin interface contract is documented (cancel-safe, session-aware).
|
||||
2. **Lazy allocation plan iterator**: `xce_alloc_plan_iter_t` — generates candidate (nhosts, slots_per_host) plans one at a time, sorted by preference. Consumer tries each plan until one succeeds or all fail.
|
||||
3. **Lazy host candidate iterator**: for a given allocation plan, iterate candidate hosts filtered by select expression + resource availability. Composes with plan iterator.
|
||||
4. **Parallel job combinatorial allocation**: `parjob_allocator` implements the plan iterator. For a job needing N slots: generate plans [1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1]. Each plan: collect hosts with enough slots, try each until success.
|
||||
5. **Session-level unsatisfiable resreq cache**: if a primary resreq cannot be satisfied in the current scheduling session (all plans exhausted), fingerprint it. Skip subsequent jobs with the same fingerprint for the rest of the session.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### Approach A: Incremental Enhancement (rejected)
|
||||
|
||||
| Aspect | Detail |
|
||||
|--------|--------|
|
||||
| **Job selection** | Add `job_pick_plugin` config field, same as chosen approach |
|
||||
| **Parallel jobs** | Implement parjob allocator with pre-computed plan list. Eager generation — all plans materialized before first try |
|
||||
| **Iterator** | No new abstraction. Simple for-loop over plan array, inner for-loop over hosts |
|
||||
| **Shared resreq** | `xlist_t unsatisfiable_resreqs` in session state, same as chosen approach |
|
||||
|
||||
**Rejected because:** Eager plan generation causes memory pressure for large combinatorial spaces (e.g., 64 slots on 128 hosts = thousands of plans). No reusable abstraction — each new allocator reimplements the loop. Short-term win, long-term rework cost.
|
||||
|
||||
### Approach B: Iterator-Driven Allocation Architecture ★ Chosen
|
||||
|
||||
| Aspect | Detail |
|
||||
|--------|--------|
|
||||
| **Job selection** | Configurable plugin name via `job_pick_plugin` field |
|
||||
| **Iterator abstraction** | `xce_alloc_plan_iter_t` + `xce_host_iter_t` — lazy, cancel-safe, composable |
|
||||
| **Parallel jobs** | Combinatorial plan generation via plan iterator: 1host×N → 2hosts×⌈N/2⌉ → … → Nhhosts×1 |
|
||||
| **Host iterator** | Per-plan lazy candidate host iteration using existing select expression eval |
|
||||
| **Shared resreq** | Session-level fingerprint cache: `{select, rusage, user, queue, project}` |
|
||||
|
||||
**Chosen because:** Lazy generation avoids memory pressure. Iterator composition matches the user's vision: plan iterator → host iterator → try alloc. Clean separation of concerns. Each iterator implementation is independently testable.
|
||||
|
||||
### Approach C: Full Scheduler Framework (deferred)
|
||||
|
||||
| Aspect | Detail |
|
||||
|--------|--------|
|
||||
| **Job selection** | Policy chain — multiple plugins score/veto jobs. Composable: priority + fair-share + preemption |
|
||||
| **Iterator** | Triple composition: job iterator → plan iterator → host iterator |
|
||||
| **Resreq registry** | Named primary resreqs registered globally, referenced by name, reference-counted |
|
||||
| **Allocation plugins** | Different strategies as plugins: greedy, bin-packing, round-robin, topology-aware |
|
||||
| **Session state machine** | Formalized STAGING → ELECT → ALLOCATE → COMMIT with well-defined transitions |
|
||||
|
||||
**Deferred (not rejected):** The policy chain and resreq registry are valuable but lack current demand. Approach B's iterator abstraction is designed to compose upward — the plan iterator and host iterator are building blocks that a future job iterator or policy chain can consume. The template registry can layer on top without breaking the iterator interfaces.
|
||||
|
||||
### Decision rationale
|
||||
|
||||
Approach B chosen over A because the iterator abstraction is a one-time design cost that pays off immediately (parjob combinatorial allocation needs it) and compounds (every future allocator reuses it). Approach C's additional layers (policy chains, resreq registry, state machine) are deferred until real use cases emerge — the iterator interfaces are designed to compose upward when needed.
|
||||
|
||||
## Approach
|
||||
|
||||
**Iterator-Driven Allocation Architecture** (Approach B):
|
||||
|
||||
### Core abstractions
|
||||
|
||||
```
|
||||
xce_alloc_plan_iter_t xce_host_iter_t
|
||||
┌─────────────────────┐ ┌──────────────────────┐
|
||||
│ init(resreq, sched) │ │ init(plan, workers) │
|
||||
│ next() → plan │───────▶│ next() → worker │
|
||||
│ fini() │ │ fini() │
|
||||
└─────────────────────┘ └──────────────────────┘
|
||||
```
|
||||
|
||||
- **Plan iterator**: owned by allocator plugin (`parjob_allocator` or `seqjob_allocator`). Generates `(nhosts, slots_per_host, host_filter)` plans lazily.
|
||||
- **Host iterator**: per-plan. Wraps `sched_collect_workers()` + select expression filtering into a lazy interface. Each `next()` returns the next candidate host or NULL.
|
||||
|
||||
### Scheduling loop (modified)
|
||||
|
||||
```
|
||||
for each elected job:
|
||||
plan_iter = job.allocator.make_plan_iter(resreq)
|
||||
while (plan = plan_iter.next()):
|
||||
host_iter = make_host_iter(plan, all_workers)
|
||||
while (host = host_iter.next()):
|
||||
if try_alloc(host, plan) succeeds:
|
||||
dispatch(job, host, plan)
|
||||
goto next_job
|
||||
host_iter.advance()
|
||||
host_iter.fini()
|
||||
plan_iter.fini()
|
||||
pend(job, reason)
|
||||
```
|
||||
|
||||
### Configurable job picker
|
||||
|
||||
- Add `char* job_pick_plugin` to `xce_scheduler_t` (default `"xce_sched_fcfs"`)
|
||||
- `sched_job_picker()` reads this field instead of hardcoded string
|
||||
- Set via scheduler config or command-line flag
|
||||
- Plugin interface contract: `job_elect_fn` must be cancel-safe (respond to `pthread_cancel`), session-aware (reset state on `end_session`), and communicate via `jbuf` cbuf
|
||||
|
||||
### Unsatisfiable resreq cache
|
||||
|
||||
- Per scheduling session (`xce_scheduler_t`), cleared on `begin_session`
|
||||
- Key: hash of `{select_string, rusage_string, user, queue, project}`
|
||||
- Before full allocation: quick lookup. If present → skip to next job immediately
|
||||
- Populated when a job's plan iterator exhausts all plans without success
|
||||
|
||||
### What stays unchanged
|
||||
- Existing plugin hooks (config, queue, job, worker, state_change, commit, election, prepare_allocation, rusage) — all preserved
|
||||
- `seqjob_allocator` — wrapped with iterator interface, internally still 1host×1slot
|
||||
- `fcfs.c` — continues as default job picker, no changes needed
|
||||
- Select expression system — no changes
|
||||
- COMPOUND and ALTER resreq types — unchanged (PRIMARY focus)
|
||||
- Controller communication (handler.c opcodes, decision cbuf) — unchanged
|
||||
|
||||
## Scope
|
||||
|
||||
### In
|
||||
- `xce_alloc_plan_iter_t` interface (header + implementation)
|
||||
- `xce_host_iter_t` interface (header + implementation)
|
||||
- `parjob_allocator` combinatorial plan generation (1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1)
|
||||
- Configurable job picker: `job_pick_plugin` field + config wiring
|
||||
- Session-level unsatisfiable resreq fingerprint cache
|
||||
- Plugin interface contract documentation (cancel-safe, session-aware)
|
||||
- Unit tests for plan generation, host iteration, cache hit/miss
|
||||
- Integration test: multi-slot parallel job allocation end-to-end
|
||||
|
||||
### Out
|
||||
- New job selection policies (priority-only, fair-share, preemption, backfill) — framework enables them but they ship separately
|
||||
- Resreq template registry (named/shared resreq objects) — future enhancement
|
||||
- Select expression language changes
|
||||
- Controller-side scheduling state replication (separate concern: commit barrier wiring, see roadmap hardening item)
|
||||
- Compound/ALTER resreq allocation changes
|
||||
- GPU/affinity-aware allocation (pluggable later via allocator plugins)
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
The work naturally splits into these implementation phases (single spec, sequential):
|
||||
|
||||
1. **Iterator interfaces** — define `xce_alloc_plan_iter_t` and `xce_host_iter_t`, implement for seqjob path first (trivial wrapper), validate the abstraction
|
||||
2. **Parallel job allocator** — implement combinatorial plan generation in `parjob_allocator`, wire through host iterator
|
||||
3. **Configurable job picker** — add `job_pick_plugin` config, document plugin contract
|
||||
4. **Unsatisfiable resreq cache** — session-level fingerprint cache, wire into scheduling loop
|
||||
5. **Integration + testing** — end-to-end multi-slot job allocation, cache behavior, config swapping
|
||||
|
||||
## Out of Boundary
|
||||
- New job selection policy implementations (priority, fair-share, etc.) — these are new plugins that USE the configurable picker, not part of this spec
|
||||
- Resreq template sharing across jobs (future: `resreq_registry` spec)
|
||||
- Commit barrier wiring for scheduler decisions (separate hardening item in roadmap)
|
||||
- Controller-side changes (the scheduler runs independently, receives updates via existing opcodes)
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
### Upstream (dependencies)
|
||||
- Existing plugin framework (`plugin.h`, `plugin.c`) — NO changes needed
|
||||
- Existing select expression evaluator (`resreq.h` eval/Gc) — NO changes needed
|
||||
- Existing shared resource tracking (`sharedInst`/`sharedInUse`) — NO changes needed
|
||||
- Existing scheduler data structures (`xce_scheduler_t`, `xce_sched_ctx_t`) — minimal additions
|
||||
|
||||
### Downstream (future consumers)
|
||||
- **New job selection policies** — priority, fair-share, preemption plugins register via the configurable picker
|
||||
- **Affinity/GPU allocators** — register as allocator plugins, implement plan iterator for topology-aware placement
|
||||
- **Commit barrier wiring** — scheduling decisions flow through majority commit (separate hardening item)
|
||||
- **Resreq template registry** — named resreqs shared across jobs, reference-counted
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
### Extends
|
||||
- **scheduler-controller-msg-gap** — adjacent (message handling). The framework enhancement operates on the same `xce_scheduler_t` state. No conflicts; the msg-gap spec fills handler gaps this spec reads from.
|
||||
|
||||
### Adjacent
|
||||
- **ha-consistency-invariants** — commit barrier that scheduling decisions will eventually flow through. This spec does NOT wire the barrier; it only makes allocation decisions that the barrier WILL consume.
|
||||
- **worker-join-refactor** — worker state management. Allocation consumes worker resource availability pushed via the existing join/update path.
|
||||
|
||||
## Constraints
|
||||
- **C (gnu17)** — no C++, no external libraries beyond existing (capnp, zlib, pthreads)
|
||||
- **Non-blocking in plugin hooks** — allocation happens on allocator thread, not iomgr. No new blocking concerns.
|
||||
- **Cancel-safe** — job picker thread is `pthread_cancel`-able. Iterator `next()` calls must have cancellation points.
|
||||
- **Session isolation** — iterator state and resreq cache are per-session, cleared on `begin_session`.
|
||||
- **Backward compatible** — existing FCFS + seqjob behavior unchanged. New functionality is additive.
|
||||
- **Memory** — lazy generation avoids materializing all combinations. Worst-case for a job needing N slots on M hosts: M×N candidate evaluations (acceptably bounded).
|
||||
Loading…
Add table
Add a link
Reference in a new issue