lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

spec(scheduler-framework): finalize 4-level job-picker design

Browse source 
Redesign the plugin picker hierarchy to project→queue→job+preempt
(was 3-level). Adds LSF-style cross-queue preemption (SIGSTP) and a
unified v1/v2 dispatch loop. Requirements grow 6→9 to cover the new
preemption and cross-queue levels.
This commit is contained in:
Rongsong Shen 2026-07-12 04:25:49 +00:00
parent 6529dcf08f
commit 2b15c09646
3 changed files with 491 additions and 55 deletions
Download patch file Download diff file Expand all files Collapse all files

248
.kiro/specs/scheduler-framework/brief.md
View file

@ -1,12 +1,14 @@
# Brief: scheduler-framework
## Problem
The XCE scheduler has a plugin architecture but three critical gaps prevent it from being a real framework:
1. **Job selection policy is hardcoded** — `sched_job_picker()` in `scheduler.c:48` uses the literal string `"xce_sched_fcfs"`. No way to swap policies without recompilation.
The XCE scheduler has a plugin architecture but critical gaps prevent it from being a real framework:
1. **Job selection policy is hardcoded** — `sched_job_picker()` in `scheduler.c:39` uses the literal string `"xce_sched_fcfs"`. No way to swap policies without recompilation. The picker is monolithic — it both chooses which queue to serve and which job within the queue.
2. **Parallel job allocation is a stub** — `parjob_allocator.c:my_alloc()` returns 0 with no logic. Jobs requiring multiple slots (`maxTasks > 1`) produce no allocation.
3. **Allocation is one-shot** — `sched_make_allocation()` tries the first matching host and stops. No mechanism to try alternative allocation plans (different host×slot combinations) when the first fails.
4. **No preemption** — when a high-priority job cannot be scheduled due to insufficient free resources, there is no mechanism to preempt lower-priority running jobs to free resources.
5. **No project-level scheduling** — `xce_sched_project_t` is just `{ char* name }`. Projects have no resource ownership, quota, or scheduling policy. Queue and project roles are conflated.
Operators need configurable scheduling policies and parallel job support. The scheduler must generate multiple candidate allocation plans and try them in preference order.
Operators need configurable scheduling policies at multiple levels, parallel job support, and preemption. The scheduler must generate multiple candidate allocation plans and try them in preference order.
## Current State
@ -21,18 +23,46 @@ Operators need configurable scheduling policies and parallel job support. The sc
- **Scheduling loop** (`scheduler.c:sched_allocator()`): pick job → `sched_make_allocation()` → pend or dispatch → next.
### What's missing
- Configurable job picker: the plugin name is a hardcoded string literal
- Configurable job picker: the plugin name is a hardcoded string literal; a monolithic picker conflates inter-queue and intra-queue selection
- Parallel job allocation: no combinatorial plan generation, no multi-slot host matching
- Iterator/lazy generation: allocation is eager one-shot, no retry with alternative plans
- Preemption: no mechanism to preempt lower-priority running jobs when a high-priority job cannot be scheduled
- Project-level scheduling: `xce_sched_project_t` is just `{ char* name }` — no resource ownership, quota, or scheduling policy
- Cross-job resreq optimization: no mechanism to skip already-failed resreq patterns within a session
- Picker composition: no way to assign different pickers to different queues or projects
## Desired Outcome
1. **Configurable job selection**: operator can set `job_pick_plugin` in scheduler config to swap policies (FCFS, priority, custom) without recompilation. Plugin interface contract is documented (cancel-safe, session-aware).
2. **Lazy allocation plan iterator**: `xce_alloc_plan_iter_t` — generates candidate (nhosts, slots_per_host) plans one at a time, sorted by preference. Consumer tries each plan until one succeeds or all fail.
3. **Lazy host candidate iterator**: for a given allocation plan, iterate candidate hosts filtered by select expression + resource availability. Composes with plan iterator.
4. **Parallel job combinatorial allocation**: `parjob_allocator` implements the plan iterator. For a job needing N slots: generate plans [1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1]. Each plan: collect hosts with enough slots, try each until success.
5. **Session-level unsatisfiable resreq cache**: if a primary resreq cannot be satisfied in the current scheduling session (all plans exhausted), fingerprint it. Skip subsequent jobs with the same fingerprint for the rest of the session.
### Four-Level Scheduling Architecture
The scheduling loop is always `project → queue → job`, composing pickers at each level:
| Level | Picker | Plugin slot | Default (v1) |
|-------|--------|-------------|---------------|
| 0 | Project picker | `sched->project_pick_plugin` | `builtin_flat` — single "default" project |
| 1 | Inter-queue picker | `sched->queue_pick_plugin` | `builtin_strict_priority` |
| 2 | Intra-queue picker | `queue→project→sched` chain | `builtin_fcfs` |
| 3 | Preemption policy | `queue->preempt_policy` | `lowest_priority_first` |
1. **Unified picker loop**: A single scheduling loop that always follows `project → queue → job`. In v1 (flat project space), the project picker `builtin_flat` yields one "default" project containing all queues — identical to today's behavior. In v2 (future), the project picker becomes quota-aware and hierarchical, without changing the loop code.
2. **Configurable job selection at every level**: each picker slot is a plugin. Level 1 (inter-queue) picks which queue to serve next. Level 2 (intra-queue) picks which job within that queue, with per-queue customization via the `queue → project → global` precedence chain.
3. **Lazy allocation plan iterator**: `xce_alloc_plan_iter_t` — generates candidate (nhosts, slots_per_host) plans one at a time, sorted by preference. Consumer tries each plan until one succeeds or all fail.
4. **Lazy host candidate iterator**: for a given allocation plan, iterate candidate hosts filtered by select expression + resource availability. Composes with plan iterator.
5. **Parallel job combinatorial allocation**: `parjob_allocator` implements the plan iterator. For a job needing N slots: generate plans [1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1]. Each plan: collect hosts with enough slots, try each until success.
6. **Preemption**: when allocation fails for a high-priority job, select victim jobs from lower-priority queues (LSF-style per-queue `preemptible_queues` list), send SIGSTP via controller→worker pipeline, and retry allocation next round.
7. **Session-level unsatisfiable resreq cache**: if a primary resreq cannot be satisfied in the current scheduling session (all plans + preemption exhausted), fingerprint it. Skip subsequent jobs with the same fingerprint for the rest of the session.
### Queue vs Project — Roles
| | Queue | Project |
|---|---|---|
| **Role** | Scheduling mechanism | Resource ownership |
| **Question** | HOW are jobs ordered and constrained? | WHO owns resources and gets charged? |
| **Scope** | One hardware/scheduling profile | Cross-queue, cross-cluster |
| **Key attributes** | Priority, limits, time windows, picker, preemption config | Quota, usage, hierarchy, members (future) |
| **Cardinality** | 1 job → 1 queue, many-to-many | 1 job → 1 project, many-to-many |
One project submits to many queues (different hardware profiles). One queue receives from many projects (shared hardware).
## Alternatives Considered
@ -73,94 +103,202 @@ Operators need configurable scheduling policies and parallel job support. The sc
### Decision rationale
Approach B chosen over A because the iterator abstraction is a one-time design cost that pays off immediately (parjob combinatorial allocation needs it) and compounds (every future allocator reuses it). Approach C's additional layers (policy chains, resreq registry, state machine) are deferred until real use cases emerge — the iterator interfaces are designed to compose upward when needed.
Approach B chosen over A because the iterator abstraction is a one-time design cost that pays off immediately (parjob combinatorial allocation needs it) and compounds (every future allocator reuses it). The four-level picker architecture (project → queue → job + preemption) extends Approach B with design elements from Approach C (policy composition, formalized level separation) that are needed NOW (per-queue pickers, preemption), while deferring the remaining Approach C layers (policy chains with scoring, resreq registry, formal state machine) until real use cases emerge. The iterator interfaces and picker slots are designed to compose upward when needed.
**Updated decision (2026-07-10):** The original Approach B treated the job picker as a single global plugin (`job_pick_plugin` on `xce_scheduler_t`). Design discussion identified that different queues/projects need different pickers, and that preemption is an essential missing mechanism. The architecture now specifies four composable levels (project, queue, job, preemption) with per-queue configuration, LSF-style cross-queue preemption, and a unified loop that works identically for v1 (flat projects) and v2 (hierarchical projects). The project picker level is deferred to a future spec but the architecture slot and interface contract are defined now.
## Approach
**Iterator-Driven Allocation Architecture** (Approach B):
**Iterator-Driven Allocation Architecture with Four-Level Picker Composition** (extended from original Approach B):
### Core abstractions
```
xce_alloc_plan_iter_t xce_host_iter_t
┌─────────────────────┐ ┌──────────────────────┐
│ init(resreq, sched) │ │ init(plan, workers) │
│ next() → plan │───────▶│ next() → worker │
│ fini() │ │ fini() │
└─────────────────────┘ └──────────────────────┘
┌──────────────────────────────────────────────────────────┐
│ Level 0: Project Picker [sched->project_pick_plugin] │
│ │
│ v1 (builtin_flat): yields one "default" project │
│ v2 (future): quota-aware project selection │
└──────────────────────┬───────────────────────────────────┘
│ project scope
▼
┌──────────────────────────────────────────────────────────┐
│ Level 1: Inter-Queue Picker [sched->queue_pick_plugin] │
│ │
│ Iterates queues scoped to the selected project │
│ Default: strict_priority (descending queue->priority) │
└──────────────────────┬───────────────────────────────────┘
│ queue
▼
┌──────────────────────────────────────────────────────────┐
│ Level 2: Intra-Queue Picker [queue→project→sched] │
│ │
│ Picks one job from the selected queue │
│ Default: builtin_fcfs (priority desc, queue_time asc) │
└──────────────────────┬───────────────────────────────────┘
│ job
▼
┌──────────────────────────────────────────────────────────┐
│ Allocation │
│ │
│ xce_alloc_plan_iter_t xce_host_iter_t │
│ ┌─────────────────────┐ ┌──────────────────────┐ │
│ │ init(resreq, sched) │ │ init(plan, workers) │ │
│ │ next() → plan │───────▶│ next() → worker │ │
│ │ fini() │ │ fini() │ │
│ └─────────────────────┘ └──────────────────────┘ │
│ │
│ success → dispatch │
│ fail → Level 3 (preemption) │
└──────────────────────┬──────────────────────────────────┘
│ fail
▼
┌──────────────────────────────────────────────────────────┐
│ Level 3: Preemption │
│ │
│ if queue->preemptible_queues: │
│ victims = preempt_policy.select_victims(job) │
│ if victims: commit XCE_DECISION_PREEMPT │
│ → controller → worker → SIGSTP victims │
│ → victims → JOB_SUSPENDED, release resources │
│ → pend job for next round │
│ else: pend(job, "Resources") │
│ else: pend(job, "Resources") │
└──────────────────────────────────────────────────────────┘
```
- **Plan iterator**: owned by allocator plugin (`parjob_allocator` or `seqjob_allocator`). Generates `(nhosts, slots_per_host, host_filter)` plans lazily.
- **Host iterator**: per-plan. Wraps `sched_collect_workers()` + select expression filtering into a lazy interface. Each `next()` returns the next candidate host or NULL.
### Picker plugin interface
### Scheduling loop (modified)
```
for each elected job:
plan_iter = job.allocator.make_plan_iter(resreq)
while (plan = plan_iter.next()):
host_iter = make_host_iter(plan, all_workers)
while (host = host_iter.next()):
if try_alloc(host, plan) succeeds:
dispatch(job, host, plan)
goto next_job
host_iter.advance()
host_iter.fini()
plan_iter.fini()
pend(job, reason)
Each picker level follows the same contract:
```c
typedef struct {
void* (*init)(xce_scheduler_t* sched, void* parent_scope);
void* (*next)(void* iter); // returns scope or NULL
void (*fini)(void* iter);
} picker_iface_t;
```
### Configurable job picker
### Unified scheduling loop
- Add `char* job_pick_plugin` to `xce_scheduler_t` (default `"xce_sched_fcfs"`)
- `sched_job_picker()` reads this field instead of hardcoded string
- Set via scheduler config or command-line flag
- Plugin interface contract: `job_elect_fn` must be cancel-safe (respond to `pthread_cancel`), session-aware (reset state on `end_session`), and communicate via `jbuf` cbuf
```c
void* sched_allocator(void* arg) {
while (1) {
// Level 0: project scope
scope = project_picker.next(sched);
if (!scope) { sleep(1); continue; }
### Unsatisfiable resreq cache
// Level 1: queue (scoped to project)
queue = queue_picker.next(sched, scope);
if (!queue) { project_picker.advance(); continue; }
- Per scheduling session (`xce_scheduler_t`), cleared on `begin_session`
- Key: hash of `{select_string, rusage_string, user, queue, project}`
- Before full allocation: quick lookup. If present → skip to next job immediately
- Populated when a job's plan iterator exhausts all plans without success
// Level 2: job (within queue)
// Precedence: queue->job_pick_plugin → project->job_pick_plugin → sched->job_pick_plugin
picker = resolve_job_picker(sched, queue);
job = picker->elect_one(queue);
if (!job) { queue_picker.advance(); continue; }
// Allocation + preemption
if (try_allocate(sched, job)) {
dispatch(sched, job);
} else if (can_preempt(sched, job)) {
preempt_and_pend(sched, job);
} else {
pend(sched, job, "Resources");
}
}
}
```
This loop is identical for v1 and v2. v1 uses `builtin_flat` for Level 0. v2 replaces it with a quota-aware project picker.
### Preemption flow
```
scheduler controller worker
─────────── ────────── ──────
preempt_policy.select_victims(J)
→ [victim1, victim2]
XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim
.victim_jobids[] deliver to each job → JOB_SUSPENDED
.signal = SIGSTP victim's worker release resources
.preempting_jobid = J
J pended → next round → retry alloc
```
V1 uses SIGSTP (terminal stop — process groups paused, job stays in queue). Future escalation to SIGKILL kill-and-requeue is deferred.
### What stays unchanged
- Existing plugin hooks (config, queue, job, worker, state_change, commit, election, prepare_allocation, rusage) — all preserved
- `seqjob_allocator` — wrapped with iterator interface, internally still 1host×1slot
- `fcfs.c` — continues as default job picker, no changes needed
- Select expression system — no changes
- COMPOUND and ALTER resreq types — unchanged (PRIMARY focus)
- Controller communication (handler.c opcodes, decision cbuf) — unchanged
- Controller communication (handler.c opcodes, decision cbuf) — extended, not replaced
- `HOOK_DETAIL_JOB_STOP` — reused for preemption stop notifications
### What's new
- `XCE_DECISION_PREEMPT` decision type
- `xce_preempt_decision_t` payload
- Three builtin plugins: `builtin_flat`, `builtin_strict_priority`, `builtin_fcfs`
- `project_name` on queue structs
- `job_pick_plugin` on queue + project structs
- `preemptible_queues`, `preemptible`, `preempt_policy` on queue struct
- `project_pick_plugin`, `queue_pick_plugin`, `job_pick_plugin` on scheduler struct
## Scope
### In
- Unified picker loop: `project → queue → job` with plugin slots at each level
- Three builtin plugins: `builtin_flat` (Level 0), `builtin_strict_priority` (Level 1), `builtin_fcfs` (Level 2)
- Picker plugin interface: `init/next/fini` — cancel-safe, session-aware
- Picker precedence chain: `queue → project → global`
- `xce_alloc_plan_iter_t` interface (header + implementation)
- `xce_host_iter_t` interface (header + implementation)
- `parjob_allocator` combinatorial plan generation (1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1)
- Configurable job picker: `job_pick_plugin` field + config wiring
- Preemption: `XCE_DECISION_PREEMPT` decision type, `xce_preempt_decision_t` payload, LSF-style cross-queue victim selection, SIGSTP mechanism via controller→worker pipeline
- Per-queue preemption config: `preemptible_queues`, `preemptible`, `preempt_policy`
- Session-level unsatisfiable resreq fingerprint cache
- Data structure changes to `xce_scheduler_t`, `xce_sched_queue_t`, `xce_sched_project_t`, `decision.h`
- Plugin interface contract documentation (cancel-safe, session-aware)
- Unit tests for plan generation, host iteration, cache hit/miss
- Integration test: multi-slot parallel job allocation end-to-end
- Unit tests for plan generation, host iteration, cache hit/miss, picker resolution, preemption victim selection
- Integration test: multi-slot parallel job allocation and preemption flow end-to-end
### Out
- New job selection policies (priority-only, fair-share, preemption, backfill) — framework enables them but they ship separately
- New job selection policy implementations beyond the three builtins (priority, fair-share, deadline, preemption, backfill, etc.) — the configurable slots enable them, but none ship in this spec
- Hierarchical project resource quotas (Level 0 full) — deferred to `project-resource-ownership` spec
- Resreq template registry (named/shared resreq objects) — future enhancement
- Select expression language changes
- Controller-side scheduling state replication (separate concern: commit barrier wiring, see roadmap hardening item)
- Compound/ALTER resreq allocation changes
- GPU/affinity-aware allocation (pluggable later via allocator plugins)
- Kill-and-requeue preemption escalation (v1 is SIGSTP only, suspend-and-retry)
- Deadline field on `xce_sched_job_t` (needed for deadline-based pickers — deferred)
## Future: Project Resource Ownership
Deferred to a separate `project-resource-ownership` spec. When implemented:
- Projects form a tree with resource quota at each node
- Level 0 project picker selects which project gets resources (fair-share, priority, proportional)
- Level 1 queue picker is scoped to the selected project's queues
- Level 2 job picker sees only jobs from the selected project within each queue
- Project quota enforcement: projects at or over quota are skipped by Level 0
- The unified picker loop (above) requires zero code changes — only a new project picker plugin replaces `builtin_flat`
The architecture is designed so that v1 (flat project space) and v2 (hierarchical projects) share the identical scheduling loop.
## Boundary Candidates
The work naturally splits into these implementation phases (single spec, sequential):
1. **Iterator interfaces** — define `xce_alloc_plan_iter_t` and `xce_host_iter_t`, implement for seqjob path first (trivial wrapper), validate the abstraction
2. **Parallel job allocator** — implement combinatorial plan generation in `parjob_allocator`, wire through host iterator
3. **Configurable job picker** — add `job_pick_plugin` config, document plugin contract
4. **Unsatisfiable resreq cache** — session-level fingerprint cache, wire into scheduling loop
5. **Integration + testing** — end-to-end multi-slot job allocation, cache behavior, config swapping
1. **Unified picker loop + builtin plugins** — implement the `project → queue → job` loop with `builtin_flat`, `builtin_strict_priority`, `builtin_fcfs`; retire `sched_job_picker()`; add plugin slots and precedence chain to structs
2. **Iterator interfaces** — define `xce_alloc_plan_iter_t` and `xce_host_iter_t`, implement for seqjob path first (trivial wrapper), validate the abstraction
3. **Parallel job allocator** — implement combinatorial plan generation in `parjob_allocator`, wire through host iterator
4. **Preemption** — implement `XCE_DECISION_PREEMPT` decision type, cross-queue victim selection policy, SIGSTP pipeline (scheduler→controller→worker), queue preemption config fields
5. **Unsatisfiable resreq cache** — session-level fingerprint cache, wire into scheduling loop (skip before allocation + preemption attempt)
6. **Integration + testing** — end-to-end multi-slot parallel job allocation, preemption flow, picker resolution, cache behavior, HA soak regression
## Out of Boundary
- New job selection policy implementations (priority, fair-share, etc.) — these are new plugins that USE the configurable picker, not part of this spec