Rongsong Shen
2b15c09646
Redesign the plugin picker hierarchy to project→queue→job+preempt (was 3-level). Adds LSF-style cross-queue preemption (SIGSTP) and a unified v1/v2 dispatch loop. Requirements grow 6→9 to cover the new preemption and cross-queue levels.
338 lines
24 KiB
Markdown
338 lines
24 KiB
Markdown
# Brief: scheduler-framework
|
||
|
||
## Problem
|
||
The XCE scheduler has a plugin architecture but critical gaps prevent it from being a real framework:
|
||
1. **Job selection policy is hardcoded** — `sched_job_picker()` in `scheduler.c:39` uses the literal string `"xce_sched_fcfs"`. No way to swap policies without recompilation. The picker is monolithic — it both chooses which queue to serve and which job within the queue.
|
||
2. **Parallel job allocation is a stub** — `parjob_allocator.c:my_alloc()` returns 0 with no logic. Jobs requiring multiple slots (`maxTasks > 1`) produce no allocation.
|
||
3. **Allocation is one-shot** — `sched_make_allocation()` tries the first matching host and stops. No mechanism to try alternative allocation plans (different host×slot combinations) when the first fails.
|
||
4. **No preemption** — when a high-priority job cannot be scheduled due to insufficient free resources, there is no mechanism to preempt lower-priority running jobs to free resources.
|
||
5. **No project-level scheduling** — `xce_sched_project_t` is just `{ char* name }`. Projects have no resource ownership, quota, or scheduling policy. Queue and project roles are conflated.
|
||
|
||
Operators need configurable scheduling policies at multiple levels, parallel job support, and preemption. The scheduler must generate multiple candidate allocation plans and try them in preference order.
|
||
|
||
## Current State
|
||
|
||
### What exists
|
||
- **Plugin framework** (`plugin.h`, `plugin.c`): `xce_sched_plugin_t` with 9 hooks + `job_elect_fn` / `job_alloc_fn`. Plugins registered in key-value map by name.
|
||
- **FCFS job picker** (`fcfs.c`): dedicated `jobpicker` thread. Iterates queues by priority, jobs within queue by (priority desc, queue_time asc). Communicates picked jobid via `jbuf` cbuf.
|
||
- **Seqjob allocator** (`seqjob_allocator.c`): handles `maxTasks == 1`. Checks shared resources → collects workers → tries one host → returns first match.
|
||
- **Parjob allocator stub** (`parjob_allocator.c`): registered but `my_alloc()` is empty.
|
||
- **Resource request types** (`resreq.h`): PRIMARY (minTasks/maxTasks/select/rusages/span/same/affinity), COMPOUND (list of primaries), ALTER (list of compounds).
|
||
- **Select expression AST** (`resreq.h`): full parser/evaluator with per-session memoization (`select_eval_ctx_t`) and inter-session GC.
|
||
- **Shared resources** (`xce_scheduler_t.sharedInst`/`sharedInUse`): zarray of resource values tracked per scheduling session.
|
||
- **Scheduling loop** (`scheduler.c:sched_allocator()`): pick job → `sched_make_allocation()` → pend or dispatch → next.
|
||
|
||
### What's missing
|
||
- Configurable job picker: the plugin name is a hardcoded string literal; a monolithic picker conflates inter-queue and intra-queue selection
|
||
- Parallel job allocation: no combinatorial plan generation, no multi-slot host matching
|
||
- Iterator/lazy generation: allocation is eager one-shot, no retry with alternative plans
|
||
- Preemption: no mechanism to preempt lower-priority running jobs when a high-priority job cannot be scheduled
|
||
- Project-level scheduling: `xce_sched_project_t` is just `{ char* name }` — no resource ownership, quota, or scheduling policy
|
||
- Cross-job resreq optimization: no mechanism to skip already-failed resreq patterns within a session
|
||
- Picker composition: no way to assign different pickers to different queues or projects
|
||
|
||
## Desired Outcome
|
||
|
||
### Four-Level Scheduling Architecture
|
||
|
||
The scheduling loop is always `project → queue → job`, composing pickers at each level:
|
||
|
||
| Level | Picker | Plugin slot | Default (v1) |
|
||
|-------|--------|-------------|---------------|
|
||
| 0 | Project picker | `sched->project_pick_plugin` | `builtin_flat` — single "default" project |
|
||
| 1 | Inter-queue picker | `sched->queue_pick_plugin` | `builtin_strict_priority` |
|
||
| 2 | Intra-queue picker | `queue→project→sched` chain | `builtin_fcfs` |
|
||
| 3 | Preemption policy | `queue->preempt_policy` | `lowest_priority_first` |
|
||
|
||
1. **Unified picker loop**: A single scheduling loop that always follows `project → queue → job`. In v1 (flat project space), the project picker `builtin_flat` yields one "default" project containing all queues — identical to today's behavior. In v2 (future), the project picker becomes quota-aware and hierarchical, without changing the loop code.
|
||
2. **Configurable job selection at every level**: each picker slot is a plugin. Level 1 (inter-queue) picks which queue to serve next. Level 2 (intra-queue) picks which job within that queue, with per-queue customization via the `queue → project → global` precedence chain.
|
||
3. **Lazy allocation plan iterator**: `xce_alloc_plan_iter_t` — generates candidate (nhosts, slots_per_host) plans one at a time, sorted by preference. Consumer tries each plan until one succeeds or all fail.
|
||
4. **Lazy host candidate iterator**: for a given allocation plan, iterate candidate hosts filtered by select expression + resource availability. Composes with plan iterator.
|
||
5. **Parallel job combinatorial allocation**: `parjob_allocator` implements the plan iterator. For a job needing N slots: generate plans [1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1]. Each plan: collect hosts with enough slots, try each until success.
|
||
6. **Preemption**: when allocation fails for a high-priority job, select victim jobs from lower-priority queues (LSF-style per-queue `preemptible_queues` list), send SIGSTP via controller→worker pipeline, and retry allocation next round.
|
||
7. **Session-level unsatisfiable resreq cache**: if a primary resreq cannot be satisfied in the current scheduling session (all plans + preemption exhausted), fingerprint it. Skip subsequent jobs with the same fingerprint for the rest of the session.
|
||
|
||
### Queue vs Project — Roles
|
||
|
||
| | Queue | Project |
|
||
|---|---|---|
|
||
| **Role** | Scheduling mechanism | Resource ownership |
|
||
| **Question** | HOW are jobs ordered and constrained? | WHO owns resources and gets charged? |
|
||
| **Scope** | One hardware/scheduling profile | Cross-queue, cross-cluster |
|
||
| **Key attributes** | Priority, limits, time windows, picker, preemption config | Quota, usage, hierarchy, members (future) |
|
||
| **Cardinality** | 1 job → 1 queue, many-to-many | 1 job → 1 project, many-to-many |
|
||
|
||
One project submits to many queues (different hardware profiles). One queue receives from many projects (shared hardware).
|
||
|
||
## Alternatives Considered
|
||
|
||
### Approach A: Incremental Enhancement (rejected)
|
||
|
||
| Aspect | Detail |
|
||
|--------|--------|
|
||
| **Job selection** | Add `job_pick_plugin` config field, same as chosen approach |
|
||
| **Parallel jobs** | Implement parjob allocator with pre-computed plan list. Eager generation — all plans materialized before first try |
|
||
| **Iterator** | No new abstraction. Simple for-loop over plan array, inner for-loop over hosts |
|
||
| **Shared resreq** | `xlist_t unsatisfiable_resreqs` in session state, same as chosen approach |
|
||
|
||
**Rejected because:** Eager plan generation causes memory pressure for large combinatorial spaces (e.g., 64 slots on 128 hosts = thousands of plans). No reusable abstraction — each new allocator reimplements the loop. Short-term win, long-term rework cost.
|
||
|
||
### Approach B: Iterator-Driven Allocation Architecture ★ Chosen
|
||
|
||
| Aspect | Detail |
|
||
|--------|--------|
|
||
| **Job selection** | Configurable plugin name via `job_pick_plugin` field |
|
||
| **Iterator abstraction** | `xce_alloc_plan_iter_t` + `xce_host_iter_t` — lazy, cancel-safe, composable |
|
||
| **Parallel jobs** | Combinatorial plan generation via plan iterator: 1host×N → 2hosts×⌈N/2⌉ → … → Nhhosts×1 |
|
||
| **Host iterator** | Per-plan lazy candidate host iteration using existing select expression eval |
|
||
| **Shared resreq** | Session-level fingerprint cache: `{select, rusage, user, queue, project}` |
|
||
|
||
**Chosen because:** Lazy generation avoids memory pressure. Iterator composition matches the user's vision: plan iterator → host iterator → try alloc. Clean separation of concerns. Each iterator implementation is independently testable.
|
||
|
||
### Approach C: Full Scheduler Framework (deferred)
|
||
|
||
| Aspect | Detail |
|
||
|--------|--------|
|
||
| **Job selection** | Policy chain — multiple plugins score/veto jobs. Composable: priority + fair-share + preemption |
|
||
| **Iterator** | Triple composition: job iterator → plan iterator → host iterator |
|
||
| **Resreq registry** | Named primary resreqs registered globally, referenced by name, reference-counted |
|
||
| **Allocation plugins** | Different strategies as plugins: greedy, bin-packing, round-robin, topology-aware |
|
||
| **Session state machine** | Formalized STAGING → ELECT → ALLOCATE → COMMIT with well-defined transitions |
|
||
|
||
**Deferred (not rejected):** The policy chain and resreq registry are valuable but lack current demand. Approach B's iterator abstraction is designed to compose upward — the plan iterator and host iterator are building blocks that a future job iterator or policy chain can consume. The template registry can layer on top without breaking the iterator interfaces.
|
||
|
||
### Decision rationale
|
||
|
||
Approach B chosen over A because the iterator abstraction is a one-time design cost that pays off immediately (parjob combinatorial allocation needs it) and compounds (every future allocator reuses it). The four-level picker architecture (project → queue → job + preemption) extends Approach B with design elements from Approach C (policy composition, formalized level separation) that are needed NOW (per-queue pickers, preemption), while deferring the remaining Approach C layers (policy chains with scoring, resreq registry, formal state machine) until real use cases emerge. The iterator interfaces and picker slots are designed to compose upward when needed.
|
||
|
||
**Updated decision (2026-07-10):** The original Approach B treated the job picker as a single global plugin (`job_pick_plugin` on `xce_scheduler_t`). Design discussion identified that different queues/projects need different pickers, and that preemption is an essential missing mechanism. The architecture now specifies four composable levels (project, queue, job, preemption) with per-queue configuration, LSF-style cross-queue preemption, and a unified loop that works identically for v1 (flat projects) and v2 (hierarchical projects). The project picker level is deferred to a future spec but the architecture slot and interface contract are defined now.
|
||
|
||
## Approach
|
||
|
||
**Iterator-Driven Allocation Architecture with Four-Level Picker Composition** (extended from original Approach B):
|
||
|
||
### Core abstractions
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────┐
|
||
│ Level 0: Project Picker [sched->project_pick_plugin] │
|
||
│ │
|
||
│ v1 (builtin_flat): yields one "default" project │
|
||
│ v2 (future): quota-aware project selection │
|
||
└──────────────────────┬───────────────────────────────────┘
|
||
│ project scope
|
||
▼
|
||
┌──────────────────────────────────────────────────────────┐
|
||
│ Level 1: Inter-Queue Picker [sched->queue_pick_plugin] │
|
||
│ │
|
||
│ Iterates queues scoped to the selected project │
|
||
│ Default: strict_priority (descending queue->priority) │
|
||
└──────────────────────┬───────────────────────────────────┘
|
||
│ queue
|
||
▼
|
||
┌──────────────────────────────────────────────────────────┐
|
||
│ Level 2: Intra-Queue Picker [queue→project→sched] │
|
||
│ │
|
||
│ Picks one job from the selected queue │
|
||
│ Default: builtin_fcfs (priority desc, queue_time asc) │
|
||
└──────────────────────┬───────────────────────────────────┘
|
||
│ job
|
||
▼
|
||
┌──────────────────────────────────────────────────────────┐
|
||
│ Allocation │
|
||
│ │
|
||
│ xce_alloc_plan_iter_t xce_host_iter_t │
|
||
│ ┌─────────────────────┐ ┌──────────────────────┐ │
|
||
│ │ init(resreq, sched) │ │ init(plan, workers) │ │
|
||
│ │ next() → plan │───────▶│ next() → worker │ │
|
||
│ │ fini() │ │ fini() │ │
|
||
│ └─────────────────────┘ └──────────────────────┘ │
|
||
│ │
|
||
│ success → dispatch │
|
||
│ fail → Level 3 (preemption) │
|
||
└──────────────────────┬──────────────────────────────────┘
|
||
│ fail
|
||
▼
|
||
┌──────────────────────────────────────────────────────────┐
|
||
│ Level 3: Preemption │
|
||
│ │
|
||
│ if queue->preemptible_queues: │
|
||
│ victims = preempt_policy.select_victims(job) │
|
||
│ if victims: commit XCE_DECISION_PREEMPT │
|
||
│ → controller → worker → SIGSTP victims │
|
||
│ → victims → JOB_SUSPENDED, release resources │
|
||
│ → pend job for next round │
|
||
│ else: pend(job, "Resources") │
|
||
│ else: pend(job, "Resources") │
|
||
└──────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### Picker plugin interface
|
||
|
||
Each picker level follows the same contract:
|
||
```c
|
||
typedef struct {
|
||
void* (*init)(xce_scheduler_t* sched, void* parent_scope);
|
||
void* (*next)(void* iter); // returns scope or NULL
|
||
void (*fini)(void* iter);
|
||
} picker_iface_t;
|
||
```
|
||
|
||
### Unified scheduling loop
|
||
|
||
```c
|
||
void* sched_allocator(void* arg) {
|
||
while (1) {
|
||
// Level 0: project scope
|
||
scope = project_picker.next(sched);
|
||
if (!scope) { sleep(1); continue; }
|
||
|
||
// Level 1: queue (scoped to project)
|
||
queue = queue_picker.next(sched, scope);
|
||
if (!queue) { project_picker.advance(); continue; }
|
||
|
||
// Level 2: job (within queue)
|
||
// Precedence: queue->job_pick_plugin → project->job_pick_plugin → sched->job_pick_plugin
|
||
picker = resolve_job_picker(sched, queue);
|
||
job = picker->elect_one(queue);
|
||
|
||
if (!job) { queue_picker.advance(); continue; }
|
||
|
||
// Allocation + preemption
|
||
if (try_allocate(sched, job)) {
|
||
dispatch(sched, job);
|
||
} else if (can_preempt(sched, job)) {
|
||
preempt_and_pend(sched, job);
|
||
} else {
|
||
pend(sched, job, "Resources");
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
This loop is identical for v1 and v2. v1 uses `builtin_flat` for Level 0. v2 replaces it with a quota-aware project picker.
|
||
|
||
### Preemption flow
|
||
|
||
```
|
||
scheduler controller worker
|
||
─────────── ────────── ──────
|
||
preempt_policy.select_victims(J)
|
||
→ [victim1, victim2]
|
||
|
||
XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim
|
||
.victim_jobids[] deliver to each job → JOB_SUSPENDED
|
||
.signal = SIGSTP victim's worker release resources
|
||
.preempting_jobid = J
|
||
|
||
J pended → next round → retry alloc
|
||
```
|
||
|
||
V1 uses SIGSTP (terminal stop — process groups paused, job stays in queue). Future escalation to SIGKILL kill-and-requeue is deferred.
|
||
|
||
### What stays unchanged
|
||
- Existing plugin hooks (config, queue, job, worker, state_change, commit, election, prepare_allocation, rusage) — all preserved
|
||
- `seqjob_allocator` — wrapped with iterator interface, internally still 1host×1slot
|
||
- Select expression system — no changes
|
||
- COMPOUND and ALTER resreq types — unchanged (PRIMARY focus)
|
||
- Controller communication (handler.c opcodes, decision cbuf) — extended, not replaced
|
||
- `HOOK_DETAIL_JOB_STOP` — reused for preemption stop notifications
|
||
|
||
### What's new
|
||
- `XCE_DECISION_PREEMPT` decision type
|
||
- `xce_preempt_decision_t` payload
|
||
- Three builtin plugins: `builtin_flat`, `builtin_strict_priority`, `builtin_fcfs`
|
||
- `project_name` on queue structs
|
||
- `job_pick_plugin` on queue + project structs
|
||
- `preemptible_queues`, `preemptible`, `preempt_policy` on queue struct
|
||
- `project_pick_plugin`, `queue_pick_plugin`, `job_pick_plugin` on scheduler struct
|
||
|
||
## Scope
|
||
|
||
### In
|
||
- Unified picker loop: `project → queue → job` with plugin slots at each level
|
||
- Three builtin plugins: `builtin_flat` (Level 0), `builtin_strict_priority` (Level 1), `builtin_fcfs` (Level 2)
|
||
- Picker plugin interface: `init/next/fini` — cancel-safe, session-aware
|
||
- Picker precedence chain: `queue → project → global`
|
||
- `xce_alloc_plan_iter_t` interface (header + implementation)
|
||
- `xce_host_iter_t` interface (header + implementation)
|
||
- `parjob_allocator` combinatorial plan generation (1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1)
|
||
- Preemption: `XCE_DECISION_PREEMPT` decision type, `xce_preempt_decision_t` payload, LSF-style cross-queue victim selection, SIGSTP mechanism via controller→worker pipeline
|
||
- Per-queue preemption config: `preemptible_queues`, `preemptible`, `preempt_policy`
|
||
- Session-level unsatisfiable resreq fingerprint cache
|
||
- Data structure changes to `xce_scheduler_t`, `xce_sched_queue_t`, `xce_sched_project_t`, `decision.h`
|
||
- Plugin interface contract documentation (cancel-safe, session-aware)
|
||
- Unit tests for plan generation, host iteration, cache hit/miss, picker resolution, preemption victim selection
|
||
- Integration test: multi-slot parallel job allocation and preemption flow end-to-end
|
||
|
||
### Out
|
||
- New job selection policy implementations beyond the three builtins (priority, fair-share, deadline, preemption, backfill, etc.) — the configurable slots enable them, but none ship in this spec
|
||
- Hierarchical project resource quotas (Level 0 full) — deferred to `project-resource-ownership` spec
|
||
- Resreq template registry (named/shared resreq objects) — future enhancement
|
||
- Select expression language changes
|
||
- Controller-side scheduling state replication (separate concern: commit barrier wiring, see roadmap hardening item)
|
||
- Compound/ALTER resreq allocation changes
|
||
- GPU/affinity-aware allocation (pluggable later via allocator plugins)
|
||
- Kill-and-requeue preemption escalation (v1 is SIGSTP only, suspend-and-retry)
|
||
- Deadline field on `xce_sched_job_t` (needed for deadline-based pickers — deferred)
|
||
|
||
## Future: Project Resource Ownership
|
||
|
||
Deferred to a separate `project-resource-ownership` spec. When implemented:
|
||
|
||
- Projects form a tree with resource quota at each node
|
||
- Level 0 project picker selects which project gets resources (fair-share, priority, proportional)
|
||
- Level 1 queue picker is scoped to the selected project's queues
|
||
- Level 2 job picker sees only jobs from the selected project within each queue
|
||
- Project quota enforcement: projects at or over quota are skipped by Level 0
|
||
- The unified picker loop (above) requires zero code changes — only a new project picker plugin replaces `builtin_flat`
|
||
|
||
The architecture is designed so that v1 (flat project space) and v2 (hierarchical projects) share the identical scheduling loop.
|
||
|
||
## Boundary Candidates
|
||
|
||
The work naturally splits into these implementation phases (single spec, sequential):
|
||
|
||
1. **Unified picker loop + builtin plugins** — implement the `project → queue → job` loop with `builtin_flat`, `builtin_strict_priority`, `builtin_fcfs`; retire `sched_job_picker()`; add plugin slots and precedence chain to structs
|
||
2. **Iterator interfaces** — define `xce_alloc_plan_iter_t` and `xce_host_iter_t`, implement for seqjob path first (trivial wrapper), validate the abstraction
|
||
3. **Parallel job allocator** — implement combinatorial plan generation in `parjob_allocator`, wire through host iterator
|
||
4. **Preemption** — implement `XCE_DECISION_PREEMPT` decision type, cross-queue victim selection policy, SIGSTP pipeline (scheduler→controller→worker), queue preemption config fields
|
||
5. **Unsatisfiable resreq cache** — session-level fingerprint cache, wire into scheduling loop (skip before allocation + preemption attempt)
|
||
6. **Integration + testing** — end-to-end multi-slot parallel job allocation, preemption flow, picker resolution, cache behavior, HA soak regression
|
||
|
||
## Out of Boundary
|
||
- New job selection policy implementations (priority, fair-share, etc.) — these are new plugins that USE the configurable picker, not part of this spec
|
||
- Resreq template sharing across jobs (future: `resreq_registry` spec)
|
||
- Commit barrier wiring for scheduler decisions (separate hardening item in roadmap)
|
||
- Controller-side changes (the scheduler runs independently, receives updates via existing opcodes)
|
||
|
||
## Upstream / Downstream
|
||
|
||
### Upstream (dependencies)
|
||
- Existing plugin framework (`plugin.h`, `plugin.c`) — NO changes needed
|
||
- Existing select expression evaluator (`resreq.h` eval/Gc) — NO changes needed
|
||
- Existing shared resource tracking (`sharedInst`/`sharedInUse`) — NO changes needed
|
||
- Existing scheduler data structures (`xce_scheduler_t`, `xce_sched_ctx_t`) — minimal additions
|
||
|
||
### Downstream (future consumers)
|
||
- **New job selection policies** — priority, fair-share, preemption plugins register via the configurable picker
|
||
- **Affinity/GPU allocators** — register as allocator plugins, implement plan iterator for topology-aware placement
|
||
- **Commit barrier wiring** — scheduling decisions flow through majority commit (separate hardening item)
|
||
- **Resreq template registry** — named resreqs shared across jobs, reference-counted
|
||
|
||
## Existing Spec Touchpoints
|
||
|
||
### Extends
|
||
- **scheduler-controller-msg-gap** — adjacent (message handling). The framework enhancement operates on the same `xce_scheduler_t` state. No conflicts; the msg-gap spec fills handler gaps this spec reads from.
|
||
|
||
### Adjacent
|
||
- **ha-consistency-invariants** — commit barrier that scheduling decisions will eventually flow through. This spec does NOT wire the barrier; it only makes allocation decisions that the barrier WILL consume.
|
||
- **worker-join-refactor** — worker state management. Allocation consumes worker resource availability pushed via the existing join/update path.
|
||
|
||
## Constraints
|
||
- **C (gnu17)** — no C++, no external libraries beyond existing (capnp, zlib, pthreads)
|
||
- **Non-blocking in plugin hooks** — allocation happens on allocator thread, not iomgr. No new blocking concerns.
|
||
- **Cancel-safe** — job picker thread is `pthread_cancel`-able. Iterator `next()` calls must have cancellation points.
|
||
- **Session isolation** — iterator state and resreq cache are per-session, cleared on `begin_session`.
|
||
- **Backward compatible** — existing FCFS + seqjob behavior unchanged. New functionality is additive.
|
||
- **Memory** — lazy generation avoids materializing all combinations. Worst-case for a job needing N slots on M hosts: M×N candidate evaluations (acceptably bounded).
|