add scheduler related projects
This commit is contained in:
parent
986474eb65
commit
5e8a20178b
12 changed files with 3602 additions and 265 deletions
|
|
@ -2,13 +2,14 @@
|
|||
|
||||
## Problem
|
||||
The XCE scheduler has a plugin architecture but critical gaps prevent it from being a real framework:
|
||||
1. **Job selection policy is hardcoded** — `sched_job_picker()` in `scheduler.c:39` uses the literal string `"xce_sched_fcfs"`. No way to swap policies without recompilation. The picker is monolithic — it both chooses which queue to serve and which job within the queue.
|
||||
1. **Job selection policy is hardcoded** — `sched_job_picker()` in `scheduler.c:39` uses the literal string `"xce_sched_fcfs"`. The picker runs in a dedicated thread with `jbuf` cbuf communication. No way to swap policies without recompilation.
|
||||
2. **Parallel job allocation is a stub** — `parjob_allocator.c:my_alloc()` returns 0 with no logic. Jobs requiring multiple slots (`maxTasks > 1`) produce no allocation.
|
||||
3. **Allocation is one-shot** — `sched_make_allocation()` tries the first matching host and stops. No mechanism to try alternative allocation plans (different host×slot combinations) when the first fails.
|
||||
4. **No preemption** — when a high-priority job cannot be scheduled due to insufficient free resources, there is no mechanism to preempt lower-priority running jobs to free resources.
|
||||
5. **No project-level scheduling** — `xce_sched_project_t` is just `{ char* name }`. Projects have no resource ownership, quota, or scheduling policy. Queue and project roles are conflated.
|
||||
5. **No project-level scheduling** — `xce_sched_project_t` is just `{ char* name }`. Projects have no resource ownership, quota, or scheduling policy.
|
||||
6. **Monolithic scheduler** — the scheduler is tightly coupled to controller I/O (capnp, `apictx`). No way to replay recorded events for debugging or test scheduling policies without a live cluster.
|
||||
|
||||
Operators need configurable scheduling policies at multiple levels, parallel job support, and preemption. The scheduler must generate multiple candidate allocation plans and try them in preference order.
|
||||
Operators need configurable scheduling policies, parallel job support, preemption, and a way to debug/tune scheduling behavior without a live cluster.
|
||||
|
||||
## Current State
|
||||
|
||||
|
|
@ -21,36 +22,49 @@ Operators need configurable scheduling policies at multiple levels, parallel job
|
|||
- **Select expression AST** (`resreq.h`): full parser/evaluator with per-session memoization (`select_eval_ctx_t`) and inter-session GC.
|
||||
- **Shared resources** (`xce_scheduler_t.sharedInst`/`sharedInUse`): zarray of resource values tracked per scheduling session.
|
||||
- **Scheduling loop** (`scheduler.c:sched_allocator()`): pick job → `sched_make_allocation()` → pend or dispatch → next.
|
||||
- **Controller integration** (`handler.c`): receives capnp push/notify → enqueues `xce_sched_update_request_t` → `sched_update()` dequeues. Decisions published via `sched->apictx`.
|
||||
|
||||
### What's missing
|
||||
- Configurable job picker: the plugin name is a hardcoded string literal; a monolithic picker conflates inter-queue and intra-queue selection
|
||||
- Configurable job picker: the plugin name is a hardcoded string literal
|
||||
- Parallel job allocation: no combinatorial plan generation, no multi-slot host matching
|
||||
- Iterator/lazy generation: allocation is eager one-shot, no retry with alternative plans
|
||||
- Preemption: no mechanism to preempt lower-priority running jobs when a high-priority job cannot be scheduled
|
||||
- Project-level scheduling: `xce_sched_project_t` is just `{ char* name }` — no resource ownership, quota, or scheduling policy
|
||||
- Preemption: no mechanism to preempt lower-priority running jobs
|
||||
- Project-level scheduling: `xce_sched_project_t` is just `{ char* name }`
|
||||
- Cross-job resreq optimization: no mechanism to skip already-failed resreq patterns within a session
|
||||
- Picker composition: no way to assign different pickers to different queues or projects
|
||||
- Scheduler simulation/debugging: no way to replay recorded events, no virtual clock, no deterministic mode
|
||||
- Core/driver separation: scheduling logic is tightly coupled to controller I/O
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
### Four-Level Scheduling Architecture
|
||||
### Three Architectural Foundations
|
||||
|
||||
The scheduling loop is always `project → queue → job`, composing pickers at each level:
|
||||
**1. Pluggable Job Picker** — the core scheduling loop calls `picker->next_job()` to get the next schedulable job, with zero knowledge of how jobs are selected. The default `builtin_fcfs` picker internally implements queue-priority ordering and FCFS job ordering, replicating today's behavior exactly. Per-queue picker overrides are supported. The picker runs synchronously in the core's scheduling thread — no dedicated picker thread, no `jbuf` cbuf.
|
||||
|
||||
| Level | Picker | Plugin slot | Default (v1) |
|
||||
|-------|--------|-------------|---------------|
|
||||
| 0 | Project picker | `sched->project_pick_plugin` | `builtin_flat` — single "default" project |
|
||||
| 1 | Inter-queue picker | `sched->queue_pick_plugin` | `builtin_strict_priority` |
|
||||
| 2 | Intra-queue picker | `queue→project→sched` chain | `builtin_fcfs` |
|
||||
| 3 | Preemption policy | `queue->preempt_policy` | `lowest_priority_first` |
|
||||
The picker plugin interface (`xce_job_picker_iface_t`):
|
||||
|
||||
1. **Unified picker loop**: A single scheduling loop that always follows `project → queue → job`. In v1 (flat project space), the project picker `builtin_flat` yields one "default" project containing all queues — identical to today's behavior. In v2 (future), the project picker becomes quota-aware and hierarchical, without changing the loop code.
|
||||
2. **Configurable job selection at every level**: each picker slot is a plugin. Level 1 (inter-queue) picks which queue to serve next. Level 2 (intra-queue) picks which job within that queue, with per-queue customization via the `queue → project → global` precedence chain.
|
||||
3. **Lazy allocation plan iterator**: `xce_alloc_plan_iter_t` — generates candidate (nhosts, slots_per_host) plans one at a time, sorted by preference. Consumer tries each plan until one succeeds or all fail.
|
||||
4. **Lazy host candidate iterator**: for a given allocation plan, iterate candidate hosts filtered by select expression + resource availability. Composes with plan iterator.
|
||||
5. **Parallel job combinatorial allocation**: `parjob_allocator` implements the plan iterator. For a job needing N slots: generate plans [1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1]. Each plan: collect hosts with enough slots, try each until success.
|
||||
6. **Preemption**: when allocation fails for a high-priority job, select victim jobs from lower-priority queues (LSF-style per-queue `preemptible_queues` list), send SIGSTP via controller→worker pipeline, and retry allocation next round.
|
||||
7. **Session-level unsatisfiable resreq cache**: if a primary resreq cannot be satisfied in the current scheduling session (all plans + preemption exhausted), fingerprint it. Skip subsequent jobs with the same fingerprint for the rest of the session.
|
||||
| Category | Hooks | Purpose |
|
||||
|----------|-------|---------|
|
||||
| Lifecycle | `init`, `fini` | Setup and teardown |
|
||||
| State-change | `on_queue_update`, `on_job_push`, `on_job_state_change`, `on_job_update` | Called by core when events arrive from driver |
|
||||
| Session | `begin_session`, `next_job`, `end_session` | Scheduling round lifecycle |
|
||||
| Post-allocation | `on_dispatch`, `on_pend` | Notification after allocation attempt |
|
||||
| Optional | `on_config`, `on_job_rusage` | Advanced pickers (NULL if unused) |
|
||||
|
||||
**2. Iterator-Driven Allocation** — lazy `xce_alloc_plan_iter_t` (candidate (nhosts, slots_per_host) plans, most-preferred first) composed with lazy `xce_host_iter_t` (candidate hosts for a given plan). `parjob_allocator` implements real combinatorial multi-slot allocation. Session-level unsatisfiable resreq fingerprint cache avoids re-evaluating already-proven-unsatisfiable resreqs.
|
||||
|
||||
**3. Core/Driver Separation** — the scheduler core is a passive, event-driven engine with zero controller I/O dependencies. The driver connects it to the outside world:
|
||||
|
||||
| Mode | Driver | Input | Output |
|
||||
|------|--------|-------|--------|
|
||||
| Normal | Controller-connected | capnp push/notify → core events | Core decisions → controller |
|
||||
| Simulation | Replay file | JSONL event file → core events | Core decisions → log (with `--sim-verify` diff against recorded) |
|
||||
|
||||
- **Discrete event-driven virtual clock**: jumps directly to next event timestamp, no fixed tick, monotonic
|
||||
- **Snapshot + delta replay**: load state snapshot, replay only subsequent delta events
|
||||
- **Deterministic guarantee**: same initial state + same events = same decisions
|
||||
- **Runtime recording control**: xceadmin `scheduler record start/stop` commands
|
||||
- **JSONL replay format**: one event per line, debuggable, plain text
|
||||
- **What-if branching**: load snapshot, replay alternative delta events
|
||||
|
||||
### Queue vs Project — Roles
|
||||
|
||||
|
|
@ -59,10 +73,10 @@ The scheduling loop is always `project → queue → job`, composing pickers at
|
|||
| **Role** | Scheduling mechanism | Resource ownership |
|
||||
| **Question** | HOW are jobs ordered and constrained? | WHO owns resources and gets charged? |
|
||||
| **Scope** | One hardware/scheduling profile | Cross-queue, cross-cluster |
|
||||
| **Key attributes** | Priority, limits, time windows, picker, preemption config | Quota, usage, hierarchy, members (future) |
|
||||
| **Key attributes** | Priority, limits, time windows, per-queue picker override, preemption config | Quota, usage, hierarchy, members (future) |
|
||||
| **Cardinality** | 1 job → 1 queue, many-to-many | 1 job → 1 project, many-to-many |
|
||||
|
||||
One project submits to many queues (different hardware profiles). One queue receives from many projects (shared hardware).
|
||||
One project submits to many queues (different hardware profiles). One queue receives from many projects (shared hardware). The picker plugin may use `queue->project_name` internally for project-scoped iteration (v2).
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
|
|
@ -71,237 +85,267 @@ One project submits to many queues (different hardware profiles). One queue rece
|
|||
| Aspect | Detail |
|
||||
|--------|--------|
|
||||
| **Job selection** | Add `job_pick_plugin` config field, same as chosen approach |
|
||||
| **Parallel jobs** | Implement parjob allocator with pre-computed plan list. Eager generation — all plans materialized before first try |
|
||||
| **Iterator** | No new abstraction. Simple for-loop over plan array, inner for-loop over hosts |
|
||||
| **Shared resreq** | `xlist_t unsatisfiable_resreqs` in session state, same as chosen approach |
|
||||
| **Parallel jobs** | Implement parjob allocator with pre-computed plan list. Eager generation |
|
||||
| **Iterator** | No new abstraction. Simple for-loop over plan array |
|
||||
| **Shared resreq** | `xlist_t unsatisfiable_resreqs` in session state |
|
||||
|
||||
**Rejected because:** Eager plan generation causes memory pressure for large combinatorial spaces (e.g., 64 slots on 128 hosts = thousands of plans). No reusable abstraction — each new allocator reimplements the loop. Short-term win, long-term rework cost.
|
||||
**Rejected because:** Eager plan generation causes memory pressure for large combinatorial spaces. No reusable abstraction — each new allocator reimplements the loop. Short-term win, long-term rework cost.
|
||||
|
||||
### Approach B: Iterator-Driven Allocation Architecture ★ Chosen
|
||||
### Approach B: Iterator-Driven Allocation + Picker Plugin ★ Chosen
|
||||
|
||||
| Aspect | Detail |
|
||||
|--------|--------|
|
||||
| **Job selection** | Configurable plugin name via `job_pick_plugin` field |
|
||||
| **Job selection** | Pluggable `xce_job_picker_iface_t` with `next_job()`. Single `builtin_fcfs` default. Per-queue picker override. Synchronous, no dedicated thread. |
|
||||
| **Iterator abstraction** | `xce_alloc_plan_iter_t` + `xce_host_iter_t` — lazy, cancel-safe, composable |
|
||||
| **Parallel jobs** | Combinatorial plan generation via plan iterator: 1host×N → 2hosts×⌈N/2⌉ → … → Nhhosts×1 |
|
||||
| **Host iterator** | Per-plan lazy candidate host iteration using existing select expression eval |
|
||||
| **Preemption** | `XCE_DECISION_PREEMPT` decision type, LSF-style cross-queue victim selection, SIGSTP via controller→worker |
|
||||
| **Shared resreq** | Session-level fingerprint cache: `{select, rusage, user, queue, project}` |
|
||||
| **Core/driver split** | Passive event-driven core + dual-mode driver (normal/simulation) |
|
||||
| **Simulation** | JSONL replay, snapshot + delta, discrete event-driven virtual clock, deterministic |
|
||||
|
||||
**Chosen because:** Lazy generation avoids memory pressure. Iterator composition matches the user's vision: plan iterator → host iterator → try alloc. Clean separation of concerns. Each iterator implementation is independently testable.
|
||||
**Chosen because:** Lazy generation avoids memory pressure. Single `next_job()` interface makes pickers truly pluggable — the core loop never changes. Core/driver separation enables simulation mode for debugging/tuning without a live cluster. Iterator composition matches the user's vision: plan iterator → host iterator → try alloc.
|
||||
|
||||
### Approach C: Full Scheduler Framework (deferred)
|
||||
|
||||
| Aspect | Detail |
|
||||
|--------|--------|
|
||||
| **Job selection** | Policy chain — multiple plugins score/veto jobs. Composable: priority + fair-share + preemption |
|
||||
| **Job selection** | Policy chain — multiple plugins score/veto jobs |
|
||||
| **Iterator** | Triple composition: job iterator → plan iterator → host iterator |
|
||||
| **Resreq registry** | Named primary resreqs registered globally, referenced by name, reference-counted |
|
||||
| **Allocation plugins** | Different strategies as plugins: greedy, bin-packing, round-robin, topology-aware |
|
||||
| **Session state machine** | Formalized STAGING → ELECT → ALLOCATE → COMMIT with well-defined transitions |
|
||||
| **Resreq registry** | Named primary resreqs registered globally, reference-counted |
|
||||
| **Allocation plugins** | Different strategies as plugins: greedy, bin-packing, round-robin |
|
||||
| **Session state machine** | Formalized STAGING → ELECT → ALLOCATE → COMMIT |
|
||||
|
||||
**Deferred (not rejected):** The policy chain and resreq registry are valuable but lack current demand. Approach B's iterator abstraction is designed to compose upward — the plan iterator and host iterator are building blocks that a future job iterator or policy chain can consume. The template registry can layer on top without breaking the iterator interfaces.
|
||||
**Deferred (not rejected):** The policy chain and resreq registry are valuable but lack current demand. Approach B's picker interface is designed to compose upward — a future policy-chain picker implements the same `next_job()` interface.
|
||||
|
||||
### Decision rationale
|
||||
### Decision Rationale
|
||||
|
||||
Approach B chosen over A because the iterator abstraction is a one-time design cost that pays off immediately (parjob combinatorial allocation needs it) and compounds (every future allocator reuses it). The four-level picker architecture (project → queue → job + preemption) extends Approach B with design elements from Approach C (policy composition, formalized level separation) that are needed NOW (per-queue pickers, preemption), while deferring the remaining Approach C layers (policy chains with scoring, resreq registry, formal state machine) until real use cases emerge. The iterator interfaces and picker slots are designed to compose upward when needed.
|
||||
Approach B chosen over A because the iterator abstraction pays off immediately (parjob combinatorial allocation) and compounds (every future allocator reuses it). The picker plugin interface with `next_job()` keeps the core loop simple (5 lines) and makes pickers truly pluggable. Core/driver separation enables simulation mode — a capability that would be prohibitively expensive to retrofit later.
|
||||
|
||||
**Updated decision (2026-07-10):** The original Approach B treated the job picker as a single global plugin (`job_pick_plugin` on `xce_scheduler_t`). Design discussion identified that different queues/projects need different pickers, and that preemption is an essential missing mechanism. The architecture now specifies four composable levels (project, queue, job, preemption) with per-queue configuration, LSF-style cross-queue preemption, and a unified loop that works identically for v1 (flat projects) and v2 (hierarchical projects). The project picker level is deferred to a future spec but the architecture slot and interface contract are defined now.
|
||||
**Design evolution (2026-07-10 → 2026-07-27):** The original design specified a four-level architecture with three separate builtin plugins (`builtin_flat`, `builtin_strict_priority`, `builtin_fcfs`) and a three-level `project→queue→job` loop hardcoded in the core. Analysis of the existing FCFS plugin (`fcfs.c`) showed it already encapsulates all picking logic internally. The architecture was simplified to a single `next_job()` plugin interface with one builtin picker (`builtin_fcfs`) — the three-level decomposition is now an internal implementation detail of the plugin, not the core loop. Core/driver separation was added (Req 10) to enable simulation mode for policy debugging and tuning.
|
||||
|
||||
## Approach
|
||||
|
||||
**Iterator-Driven Allocation Architecture with Four-Level Picker Composition** (extended from original Approach B):
|
||||
**Iterator-Driven Allocation with Pluggable Picker + Core/Driver Separation**:
|
||||
|
||||
### Core abstractions
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ Level 0: Project Picker [sched->project_pick_plugin] │
|
||||
│ │
|
||||
│ v1 (builtin_flat): yields one "default" project │
|
||||
│ v2 (future): quota-aware project selection │
|
||||
└──────────────────────┬───────────────────────────────────┘
|
||||
│ project scope
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ Level 1: Inter-Queue Picker [sched->queue_pick_plugin] │
|
||||
│ │
|
||||
│ Iterates queues scoped to the selected project │
|
||||
│ Default: strict_priority (descending queue->priority) │
|
||||
└──────────────────────┬───────────────────────────────────┘
|
||||
│ queue
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ Level 2: Intra-Queue Picker [queue→project→sched] │
|
||||
│ │
|
||||
│ Picks one job from the selected queue │
|
||||
│ Default: builtin_fcfs (priority desc, queue_time asc) │
|
||||
└──────────────────────┬───────────────────────────────────┘
|
||||
│ job
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ Allocation │
|
||||
│ │
|
||||
│ xce_alloc_plan_iter_t xce_host_iter_t │
|
||||
│ ┌─────────────────────┐ ┌──────────────────────┐ │
|
||||
│ │ init(resreq, sched) │ │ init(plan, workers) │ │
|
||||
│ │ next() → plan │───────▶│ next() → worker │ │
|
||||
│ │ fini() │ │ fini() │ │
|
||||
│ └─────────────────────┘ └──────────────────────┘ │
|
||||
│ │
|
||||
│ success → dispatch │
|
||||
│ fail → Level 3 (preemption) │
|
||||
└──────────────────────┬──────────────────────────────────┘
|
||||
│ fail
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ Level 3: Preemption │
|
||||
│ │
|
||||
│ if queue->preemptible_queues: │
|
||||
│ victims = preempt_policy.select_victims(job) │
|
||||
│ if victims: commit XCE_DECISION_PREEMPT │
|
||||
│ → controller → worker → SIGSTP victims │
|
||||
│ → victims → JOB_SUSPENDED, release resources │
|
||||
│ → pend job for next round │
|
||||
│ else: pend(job, "Resources") │
|
||||
│ else: pend(job, "Resources") │
|
||||
└──────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Picker plugin interface
|
||||
|
||||
Each picker level follows the same contract:
|
||||
```c
|
||||
typedef struct {
|
||||
void* (*init)(xce_scheduler_t* sched, void* parent_scope);
|
||||
void* (*next)(void* iter); // returns scope or NULL
|
||||
void (*fini)(void* iter);
|
||||
} picker_iface_t;
|
||||
```
|
||||
|
||||
### Unified scheduling loop
|
||||
### Core scheduling loop
|
||||
|
||||
```c
|
||||
void* sched_allocator(void* arg) {
|
||||
while (1) {
|
||||
// Level 0: project scope
|
||||
scope = project_picker.next(sched);
|
||||
if (!scope) { sleep(1); continue; }
|
||||
picker->begin_session(picker);
|
||||
|
||||
// Level 1: queue (scoped to project)
|
||||
queue = queue_picker.next(sched, scope);
|
||||
if (!queue) { project_picker.advance(); continue; }
|
||||
|
||||
// Level 2: job (within queue)
|
||||
// Precedence: queue->job_pick_plugin → project->job_pick_plugin → sched->job_pick_plugin
|
||||
picker = resolve_job_picker(sched, queue);
|
||||
job = picker->elect_one(queue);
|
||||
|
||||
if (!job) { queue_picker.advance(); continue; }
|
||||
|
||||
// Allocation + preemption
|
||||
if (try_allocate(sched, job)) {
|
||||
dispatch(sched, job);
|
||||
} else if (can_preempt(sched, job)) {
|
||||
preempt_and_pend(sched, job);
|
||||
} else {
|
||||
pend(sched, job, "Resources");
|
||||
xce_sched_job_t* job;
|
||||
while ((job = picker->next_job(picker)) != NULL) {
|
||||
if (try_allocate(job) > 0) {
|
||||
picker->on_dispatch(picker, job->jobid);
|
||||
emit_decision(DISPATCH, job);
|
||||
} else if (can_preempt(job)) {
|
||||
emit_decision(PREEMPT, job, victims);
|
||||
picker->on_pend(picker, job->jobid);
|
||||
} else {
|
||||
emit_decision(PEND, job);
|
||||
picker->on_pend(picker, job->jobid);
|
||||
}
|
||||
}
|
||||
|
||||
picker->end_session(picker);
|
||||
sleep(1);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This loop is identical for v1 and v2. v1 uses `builtin_flat` for Level 0. v2 replaces it with a quota-aware project picker.
|
||||
The core loop has zero knowledge of project/queue/job ordering — all selection logic lives behind `next_job()`.
|
||||
|
||||
### Picker plugin interface
|
||||
|
||||
```c
|
||||
typedef struct xce_job_picker {
|
||||
// Lifecycle
|
||||
int (*init)(void* picker, xce_sched_core_t* core);
|
||||
void (*fini)(void* picker);
|
||||
|
||||
// State-change hooks (called by core when events arrive)
|
||||
int (*on_queue_update)(void* picker, xce_sched_queue_t* queue);
|
||||
int (*on_job_push)(void* picker, xce_sched_job_t* job);
|
||||
int (*on_job_state_change)(void* picker, xce_sched_job_t* job, uint32_t state);
|
||||
int (*on_job_update)(void* picker, xce_sched_job_t* job);
|
||||
|
||||
// Session lifecycle
|
||||
int (*begin_session)(void* picker);
|
||||
int (*end_session)(void* picker);
|
||||
|
||||
// Job selection
|
||||
xce_sched_job_t* (*next_job)(void* picker);
|
||||
|
||||
// Post-allocation notification
|
||||
int (*on_dispatch)(void* picker, uint64_t jobid);
|
||||
int (*on_pend)(void* picker, uint64_t jobid);
|
||||
|
||||
// Optional hooks (NULL = not implemented)
|
||||
int (*on_config)(void* picker, xce_conf4sched_t* config);
|
||||
int (*on_job_rusage)(void* picker, xce_sched_job_t* job, xce_job_rusages_t* rusages);
|
||||
} xce_job_picker_iface_t;
|
||||
```
|
||||
|
||||
### builtin_fcfs internal design
|
||||
|
||||
The default picker internally implements the equivalent of today's selection:
|
||||
|
||||
```
|
||||
builtin_fcfs.next_job():
|
||||
Iterate queues by priority (descending)
|
||||
Skip queues with no ready jobs
|
||||
For each queue:
|
||||
If queue has per-queue picker override:
|
||||
Delegate to that picker
|
||||
Else:
|
||||
Iterate jobs by priority desc, queue_time asc (FCFS)
|
||||
Skip jobs not ready
|
||||
Return first ready job
|
||||
Return NULL if all queues exhausted
|
||||
```
|
||||
|
||||
This is an implementation detail of the plugin, not visible to the core loop.
|
||||
|
||||
### Preemption flow
|
||||
|
||||
```
|
||||
scheduler controller worker
|
||||
─────────── ────────── ──────
|
||||
preempt_policy.select_victims(J)
|
||||
alloc fails for high-prio job J
|
||||
|
||||
preempt_policy.select_victims(J):
|
||||
collect running jobs from
|
||||
Q_aggressor.preemptible_queues
|
||||
filter: victim.prio < J.prio
|
||||
filter: victim.queue.preemptible
|
||||
rank: lowest priority first
|
||||
select minimum set satisfying J.resreq
|
||||
|
||||
→ [victim1, victim2]
|
||||
|
||||
XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim
|
||||
.victim_jobids[] deliver to each job → JOB_SUSPENDED
|
||||
.signal = SIGSTP victim's worker release resources
|
||||
XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim1
|
||||
.victim_jobids = [v1, v2] deliver to each job → JOB_SUSPENDED
|
||||
.signal = SIGSTP victim's worker release resources
|
||||
.preempting_jobid = J
|
||||
|
||||
J pended → next round → retry alloc
|
||||
```
|
||||
|
||||
V1 uses SIGSTP (terminal stop — process groups paused, job stays in queue). Future escalation to SIGKILL kill-and-requeue is deferred.
|
||||
### Core/Driver Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ DRIVER LAYER │
|
||||
│ │
|
||||
│ Normal Mode Simulation Mode │
|
||||
│ controller capnp ──┐ replay.jsonl ──┐ │
|
||||
│ handler → events │ virtual clock │ │
|
||||
│ → publish decisions│ → record/log │ │
|
||||
│ │ ▲ │ │ ▲ │ │
|
||||
└───────┼───────┼─────┴──────────────┼───────┼───┴──────────────┘
|
||||
│ │ decisions │ │ decisions
|
||||
│ │ (callback) │ │ (callback)
|
||||
▼ │ ▼ │
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ SCHEDULER CORE │
|
||||
│ │
|
||||
│ core_push_event() → picker hooks → picker->next_job() │
|
||||
│ → try_allocate() → DISPATCH / PREEMPT / PEND │
|
||||
│ │
|
||||
│ ZERO dependencies on: controller, capnp, network, wall-clock │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### What stays unchanged
|
||||
- Existing plugin hooks (config, queue, job, worker, state_change, commit, election, prepare_allocation, rusage) — all preserved
|
||||
- Existing plugin hooks (config, queue, job, worker, state_change, commit, election, prepare_allocation, rusage) — preserved, but the picker no longer uses the dedicated `job_elect_fn` thread
|
||||
- `seqjob_allocator` — wrapped with iterator interface, internally still 1host×1slot
|
||||
- Select expression system — no changes
|
||||
- COMPOUND and ALTER resreq types — unchanged (PRIMARY focus)
|
||||
- Controller communication (handler.c opcodes, decision cbuf) — extended, not replaced
|
||||
- Controller communication (handler.c opcodes, decision cbuf in driver) — extended, not replaced
|
||||
- `HOOK_DETAIL_JOB_STOP` — reused for preemption stop notifications
|
||||
|
||||
### What's new
|
||||
- `XCE_DECISION_PREEMPT` decision type
|
||||
- `xce_preempt_decision_t` payload
|
||||
- Three builtin plugins: `builtin_flat`, `builtin_strict_priority`, `builtin_fcfs`
|
||||
- `xce_job_picker_iface_t` — pluggable picker interface with full hook specification
|
||||
- `xce_alloc_plan_iter_t` + `xce_host_iter_t` — lazy iterator abstractions
|
||||
- `XCE_DECISION_PREEMPT` decision type + `xce_preempt_decision_t` payload
|
||||
- `builtin_fcfs` — single builtin picker (replaces three separate builtins from original design)
|
||||
- `xce_sched_core_t` / `xce_sched_driver_t` / `xce_sched_sim_driver_t` — core/driver struct split
|
||||
- `xce_core_event_t` — typed event interface between driver and core
|
||||
- `xce_core_decision_cb_t` — decision callback (core → driver)
|
||||
- JSONL replay file format + snapshot/restore
|
||||
- Discrete event-driven virtual clock
|
||||
- `project_name` on queue structs
|
||||
- `job_pick_plugin` on queue + project structs
|
||||
- `job_pick_plugin` on scheduler + queue structs (per-queue override)
|
||||
- `preemptible_queues`, `preemptible`, `preempt_policy` on queue struct
|
||||
- `project_pick_plugin`, `queue_pick_plugin`, `job_pick_plugin` on scheduler struct
|
||||
|
||||
## Scope
|
||||
|
||||
### In
|
||||
- Unified picker loop: `project → queue → job` with plugin slots at each level
|
||||
- Three builtin plugins: `builtin_flat` (Level 0), `builtin_strict_priority` (Level 1), `builtin_fcfs` (Level 2)
|
||||
- Picker plugin interface: `init/next/fini` — cancel-safe, session-aware
|
||||
- Picker precedence chain: `queue → project → global`
|
||||
- `xce_alloc_plan_iter_t` interface (header + implementation)
|
||||
- `xce_host_iter_t` interface (header + implementation)
|
||||
- Pluggable job picker interface (`xce_job_picker_iface_t` with `next_job()`) hiding all selection logic
|
||||
- Single builtin picker (`builtin_fcfs`) replicating today's behavior
|
||||
- Configurable picker plugin with per-queue override (`queue->job_pick_plugin`)
|
||||
- Picker hook specification: lifecycle, state-change, session, post-allocation, optional hooks
|
||||
- `xce_alloc_plan_iter_t` + `xce_host_iter_t` interfaces (lazy, cancel-safe, composable)
|
||||
- `parjob_allocator` combinatorial plan generation (1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1)
|
||||
- Preemption: `XCE_DECISION_PREEMPT` decision type, `xce_preempt_decision_t` payload, LSF-style cross-queue victim selection, SIGSTP mechanism via controller→worker pipeline
|
||||
- Preemption: `XCE_DECISION_PREEMPT` decision type, LSF-style cross-queue victim selection, SIGSTP mechanism
|
||||
- Per-queue preemption config: `preemptible_queues`, `preemptible`, `preempt_policy`
|
||||
- Session-level unsatisfiable resreq fingerprint cache
|
||||
- Data structure changes to `xce_scheduler_t`, `xce_sched_queue_t`, `xce_sched_project_t`, `decision.h`
|
||||
- Plugin interface contract documentation (cancel-safe, session-aware)
|
||||
- Unit tests for plan generation, host iteration, cache hit/miss, picker resolution, preemption victim selection
|
||||
- Core/driver separation: passive event-driven core, normal-mode driver, simulation-mode driver
|
||||
- Discrete event-driven virtual clock (monotonic, jumps to next event timestamp)
|
||||
- Runtime recording control via xceadmin commands (`scheduler record start/stop`)
|
||||
- JSONL replay file format (HEADER + SNAPSHOT + events + DECISIONS)
|
||||
- State snapshot/restore for replay efficiency
|
||||
- Deterministic replay guarantee for debugging/tuning
|
||||
- What-if branching (load snapshot, replay alternative delta events)
|
||||
- Data structure changes: `xce_scheduler_t`, `xce_sched_queue_t`, `xce_sched_project_t`, `decision.h`, `xce_sched_core_t`, `xce_sched_driver_t`, `xce_sched_sim_driver_t`, `xce_core_event_t`, `xce_job_picker_iface_t`
|
||||
- Picker plugin interface contract documentation
|
||||
- Unit tests: plan generation, host iteration, cache hit/miss, picker hook behavior, preemption victim selection, event recording/replay, snapshot/restore, deterministic replay
|
||||
- Integration test: multi-slot parallel job allocation and preemption flow end-to-end
|
||||
|
||||
### Out
|
||||
- New job selection policy implementations beyond the three builtins (priority, fair-share, deadline, preemption, backfill, etc.) — the configurable slots enable them, but none ship in this spec
|
||||
- Hierarchical project resource quotas (Level 0 full) — deferred to `project-resource-ownership` spec
|
||||
- Resreq template registry (named/shared resreq objects) — future enhancement
|
||||
- New job selection policies beyond the builtin FCFS picker — the pluggable interface enables them, but none ship here
|
||||
- Hierarchical project resource quotas — deferred to `project-resource-ownership` spec
|
||||
- Resreq template registry — future enhancement
|
||||
- Select expression language changes
|
||||
- Controller-side scheduling state replication (separate concern: commit barrier wiring, see roadmap hardening item)
|
||||
- Controller-side scheduling state replication (commit barrier wiring)
|
||||
- Compound/ALTER resreq allocation changes
|
||||
- GPU/affinity-aware allocation (pluggable later via allocator plugins)
|
||||
- Kill-and-requeue preemption escalation (v1 is SIGSTP only, suspend-and-retry)
|
||||
- Deadline field on `xce_sched_job_t` (needed for deadline-based pickers — deferred)
|
||||
- Kill-and-requeue preemption escalation (v1 is SIGSTP only)
|
||||
- Deadline field on `xce_sched_job_t`
|
||||
- Event payload JSON schema (open design question — deferred to design phase)
|
||||
- Distributed/cluster-mode simulation (v1 is single-scheduler replay only)
|
||||
|
||||
## Future: Project Resource Ownership
|
||||
|
||||
Deferred to a separate `project-resource-ownership` spec. When implemented:
|
||||
|
||||
- Projects form a tree with resource quota at each node
|
||||
- Level 0 project picker selects which project gets resources (fair-share, priority, proportional)
|
||||
- Level 1 queue picker is scoped to the selected project's queues
|
||||
- Level 2 job picker sees only jobs from the selected project within each queue
|
||||
- Project quota enforcement: projects at or over quota are skipped by Level 0
|
||||
- The unified picker loop (above) requires zero code changes — only a new project picker plugin replaces `builtin_flat`
|
||||
- A project-aware picker plugin selects which project gets resources, then iterates queues scoped to that project, then jobs within each queue
|
||||
- Project quota enforcement: projects over quota are skipped by the picker
|
||||
- The core scheduling loop requires zero code changes — only a new picker plugin replaces `builtin_fcfs`
|
||||
- `queue->project_name` (added here) provides the queue→project mapping the v2 picker consumes
|
||||
- `xce_sched_project_t` scaffolding (added here) provides the struct the v2 picker extends
|
||||
|
||||
The architecture is designed so that v1 (flat project space) and v2 (hierarchical projects) share the identical scheduling loop.
|
||||
The architecture is designed so that v1 (flat, `builtin_fcfs` ignores projects) and v2 (hierarchical, project-aware picker) share the identical core scheduling loop.
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
The work naturally splits into these implementation phases (single spec, sequential):
|
||||
|
||||
1. **Unified picker loop + builtin plugins** — implement the `project → queue → job` loop with `builtin_flat`, `builtin_strict_priority`, `builtin_fcfs`; retire `sched_job_picker()`; add plugin slots and precedence chain to structs
|
||||
2. **Iterator interfaces** — define `xce_alloc_plan_iter_t` and `xce_host_iter_t`, implement for seqjob path first (trivial wrapper), validate the abstraction
|
||||
3. **Parallel job allocator** — implement combinatorial plan generation in `parjob_allocator`, wire through host iterator
|
||||
4. **Preemption** — implement `XCE_DECISION_PREEMPT` decision type, cross-queue victim selection policy, SIGSTP pipeline (scheduler→controller→worker), queue preemption config fields
|
||||
5. **Unsatisfiable resreq cache** — session-level fingerprint cache, wire into scheduling loop (skip before allocation + preemption attempt)
|
||||
6. **Integration + testing** — end-to-end multi-slot parallel job allocation, preemption flow, picker resolution, cache behavior, HA soak regression
|
||||
1. **Core/driver separation** — split `xce_scheduler_t` into `xce_sched_core_t` + `xce_sched_driver_t`; define `xce_core_event_t` and `xce_core_decision_cb_t`; implement passive event-driven core interface; wire normal-mode driver to existing handler.c
|
||||
2. **Picker plugin interface + builtin_fcfs** — define `xce_job_picker_iface_t`; implement `builtin_fcfs` with all MUST/SHOULD hooks; retire dedicated picker thread + `jbuf`; make core loop call `next_job()` synchronously; add `job_pick_plugin` config with per-queue override
|
||||
3. **Iterator interfaces** — define `xce_alloc_plan_iter_t` and `xce_host_iter_t`, implement for seqjob path first (trivial wrapper), validate the abstraction
|
||||
4. **Parallel job allocator** — implement combinatorial plan generation in `parjob_allocator`, wire through host iterator
|
||||
5. **Preemption** — implement `XCE_DECISION_PREEMPT` decision type, cross-queue victim selection policy, SIGSTP pipeline, queue preemption config fields
|
||||
6. **Unsatisfiable resreq cache** — session-level fingerprint cache, wire into scheduling loop
|
||||
7. **Simulation driver** — implement simulation-mode driver with JSONL replay, virtual clock, snapshot/restore, deterministic verification
|
||||
8. **Recording + xceadmin commands** — implement runtime recording control, periodic snapshots during recording
|
||||
9. **Integration + testing** — end-to-end multi-slot parallel job allocation, preemption flow, picker resolution, cache behavior, simulation replay, HA soak regression
|
||||
|
||||
## Out of Boundary
|
||||
- New job selection policy implementations (priority, fair-share, etc.) — these are new plugins that USE the configurable picker, not part of this spec
|
||||
- New job selection policy implementations (priority, fair-share, etc.) — these are new plugins that implement `xce_job_picker_iface_t`
|
||||
- Resreq template sharing across jobs (future: `resreq_registry` spec)
|
||||
- Commit barrier wiring for scheduler decisions (separate hardening item in roadmap)
|
||||
- Controller-side changes (the scheduler runs independently, receives updates via existing opcodes)
|
||||
|
|
@ -312,10 +356,11 @@ The work naturally splits into these implementation phases (single spec, sequent
|
|||
- Existing plugin framework (`plugin.h`, `plugin.c`) — NO changes needed
|
||||
- Existing select expression evaluator (`resreq.h` eval/Gc) — NO changes needed
|
||||
- Existing shared resource tracking (`sharedInst`/`sharedInUse`) — NO changes needed
|
||||
- Existing scheduler data structures (`xce_scheduler_t`, `xce_sched_ctx_t`) — minimal additions
|
||||
- Existing scheduler data structures (`xce_scheduler_t`, `xce_sched_ctx_t`) — split, not replaced
|
||||
|
||||
### Downstream (future consumers)
|
||||
- **New job selection policies** — priority, fair-share, preemption plugins register via the configurable picker
|
||||
- **New job selection policies** — priority, fair-share, deadline, backfill plugins implement `xce_job_picker_iface_t`
|
||||
- **Project-resource-ownership** — project-aware picker plugin consumes `queue->project_name` and `xce_sched_project_t`
|
||||
- **Affinity/GPU allocators** — register as allocator plugins, implement plan iterator for topology-aware placement
|
||||
- **Commit barrier wiring** — scheduling decisions flow through majority commit (separate hardening item)
|
||||
- **Resreq template registry** — named resreqs shared across jobs, reference-counted
|
||||
|
|
@ -323,7 +368,7 @@ The work naturally splits into these implementation phases (single spec, sequent
|
|||
## Existing Spec Touchpoints
|
||||
|
||||
### Extends
|
||||
- **scheduler-controller-msg-gap** — adjacent (message handling). The framework enhancement operates on the same `xce_scheduler_t` state. No conflicts; the msg-gap spec fills handler gaps this spec reads from.
|
||||
- **scheduler-controller-msg-gap** — adjacent (message handling). The framework enhancement operates on the same scheduling state. No conflicts; the msg-gap spec fills handler gaps this spec reads from.
|
||||
|
||||
### Adjacent
|
||||
- **ha-consistency-invariants** — commit barrier that scheduling decisions will eventually flow through. This spec does NOT wire the barrier; it only makes allocation decisions that the barrier WILL consume.
|
||||
|
|
@ -332,7 +377,8 @@ The work naturally splits into these implementation phases (single spec, sequent
|
|||
## Constraints
|
||||
- **C (gnu17)** — no C++, no external libraries beyond existing (capnp, zlib, pthreads)
|
||||
- **Non-blocking in plugin hooks** — allocation happens on allocator thread, not iomgr. No new blocking concerns.
|
||||
- **Cancel-safe** — job picker thread is `pthread_cancel`-able. Iterator `next()` calls must have cancellation points.
|
||||
- **Session isolation** — iterator state and resreq cache are per-session, cleared on `begin_session`.
|
||||
- **Cancel-safe** — `next_job()` must have `pthread_cancel` cancellation points
|
||||
- **Session isolation** — iterator state and resreq cache are per-session, cleared on `begin_session`
|
||||
- **Backward compatible** — existing FCFS + seqjob behavior unchanged. New functionality is additive.
|
||||
- **Deterministic replay** — same initial state + same event sequence → same decisions. No wall-clock time, no unseeded randomness, stable iteration order.
|
||||
- **Memory** — lazy generation avoids materializing all combinations. Worst-case for a job needing N slots on M hosts: M×N candidate evaluations (acceptably bounded).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue