myxce/.kiro/specs/scheduler-preemption/brief.md

# Brief: scheduler-preemption

## Problem

The XCE scheduler has no preemption mechanism. When a high-priority job cannot be scheduled because all resources are consumed by lower-priority running jobs, the scheduler has no way to free those resources. The job is simply pended, and it waits indefinitely — regardless of its priority — until running jobs naturally complete.

Three specific gaps exist:

1. **No preemption decision type** — the `xce_decision_op_t` enum has `DISPATCH` and `PEND` but no `PREEMPT`. The scheduler cannot communicate a preemption intent to the controller.
2. **No victim selection mechanism** — even if a preemption decision type existed, there is no logic to select which running jobs should be preempted. Cross-queue victim selection (a high-priority job in queue A preempting a low-priority job in queue B) is entirely absent.
3. **No SIGSTP pipeline** — the controller and worker have no path to deliver a stop signal to a running job and transition its state to `JOB_SUSPENDED`, freeing its resources for the preempting job.

Operators with mixed-priority workloads need preemption to ensure critical work proceeds without manual intervention — the operator should not have to manually kill low-priority jobs to make room.

## Current State

### What exists
- **Decision framework** (`decision.h`): `xce_decision_op_t` with `DISPATCH`, `PEND`, and `NONE` ops. Core/decision callback flows to driver/controller.
- **Queue structure** (`xce_sched_queue_t`): has `priority` field for inter-queue ordering within the picker, but no preemption-related fields.
- **HOOK_DETAIL_JOB_STOP** (enum value 65 in `cluster_def.h`): existing hook detail enum, reused for preemption stop notifications.
- **Controller→worker messaging**: existing message pipeline for dispatch/pend decisions; the same path can carry preemption actions to workers.

### What's missing
- `XCE_DECISION_PREEMPT` opcode and `xce_preempt_decision_t` payload in `decision.h`
- Victim selection policy: which running jobs get preempted for a given high-priority job
- Per-queue preemption configuration: which queues can preempt from which other queues, which queues are immune to preemption
- SIGSTP delivery pipeline: scheduler → controller → worker → process group
- Job state transition to `JOB_SUSPENDED` triggered by preemption
- Integration of preemption into the scheduling loop: when allocation fails, attempt preemption before pended

## Desired Outcome

A job preemption system with three layers:

**1. Decision type** — `XCE_DECISION_PREEMPT` carries victim jobids, signal type, and preempting job id through the core's decision callback → normal-mode driver → controller → worker pipeline, reusing the same callback infrastructure (`xce_core_decision_cb_t`) used by `DISPATCH` and `PEND`.

**2. Victim selection** — a `preempt_policy` plugin on each queue selects which running jobs to preempt. The default policy `"lowest_priority_first"` implements LSF-style cross-queue victim selection:
- Collect running jobs from the preempting job's `preemptible_queues` list
- Filter: victim priority < preempting job priority AND victim's queue has `preemptible == true`
- Rank by priority ascending (lowest priority first)
- Select the minimum subset whose freed resources satisfy the preempting job's resreq

**3. SIGSTP pipeline** — the controller receives `XCE_DECISION_PREEMPT`, delivers `HOOK_DETAIL_JOB_STOP` to each victim's worker, the worker sends SIGSTP to the job's process group, the controller transitions the job to `JOB_SUSPENDED`, and resources are released. The preempting job is pended and retried in the next scheduling round (async preemption).

### Queue Preemption Configuration

Each queue carries three new fields:

| Field | Type | Purpose |
|-------|------|---------|
| `preemptible_queues` | `char**` | List of queue names this queue CAN preempt from |
| `preemptible` | `bool` | Whether this queue's jobs CAN BE preempted by other queues |
| `preempt_policy` | `char*` | Victim-selection plugin name (NULL = use default) |

## Alternatives Considered

### Approach A: Kill-and-requeue only (rejected)

| Aspect | Detail |
|--------|--------|
| **Mechanism** | SIGKILL victims, requeue from scratch |
| **Pros** | Simple, no suspended state, no resume logic |
| **Cons** | Destroys progress, lose all intermediate computation. Requeued jobs restart from zero. Not viable for long-running HPC workloads where checkpointing is explicit. |

**Rejected because:** HPC jobs can run for hours or days. Killing a job that is 90% complete to make room for a higher-priority job wastes all accumulated work. SIGSTP + resume preserves progress when the preempted job is later resumed.

### Approach B: SIGSTP with suspend/resume (chosen)

| Aspect | Detail |
|--------|--------|
| **Mechanism** | SIGSTP to process group → job → `JOB_SUSPENDED` → resources freed → preempting job runs → preempted job optionally resumed later |
| **Victim selection** | Policy plugin per queue, default `"lowest_priority_first"` |
| **Cross-queue** | Per-queue `preemptible_queues` list enables cross-queue preemption |
| **Scope** | v1 is SIGSTP only; kill-and-requeue escalation deferred |

**Chosen because:** Preserves victim job progress. Enables later resume (outside this spec's scope). Aligns with LSF-style preemption semantics familiar to HPC operators. The async pattern (pend preempting job, retry next round after SIGSTP) keeps the scheduling loop simple.

## Approach

### Preemption Flow

```
scheduler                          controller                  worker
───────────                        ──────────                  ──────
alloc fails for high-prio job J

preempt_policy.select_victims(J):
  collect running jobs from
    Q_aggressor.preemptible_queues
  filter: victim.prio < J.prio
  filter: victim.queue.preemptible
  rank: lowest priority first
  select minimum set satisfying J.resreq

  → [victim1, victim2]

XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim1
  .victim_jobids = [v1, v2]        deliver to each               job → JOB_SUSPENDED
  .signal = SIGSTP                 victim's worker               release resources
  .preempting_jobid = J

J pended → next round → retry alloc
```

### Integration into Scheduling Loop

Preemption fires at Level 3 in the core scheduling loop — after allocation failure, before pending:

```c
while ((job = picker->next_job(picker)) != NULL) {
    if (try_allocate(job) > 0) {
        emit_decision(DISPATCH, job);
        picker->on_dispatch(picker, job->jobid);
    } else if (can_preempt(job)) {
        victims = select_victims(job);
        emit_decision(PREEMPT, job, victims);
        picker->on_pend(picker, job->jobid);  // retry next round
    } else {
        emit_decision(PEND, job);
        picker->on_pend(picker, job->jobid);
    }
}
```

The core loop itself contains no victim selection logic — that lives in the `preempt_policy` plugin. The core only checks `can_preempt(job)` (does the job's queue have a non-NULL `preemptible_queues` list?) and delegates to the policy plugin for victim selection.

### What stays unchanged
- Core scheduling loop — preemption is an additional branch in the alloc-fail path (Level 3), not a restructure
- Decision callback (`xce_core_decision_cb_t`) — `PREEMPT` flows through the same callback as `DISPATCH` and `PEND`
- Existing plugin framework — `preempt_policy` is registered, loaded, and invoked through the existing `plugin_register`/`key_find` mechanism
- `HOOK_DETAIL_JOB_STOP` — existing enum value, reused without change

### What's new
- `XCE_DECISION_PREEMPT` in `xce_decision_op_t` (`decision.h`)
- `xce_preempt_decision_t` payload carrying `victim_jobids[]`, `n_victims`, `signal`, `preempting_jobid`
- `preemptible_queues`, `preemptible`, `preempt_policy` fields on `xce_sched_queue_t`
- Default `"lowest_priority_first"` preemption policy plugin
- Controller handler for `XCE_DECISION_PREEMPT` → SIGSTP delivery to workers
- Worker-side SIGSTP handler → process group signal + `JOB_SUSPENDED` state transition

## Scope

### In
- `XCE_DECISION_PREEMPT` decision type and `xce_preempt_decision_t` payload
- `preempt_policy` plugin interface (victim selection)
- Default `"lowest_priority_first"` preemption policy
- LSF-style cross-queue victim selection via `preemptible_queues` list
- SIGSTP pipeline: scheduler → controller → worker → process group
- Per-queue preemption config: `preemptible_queues`, `preemptible`, `preempt_policy`
- Victim safety guards: never preempt same-or-higher priority, never preempt non-`preemptible` queues
- `JOB_SUSPENDED` state transition on victim jobs
- Reuse of `HOOK_DETAIL_JOB_STOP` for preemption stop notifications
- Unit tests: victim selection policy, safety guard enforcement, edge cases (empty target queues, no eligible victims, insufficient resources from any victim set)
- Integration test: end-to-end preemption flow — high-priority job submits, preemption fires, victims get SIGSTP, resources freed, preempting job dispatched in next round

### Out
- Kill-and-requeue escalation (v1 is SIGSTP only)
- Deadline-based preemption (requires `deadline` field on jobs — deferred)
- Preempted job resume logic (outside this spec's scope — the `JOB_SUSPENDED` state is defined, but the resume mechanism is a separate feature)
- Preemption priority inversion guards beyond the priority comparison already specified
- Fair-share or proportional victim selection beyond the default `"lowest_priority_first"` policy (the plugin interface enables custom policies, but none beyond the default ship here)

## Upstream / Downstream

### Upstream (dependencies)
- **scheduler-core** (sibling spec): provides the decision callback (`xce_core_decision_cb_t`) through which `XCE_DECISION_PREEMPT` flows. The core scheduling loop and `xce_sched_core_t` are defined there. Preemption hooks into the core loop at Level 3 (alloc-fail branch).
- **scheduler-allocation** (sibling spec): provides `try_allocate()` — preemption fires only when allocation fails. The resreq fingerprint cache interacts with preemption (a cached-unsatisfiable resreq skips allocation AND preemption).
- **Controller message handling**: existing handler dispatch path for decisions — extended to handle `XCE_DECISION_PREEMPT`.
- **Worker job control**: existing job lifecycle management — extended to handle SIGSTP delivery and `JOB_SUSPENDED`.

### Downstream (future consumers)
- **Resume mechanism** — a future spec for resuming `JOB_SUSPENDED` jobs, potentially with priority-based ordering
- **Kill-and-requeue escalation** — a future enhancement where repeated preemption failure escalates to SIGKILL
- **Custom preemption policies** — deadline-aware, fair-share, or resource-specific victim selection implemented as new `preempt_policy` plugins

## Constraints
- **C (gnu17)** — no C++, no external libraries beyond existing (pthreads)
- **Async preemption** — the preempting job is pended and retried in the next scheduling round; preemption is not synchronous
- **Priority guard** — victim priority must be strictly less than preempting job priority (never preempt equal or higher)
- **Preemptible guard** — victim queue must have `preemptible == true` (opt-in — queues are not preemptible by default)
- **Minimal victim set** — select the minimum subset of victims whose freed resources satisfy the preempting job's resreq (avoid over-preemption)
- **No self-preemption** — the preempting job's own queue is implicitly excluded from victim selection
- **Cancel-safe** — `preempt_policy.select_victims()` must have `pthread_cancel` cancellation points