myxce/.kiro/specs/scheduler-preemption/requirements.md

# Requirements Document

## Introduction

The XCE scheduler currently has no job preemption mechanism. When a high-priority job cannot be scheduled because all resources are consumed by lower-priority running jobs, the only available action is to pend the job — it waits indefinitely until running jobs complete naturally, regardless of priority.

This feature closes the preemption gap with three additions:

1. A `XCE_DECISION_PREEMPT` decision type flowing through the same callback as `DISPATCH` and `PEND`, carrying victim job ids and signal type to the controller→worker pipeline.
2. A configurable `preempt_policy` plugin mechanism for victim selection, with a built-in `"lowest_priority_first"` default policy implementing LSF-style cross-queue victim selection.
3. A SIGSTP delivery pipeline: scheduler emits `XCE_DECISION_PREEMPT` → controller delivers `HOOK_DETAIL_JOB_STOP` to each victim's worker → worker sends SIGSTP to the job's process group → controller transitions the job to `JOB_SUSPENDED` and releases resources.

The design uses async preemption: the preempting job is pended and retried in the next scheduling round after resources are freed, rather than blocking the scheduling loop waiting for victims to stop.

> Full discovery context (problem, current state, alternatives considered, approach, boundary) is in `.kiro/specs/scheduler-preemption/brief.md`.
>
> This spec is a child of `scheduler-framework`. It depends on **scheduler-core** for the decision callback (`xce_core_decision_cb_t`) through which `XCE_DECISION_PREEMPT` flows. The preemption branch (Level 3) fires in the core scheduling loop after allocation failure, before pending — this integration point is defined in scheduler-core's requirements.

## Boundary Context

- **In scope**: `XCE_DECISION_PREEMPT` opcode and `xce_preempt_decision_t` payload in `decision.h`; `preempt_policy` plugin interface for victim selection; default `"lowest_priority_first"` preemption policy implementing LSF-style cross-queue victim selection; per-queue preemption configuration (`preemptible_queues`, `preemptible`, `preempt_policy` fields on `xce_sched_queue_t`); SIGSTP delivery pipeline through controller→worker; `JOB_SUSPENDED` state transition on victim jobs; reuse of `HOOK_DETAIL_JOB_STOP` for preemption stop notifications; victim safety guards (never preempt same-or-higher priority, never preempt non-`preemptible` queues, never over-preempt); integration with the core scheduling loop at Level 3 (alloc-fail branch).

- **Out of scope**: Kill-and-requeue escalation beyond SIGSTP (deferred — v1 is SIGSTP only); deadline-based preemption (requires `deadline` field on `xce_sched_job_t`, deferred); preempted job resume logic (the `JOB_SUSPENDED` state is defined but resume is a separate feature); fair-share or proportional victim selection policies beyond the default (the plugin interface enables them, but no additional policies ship here); changes to the picker interface, iterator abstractions, core/driver separation, or the scheduling loop structure itself (those belong to sibling specs `scheduler-core` and `scheduler-allocation`).

- **Adjacent expectations**: This spec depends on `scheduler-core` for the decision callback (`xce_core_decision_cb_t`) and the core scheduling loop into which the preemption branch integrates. The existing controller→worker message pipeline for dispatch/pend is extended to carry `XCE_DECISION_PREEMPT`. The existing `HOOK_DETAIL_JOB_STOP` hook detail enum (value 65 in `cluster_def.h`) is reused for preemption stop actions. The existing `plugin_register`/`key_find` mechanism loads the `preempt_policy` plugin.

## Data Structure Changes

### `decision.h` additions
```c
typedef enum {
    XCE_DECISION_NONE = 0,
    XCE_DECISION_DISPATCH,
    XCE_DECISION_PEND,
    XCE_DECISION_PREEMPT,     // NEW — preemption decision
    XCE_DECISION_NEW_SESSION
} xce_decision_op_t;

typedef struct {
    uint64_t* victim_jobids;    // jobs to preempt
    int       n_victims;        // count of victims
    int       signal;           // signal to send (SIGSTP in v1, SIGKILL in future)
    uint64_t  preempting_jobid; // job that triggered preemption
} xce_preempt_decision_t;
```

### `xce_sched_queue_t` additions
```c
char**   preemptible_queues;    // queue names this queue CAN preempt from (NULL = none)
int      n_preemptible_queues;  // count of preemptible_queues entries
bool     preemptible;           // CAN this queue's jobs BE preempted by other queues?
char*    preempt_policy;        // victim-selection plugin name (NULL = use default)
```

## Preemption Flow

```
scheduler                          controller                  worker
───────────                        ──────────                  ──────
alloc fails for high-prio job J

preempt_policy.select_victims(J):
  collect running jobs from
    Q_aggressor.preemptible_queues
  filter: victim.prio < J.prio
  filter: victim.queue.preemptible
  rank: lowest priority first
  select minimum set satisfying J.resreq

  → [victim1, victim2]

XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim1
  .victim_jobids = [v1, v2]        deliver to each               job → JOB_SUSPENDED
  .signal = SIGSTP                 victim's worker               release resources
  .preempting_jobid = J

J pended → next scheduling round → allocation retries on freed resources
```

## Requirements

### Requirement: Preemption policy and execution

**Objective:** As a cluster operator, I want high-priority jobs to trigger preemption of lower-priority running jobs when resources are insufficient, so that critical work can proceed without manual intervention.

#### Acceptance Criteria

1. The Scheduler shall add a `XCE_DECISION_PREEMPT` opcode to `xce_decision_op_t` in `decision.h`, with a payload type `xce_preempt_decision_t` carrying `victim_jobids[]`, `n_victims`, `signal` (SIGSTP in v1), and `preempting_jobid`. The `XCE_DECISION_PREEMPT` decision shall flow through the same `xce_core_decision_cb_t` callback as `DISPATCH` and `PEND`; the core scheduling loop shall contain zero knowledge of how the driver publishes preemption decisions.

2. When allocation fails for a job J whose queue has a non-NULL `preemptible_queues` list, the Scheduler shall invoke the queue's `preempt_policy` plugin to select victim jobs from the target queues. When `preempt_policy` is NULL, the Scheduler shall use the default `"lowest_priority_first"` policy.

3. The default preemption policy `"lowest_priority_first"` shall: WHILE the preempting job's resource request is not yet satisfied DO collect running jobs from target queues; filter to those with `priority < J.priority` AND `queue->preemptible == true`; rank by priority ascending (lowest priority first); select the next victim candidate and add to the victim set. The policy shall return the minimum subset of victims whose freed resources satisfy J's resource request. When multiple victim subsets of equal minimum size exist, the policy shall prefer the subset whose victims have the lowest summed priority.

4. When a sufficient victim set exists, the Scheduler shall commit the preemption decision via the decision callback, and shall pend job J for the next scheduling round (async preemption — the job is retried after victim resources are freed). The Scheduler shall not wait synchronously for victim jobs to transition to `JOB_SUSPENDED`.

5. When no sufficient victim set exists after exhausting all eligible running jobs from the target queues, the Scheduler shall pend job J with a resource-shortage pend reason without invoking preemption. The decision callback shall not be called for the preemption path in this case.

6. The worker receiving a preemption action shall send the specified signal (SIGSTP in v1) to the victim job's process group. The controller shall transition the victim job's state to `JOB_SUSPENDED` after confirming the signal was delivered. Resources held by the victim job shall be released upon suspension.

7. The Scheduler shall add `preemptible_queues` (list of queue names this queue CAN preempt from, NULL = none), `preemptible` (bool — CAN this queue's jobs BE preempted by other queues, default false), and `preempt_policy` (victim-selection plugin name, NULL = use default `"lowest_priority_first"`) fields to `xce_sched_queue_t`. These fields shall be populated from queue configuration at `QUEUE_UPDATE` time.

8. The existing `HOOK_DETAIL_JOB_STOP` hook detail enum (value 65 in `cluster_def.h`) shall be reused for preemption stop notifications, with no changes to the enum definition. The controller shall deliver this hook detail to each victim's worker as part of the preemption action.

9. Preemption victim selection shall never select:
   - A job whose priority is greater than or equal to the preempting job's priority.
   - A job whose queue has `preemptible == false`.
   - A job that is not in running state.
   - A job from a queue not listed in the preempting job's `preemptible_queues` list.
   - More victims than necessary to satisfy the preempting job's resource request (minimum sufficient victim set).

> **Pending revision (2026-08-23):** the sibling spec `scheduler-dispatch-commit` (not yet designed) redefines "running state" to begin at dispatch-decision-commit time rather than worker-confirmation time — meaning a job could reach RUN with resources committed but no live, signalable process yet. AC6's SIGSTP-to-process-group delivery and AC9's "not in running state" exclusion both assume every RUN job has a live process today; once `scheduler-dispatch-commit` lands, this spec's design phase needs to add a second victim action (cancel an in-flight, unconfirmed dispatch) alongside SIGSTP, and decide whether a dispatched-but-unconfirmed job is eligible as a victim at all. Re-check this note before `scheduler-preemption` proceeds to `/kiro-spec-design`.