myxce/.kiro/specs/scheduler-preemption/requirements.md

Requirements Document

Introduction

The XCE scheduler currently has no job preemption mechanism. When a high-priority job cannot be scheduled because all resources are consumed by lower-priority running jobs, the only available action is to pend the job — it waits indefinitely until running jobs complete naturally, regardless of priority.

This feature closes the preemption gap with three additions:

1. A XCE_DECISION_PREEMPT decision type flowing through the same callback as DISPATCH and PEND, carrying victim job ids and signal type to the controller→worker pipeline.
2. A configurable preempt_policy plugin mechanism for victim selection, with a built-in "lowest_priority_first" default policy implementing LSF-style cross-queue victim selection.
3. A SIGSTP delivery pipeline: scheduler emits XCE_DECISION_PREEMPT → controller delivers HOOK_DETAIL_JOB_STOP to each victim's worker → worker sends SIGSTP to the job's process group → controller transitions the job to JOB_SUSPENDED and releases resources.

The design uses async preemption: the preempting job is pended and retried in the next scheduling round after resources are freed, rather than blocking the scheduling loop waiting for victims to stop.

Full discovery context (problem, current state, alternatives considered, approach, boundary) is in .kiro/specs/scheduler-preemption/brief.md.

This spec is a child of scheduler-framework. It depends on scheduler-core for the decision callback (xce_core_decision_cb_t) through which XCE_DECISION_PREEMPT flows. The preemption branch (Level 3) fires in the core scheduling loop after allocation failure, before pending — this integration point is defined in scheduler-core's requirements.

Boundary Context

• In scope: XCE_DECISION_PREEMPT opcode and xce_preempt_decision_t payload in decision.h; preempt_policy plugin interface for victim selection; default "lowest_priority_first" preemption policy implementing LSF-style cross-queue victim selection; per-queue preemption configuration (preemptible_queues, preemptible, preempt_policy fields on xce_sched_queue_t); SIGSTP delivery pipeline through controller→worker; JOB_SUSPENDED state transition on victim jobs; reuse of HOOK_DETAIL_JOB_STOP for preemption stop notifications; victim safety guards (never preempt same-or-higher priority, never preempt non-preemptible queues, never over-preempt); integration with the core scheduling loop at Level 3 (alloc-fail branch).

• Out of scope: Kill-and-requeue escalation beyond SIGSTP (deferred — v1 is SIGSTP only); deadline-based preemption (requires deadline field on xce_sched_job_t, deferred); preempted job resume logic (the JOB_SUSPENDED state is defined but resume is a separate feature); fair-share or proportional victim selection policies beyond the default (the plugin interface enables them, but no additional policies ship here); changes to the picker interface, iterator abstractions, core/driver separation, or the scheduling loop structure itself (those belong to sibling specs scheduler-core and scheduler-allocation).

• Adjacent expectations: This spec depends on scheduler-core for the decision callback (xce_core_decision_cb_t) and the core scheduling loop into which the preemption branch integrates. The existing controller→worker message pipeline for dispatch/pend is extended to carry XCE_DECISION_PREEMPT. The existing HOOK_DETAIL_JOB_STOP hook detail enum (value 65 in cluster_def.h) is reused for preemption stop actions. The existing plugin_register/key_find mechanism loads the preempt_policy plugin.

Data Structure Changes

decision.h additions

typedef enum {
    XCE_DECISION_NONE = 0,
    XCE_DECISION_DISPATCH,
    XCE_DECISION_PEND,
    XCE_DECISION_PREEMPT,     // NEW — preemption decision
    XCE_DECISION_NEW_SESSION
} xce_decision_op_t;

typedef struct {
    uint64_t* victim_jobids;    // jobs to preempt
    int       n_victims;        // count of victims
    int       signal;           // signal to send (SIGSTP in v1, SIGKILL in future)
    uint64_t  preempting_jobid; // job that triggered preemption
} xce_preempt_decision_t;

xce_sched_queue_t additions

char**   preemptible_queues;    // queue names this queue CAN preempt from (NULL = none)
int      n_preemptible_queues;  // count of preemptible_queues entries
bool     preemptible;           // CAN this queue's jobs BE preempted by other queues?
char*    preempt_policy;        // victim-selection plugin name (NULL = use default)

Preemption Flow

scheduler                          controller                  worker
───────────                        ──────────                  ──────
alloc fails for high-prio job J

preempt_policy.select_victims(J):
  collect running jobs from
    Q_aggressor.preemptible_queues
  filter: victim.prio < J.prio
  filter: victim.queue.preemptible
  rank: lowest priority first
  select minimum set satisfying J.resreq

  → [victim1, victim2]

XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim1
  .victim_jobids = [v1, v2]        deliver to each               job → JOB_SUSPENDED
  .signal = SIGSTP                 victim's worker               release resources
  .preempting_jobid = J

J pended → next scheduling round → allocation retries on freed resources

Requirements

Requirement: Preemption policy and execution

Objective: As a cluster operator, I want high-priority jobs to trigger preemption of lower-priority running jobs when resources are insufficient, so that critical work can proceed without manual intervention.

Acceptance Criteria

1. The Scheduler shall add a XCE_DECISION_PREEMPT opcode to xce_decision_op_t in decision.h, with a payload type xce_preempt_decision_t carrying victim_jobids[], n_victims, signal (SIGSTP in v1), and preempting_jobid. The XCE_DECISION_PREEMPT decision shall flow through the same xce_core_decision_cb_t callback as DISPATCH and PEND; the core scheduling loop shall contain zero knowledge of how the driver publishes preemption decisions.

2. When allocation fails for a job J whose queue has a non-NULL preemptible_queues list, the Scheduler shall invoke the queue's preempt_policy plugin to select victim jobs from the target queues. When preempt_policy is NULL, the Scheduler shall use the default "lowest_priority_first" policy.

3. The default preemption policy "lowest_priority_first" shall: WHILE the preempting job's resource request is not yet satisfied DO collect running jobs from target queues; filter to those with priority < J.priority AND queue->preemptible == true; rank by priority ascending (lowest priority first); select the next victim candidate and add to the victim set. The policy shall return the minimum subset of victims whose freed resources satisfy J's resource request. When multiple victim subsets of equal minimum size exist, the policy shall prefer the subset whose victims have the lowest summed priority.

4. When a sufficient victim set exists, the Scheduler shall commit the preemption decision via the decision callback, and shall pend job J for the next scheduling round (async preemption — the job is retried after victim resources are freed). The Scheduler shall not wait synchronously for victim jobs to transition to JOB_SUSPENDED.

5. When no sufficient victim set exists after exhausting all eligible running jobs from the target queues, the Scheduler shall pend job J with a resource-shortage pend reason without invoking preemption. The decision callback shall not be called for the preemption path in this case.

6. The worker receiving a preemption action shall send the specified signal (SIGSTP in v1) to the victim job's process group. The controller shall transition the victim job's state to JOB_SUSPENDED after confirming the signal was delivered. Resources held by the victim job shall be released upon suspension.

7. The Scheduler shall add preemptible_queues (list of queue names this queue CAN preempt from, NULL = none), preemptible (bool — CAN this queue's jobs BE preempted by other queues, default false), and preempt_policy (victim-selection plugin name, NULL = use default "lowest_priority_first") fields to xce_sched_queue_t. These fields shall be populated from queue configuration at QUEUE_UPDATE time.

8. The existing HOOK_DETAIL_JOB_STOP hook detail enum (value 65 in cluster_def.h) shall be reused for preemption stop notifications, with no changes to the enum definition. The controller shall deliver this hook detail to each victim's worker as part of the preemption action.

9. Preemption victim selection shall never select:

   • A job whose priority is greater than or equal to the preempting job's priority.
   • A job whose queue has preemptible == false.
   • A job that is not in running state.
   • A job from a queue not listed in the preempting job's preemptible_queues list.
   • More victims than necessary to satisfy the preempting job's resource request (minimum sufficient victim set).

Pending revision (2026-08-23): the sibling spec scheduler-dispatch-commit (not yet designed) redefines "running state" to begin at dispatch-decision-commit time rather than worker-confirmation time — meaning a job could reach RUN with resources committed but no live, signalable process yet. AC6's SIGSTP-to-process-group delivery and AC9's "not in running state" exclusion both assume every RUN job has a live process today; once scheduler-dispatch-commit lands, this spec's design phase needs to add a second victim action (cancel an in-flight, unconfirmed dispatch) alongside SIGSTP, and decide whether a dispatched-but-unconfirmed job is eligible as a victim at all. Re-check this note before scheduler-preemption proceeds to /kiro-spec-design.