myxce/.kiro/specs/scheduler-preemption/brief.md

Brief: scheduler-preemption

Problem

The XCE scheduler has no preemption mechanism. When a high-priority job cannot be scheduled because all resources are consumed by lower-priority running jobs, the scheduler has no way to free those resources. The job is simply pended, and it waits indefinitely — regardless of its priority — until running jobs naturally complete.

Three specific gaps exist:

1. No preemption decision type — the xce_decision_op_t enum has DISPATCH and PEND but no PREEMPT. The scheduler cannot communicate a preemption intent to the controller.
2. No victim selection mechanism — even if a preemption decision type existed, there is no logic to select which running jobs should be preempted. Cross-queue victim selection (a high-priority job in queue A preempting a low-priority job in queue B) is entirely absent.
3. No SIGSTP pipeline — the controller and worker have no path to deliver a stop signal to a running job and transition its state to JOB_SUSPENDED, freeing its resources for the preempting job.

Operators with mixed-priority workloads need preemption to ensure critical work proceeds without manual intervention — the operator should not have to manually kill low-priority jobs to make room.

Current State

What exists

• Decision framework (decision.h): xce_decision_op_t with DISPATCH, PEND, and NONE ops. Core/decision callback flows to driver/controller.
• Queue structure (xce_sched_queue_t): has priority field for inter-queue ordering within the picker, but no preemption-related fields.
• HOOK_DETAIL_JOB_STOP (enum value 65 in cluster_def.h): existing hook detail enum, reused for preemption stop notifications.
• Controller→worker messaging: existing message pipeline for dispatch/pend decisions; the same path can carry preemption actions to workers.

What's missing

• XCE_DECISION_PREEMPT opcode and xce_preempt_decision_t payload in decision.h
• Victim selection policy: which running jobs get preempted for a given high-priority job
• Per-queue preemption configuration: which queues can preempt from which other queues, which queues are immune to preemption
• SIGSTP delivery pipeline: scheduler → controller → worker → process group
• Job state transition to JOB_SUSPENDED triggered by preemption
• Integration of preemption into the scheduling loop: when allocation fails, attempt preemption before pended

Desired Outcome

A job preemption system with three layers:

1. Decision type — XCE_DECISION_PREEMPT carries victim jobids, signal type, and preempting job id through the core's decision callback → normal-mode driver → controller → worker pipeline, reusing the same callback infrastructure (xce_core_decision_cb_t) used by DISPATCH and PEND.

2. Victim selection — a preempt_policy plugin on each queue selects which running jobs to preempt. The default policy "lowest_priority_first" implements LSF-style cross-queue victim selection:

• Collect running jobs from the preempting job's preemptible_queues list
• Filter: victim priority < preempting job priority AND victim's queue has preemptible == true
• Rank by priority ascending (lowest priority first)
• Select the minimum subset whose freed resources satisfy the preempting job's resreq

3. SIGSTP pipeline — the controller receives XCE_DECISION_PREEMPT, delivers HOOK_DETAIL_JOB_STOP to each victim's worker, the worker sends SIGSTP to the job's process group, the controller transitions the job to JOB_SUSPENDED, and resources are released. The preempting job is pended and retried in the next scheduling round (async preemption).

Queue Preemption Configuration

Each queue carries three new fields:

Field	Type	Purpose
preemptible_queues	char**	List of queue names this queue CAN preempt from
preemptible	bool	Whether this queue's jobs CAN BE preempted by other queues
preempt_policy	char*	Victim-selection plugin name (NULL = use default)

Alternatives Considered

Approach A: Kill-and-requeue only (rejected)

Aspect	Detail
Mechanism	SIGKILL victims, requeue from scratch
Pros	Simple, no suspended state, no resume logic
Cons	Destroys progress, lose all intermediate computation. Requeued jobs restart from zero. Not viable for long-running HPC workloads where checkpointing is explicit.

Rejected because: HPC jobs can run for hours or days. Killing a job that is 90% complete to make room for a higher-priority job wastes all accumulated work. SIGSTP + resume preserves progress when the preempted job is later resumed.

Approach B: SIGSTP with suspend/resume (chosen)

Aspect	Detail
Mechanism	SIGSTP to process group → job → JOB_SUSPENDED → resources freed → preempting job runs → preempted job optionally resumed later
Victim selection	Policy plugin per queue, default "lowest_priority_first"
Cross-queue	Per-queue preemptible_queues list enables cross-queue preemption
Scope	v1 is SIGSTP only; kill-and-requeue escalation deferred

Chosen because: Preserves victim job progress. Enables later resume (outside this spec's scope). Aligns with LSF-style preemption semantics familiar to HPC operators. The async pattern (pend preempting job, retry next round after SIGSTP) keeps the scheduling loop simple.

Approach

Preemption Flow

scheduler                          controller                  worker
───────────                        ──────────                  ──────
alloc fails for high-prio job J

preempt_policy.select_victims(J):
  collect running jobs from
    Q_aggressor.preemptible_queues
  filter: victim.prio < J.prio
  filter: victim.queue.preemptible
  rank: lowest priority first
  select minimum set satisfying J.resreq

  → [victim1, victim2]

XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim1
  .victim_jobids = [v1, v2]        deliver to each               job → JOB_SUSPENDED
  .signal = SIGSTP                 victim's worker               release resources
  .preempting_jobid = J

J pended → next round → retry alloc

Integration into Scheduling Loop

Preemption fires at Level 3 in the core scheduling loop — after allocation failure, before pending:

while ((job = picker->next_job(picker)) != NULL) {
    if (try_allocate(job) > 0) {
        emit_decision(DISPATCH, job);
        picker->on_dispatch(picker, job->jobid);
    } else if (can_preempt(job)) {
        victims = select_victims(job);
        emit_decision(PREEMPT, job, victims);
        picker->on_pend(picker, job->jobid);  // retry next round
    } else {
        emit_decision(PEND, job);
        picker->on_pend(picker, job->jobid);
    }
}

The core loop itself contains no victim selection logic — that lives in the preempt_policy plugin. The core only checks can_preempt(job) (does the job's queue have a non-NULL preemptible_queues list?) and delegates to the policy plugin for victim selection.

What stays unchanged

• Core scheduling loop — preemption is an additional branch in the alloc-fail path (Level 3), not a restructure
• Decision callback (xce_core_decision_cb_t) — PREEMPT flows through the same callback as DISPATCH and PEND
• Existing plugin framework — preempt_policy is registered, loaded, and invoked through the existing plugin_register/key_find mechanism
• HOOK_DETAIL_JOB_STOP — existing enum value, reused without change

What's new

• XCE_DECISION_PREEMPT in xce_decision_op_t (decision.h)
• xce_preempt_decision_t payload carrying victim_jobids[], n_victims, signal, preempting_jobid
• preemptible_queues, preemptible, preempt_policy fields on xce_sched_queue_t
• Default "lowest_priority_first" preemption policy plugin
• Controller handler for XCE_DECISION_PREEMPT → SIGSTP delivery to workers
• Worker-side SIGSTP handler → process group signal + JOB_SUSPENDED state transition

Scope

In

• XCE_DECISION_PREEMPT decision type and xce_preempt_decision_t payload
• preempt_policy plugin interface (victim selection)
• Default "lowest_priority_first" preemption policy
• LSF-style cross-queue victim selection via preemptible_queues list
• SIGSTP pipeline: scheduler → controller → worker → process group
• Per-queue preemption config: preemptible_queues, preemptible, preempt_policy
• Victim safety guards: never preempt same-or-higher priority, never preempt non-preemptible queues
• JOB_SUSPENDED state transition on victim jobs
• Reuse of HOOK_DETAIL_JOB_STOP for preemption stop notifications
• Unit tests: victim selection policy, safety guard enforcement, edge cases (empty target queues, no eligible victims, insufficient resources from any victim set)
• Integration test: end-to-end preemption flow — high-priority job submits, preemption fires, victims get SIGSTP, resources freed, preempting job dispatched in next round

Out

• Kill-and-requeue escalation (v1 is SIGSTP only)
• Deadline-based preemption (requires deadline field on jobs — deferred)
• Preempted job resume logic (outside this spec's scope — the JOB_SUSPENDED state is defined, but the resume mechanism is a separate feature)
• Preemption priority inversion guards beyond the priority comparison already specified
• Fair-share or proportional victim selection beyond the default "lowest_priority_first" policy (the plugin interface enables custom policies, but none beyond the default ship here)

Upstream / Downstream

Upstream (dependencies)

• scheduler-core (sibling spec): provides the decision callback (xce_core_decision_cb_t) through which XCE_DECISION_PREEMPT flows. The core scheduling loop and xce_sched_core_t are defined there. Preemption hooks into the core loop at Level 3 (alloc-fail branch).
• scheduler-allocation (sibling spec): provides try_allocate() — preemption fires only when allocation fails. The resreq fingerprint cache interacts with preemption (a cached-unsatisfiable resreq skips allocation AND preemption).
• Controller message handling: existing handler dispatch path for decisions — extended to handle XCE_DECISION_PREEMPT.
• Worker job control: existing job lifecycle management — extended to handle SIGSTP delivery and JOB_SUSPENDED.

Downstream (future consumers)

• Resume mechanism — a future spec for resuming JOB_SUSPENDED jobs, potentially with priority-based ordering
• Kill-and-requeue escalation — a future enhancement where repeated preemption failure escalates to SIGKILL
• Custom preemption policies — deadline-aware, fair-share, or resource-specific victim selection implemented as new preempt_policy plugins

Constraints

• C (gnu17) — no C++, no external libraries beyond existing (pthreads)
• Async preemption — the preempting job is pended and retried in the next scheduling round; preemption is not synchronous
• Priority guard — victim priority must be strictly less than preempting job priority (never preempt equal or higher)
• Preemptible guard — victim queue must have preemptible == true (opt-in — queues are not preemptible by default)
• Minimal victim set — select the minimum subset of victims whose freed resources satisfy the preempting job's resreq (avoid over-preemption)
• No self-preemption — the preempting job's own queue is implicitly excluded from victim selection
• Cancel-safe — preempt_policy.select_victims() must have pthread_cancel cancellation points