lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(worker): graceful job kill and catchable stop, as LSF (bug-1957)

Browse source 
A plain kill sent one SIGTERM and never escalated, and a plain stop sent
one uncatchable SIGSTOP. Following LSF's TERMINATE and SUSPEND actions, a
plain kill now sends SIGINT, then SIGTERM, then SIGKILL, and a plain stop
sends SIGTSTP (the job can catch it, prepare and stop itself), then SIGSTOP
if it has not stopped; each next signal follows after the grace period and
only while the job is alive (or not yet stopped). A resume cancels a
pending stop; the job's exit ends any sequence. An explicit signal is
delivered once; a force kill is SIGKILL at once.

- JobSignal gains grace (capnp @2, xce_signal_t.grace): > 0 asks for the
  sequence, in seconds between signals.
- apiserver: job.terminate_interval (default 10 s, LSF's
  JOB_TERMINATE_INTERVAL); jobCtrlParams sends kill = SIGKILL and stop =
  SIGSTOP with that grace when no signal is given; the handler no longer
  pre-fills a default signal.
- worker: grace travels through the job-control task, the kept-signal table
  (pending_signal_*_ex) and the IPC message; the job process runs the
  sequence (job_signal_seq_*, uv timer) and checks stop state in /proc.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-10 10:00:36 +00:00
parent e5a0e42b66
commit 367af11af1
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
24 changed files with 580 additions and 39 deletions
Download patch file Download diff file Expand all files Collapse all files

27
src/frontend/apiserver/backend/convert_pure.go
View file

@ -2,6 +2,7 @@ package backend
import (
"apiserver/models"
"apiserver/utils"
"encoding/json"
"errors"
"fmt"
@ -222,6 +223,7 @@ func jobCtrlUnionDiscriminant(action int) int {
type jobCtrlArgs struct {
Signo int
Force int
Grace int // seconds; > 0: graceful kill/stop sequence (bug-1957)
CkptKill int
Method string
}
@ -266,6 +268,11 @@ func jobCtrlParams(action int, content string) (jobCtrlArgs, error) {
case models.JOB_STOP, models.JOB_RESUME, models.JOB_KILL:
if signo == 0 {
signo = defaultActionSignal(action)
// bug-1957: a plain kill or stop is graceful, as LSF's
// TERMINATE and SUSPEND actions; a force kill is immediate
if action == models.JOB_STOP || (action == models.JOB_KILL && !forceKill) {
args.Grace = jobTerminateInterval()
}
}
args.Signo = signo
if action == models.JOB_KILL && forceKill {
@ -281,16 +288,30 @@ func jobCtrlParams(action int, content string) (jobCtrlArgs, error) {
// defaultActionSignal is the signal a stop/resume/kill sends when the caller
// gave none.
//
// bug-1957: kill and stop name the sequence's last signal (SIGKILL,
// SIGSTOP); with a grace period the worker sends SIGINT/SIGTERM or SIGTSTP
// first, so the job can clean up or prepare to stop.
func defaultActionSignal(action int) int {
switch action {
case models.JOB_STOP:
// bug-1954: the worker delivers the requested signal, so a plain
// stop asks for the uncatchable SIGSTOP
return models.JobSignalMapTable["STOP"] // 19
case models.JOB_RESUME:
return models.JobSignalMapTable["CONT"] // 18
}
return models.JobSignalMapTable["TERM"] // 15
return models.JobSignalMapTable["KILL"] // 9
}
// defaultJobTerminateInterval is LSF's JOB_TERMINATE_INTERVAL default.
const defaultJobTerminateInterval = 10
// jobTerminateInterval is job.terminate_interval, or the default when it is
// unset or 0.
func jobTerminateInterval() int {
if utils.JobYamlConfig != nil && utils.JobYamlConfig.TerminateInterval > 0 {
return utils.JobYamlConfig.TerminateInterval
}
return defaultJobTerminateInterval
}
// isKnownSignal reports whether n is one of models.JobSignalMapTable's values.