fix(worker): graceful job kill and catchable stop, as LSF (bug-1957)

A plain kill sent one SIGTERM and never escalated, and a plain stop sent
one uncatchable SIGSTOP. Following LSF's TERMINATE and SUSPEND actions, a
plain kill now sends SIGINT, then SIGTERM, then SIGKILL, and a plain stop
sends SIGTSTP (the job can catch it, prepare and stop itself), then SIGSTOP
if it has not stopped; each next signal follows after the grace period and
only while the job is alive (or not yet stopped). A resume cancels a
pending stop; the job's exit ends any sequence. An explicit signal is
delivered once; a force kill is SIGKILL at once.

- JobSignal gains grace (capnp @2, xce_signal_t.grace): > 0 asks for the
  sequence, in seconds between signals.
- apiserver: job.terminate_interval (default 10 s, LSF's
  JOB_TERMINATE_INTERVAL); jobCtrlParams sends kill = SIGKILL and stop =
  SIGSTOP with that grace when no signal is given; the handler no longer
  pre-fills a default signal.
- worker: grace travels through the job-control task, the kept-signal table
  (pending_signal_*_ex) and the IPC message; the job process runs the
  sequence (job_signal_seq_*, uv timer) and checks stop state in /proc.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-10 10:00:36 +00:00
parent e5a0e42b66
commit 367af11af1
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
24 changed files with 580 additions and 39 deletions

View file

@ -125,9 +125,12 @@ var JobKillSignalControlMap = map[int][]int{
JOB_KILL: {JOBSTAT["RUN"], JOBSTAT["PEND"], JOBSTAT["SSUSP"], JOBSTAT["USUSP"], JOBSTAT["PSUSP"]},
}
// DefaultActionSignal names each action's default signal. bug-1957: kill and
// stop name the last signal of their graceful sequences (backend
// jobCtrlParams sends them with the configured grace period).
var DefaultActionSignal = map[string]string{
"kill": "TERM",
"stop": "STOP", // bug-1954: the worker delivers exactly this signal
"kill": "KILL",
"stop": "STOP",
"resume": "CONT",
}