121 lines
6.3 KiB
Markdown
121 lines
6.3 KiB
Markdown
---
|
|
id: "bug-1957"
|
|
status: "fixed"
|
|
status_updated: "2026-10-10"
|
|
status_note: ""
|
|
component: "worker"
|
|
severity: "wrong-result"
|
|
timestamp: "2026-10-10T09:35:48Z"
|
|
last_seen: "2026-10-10T09:35:48Z"
|
|
occurrences: 1
|
|
files: ["src/worker/src/xcecd_worker.c", "src/frontend/apiserver/backend/convert_pure.go", "src/schema/controller.capnp"]
|
|
tags: ["worker", "job-control", "signal", "lsf"]
|
|
related: ["bug-1954", "bug-1940"]
|
|
fix_commits: ["367af11a", "97f7c800"]
|
|
gitea_issue: 388
|
|
gitea_synced_status: "fixed"
|
|
gitea_synced_hash: "0c07e8351f9e"
|
|
---
|
|
|
|
# bug-1957: plain job kill and stop are single signals: no SIGKILL escalation, no catchable stop
|
|
|
|
## Description
|
|
**Observed:** (source reading, 2026-10-10) a kill without an explicit signal
|
|
sends SIGTERM once: the apiserver defaults kill to TERM
|
|
(`defaultActionSignal`, `src/frontend/apiserver/backend/convert_pure.go`)
|
|
and the worker delivers it once (`job_signal_to_deliver`,
|
|
`src/worker/src/job_signal.c`, after bug-1954; the old worker did the same for
|
|
a non-forced kill). Nothing follows if the job ignores or hangs on SIGTERM.
|
|
|
|
**Expected:** like LSF's TERMINATE action (IBM Spectrum LSF 10.1 Administering
|
|
LSF, "Kill jobs" p. 88 and "TERMINATE action"): a plain kill sends SIGINT,
|
|
then SIGTERM after `JOB_TERMINATE_INTERVAL` (default 10 s), then SIGKILL after
|
|
another interval, each only while the job is still alive; `bkill -s SIG`
|
|
sends just that signal; a forced kill sends SIGKILL at once.
|
|
Likewise a plain stop sends one SIGSTOP (apiserver default since bug-1954),
|
|
which the job cannot catch, so it gets no chance to do pre-stop work; LSF's
|
|
SUSPEND action and the user's requirement (2026-10-10) want a catchable
|
|
SIGTSTP first, with SIGSTOP only if the job has not stopped by itself.
|
|
**Impact:** wrong result: a job that ignores SIGTERM survives a plain kill
|
|
indefinitely (no clean-up window followed by a guaranteed end), and a plain
|
|
stop gives the job no chance to prepare.
|
|
**Found by:** user review of the bug-1954 fix, 2026-10-10.
|
|
|
|
## Environment
|
|
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
|
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: asan (xce.build)
|
|
- Code: scheduler-framework @ e5a0e42b-dirty
|
|
- Deployment: any (real `xcecd` worker; `xcecd_virtual` simulates kills)
|
|
|
|
## Reproduction
|
|
1. Run a job that ignores SIGTERM: `trap '' TERM INT; sleep 600`.
|
|
2. Kill it with no signal option (`xcecli jobs kill <id>`).
|
|
3. The job keeps running.
|
|
- Frequency: deterministic (by code reading; confirmed by the RED test of the fix)
|
|
- Evidence: the code sites in Description; no logs kept.
|
|
|
|
## Root cause
|
|
The job-control path had no notion of a graceful action: `JobSignal`
|
|
carried only `signo` and `force`; the apiserver handler pre-filled a single
|
|
default signal (kill TERM, stop STOP; `controllers/jobs.go`,
|
|
`models.DefaultActionSignal`), and the job process delivered exactly one
|
|
signal (`worker_signal_callback()`, `src/worker/src/xcecd_worker.c`). Nothing
|
|
followed up a TERM the job ignored, and a stop was never catchable.
|
|
Pre-existing design gap (sharpened by bug-1954, which made the delivered
|
|
signal exact).
|
|
|
|
## Fix
|
|
- Wire: `JobSignal.grace @2` (`src/schema/controller.capnp`) /
|
|
`xce_signal_t.grace` (`include/xce/types.h`): > 0 asks for the graceful
|
|
sequence, seconds between signals; 0 keeps a single signal (all existing
|
|
senders).
|
|
- apiserver: `job.terminate_interval` (`utils/config.go`, `config/app.yaml`,
|
|
default 10 s as LSF's JOB_TERMINATE_INTERVAL); `jobCtrlParams()`
|
|
(`backend/convert_pure.go`) sends a plain kill as SIGKILL and a plain stop
|
|
as SIGSTOP with that grace (a force kill and an explicit signal get none);
|
|
`backend/clib.go` passes it; the handler no longer pre-fills a default
|
|
signal; `DefaultActionSignal` updated.
|
|
- worker: grace travels through the job-control task (`thread_pool.h`), the
|
|
kept-signal table (`pending_signal_record_ex/take_ex`), the IPC message
|
|
and callback (`uv_ipc.*`) and the kept-signal delivery
|
|
(`xcecd_worker_utils.c`); `job_signal_seq_*` (`src/worker/src/job_signal.c`,
|
|
libuv timer) runs kill = SIGINT, SIGTERM, SIGKILL and stop = SIGTSTP,
|
|
SIGSTOP, the next signal only while the job is alive / not yet stopped
|
|
(`job_pid_is_stopped()`, /proc state); `xcecd_worker.c` starts it for a
|
|
graceful request (SIGINT is followed by SIGCONT so a stopped job can act),
|
|
a resume cancels a pending stop, and the job's exit cancels any sequence.
|
|
- `xcecd_virtual` has no process: a graceful kill there still ends the job at
|
|
once (its killed-signal simulation is unchanged).
|
|
- Briefs `job-time-window` and `host-failure-job-recovery` updated to reuse
|
|
this sequence for end-time kills and requeue.
|
|
|
|
UPDATE (2026-10-10): Config completed in 97f7c800: job.terminate_interval is defaulted (10 s) and validated by the loader (TestInitConf_TerminateInterval*, RED on the default case before), described in app_schema.json and documented in src/docs/xce.md. It is an API-server setting; cluster-internal kills (e.g. requeue in host-failure-job-recovery) will need the interval from cluster config.
|
|
|
|
## Verification
|
|
- Test: `src/worker/testsuite/test_job_signal.c`
|
|
`test_kill_escalates_to_sigkill_when_ignored`,
|
|
`test_kill_stops_when_job_exits_on_sigint`,
|
|
`test_stop_is_catchable_then_enforced`,
|
|
`test_stop_not_enforced_when_job_stops_itself`,
|
|
`test_cancel_stops_the_sequence`, `test_other_signals_have_no_sequence`
|
|
(real forked children on a libuv loop): RED against the stub (6 failures,
|
|
e.g. `test_job_signal.c:133: FAIL: Expected 0 Was -1`), GREEN after, 5/5
|
|
repeated meson runs; `backend/jobctrl_params_test.go`
|
|
`TestJobCtrlParams_DefaultSignalPerAction` (kill 9 + grace 10, stop 19 +
|
|
grace 10), `TestJobCtrlParams_ExplicitOrForcedKillIsNotGraceful`,
|
|
`TestJobCtrlParams_GraceFromConfig`: failed to build before (no `Grace`),
|
|
GREEN after.
|
|
- Suites: `xce-build.sh all asan xce.build --incremental`, 2026-10-10: C suites
|
|
Ok 155 / Fail 0, frontend Go unit tests ok; apiserver middleware, router,
|
|
cmd ok.
|
|
- Live: not run: a real `xcecd` kill/stop of a TERM-ignoring and a
|
|
TSTP-catching job needs a host with systemd/dbus (follow-up).
|
|
|
|
## References
|
|
- Fix commit: `367af11a`
|
|
- Related: bug-1954 (exact signal delivery), bug-1940 (signal reaches the
|
|
controller)
|
|
- LSF 10.1 Administering LSF: "Kill jobs" (p. 88), "TERMINATE action"
|
|
(JOB_TERMINATE_INTERVAL)
|