--- id: "bug-1957" status: "fixed" status_updated: "2026-10-10" status_note: "" component: "worker" severity: "wrong-result" timestamp: "2026-10-10T09:35:48Z" last_seen: "2026-10-10T09:35:48Z" occurrences: 1 files: ["src/worker/src/xcecd_worker.c", "src/frontend/apiserver/backend/convert_pure.go", "src/schema/controller.capnp"] tags: ["worker", "job-control", "signal", "lsf"] related: ["bug-1954", "bug-1940"] fix_commits: ["367af11a", "97f7c800"] gitea_issue: 388 gitea_synced_status: "fixed" gitea_synced_hash: "0c07e8351f9e" --- # bug-1957: plain job kill and stop are single signals: no SIGKILL escalation, no catchable stop ## Description **Observed:** (source reading, 2026-10-10) a kill without an explicit signal sends SIGTERM once: the apiserver defaults kill to TERM (`defaultActionSignal`, `src/frontend/apiserver/backend/convert_pure.go`) and the worker delivers it once (`job_signal_to_deliver`, `src/worker/src/job_signal.c`, after bug-1954; the old worker did the same for a non-forced kill). Nothing follows if the job ignores or hangs on SIGTERM. **Expected:** like LSF's TERMINATE action (IBM Spectrum LSF 10.1 Administering LSF, "Kill jobs" p. 88 and "TERMINATE action"): a plain kill sends SIGINT, then SIGTERM after `JOB_TERMINATE_INTERVAL` (default 10 s), then SIGKILL after another interval, each only while the job is still alive; `bkill -s SIG` sends just that signal; a forced kill sends SIGKILL at once. Likewise a plain stop sends one SIGSTOP (apiserver default since bug-1954), which the job cannot catch, so it gets no chance to do pre-stop work; LSF's SUSPEND action and the user's requirement (2026-10-10) want a catchable SIGTSTP first, with SIGSTOP only if the job has not stopped by itself. **Impact:** wrong result: a job that ignores SIGTERM survives a plain kill indefinitely (no clean-up window followed by a guaranteed end), and a plain stop gives the job no chance to prepare. **Found by:** user review of the bug-1954 fix, 2026-10-10. ## Environment - OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 - libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) - Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 - Build: asan (xce.build) - Code: scheduler-framework @ e5a0e42b-dirty - Deployment: any (real `xcecd` worker; `xcecd_virtual` simulates kills) ## Reproduction 1. Run a job that ignores SIGTERM: `trap '' TERM INT; sleep 600`. 2. Kill it with no signal option (`xcecli jobs kill `). 3. The job keeps running. - Frequency: deterministic (by code reading; confirmed by the RED test of the fix) - Evidence: the code sites in Description; no logs kept. ## Root cause The job-control path had no notion of a graceful action: `JobSignal` carried only `signo` and `force`; the apiserver handler pre-filled a single default signal (kill TERM, stop STOP; `controllers/jobs.go`, `models.DefaultActionSignal`), and the job process delivered exactly one signal (`worker_signal_callback()`, `src/worker/src/xcecd_worker.c`). Nothing followed up a TERM the job ignored, and a stop was never catchable. Pre-existing design gap (sharpened by bug-1954, which made the delivered signal exact). ## Fix - Wire: `JobSignal.grace @2` (`src/schema/controller.capnp`) / `xce_signal_t.grace` (`include/xce/types.h`): > 0 asks for the graceful sequence, seconds between signals; 0 keeps a single signal (all existing senders). - apiserver: `job.terminate_interval` (`utils/config.go`, `config/app.yaml`, default 10 s as LSF's JOB_TERMINATE_INTERVAL); `jobCtrlParams()` (`backend/convert_pure.go`) sends a plain kill as SIGKILL and a plain stop as SIGSTOP with that grace (a force kill and an explicit signal get none); `backend/clib.go` passes it; the handler no longer pre-fills a default signal; `DefaultActionSignal` updated. - worker: grace travels through the job-control task (`thread_pool.h`), the kept-signal table (`pending_signal_record_ex/take_ex`), the IPC message and callback (`uv_ipc.*`) and the kept-signal delivery (`xcecd_worker_utils.c`); `job_signal_seq_*` (`src/worker/src/job_signal.c`, libuv timer) runs kill = SIGINT, SIGTERM, SIGKILL and stop = SIGTSTP, SIGSTOP, the next signal only while the job is alive / not yet stopped (`job_pid_is_stopped()`, /proc state); `xcecd_worker.c` starts it for a graceful request (SIGINT is followed by SIGCONT so a stopped job can act), a resume cancels a pending stop, and the job's exit cancels any sequence. - `xcecd_virtual` has no process: a graceful kill there still ends the job at once (its killed-signal simulation is unchanged). - Briefs `job-time-window` and `host-failure-job-recovery` updated to reuse this sequence for end-time kills and requeue. UPDATE (2026-10-10): Config completed in 97f7c800: job.terminate_interval is defaulted (10 s) and validated by the loader (TestInitConf_TerminateInterval*, RED on the default case before), described in app_schema.json and documented in src/docs/xce.md. It is an API-server setting; cluster-internal kills (e.g. requeue in host-failure-job-recovery) will need the interval from cluster config. ## Verification - Test: `src/worker/testsuite/test_job_signal.c` `test_kill_escalates_to_sigkill_when_ignored`, `test_kill_stops_when_job_exits_on_sigint`, `test_stop_is_catchable_then_enforced`, `test_stop_not_enforced_when_job_stops_itself`, `test_cancel_stops_the_sequence`, `test_other_signals_have_no_sequence` (real forked children on a libuv loop): RED against the stub (6 failures, e.g. `test_job_signal.c:133: FAIL: Expected 0 Was -1`), GREEN after, 5/5 repeated meson runs; `backend/jobctrl_params_test.go` `TestJobCtrlParams_DefaultSignalPerAction` (kill 9 + grace 10, stop 19 + grace 10), `TestJobCtrlParams_ExplicitOrForcedKillIsNotGraceful`, `TestJobCtrlParams_GraceFromConfig`: failed to build before (no `Grace`), GREEN after. - Suites: `xce-build.sh all asan xce.build --incremental`, 2026-10-10: C suites Ok 155 / Fail 0, frontend Go unit tests ok; apiserver middleware, router, cmd ok. - Live: not run: a real `xcecd` kill/stop of a TERM-ignoring and a TSTP-catching job needs a host with systemd/dbus (follow-up). ## References - Fix commit: `367af11a` - Related: bug-1954 (exact signal delivery), bug-1940 (signal reaches the controller) - LSF 10.1 Administering LSF: "Kill jobs" (p. 88), "TERMINATE action" (JOB_TERMINATE_INTERVAL)