myxce/.wolf/bugs/bug-1957.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	gitea_issue	gitea_synced_status	gitea_synced_hash
bug-1957	fixed	2026-10-10		worker	wrong-result	2026-10-10T09:35:48Z	2026-10-10T09:35:48Z	1	src/worker/src/xcecd_worker.c src/frontend/apiserver/backend/convert_pure.go src/schema/controller.capnp	worker job-control signal lsf	bug-1954 bug-1940	367af11a 97f7c800	388	fixed	0c07e8351f9e

bug-1957: plain job kill and stop are single signals: no SIGKILL escalation, no catchable stop

Description

Observed: (source reading, 2026-10-10) a kill without an explicit signal sends SIGTERM once: the apiserver defaults kill to TERM (defaultActionSignal, src/frontend/apiserver/backend/convert_pure.go) and the worker delivers it once (job_signal_to_deliver, src/worker/src/job_signal.c, after bug-1954; the old worker did the same for a non-forced kill). Nothing follows if the job ignores or hangs on SIGTERM.

Expected: like LSF's TERMINATE action (IBM Spectrum LSF 10.1 Administering LSF, "Kill jobs" p. 88 and "TERMINATE action"): a plain kill sends SIGINT, then SIGTERM after JOB_TERMINATE_INTERVAL (default 10 s), then SIGKILL after another interval, each only while the job is still alive; bkill -s SIG sends just that signal; a forced kill sends SIGKILL at once. Likewise a plain stop sends one SIGSTOP (apiserver default since bug-1954), which the job cannot catch, so it gets no chance to do pre-stop work; LSF's SUSPEND action and the user's requirement (2026-10-10) want a catchable SIGTSTP first, with SIGSTOP only if the job has not stopped by itself. Impact: wrong result: a job that ignores SIGTERM survives a plain kill indefinitely (no clean-up window followed by a guaranteed end), and a plain stop gives the job no chance to prepare. Found by: user review of the bug-1954 fix, 2026-10-10.

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (xce.build)
• Code: scheduler-framework @ e5a0e42b-dirty
• Deployment: any (real xcecd worker; xcecd_virtual simulates kills)

Reproduction

1. Run a job that ignores SIGTERM: trap '' TERM INT; sleep 600.
2. Kill it with no signal option (xcecli jobs kill <id>).
3. The job keeps running.

• Frequency: deterministic (by code reading; confirmed by the RED test of the fix)
• Evidence: the code sites in Description; no logs kept.

Root cause

The job-control path had no notion of a graceful action: JobSignal carried only signo and force; the apiserver handler pre-filled a single default signal (kill TERM, stop STOP; controllers/jobs.go, models.DefaultActionSignal), and the job process delivered exactly one signal (worker_signal_callback(), src/worker/src/xcecd_worker.c). Nothing followed up a TERM the job ignored, and a stop was never catchable. Pre-existing design gap (sharpened by bug-1954, which made the delivered signal exact).

Fix

• Wire: JobSignal.grace @2 (src/schema/controller.capnp) / xce_signal_t.grace (include/xce/types.h): > 0 asks for the graceful sequence, seconds between signals; 0 keeps a single signal (all existing senders).
• apiserver: job.terminate_interval (utils/config.go, config/app.yaml, default 10 s as LSF's JOB_TERMINATE_INTERVAL); jobCtrlParams() (backend/convert_pure.go) sends a plain kill as SIGKILL and a plain stop as SIGSTOP with that grace (a force kill and an explicit signal get none); backend/clib.go passes it; the handler no longer pre-fills a default signal; DefaultActionSignal updated.
• worker: grace travels through the job-control task (thread_pool.h), the kept-signal table (pending_signal_record_ex/take_ex), the IPC message and callback (uv_ipc.*) and the kept-signal delivery (xcecd_worker_utils.c); job_signal_seq_* (src/worker/src/job_signal.c, libuv timer) runs kill = SIGINT, SIGTERM, SIGKILL and stop = SIGTSTP, SIGSTOP, the next signal only while the job is alive / not yet stopped (job_pid_is_stopped(), /proc state); xcecd_worker.c starts it for a graceful request (SIGINT is followed by SIGCONT so a stopped job can act), a resume cancels a pending stop, and the job's exit cancels any sequence.
• xcecd_virtual has no process: a graceful kill there still ends the job at once (its killed-signal simulation is unchanged).
• Briefs job-time-window and host-failure-job-recovery updated to reuse this sequence for end-time kills and requeue.

UPDATE (2026-10-10): Config completed in 97f7c800: job.terminate_interval is defaulted (10 s) and validated by the loader (TestInitConf_TerminateInterval*, RED on the default case before), described in app_schema.json and documented in src/docs/xce.md. It is an API-server setting; cluster-internal kills (e.g. requeue in host-failure-job-recovery) will need the interval from cluster config.

Verification

• Test: src/worker/testsuite/test_job_signal.c test_kill_escalates_to_sigkill_when_ignored, test_kill_stops_when_job_exits_on_sigint, test_stop_is_catchable_then_enforced, test_stop_not_enforced_when_job_stops_itself, test_cancel_stops_the_sequence, test_other_signals_have_no_sequence (real forked children on a libuv loop): RED against the stub (6 failures, e.g. test_job_signal.c:133: FAIL: Expected 0 Was -1), GREEN after, 5/5 repeated meson runs; backend/jobctrl_params_test.go TestJobCtrlParams_DefaultSignalPerAction (kill 9 + grace 10, stop 19 + grace 10), TestJobCtrlParams_ExplicitOrForcedKillIsNotGraceful, TestJobCtrlParams_GraceFromConfig: failed to build before (no Grace), GREEN after.
• Suites: xce-build.sh all asan xce.build --incremental, 2026-10-10: C suites Ok 155 / Fail 0, frontend Go unit tests ok; apiserver middleware, router, cmd ok.
• Live: not run: a real xcecd kill/stop of a TERM-ignoring and a TSTP-catching job needs a host with systemd/dbus (follow-up).

References

• Fix commit: 367af11a
• Related: bug-1954 (exact signal delivery), bug-1940 (signal reaches the controller)
• LSF 10.1 Administering LSF: "Kill jobs" (p. 88), "TERMINATE action" (JOB_TERMINATE_INTERVAL)