docs(buglog,specs): bug-1957 fixed (367af11a); time-window and host-failure briefs reuse the graceful kill

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.wolf/bugs/bug-1957.md

@@ -0,0 +1,114 @@
+---
+id: "bug-1957"
+status: "fixed"
+status_updated: "2026-10-10"
+status_note: ""
+component: "worker"
+severity: "wrong-result"
+timestamp: "2026-10-10T09:35:48Z"
+last_seen: "2026-10-10T09:35:48Z"
+occurrences: 1
+files: ["src/worker/src/xcecd_worker.c", "src/frontend/apiserver/backend/convert_pure.go", "src/schema/controller.capnp"]
+tags: ["worker", "job-control", "signal", "lsf"]
+related: ["bug-1954", "bug-1940"]
+fix_commits: ["367af11a"]
+---
+
+# bug-1957: plain job kill and stop are single signals: no SIGKILL escalation, no catchable stop
+
+## Description
+**Observed:** (source reading, 2026-10-10) a kill without an explicit signal
+sends SIGTERM once: the apiserver defaults kill to TERM
+(`defaultActionSignal`, `src/frontend/apiserver/backend/convert_pure.go`)
+and the worker delivers it once (`job_signal_to_deliver`,
+`src/worker/src/job_signal.c`, after bug-1954; the old worker did the same for
+a non-forced kill). Nothing follows if the job ignores or hangs on SIGTERM.
+
+**Expected:** like LSF's TERMINATE action (IBM Spectrum LSF 10.1 Administering
+LSF, "Kill jobs" p. 88 and "TERMINATE action"): a plain kill sends SIGINT,
+then SIGTERM after `JOB_TERMINATE_INTERVAL` (default 10 s), then SIGKILL after
+another interval, each only while the job is still alive; `bkill -s SIG`
+sends just that signal; a forced kill sends SIGKILL at once.
+Likewise a plain stop sends one SIGSTOP (apiserver default since bug-1954),
+which the job cannot catch, so it gets no chance to do pre-stop work; LSF's
+SUSPEND action and the user's requirement (2026-10-10) want a catchable
+SIGTSTP first, with SIGSTOP only if the job has not stopped by itself.
+**Impact:** wrong result: a job that ignores SIGTERM survives a plain kill
+indefinitely (no clean-up window followed by a guaranteed end), and a plain
+stop gives the job no chance to prepare.
+**Found by:** user review of the bug-1954 fix, 2026-10-10.
+## Environment
+- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
+- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
+- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
+- Build: asan (xce.build)
+- Code: scheduler-framework @ e5a0e42b-dirty
+- Deployment: any (real `xcecd` worker; `xcecd_virtual` simulates kills)
+
+## Reproduction
+1. Run a job that ignores SIGTERM: `trap '' TERM INT; sleep 600`.
+2. Kill it with no signal option (`xcecli jobs kill <id>`).
+3. The job keeps running.
+- Frequency: deterministic (by code reading; confirmed by the RED test of the fix)
+- Evidence: the code sites in Description; no logs kept.
+## Root cause
+The job-control path had no notion of a graceful action: `JobSignal`
+carried only `signo` and `force`; the apiserver handler pre-filled a single
+default signal (kill TERM, stop STOP; `controllers/jobs.go`,
+`models.DefaultActionSignal`), and the job process delivered exactly one
+signal (`worker_signal_callback()`, `src/worker/src/xcecd_worker.c`). Nothing
+followed up a TERM the job ignored, and a stop was never catchable.
+Pre-existing design gap (sharpened by bug-1954, which made the delivered
+signal exact).
+
+## Fix
+- Wire: `JobSignal.grace @2` (`src/schema/controller.capnp`) /
+  `xce_signal_t.grace` (`include/xce/types.h`): > 0 asks for the graceful
+  sequence, seconds between signals; 0 keeps a single signal (all existing
+  senders).
+- apiserver: `job.terminate_interval` (`utils/config.go`, `config/app.yaml`,
+  default 10 s as LSF's JOB_TERMINATE_INTERVAL); `jobCtrlParams()`
+  (`backend/convert_pure.go`) sends a plain kill as SIGKILL and a plain stop
+  as SIGSTOP with that grace (a force kill and an explicit signal get none);
+  `backend/clib.go` passes it; the handler no longer pre-fills a default
+  signal; `DefaultActionSignal` updated.
+- worker: grace travels through the job-control task (`thread_pool.h`), the
+  kept-signal table (`pending_signal_record_ex/take_ex`), the IPC message
+  and callback (`uv_ipc.*`) and the kept-signal delivery
+  (`xcecd_worker_utils.c`); `job_signal_seq_*` (`src/worker/src/job_signal.c`,
+  libuv timer) runs kill = SIGINT, SIGTERM, SIGKILL and stop = SIGTSTP,
+  SIGSTOP, the next signal only while the job is alive / not yet stopped
+  (`job_pid_is_stopped()`, /proc state); `xcecd_worker.c` starts it for a
+  graceful request (SIGINT is followed by SIGCONT so a stopped job can act),
+  a resume cancels a pending stop, and the job's exit cancels any sequence.
+- `xcecd_virtual` has no process: a graceful kill there still ends the job at
+  once (its killed-signal simulation is unchanged).
+- Briefs `job-time-window` and `host-failure-job-recovery` updated to reuse
+  this sequence for end-time kills and requeue.
+
+## Verification
+- Test: `src/worker/testsuite/test_job_signal.c`
+  `test_kill_escalates_to_sigkill_when_ignored`,
+  `test_kill_stops_when_job_exits_on_sigint`,
+  `test_stop_is_catchable_then_enforced`,
+  `test_stop_not_enforced_when_job_stops_itself`,
+  `test_cancel_stops_the_sequence`, `test_other_signals_have_no_sequence`
+  (real forked children on a libuv loop): RED against the stub (6 failures,
+  e.g. `test_job_signal.c:133: FAIL: Expected 0 Was -1`), GREEN after, 5/5
+  repeated meson runs; `backend/jobctrl_params_test.go`
+  `TestJobCtrlParams_DefaultSignalPerAction` (kill 9 + grace 10, stop 19 +
+  grace 10), `TestJobCtrlParams_ExplicitOrForcedKillIsNotGraceful`,
+  `TestJobCtrlParams_GraceFromConfig`: failed to build before (no `Grace`),
+  GREEN after.
+- Suites: `xce-build.sh all asan xce.build --incremental`, 2026-10-10: C suites
+  Ok 155 / Fail 0, frontend Go unit tests ok; apiserver middleware, router,
+  cmd ok.
+- Live: not run: a real `xcecd` kill/stop of a TERM-ignoring and a
+  TSTP-catching job needs a host with systemd/dbus (follow-up).
+
+## References
+- Fix commit: `367af11a`
+- Related: bug-1954 (exact signal delivery), bug-1940 (signal reaches the
+  controller)
+- LSF 10.1 Administering LSF: "Kill jobs" (p. 88), "TERMINATE action"
+  (JOB_TERMINATE_INTERVAL)