From 73fe8f787c1990d49bf1d398f71bf94cf93db1a3 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sat, 10 Oct 2026 10:01:02 +0000 Subject: [PATCH] docs(buglog,specs): bug-1957 fixed (367af11a); time-window and host-failure briefs reuse the graceful kill Co-Authored-By: Claude Opus 5.5 (1M context) --- .../specs/host-failure-job-recovery/brief.md | 5 + .kiro/specs/job-time-window/brief.md | 8 ++ .wolf/buglog.json | 29 +++++ .wolf/bugs/bug-1957.md | 114 ++++++++++++++++++ 4 files changed, 156 insertions(+) create mode 100644 .wolf/bugs/bug-1957.md diff --git a/.kiro/specs/host-failure-job-recovery/brief.md b/.kiro/specs/host-failure-job-recovery/brief.md index c790a722..66123bfa 100644 --- a/.kiro/specs/host-failure-job-recovery/brief.md +++ b/.kiro/specs/host-failure-job-recovery/brief.md @@ -89,3 +89,8 @@ that kills jobs whose attempt number is stale. - HA/persistence: full 3-phase workflow; additive Cap'n Proto; the new facts round-trip through WAL and snapshot (`/xce-audit-invariants`). - The timer must never block the delaymgr (`ha-async-io.md`). + +**Update 2026-10-10 (bug-1957):** killing a job's stale copy or a job being +requeued should use the worker's graceful termination (LSF TERMINATE action: +SIGINT, SIGTERM, SIGKILL with `job.terminate_interval`), as LSF's `brequeue` +does, rather than an immediate SIGKILL. diff --git a/.kiro/specs/job-time-window/brief.md b/.kiro/specs/job-time-window/brief.md index 334a4d5d..7127ff57 100644 --- a/.kiro/specs/job-time-window/brief.md +++ b/.kiro/specs/job-time-window/brief.md @@ -266,3 +266,11 @@ backstop. enforce the deadline too; the helper timer covers it. - TDD: `testsuite/` first, then `_test.go`. ASAN must be clean. - HA/persistence touch: use the full 3-phase workflow. + +**Update 2026-10-10 (bug-1957):** the worker now has a shared graceful +termination: a job-control kill with `grace > 0` runs LSF's TERMINATE action +(SIGINT, SIGTERM, SIGKILL, one grace period apart, stopping as soon as the +job is gone; `job_signal_seq_*`, `src/worker/src/job_signal.c`), with the +cluster default `job.terminate_interval` (10 s). The end-time kill should +reuse it (sequence and interval) instead of its own SIGTERM → grace → +SIGKILL helper described above. diff --git a/.wolf/buglog.json b/.wolf/buglog.json index a8acad9e..789d6b9b 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -18348,6 +18348,35 @@ "fix_commits": [], "record": ".wolf/bugs/bug-1956.md", "repro_attempts": "decoder analysis 2026-10-10: empty launchType decodes to \"\" (capn_val0 default + strdup), never NULL" + }, + { + "id": "bug-1957", + "status": "fixed", + "status_updated": "2026-10-10", + "status_note": "", + "component": "worker", + "severity": "wrong-result", + "timestamp": "2026-10-10T09:35:48Z", + "last_seen": "2026-10-10T09:35:48Z", + "occurrences": 1, + "error_message": "plain job kill and stop are single signals: no SIGKILL escalation, no catchable stop\n\n**Observed:** (source reading, 2026-10-10) a kill without an explicit signal\nsends SIGTERM once: the apiserver defaults kill to TERM\n(`defaultActionSignal`, `src/frontend/apiserver/backend/convert_pure.go`)\nand the worker delivers it once (`job_signal_to_deliver`,\n`src/worker/src/job_signal.c`, after bug-1954; the old worker did the same for\na non-forced kill). Nothing follows if the job ignores or hangs on SIGTERM.\n\n**Expected:** like LSF's TERMINATE action (IBM Spectrum LSF 10.1 Administering\nLSF, \"Kill jobs\" p. 88 and \"TERMINATE action\"): a plain kill sends SIGINT,\nthen SIGTERM after `JOB_TERMINATE_INTERVAL` (default 10 s), then SIGKILL after\nanother interval, each only while the job is still alive; `bkill -s SIG`\nsends just that signal; a forced kill sends SIGKILL at once.\nLikewise a plain stop sends one SIGSTOP (apiserver default since bug-1954),\nwhich the job cannot catch, so it gets no chance to do pre-stop work; LSF's\nSUSPEND action and the user's requirement (2026-10-10) want a catchable\nSIGTSTP first, with SIGSTOP only if the job has not stopped by itself.\n**Impact:** wrong result: a job that ignores SIGTERM survives a plain kill\nindefinitely (no clean-up window followed by a guaranteed end), and a plain\nstop gives the job no chance to prepare.\n**Found by:** user review of the bug-1954 fix, 2026-10-10.", + "file": "src/worker/src/xcecd_worker.c, src/frontend/apiserver/backend/convert_pure.go, src/schema/controller.capnp", + "root_cause": "The job-control path had no notion of a graceful action: `JobSignal`\ncarried only `signo` and `force`; the apiserver handler pre-filled a single\ndefault signal (kill TERM, stop STOP; `controllers/jobs.go`,\n`models.DefaultActionSignal`), and the job process delivered exactly one\nsignal (`worker_signal_callback()`, `src/worker/src/xcecd_worker.c`). Nothing\nfollowed up a TERM the job ignored, and a stop was never catchable.\nPre-existing design gap (sharpened by bug-1954, which made the delivered\nsignal exact).", + "fix": "- Wire: `JobSignal.grace @2` (`src/schema/controller.capnp`) /\n `xce_signal_t.grace` (`include/xce/types.h`): > 0 asks for the graceful\n sequence, seconds between signals; 0 keeps a single signal (all existing\n senders).\n- apiserver: `job.terminate_interval` (`utils/config.go`, `config/app.yaml`,\n default 10 s as LSF's JOB_TERMINATE_INTERVAL); `jobCtrlParams()`\n (`backend/convert_pure.go`) sends a plain kill as SIGKILL and a plain stop\n as SIGSTOP with that grace (a force kill and an explicit signal get none);\n `backend/clib.go` passes it; the handler no longer pre-fills a default\n signal; `DefaultActionSignal` updated.\n- worker: grace travels through the job-control task (`thread_pool.h`), the\n kept-signal table (`pending_signal_record_ex/take_ex`), the IPC message\n and callback (`uv_ipc.*`) and the kept-signal delivery\n (`xcecd_worker_utils.c`); `job_signal_seq_*` (`src/worker/src/job_signal.c`,\n libuv timer) runs kill = SIGINT, SIGTERM, SIGKILL and stop = SIGTSTP,\n SIGSTOP, the next signal only while the job is alive / not yet stopped\n (`job_pid_is_stopped()`, /proc state); `xcecd_worker.c` starts it for a\n graceful request (SIGINT is followed by SIGCONT so a stopped job can act),\n a resume cancels a pending stop, and the job's exit cancels any sequence.\n- `xcecd_virtual` has no process: a graceful kill there still ends the job at\n once (its killed-signal simulation is unchanged).\n- Briefs `job-time-window` and `host-failure-job-recovery` updated to reuse\n this sequence for end-time kills and requeue.\n\nVerification:\n- Test: `src/worker/testsuite/test_job_signal.c`\n `test_kill_escalates_to_sigkill_when_ignored`,\n `test_kill_stops_when_job_exits_on_sigint`,\n `test_stop_is_catchable_then_enforced`,\n `test_stop_not_enforced_when_job_stops_itself`,\n `test_cancel_stops_the_sequence`, `test_other_signals_have_no_sequence`\n (real forked children on a libuv loop): RED against the stub (6 failures,\n e.g. `test_job_signal.c:133: FAIL: Expected 0 Was -1`), GREEN after, 5/5\n repeated meson runs; `backend/jobctrl_params_test.go`\n `TestJobCtrlParams_DefaultSignalPerAction` (kill 9 + grace 10, stop 19 +\n grace 10), `TestJobCtrlParams_ExplicitOrForcedKillIsNotGraceful`,\n `TestJobCtrlParams_GraceFromConfig`: failed to build before (no `Grace`),\n GREEN after.\n- Suites: `xce-build.sh all asan xce.build --incremental`, 2026-10-10: C suites\n Ok 155 / Fail 0, frontend Go unit tests ok; apiserver middleware, router,\n cmd ok.\n- Live: not run: a real `xcecd` kill/stop of a TERM-ignoring and a\n TSTP-catching job needs a host with systemd/dbus (follow-up).", + "tags": [ + "worker", + "job-control", + "signal", + "lsf" + ], + "related_bugs": [ + "bug-1954", + "bug-1940" + ], + "fix_commits": [ + "367af11a" + ], + "record": ".wolf/bugs/bug-1957.md" } ] } diff --git a/.wolf/bugs/bug-1957.md b/.wolf/bugs/bug-1957.md new file mode 100644 index 00000000..a9bf8f38 --- /dev/null +++ b/.wolf/bugs/bug-1957.md @@ -0,0 +1,114 @@ +--- +id: "bug-1957" +status: "fixed" +status_updated: "2026-10-10" +status_note: "" +component: "worker" +severity: "wrong-result" +timestamp: "2026-10-10T09:35:48Z" +last_seen: "2026-10-10T09:35:48Z" +occurrences: 1 +files: ["src/worker/src/xcecd_worker.c", "src/frontend/apiserver/backend/convert_pure.go", "src/schema/controller.capnp"] +tags: ["worker", "job-control", "signal", "lsf"] +related: ["bug-1954", "bug-1940"] +fix_commits: ["367af11a"] +--- + +# bug-1957: plain job kill and stop are single signals: no SIGKILL escalation, no catchable stop + +## Description +**Observed:** (source reading, 2026-10-10) a kill without an explicit signal +sends SIGTERM once: the apiserver defaults kill to TERM +(`defaultActionSignal`, `src/frontend/apiserver/backend/convert_pure.go`) +and the worker delivers it once (`job_signal_to_deliver`, +`src/worker/src/job_signal.c`, after bug-1954; the old worker did the same for +a non-forced kill). Nothing follows if the job ignores or hangs on SIGTERM. + +**Expected:** like LSF's TERMINATE action (IBM Spectrum LSF 10.1 Administering +LSF, "Kill jobs" p. 88 and "TERMINATE action"): a plain kill sends SIGINT, +then SIGTERM after `JOB_TERMINATE_INTERVAL` (default 10 s), then SIGKILL after +another interval, each only while the job is still alive; `bkill -s SIG` +sends just that signal; a forced kill sends SIGKILL at once. +Likewise a plain stop sends one SIGSTOP (apiserver default since bug-1954), +which the job cannot catch, so it gets no chance to do pre-stop work; LSF's +SUSPEND action and the user's requirement (2026-10-10) want a catchable +SIGTSTP first, with SIGSTOP only if the job has not stopped by itself. +**Impact:** wrong result: a job that ignores SIGTERM survives a plain kill +indefinitely (no clean-up window followed by a guaranteed end), and a plain +stop gives the job no chance to prepare. +**Found by:** user review of the bug-1954 fix, 2026-10-10. +## Environment +- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 +- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan (xce.build) +- Code: scheduler-framework @ e5a0e42b-dirty +- Deployment: any (real `xcecd` worker; `xcecd_virtual` simulates kills) + +## Reproduction +1. Run a job that ignores SIGTERM: `trap '' TERM INT; sleep 600`. +2. Kill it with no signal option (`xcecli jobs kill `). +3. The job keeps running. +- Frequency: deterministic (by code reading; confirmed by the RED test of the fix) +- Evidence: the code sites in Description; no logs kept. +## Root cause +The job-control path had no notion of a graceful action: `JobSignal` +carried only `signo` and `force`; the apiserver handler pre-filled a single +default signal (kill TERM, stop STOP; `controllers/jobs.go`, +`models.DefaultActionSignal`), and the job process delivered exactly one +signal (`worker_signal_callback()`, `src/worker/src/xcecd_worker.c`). Nothing +followed up a TERM the job ignored, and a stop was never catchable. +Pre-existing design gap (sharpened by bug-1954, which made the delivered +signal exact). + +## Fix +- Wire: `JobSignal.grace @2` (`src/schema/controller.capnp`) / + `xce_signal_t.grace` (`include/xce/types.h`): > 0 asks for the graceful + sequence, seconds between signals; 0 keeps a single signal (all existing + senders). +- apiserver: `job.terminate_interval` (`utils/config.go`, `config/app.yaml`, + default 10 s as LSF's JOB_TERMINATE_INTERVAL); `jobCtrlParams()` + (`backend/convert_pure.go`) sends a plain kill as SIGKILL and a plain stop + as SIGSTOP with that grace (a force kill and an explicit signal get none); + `backend/clib.go` passes it; the handler no longer pre-fills a default + signal; `DefaultActionSignal` updated. +- worker: grace travels through the job-control task (`thread_pool.h`), the + kept-signal table (`pending_signal_record_ex/take_ex`), the IPC message + and callback (`uv_ipc.*`) and the kept-signal delivery + (`xcecd_worker_utils.c`); `job_signal_seq_*` (`src/worker/src/job_signal.c`, + libuv timer) runs kill = SIGINT, SIGTERM, SIGKILL and stop = SIGTSTP, + SIGSTOP, the next signal only while the job is alive / not yet stopped + (`job_pid_is_stopped()`, /proc state); `xcecd_worker.c` starts it for a + graceful request (SIGINT is followed by SIGCONT so a stopped job can act), + a resume cancels a pending stop, and the job's exit cancels any sequence. +- `xcecd_virtual` has no process: a graceful kill there still ends the job at + once (its killed-signal simulation is unchanged). +- Briefs `job-time-window` and `host-failure-job-recovery` updated to reuse + this sequence for end-time kills and requeue. + +## Verification +- Test: `src/worker/testsuite/test_job_signal.c` + `test_kill_escalates_to_sigkill_when_ignored`, + `test_kill_stops_when_job_exits_on_sigint`, + `test_stop_is_catchable_then_enforced`, + `test_stop_not_enforced_when_job_stops_itself`, + `test_cancel_stops_the_sequence`, `test_other_signals_have_no_sequence` + (real forked children on a libuv loop): RED against the stub (6 failures, + e.g. `test_job_signal.c:133: FAIL: Expected 0 Was -1`), GREEN after, 5/5 + repeated meson runs; `backend/jobctrl_params_test.go` + `TestJobCtrlParams_DefaultSignalPerAction` (kill 9 + grace 10, stop 19 + + grace 10), `TestJobCtrlParams_ExplicitOrForcedKillIsNotGraceful`, + `TestJobCtrlParams_GraceFromConfig`: failed to build before (no `Grace`), + GREEN after. +- Suites: `xce-build.sh all asan xce.build --incremental`, 2026-10-10: C suites + Ok 155 / Fail 0, frontend Go unit tests ok; apiserver middleware, router, + cmd ok. +- Live: not run: a real `xcecd` kill/stop of a TERM-ignoring and a + TSTP-catching job needs a host with systemd/dbus (follow-up). + +## References +- Fix commit: `367af11a` +- Related: bug-1954 (exact signal delivery), bug-1940 (signal reaches the + controller) +- LSF 10.1 Administering LSF: "Kill jobs" (p. 88), "TERMINATE action" + (JOB_TERMINATE_INTERVAL)