lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(buglog): bug-1957 config completed (97f7c800)

Browse source 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-10 10:04:47 +00:00
parent 97f7c800f1
commit ff1de90ae3
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
2 changed files with 8 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

5
.wolf/buglog.json
View file

@ -18362,7 +18362,7 @@
"error_message": "plain job kill and stop are single signals: no SIGKILL escalation, no catchable stop\n\n**Observed:** (source reading, 2026-10-10) a kill without an explicit signal\nsends SIGTERM once: the apiserver defaults kill to TERM\n(`defaultActionSignal`, `src/frontend/apiserver/backend/convert_pure.go`)\nand the worker delivers it once (`job_signal_to_deliver`,\n`src/worker/src/job_signal.c`, after bug-1954; the old worker did the same for\na non-forced kill). Nothing follows if the job ignores or hangs on SIGTERM.\n\n**Expected:** like LSF's TERMINATE action (IBM Spectrum LSF 10.1 Administering\nLSF, \"Kill jobs\" p. 88 and \"TERMINATE action\"): a plain kill sends SIGINT,\nthen SIGTERM after `JOB_TERMINATE_INTERVAL` (default 10 s), then SIGKILL after\nanother interval, each only while the job is still alive; `bkill -s SIG`\nsends just that signal; a forced kill sends SIGKILL at once.\nLikewise a plain stop sends one SIGSTOP (apiserver default since bug-1954),\nwhich the job cannot catch, so it gets no chance to do pre-stop work; LSF's\nSUSPEND action and the user's requirement (2026-10-10) want a catchable\nSIGTSTP first, with SIGSTOP only if the job has not stopped by itself.\n**Impact:** wrong result: a job that ignores SIGTERM survives a plain kill\nindefinitely (no clean-up window followed by a guaranteed end), and a plain\nstop gives the job no chance to prepare.\n**Found by:** user review of the bug-1954 fix, 2026-10-10.",
"file": "src/worker/src/xcecd_worker.c, src/frontend/apiserver/backend/convert_pure.go, src/schema/controller.capnp",
"root_cause": "The job-control path had no notion of a graceful action: `JobSignal`\ncarried only `signo` and `force`; the apiserver handler pre-filled a single\ndefault signal (kill TERM, stop STOP; `controllers/jobs.go`,\n`models.DefaultActionSignal`), and the job process delivered exactly one\nsignal (`worker_signal_callback()`, `src/worker/src/xcecd_worker.c`). Nothing\nfollowed up a TERM the job ignored, and a stop was never catchable.\nPre-existing design gap (sharpened by bug-1954, which made the delivered\nsignal exact).",
"fix": "- Wire: `JobSignal.grace @2` (`src/schema/controller.capnp`) /\n `xce_signal_t.grace` (`include/xce/types.h`): > 0 asks for the graceful\n sequence, seconds between signals; 0 keeps a single signal (all existing\n senders).\n- apiserver: `job.terminate_interval` (`utils/config.go`, `config/app.yaml`,\n default 10 s as LSF's JOB_TERMINATE_INTERVAL); `jobCtrlParams()`\n (`backend/convert_pure.go`) sends a plain kill as SIGKILL and a plain stop\n as SIGSTOP with that grace (a force kill and an explicit signal get none);\n `backend/clib.go` passes it; the handler no longer pre-fills a default\n signal; `DefaultActionSignal` updated.\n- worker: grace travels through the job-control task (`thread_pool.h`), the\n kept-signal table (`pending_signal_record_ex/take_ex`), the IPC message\n and callback (`uv_ipc.*`) and the kept-signal delivery\n (`xcecd_worker_utils.c`); `job_signal_seq_*` (`src/worker/src/job_signal.c`,\n libuv timer) runs kill = SIGINT, SIGTERM, SIGKILL and stop = SIGTSTP,\n SIGSTOP, the next signal only while the job is alive / not yet stopped\n (`job_pid_is_stopped()`, /proc state); `xcecd_worker.c` starts it for a\n graceful request (SIGINT is followed by SIGCONT so a stopped job can act),\n a resume cancels a pending stop, and the job's exit cancels any sequence.\n- `xcecd_virtual` has no process: a graceful kill there still ends the job at\n once (its killed-signal simulation is unchanged).\n- Briefs `job-time-window` and `host-failure-job-recovery` updated to reuse\n this sequence for end-time kills and requeue.\n\nVerification:\n- Test: `src/worker/testsuite/test_job_signal.c`\n `test_kill_escalates_to_sigkill_when_ignored`,\n `test_kill_stops_when_job_exits_on_sigint`,\n `test_stop_is_catchable_then_enforced`,\n `test_stop_not_enforced_when_job_stops_itself`,\n `test_cancel_stops_the_sequence`, `test_other_signals_have_no_sequence`\n (real forked children on a libuv loop): RED against the stub (6 failures,\n e.g. `test_job_signal.c:133: FAIL: Expected 0 Was -1`), GREEN after, 5/5\n repeated meson runs; `backend/jobctrl_params_test.go`\n `TestJobCtrlParams_DefaultSignalPerAction` (kill 9 + grace 10, stop 19 +\n grace 10), `TestJobCtrlParams_ExplicitOrForcedKillIsNotGraceful`,\n `TestJobCtrlParams_GraceFromConfig`: failed to build before (no `Grace`),\n GREEN after.\n- Suites: `xce-build.sh all asan xce.build --incremental`, 2026-10-10: C suites\n Ok 155 / Fail 0, frontend Go unit tests ok; apiserver middleware, router,\n cmd ok.\n- Live: not run: a real `xcecd` kill/stop of a TERM-ignoring and a\n TSTP-catching job needs a host with systemd/dbus (follow-up).",
"fix": "- Wire: `JobSignal.grace @2` (`src/schema/controller.capnp`) /\n `xce_signal_t.grace` (`include/xce/types.h`): > 0 asks for the graceful\n sequence, seconds between signals; 0 keeps a single signal (all existing\n senders).\n- apiserver: `job.terminate_interval` (`utils/config.go`, `config/app.yaml`,\n default 10 s as LSF's JOB_TERMINATE_INTERVAL); `jobCtrlParams()`\n (`backend/convert_pure.go`) sends a plain kill as SIGKILL and a plain stop\n as SIGSTOP with that grace (a force kill and an explicit signal get none);\n `backend/clib.go` passes it; the handler no longer pre-fills a default\n signal; `DefaultActionSignal` updated.\n- worker: grace travels through the job-control task (`thread_pool.h`), the\n kept-signal table (`pending_signal_record_ex/take_ex`), the IPC message\n and callback (`uv_ipc.*`) and the kept-signal delivery\n (`xcecd_worker_utils.c`); `job_signal_seq_*` (`src/worker/src/job_signal.c`,\n libuv timer) runs kill = SIGINT, SIGTERM, SIGKILL and stop = SIGTSTP,\n SIGSTOP, the next signal only while the job is alive / not yet stopped\n (`job_pid_is_stopped()`, /proc state); `xcecd_worker.c` starts it for a\n graceful request (SIGINT is followed by SIGCONT so a stopped job can act),\n a resume cancels a pending stop, and the job's exit cancels any sequence.\n- `xcecd_virtual` has no process: a graceful kill there still ends the job at\n once (its killed-signal simulation is unchanged).\n- Briefs `job-time-window` and `host-failure-job-recovery` updated to reuse\n this sequence for end-time kills and requeue.\n\nUPDATE (2026-10-10): Config completed in 97f7c800: job.terminate_interval is defaulted (10 s) and validated by the loader (TestInitConf_TerminateInterval*, RED on the default case before), described in app_schema.json and documented in src/docs/xce.md. It is an API-server setting; cluster-internal kills (e.g. requeue in host-failure-job-recovery) will need the interval from cluster config.\n\nVerification:\n- Test: `src/worker/testsuite/test_job_signal.c`\n `test_kill_escalates_to_sigkill_when_ignored`,\n `test_kill_stops_when_job_exits_on_sigint`,\n `test_stop_is_catchable_then_enforced`,\n `test_stop_not_enforced_when_job_stops_itself`,\n `test_cancel_stops_the_sequence`, `test_other_signals_have_no_sequence`\n (real forked children on a libuv loop): RED against the stub (6 failures,\n e.g. `test_job_signal.c:133: FAIL: Expected 0 Was -1`), GREEN after, 5/5\n repeated meson runs; `backend/jobctrl_params_test.go`\n `TestJobCtrlParams_DefaultSignalPerAction` (kill 9 + grace 10, stop 19 +\n grace 10), `TestJobCtrlParams_ExplicitOrForcedKillIsNotGraceful`,\n `TestJobCtrlParams_GraceFromConfig`: failed to build before (no `Grace`),\n GREEN after.\n- Suites: `xce-build.sh all asan xce.build --incremental`, 2026-10-10: C suites\n Ok 155 / Fail 0, frontend Go unit tests ok; apiserver middleware, router,\n cmd ok.\n- Live: not run: a real `xcecd` kill/stop of a TERM-ignoring and a\n TSTP-catching job needs a host with systemd/dbus (follow-up).",
"tags": [
"worker",
"job-control",
@ -18374,7 +18374,8 @@
"bug-1940"
],
"fix_commits": [
"367af11a"
"367af11a",
"97f7c800"
],
"record": ".wolf/bugs/bug-1957.md"
}

6
.wolf/bugs/bug-1957.md
View file

@ -11,7 +11,7 @@ occurrences: 1
files: ["src/worker/src/xcecd_worker.c", "src/frontend/apiserver/backend/convert_pure.go", "src/schema/controller.capnp"]
tags: ["worker", "job-control", "signal", "lsf"]
related: ["bug-1954", "bug-1940"]
fix_commits: ["367af11a"]
fix_commits: ["367af11a", "97f7c800"]
---
# bug-1957: plain job kill and stop are single signals: no SIGKILL escalation, no catchable stop
@ -37,6 +37,7 @@ SIGTSTP first, with SIGSTOP only if the job has not stopped by itself.
indefinitely (no clean-up window followed by a guaranteed end), and a plain
stop gives the job no chance to prepare.
**Found by:** user review of the bug-1954 fix, 2026-10-10.
## Environment
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
@ -51,6 +52,7 @@ stop gives the job no chance to prepare.
3. The job keeps running.
- Frequency: deterministic (by code reading; confirmed by the RED test of the fix)
- Evidence: the code sites in Description; no logs kept.
## Root cause
The job-control path had no notion of a graceful action: `JobSignal`
carried only `signo` and `force`; the apiserver handler pre-filled a single
@ -86,6 +88,8 @@ signal exact).
- Briefs `job-time-window` and `host-failure-job-recovery` updated to reuse
this sequence for end-time kills and requeue.
UPDATE (2026-10-10): Config completed in 97f7c800: job.terminate_interval is defaulted (10 s) and validated by the loader (TestInitConf_TerminateInterval*, RED on the default case before), described in app_schema.json and documented in src/docs/xce.md. It is an API-server setting; cluster-internal kills (e.g. requeue in host-failure-job-recovery) will need the interval from cluster config.
## Verification
- Test: `src/worker/testsuite/test_job_signal.c`
`test_kill_escalates_to_sigkill_when_ignored`,