docs(buglog,specs): bug-1957 fixed (367af11a); time-window and host-failure briefs reuse the graceful kill
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
367af11af1
commit
73fe8f787c
GPG key ID:
80533536B0EB651C
4 changed files with 156 additions and 0 deletions
|
|
@ -89,3 +89,8 @@ that kills jobs whose attempt number is stale.
|
|||
- HA/persistence: full 3-phase workflow; additive Cap'n Proto; the new facts
|
||||
round-trip through WAL and snapshot (`/xce-audit-invariants`).
|
||||
- The timer must never block the delaymgr (`ha-async-io.md`).
|
||||
|
||||
**Update 2026-10-10 (bug-1957):** killing a job's stale copy or a job being
|
||||
requeued should use the worker's graceful termination (LSF TERMINATE action:
|
||||
SIGINT, SIGTERM, SIGKILL with `job.terminate_interval`), as LSF's `brequeue`
|
||||
does, rather than an immediate SIGKILL.
|
||||
|
|
|
|||
|
|
@ -266,3 +266,11 @@ backstop.
|
|||
enforce the deadline too; the helper timer covers it.
|
||||
- TDD: `testsuite/` first, then `_test.go`. ASAN must be clean.
|
||||
- HA/persistence touch: use the full 3-phase workflow.
|
||||
|
||||
**Update 2026-10-10 (bug-1957):** the worker now has a shared graceful
|
||||
termination: a job-control kill with `grace > 0` runs LSF's TERMINATE action
|
||||
(SIGINT, SIGTERM, SIGKILL, one grace period apart, stopping as soon as the
|
||||
job is gone; `job_signal_seq_*`, `src/worker/src/job_signal.c`), with the
|
||||
cluster default `job.terminate_interval` (10 s). The end-time kill should
|
||||
reuse it (sequence and interval) instead of its own SIGTERM → grace →
|
||||
SIGKILL helper described above.
|
||||
|
|
|
|||
|
|
@ -18348,6 +18348,35 @@
|
|||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1956.md",
|
||||
"repro_attempts": "decoder analysis 2026-10-10: empty launchType decodes to \"\" (capn_val0 default + strdup), never NULL"
|
||||
},
|
||||
{
|
||||
"id": "bug-1957",
|
||||
"status": "fixed",
|
||||
"status_updated": "2026-10-10",
|
||||
"status_note": "",
|
||||
"component": "worker",
|
||||
"severity": "wrong-result",
|
||||
"timestamp": "2026-10-10T09:35:48Z",
|
||||
"last_seen": "2026-10-10T09:35:48Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "plain job kill and stop are single signals: no SIGKILL escalation, no catchable stop\n\n**Observed:** (source reading, 2026-10-10) a kill without an explicit signal\nsends SIGTERM once: the apiserver defaults kill to TERM\n(`defaultActionSignal`, `src/frontend/apiserver/backend/convert_pure.go`)\nand the worker delivers it once (`job_signal_to_deliver`,\n`src/worker/src/job_signal.c`, after bug-1954; the old worker did the same for\na non-forced kill). Nothing follows if the job ignores or hangs on SIGTERM.\n\n**Expected:** like LSF's TERMINATE action (IBM Spectrum LSF 10.1 Administering\nLSF, \"Kill jobs\" p. 88 and \"TERMINATE action\"): a plain kill sends SIGINT,\nthen SIGTERM after `JOB_TERMINATE_INTERVAL` (default 10 s), then SIGKILL after\nanother interval, each only while the job is still alive; `bkill -s SIG`\nsends just that signal; a forced kill sends SIGKILL at once.\nLikewise a plain stop sends one SIGSTOP (apiserver default since bug-1954),\nwhich the job cannot catch, so it gets no chance to do pre-stop work; LSF's\nSUSPEND action and the user's requirement (2026-10-10) want a catchable\nSIGTSTP first, with SIGSTOP only if the job has not stopped by itself.\n**Impact:** wrong result: a job that ignores SIGTERM survives a plain kill\nindefinitely (no clean-up window followed by a guaranteed end), and a plain\nstop gives the job no chance to prepare.\n**Found by:** user review of the bug-1954 fix, 2026-10-10.",
|
||||
"file": "src/worker/src/xcecd_worker.c, src/frontend/apiserver/backend/convert_pure.go, src/schema/controller.capnp",
|
||||
"root_cause": "The job-control path had no notion of a graceful action: `JobSignal`\ncarried only `signo` and `force`; the apiserver handler pre-filled a single\ndefault signal (kill TERM, stop STOP; `controllers/jobs.go`,\n`models.DefaultActionSignal`), and the job process delivered exactly one\nsignal (`worker_signal_callback()`, `src/worker/src/xcecd_worker.c`). Nothing\nfollowed up a TERM the job ignored, and a stop was never catchable.\nPre-existing design gap (sharpened by bug-1954, which made the delivered\nsignal exact).",
|
||||
"fix": "- Wire: `JobSignal.grace @2` (`src/schema/controller.capnp`) /\n `xce_signal_t.grace` (`include/xce/types.h`): > 0 asks for the graceful\n sequence, seconds between signals; 0 keeps a single signal (all existing\n senders).\n- apiserver: `job.terminate_interval` (`utils/config.go`, `config/app.yaml`,\n default 10 s as LSF's JOB_TERMINATE_INTERVAL); `jobCtrlParams()`\n (`backend/convert_pure.go`) sends a plain kill as SIGKILL and a plain stop\n as SIGSTOP with that grace (a force kill and an explicit signal get none);\n `backend/clib.go` passes it; the handler no longer pre-fills a default\n signal; `DefaultActionSignal` updated.\n- worker: grace travels through the job-control task (`thread_pool.h`), the\n kept-signal table (`pending_signal_record_ex/take_ex`), the IPC message\n and callback (`uv_ipc.*`) and the kept-signal delivery\n (`xcecd_worker_utils.c`); `job_signal_seq_*` (`src/worker/src/job_signal.c`,\n libuv timer) runs kill = SIGINT, SIGTERM, SIGKILL and stop = SIGTSTP,\n SIGSTOP, the next signal only while the job is alive / not yet stopped\n (`job_pid_is_stopped()`, /proc state); `xcecd_worker.c` starts it for a\n graceful request (SIGINT is followed by SIGCONT so a stopped job can act),\n a resume cancels a pending stop, and the job's exit cancels any sequence.\n- `xcecd_virtual` has no process: a graceful kill there still ends the job at\n once (its killed-signal simulation is unchanged).\n- Briefs `job-time-window` and `host-failure-job-recovery` updated to reuse\n this sequence for end-time kills and requeue.\n\nVerification:\n- Test: `src/worker/testsuite/test_job_signal.c`\n `test_kill_escalates_to_sigkill_when_ignored`,\n `test_kill_stops_when_job_exits_on_sigint`,\n `test_stop_is_catchable_then_enforced`,\n `test_stop_not_enforced_when_job_stops_itself`,\n `test_cancel_stops_the_sequence`, `test_other_signals_have_no_sequence`\n (real forked children on a libuv loop): RED against the stub (6 failures,\n e.g. `test_job_signal.c:133: FAIL: Expected 0 Was -1`), GREEN after, 5/5\n repeated meson runs; `backend/jobctrl_params_test.go`\n `TestJobCtrlParams_DefaultSignalPerAction` (kill 9 + grace 10, stop 19 +\n grace 10), `TestJobCtrlParams_ExplicitOrForcedKillIsNotGraceful`,\n `TestJobCtrlParams_GraceFromConfig`: failed to build before (no `Grace`),\n GREEN after.\n- Suites: `xce-build.sh all asan xce.build --incremental`, 2026-10-10: C suites\n Ok 155 / Fail 0, frontend Go unit tests ok; apiserver middleware, router,\n cmd ok.\n- Live: not run: a real `xcecd` kill/stop of a TERM-ignoring and a\n TSTP-catching job needs a host with systemd/dbus (follow-up).",
|
||||
"tags": [
|
||||
"worker",
|
||||
"job-control",
|
||||
"signal",
|
||||
"lsf"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1954",
|
||||
"bug-1940"
|
||||
],
|
||||
"fix_commits": [
|
||||
"367af11a"
|
||||
],
|
||||
"record": ".wolf/bugs/bug-1957.md"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
114
.wolf/bugs/bug-1957.md
Normal file
114
.wolf/bugs/bug-1957.md
Normal file
|
|
@ -0,0 +1,114 @@
|
|||
---
|
||||
id: "bug-1957"
|
||||
status: "fixed"
|
||||
status_updated: "2026-10-10"
|
||||
status_note: ""
|
||||
component: "worker"
|
||||
severity: "wrong-result"
|
||||
timestamp: "2026-10-10T09:35:48Z"
|
||||
last_seen: "2026-10-10T09:35:48Z"
|
||||
occurrences: 1
|
||||
files: ["src/worker/src/xcecd_worker.c", "src/frontend/apiserver/backend/convert_pure.go", "src/schema/controller.capnp"]
|
||||
tags: ["worker", "job-control", "signal", "lsf"]
|
||||
related: ["bug-1954", "bug-1940"]
|
||||
fix_commits: ["367af11a"]
|
||||
---
|
||||
|
||||
# bug-1957: plain job kill and stop are single signals: no SIGKILL escalation, no catchable stop
|
||||
|
||||
## Description
|
||||
**Observed:** (source reading, 2026-10-10) a kill without an explicit signal
|
||||
sends SIGTERM once: the apiserver defaults kill to TERM
|
||||
(`defaultActionSignal`, `src/frontend/apiserver/backend/convert_pure.go`)
|
||||
and the worker delivers it once (`job_signal_to_deliver`,
|
||||
`src/worker/src/job_signal.c`, after bug-1954; the old worker did the same for
|
||||
a non-forced kill). Nothing follows if the job ignores or hangs on SIGTERM.
|
||||
|
||||
**Expected:** like LSF's TERMINATE action (IBM Spectrum LSF 10.1 Administering
|
||||
LSF, "Kill jobs" p. 88 and "TERMINATE action"): a plain kill sends SIGINT,
|
||||
then SIGTERM after `JOB_TERMINATE_INTERVAL` (default 10 s), then SIGKILL after
|
||||
another interval, each only while the job is still alive; `bkill -s SIG`
|
||||
sends just that signal; a forced kill sends SIGKILL at once.
|
||||
Likewise a plain stop sends one SIGSTOP (apiserver default since bug-1954),
|
||||
which the job cannot catch, so it gets no chance to do pre-stop work; LSF's
|
||||
SUSPEND action and the user's requirement (2026-10-10) want a catchable
|
||||
SIGTSTP first, with SIGSTOP only if the job has not stopped by itself.
|
||||
**Impact:** wrong result: a job that ignores SIGTERM survives a plain kill
|
||||
indefinitely (no clean-up window followed by a guaranteed end), and a plain
|
||||
stop gives the job no chance to prepare.
|
||||
**Found by:** user review of the bug-1954 fix, 2026-10-10.
|
||||
## Environment
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: asan (xce.build)
|
||||
- Code: scheduler-framework @ e5a0e42b-dirty
|
||||
- Deployment: any (real `xcecd` worker; `xcecd_virtual` simulates kills)
|
||||
|
||||
## Reproduction
|
||||
1. Run a job that ignores SIGTERM: `trap '' TERM INT; sleep 600`.
|
||||
2. Kill it with no signal option (`xcecli jobs kill <id>`).
|
||||
3. The job keeps running.
|
||||
- Frequency: deterministic (by code reading; confirmed by the RED test of the fix)
|
||||
- Evidence: the code sites in Description; no logs kept.
|
||||
## Root cause
|
||||
The job-control path had no notion of a graceful action: `JobSignal`
|
||||
carried only `signo` and `force`; the apiserver handler pre-filled a single
|
||||
default signal (kill TERM, stop STOP; `controllers/jobs.go`,
|
||||
`models.DefaultActionSignal`), and the job process delivered exactly one
|
||||
signal (`worker_signal_callback()`, `src/worker/src/xcecd_worker.c`). Nothing
|
||||
followed up a TERM the job ignored, and a stop was never catchable.
|
||||
Pre-existing design gap (sharpened by bug-1954, which made the delivered
|
||||
signal exact).
|
||||
|
||||
## Fix
|
||||
- Wire: `JobSignal.grace @2` (`src/schema/controller.capnp`) /
|
||||
`xce_signal_t.grace` (`include/xce/types.h`): > 0 asks for the graceful
|
||||
sequence, seconds between signals; 0 keeps a single signal (all existing
|
||||
senders).
|
||||
- apiserver: `job.terminate_interval` (`utils/config.go`, `config/app.yaml`,
|
||||
default 10 s as LSF's JOB_TERMINATE_INTERVAL); `jobCtrlParams()`
|
||||
(`backend/convert_pure.go`) sends a plain kill as SIGKILL and a plain stop
|
||||
as SIGSTOP with that grace (a force kill and an explicit signal get none);
|
||||
`backend/clib.go` passes it; the handler no longer pre-fills a default
|
||||
signal; `DefaultActionSignal` updated.
|
||||
- worker: grace travels through the job-control task (`thread_pool.h`), the
|
||||
kept-signal table (`pending_signal_record_ex/take_ex`), the IPC message
|
||||
and callback (`uv_ipc.*`) and the kept-signal delivery
|
||||
(`xcecd_worker_utils.c`); `job_signal_seq_*` (`src/worker/src/job_signal.c`,
|
||||
libuv timer) runs kill = SIGINT, SIGTERM, SIGKILL and stop = SIGTSTP,
|
||||
SIGSTOP, the next signal only while the job is alive / not yet stopped
|
||||
(`job_pid_is_stopped()`, /proc state); `xcecd_worker.c` starts it for a
|
||||
graceful request (SIGINT is followed by SIGCONT so a stopped job can act),
|
||||
a resume cancels a pending stop, and the job's exit cancels any sequence.
|
||||
- `xcecd_virtual` has no process: a graceful kill there still ends the job at
|
||||
once (its killed-signal simulation is unchanged).
|
||||
- Briefs `job-time-window` and `host-failure-job-recovery` updated to reuse
|
||||
this sequence for end-time kills and requeue.
|
||||
|
||||
## Verification
|
||||
- Test: `src/worker/testsuite/test_job_signal.c`
|
||||
`test_kill_escalates_to_sigkill_when_ignored`,
|
||||
`test_kill_stops_when_job_exits_on_sigint`,
|
||||
`test_stop_is_catchable_then_enforced`,
|
||||
`test_stop_not_enforced_when_job_stops_itself`,
|
||||
`test_cancel_stops_the_sequence`, `test_other_signals_have_no_sequence`
|
||||
(real forked children on a libuv loop): RED against the stub (6 failures,
|
||||
e.g. `test_job_signal.c:133: FAIL: Expected 0 Was -1`), GREEN after, 5/5
|
||||
repeated meson runs; `backend/jobctrl_params_test.go`
|
||||
`TestJobCtrlParams_DefaultSignalPerAction` (kill 9 + grace 10, stop 19 +
|
||||
grace 10), `TestJobCtrlParams_ExplicitOrForcedKillIsNotGraceful`,
|
||||
`TestJobCtrlParams_GraceFromConfig`: failed to build before (no `Grace`),
|
||||
GREEN after.
|
||||
- Suites: `xce-build.sh all asan xce.build --incremental`, 2026-10-10: C suites
|
||||
Ok 155 / Fail 0, frontend Go unit tests ok; apiserver middleware, router,
|
||||
cmd ok.
|
||||
- Live: not run: a real `xcecd` kill/stop of a TERM-ignoring and a
|
||||
TSTP-catching job needs a host with systemd/dbus (follow-up).
|
||||
|
||||
## References
|
||||
- Fix commit: `367af11a`
|
||||
- Related: bug-1954 (exact signal delivery), bug-1940 (signal reaches the
|
||||
controller)
|
||||
- LSF 10.1 Administering LSF: "Kill jobs" (p. 88), "TERMINATE action"
|
||||
(JOB_TERMINATE_INTERVAL)
|
||||
Loading…
Add table
Add a link
Reference in a new issue