lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(buglog): bug-1908 fixed in 8b7173bb

Browse source 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-02 02:50:49 +00:00
parent 8b7173bb09
commit 167c854673
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
2 changed files with 15 additions and 12 deletions
Download patch file Download diff file Expand all files Collapse all files

12
.wolf/buglog.json
View file

@ -16969,9 +16969,9 @@
},
{
"id": "bug-1908",
"status": "investigating",
"status": "fixed",
"status_updated": "2026-10-02",
"status_note": "worker drops a kill that arrives before the job is in its worker list; controller-side handling of the worker's error (retry or not) not yet traced",
"status_note": "",
"component": "worker",
"severity": "wrong-result",
"timestamp": "2026-10-02T02:27:01Z",
@ -16979,8 +16979,8 @@
"occurrences": 1,
"error_message": "a kill sent right after dispatch is lost: the worker drops it before registering the job\n\n**Observed:**\n```\n05,850 controller_connector.c:563 Job ID: 1-0-1790907905-32768[2], Command: sleep 600 ... (dispatch received)\n07,318 controller_connector.c:1546 Received job control request to send SIGINFO for job ID: 1-0-1790907905-32768[2]\n07,318 controller_connector.c:1586 Failed to find worker for job ID: 1-0-1790907905-32768[2]\n09,690 xcecd_worker_utils.c:352 Job ID 1-0-1790907905-32768[2] running - sending status to controller\n```\n`xcecli jobs list` already showed the job RUN, `xcecli jobs kill` answered `job <...> is being <kill>` (apiserver `overallStatus accepted`), but the worker had not yet added the job to its worker list, dropped the request, and the job then ran to completion. Same for a plain job (1-0-1790907905-0: dispatch 06,876, kill 07,313 dropped, running 09,724).\n\n**Expected:** a kill accepted for a job the controller shows as RUN takes effect: the worker applies it once the job is registered (or the controller retries / reports the kill as not delivered).\n**Impact:** wrong result: a kill issued in the first seconds after dispatch is reported as accepted but silently lost; the job keeps running. The lookup branch is shared by the real worker (`xcecd`), not only `xcecd_virtual` (real worker not tested).\n**Found by:** test (live verification of bug-1905, 2026-10-01)",
"file": "src/worker/src/controller_connector.c",
"root_cause": "TODO: required for fixed/partial",
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
"root_cause": "The controller forwards a job-control signal as soon as the job has an allocation (`src/controller/job.c` job-control handler -> `xce_jobctrl_req`). The worker answers the start request by queuing it on its thread pool (`controller_connector.c` start handler -> `submit_fork_task`) and only adds the job to its worker list when `xcecd_process_command` runs (`xcecd.c`), which for the first jobs after a worker starts was ~3 s later; the real worker can also only signal a job once the job's IPC client has connected. `handle_jobcontrol` (`controller_connector.c`) dropped a signal for a job not in the list (\"Failed to find worker\", -1) and also for a listed job still PENDING/STARTING, so the kill was lost while the API had already reported it accepted.",
"fix": "New `src/worker/src/pending_signal.c` (`include/pending_signal.h`): a thread-safe table of jobs whose start the worker has accepted. The start handler calls `pending_signal_start()` before queuing the start; `handle_jobcontrol` keeps a signal for a queued or PENDING/STARTING job with `pending_signal_record()` and replies success; `update_worker_status()` takes it on the STARTING -> RUNNING transition and delivers it (virtual: `worker->killed_signal`; real: `ipc_master_send_job_signal`, whose client is connected once the job reports RUNNING). `xcecd_job_start_error()` and the DONE/EXITED transitions drop the entry. A signal for a job the worker knows nothing about still fails as before.\n\nVerification:\n- Test: `test_jobcontrol_signal_for_queued_start_is_kept` (RED before the fix: `Expected 0 Was -1`) and `test_jobcontrol_signal_for_unknown_job_fails` in `src/worker/testsuite/test_controller_connector.c`; table semantics in the new `src/worker/testsuite/test_pending_signal.c` (6 tests); GREEN after\n- Suites: worker meson suite 15/15 OK (asan, src/build.jai)\n- Live: fresh xce-deploy-ha + xcecli, kills sent as soon as the jobs listed RUN: the worker logged `has not started yet; signal 15 will be delivered when it runs` for `plain` and `arr[2]` 2.3 s before they started, then `delivering signal 15 kept while it was starting`; both ended EXIT while `arr[1]`/`arr[3]` kept running; a later kill of the running `arr[1]` still worked (/home/claude/tmp/bug-1908/fixed-run.log). The real worker's IPC delivery path is not exercised here (no systemd/dbus in this container).",
"tags": [
"job-control",
"kill",
@ -16990,7 +16990,9 @@
"related_bugs": [
"bug-1905"
],
"fix_commits": [],
"fix_commits": [
"8b7173bb"
],
"record": ".wolf/bugs/bug-1908.md"
}
]

15
.wolf/bugs/bug-1908.md
View file

@ -1,8 +1,8 @@
---
id: "bug-1908"
status: "investigating"
status: "fixed"
status_updated: "2026-10-02"
status_note: "worker drops a kill that arrives before the job is in its worker list; controller-side handling of the worker's error (retry or not) not yet traced"
status_note: ""
component: "worker"
severity: "wrong-result"
timestamp: "2026-10-02T02:27:01Z"
@ -11,6 +11,7 @@ occurrences: 1
files: ["src/worker/src/controller_connector.c"]
tags: ["job-control", "kill", "race", "dispatch"]
related: ["bug-1905"]
fix_commits: ["8b7173bb"]
---
# bug-1908: a kill sent right after dispatch is lost: the worker drops it before registering the job
@ -45,15 +46,15 @@ related: ["bug-1905"]
- Evidence: /home/claude/tmp/kill-race/worker.log
## Root cause
TODO: required for fixed/partial
The controller forwards a job-control signal as soon as the job has an allocation (`src/controller/job.c` job-control handler -> `xce_jobctrl_req`). The worker answers the start request by queuing it on its thread pool (`controller_connector.c` start handler -> `submit_fork_task`) and only adds the job to its worker list when `xcecd_process_command` runs (`xcecd.c`), which for the first jobs after a worker starts was ~3 s later; the real worker can also only signal a job once the job's IPC client has connected. `handle_jobcontrol` (`controller_connector.c`) dropped a signal for a job not in the list ("Failed to find worker", -1) and also for a listed job still PENDING/STARTING, so the kill was lost while the API had already reported it accepted.
## Fix
TODO: required for fixed/partial
New `src/worker/src/pending_signal.c` (`include/pending_signal.h`): a thread-safe table of jobs whose start the worker has accepted. The start handler calls `pending_signal_start()` before queuing the start; `handle_jobcontrol` keeps a signal for a queued or PENDING/STARTING job with `pending_signal_record()` and replies success; `update_worker_status()` takes it on the STARTING -> RUNNING transition and delivers it (virtual: `worker->killed_signal`; real: `ipc_master_send_job_signal`, whose client is connected once the job reports RUNNING). `xcecd_job_start_error()` and the DONE/EXITED transitions drop the entry. A signal for a job the worker knows nothing about still fails as before.
## Verification
- Test: TODO: name + file; RED before the fix, GREEN after
- Suites: TODO: suite results and build profile
- Live: TODO: soak/deploy run, or n/a and why
- Test: `test_jobcontrol_signal_for_queued_start_is_kept` (RED before the fix: `Expected 0 Was -1`) and `test_jobcontrol_signal_for_unknown_job_fails` in `src/worker/testsuite/test_controller_connector.c`; table semantics in the new `src/worker/testsuite/test_pending_signal.c` (6 tests); GREEN after
- Suites: worker meson suite 15/15 OK (asan, src/build.jai)
- Live: fresh xce-deploy-ha + xcecli, kills sent as soon as the jobs listed RUN: the worker logged `has not started yet; signal 15 will be delivered when it runs` for `plain` and `arr[2]` 2.3 s before they started, then `delivering signal 15 kept while it was starting`; both ended EXIT while `arr[1]`/`arr[3]` kept running; a later kill of the running `arr[1]` still worked (/home/claude/tmp/bug-1908/fixed-run.log). The real worker's IPC delivery path is not exercised here (no systemd/dbus in this container).
## References
- Related: none