docs(buglog): bug-1967, bug-1955, bug-1936 fixed; log bug-1968 (restart drops options)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
edad8c7385
commit
05465a20bf
5 changed files with 148 additions and 34 deletions
|
|
@ -1,8 +1,8 @@
|
|||
---
|
||||
id: "bug-1955"
|
||||
status: "partial"
|
||||
status: "fixed"
|
||||
status_updated: "2026-10-10"
|
||||
status_note: "covers a signal arriving after this host accepted the runtasks request; a control that reaches a remote host before its runtasks request arrives is still dropped there (needs a controller-side or pre-start record)"
|
||||
status_note: ""
|
||||
component: "worker"
|
||||
severity: "wrong-result"
|
||||
timestamp: "2026-10-10T08:46:10Z"
|
||||
|
|
@ -11,7 +11,7 @@ occurrences: 1
|
|||
files: ["src/worker/src/controller_connector.c"]
|
||||
tags: ["worker", "job-control", "parallel", "signal"]
|
||||
related: ["bug-1942", "bug-1908"]
|
||||
fix_commits: ["d3da3140"]
|
||||
fix_commits: ["d3da3140", "81e7187d"]
|
||||
gitea_issue: 386
|
||||
gitea_synced_status: "partial"
|
||||
gitea_synced_hash: "095558c6c67a"
|
||||
|
|
@ -57,14 +57,19 @@ starts can miss some of its remote tasks, which keep running.
|
|||
## Root cause
|
||||
Only the head-host start path opened the pending-signal window (`pending_signal_start`, `controller_connector.c`, bug-1908); `handle_runtasks()` queued remote tasks without it, so a signal arriving before the task's worker entry registered hit "Failed to find worker" and was dropped.
|
||||
|
||||
UPDATE (2026-10-10): the remaining window: job control reaches a remote host from the controller (`job_ctrl()`, `src/controller/job.c`, bug-1942) as soon as the job is dispatched, while that host's runtasks only arrives after the head xcecd starts and asks the controller (`job_start_remote_task()` → `do_runtasks()`, `task.c`, which keeps no per-job state). A control arriving before runtasks found neither a worker entry nor a pending start and was dropped.
|
||||
|
||||
## Fix
|
||||
`handle_runtasks()` calls `pending_signal_start(jobid)` before `submit_fork_task()` and takes it back if queuing fails; the existing RUNNING transition (`xcecd_worker_utils.c`, bug-1908) delivers the kept signal to the task.
|
||||
|
||||
UPDATE (2026-10-10, `81e7187d`): worker-side "early" pending signals. `pending_signal.c`: `pending_signal_record_early()` keeps a signal for a job with no start on this host (or records it on a start already pending); `pending_signal_start_task()` (used by `handle_runtasks()`) adopts an unexpired early signal, which the existing RUNNING transition (bug-1908) delivers, so the task exits through the normal finish report (no hung job). Early entries are capped at 256 and expire after 60 s, swept on use under the table mutex (nothing in delaymgr). A head-host start (`pending_signal_start`) discards rather than inherits an early entry. `handle_jobcontrol()` keeps the control early unless the job already finished on this host (`xcecd_finished_job_lookup`). Accepted limits: a control arriving before a rerun's runtasks on the same host within the 60 s finished-job retention is still treated as late; a control for a host the job never starts on lingers up to 60 s.
|
||||
|
||||
## Verification
|
||||
- Test: `test_jobcontrol_signal_for_queued_remote_task_is_kept` (`src/worker/testsuite/test_controller_connector.c`): RED `:1631: FAIL: Expected 0 Was -1. the kill must not be dropped`, GREEN after (32 tests, 0 failures).
|
||||
- Suites: `xce-build.sh all asan xce.build --incremental` on the merged branch, 2026-10-10: all C suites Ok 155 / Fail 0, frontend Go unit tests ok, xceadmin tests ok; apiserver `CGO_ENABLED=0 go test ./...` (incl. middleware, router, cmd) ok
|
||||
- Live: n/a: unit/fixture-level verification only; no live deployment was run (follow-up)
|
||||
- Test: `test_jobcontrol_signal_for_queued_remote_task_is_kept` (`src/worker/testsuite/test_controller_connector.c`): RED `:1631: FAIL: Expected 0 Was -1. the kill must not be dropped`, GREEN after. Second fix: `test_jobcontrol_signal_before_remote_runtasks_is_applied` RED `test_controller_connector.c:1695:...:FAIL: Expected 0 Was -1. a kill arriving before runtasks must not be dropped`, GREEN after; also `test_jobcontrol_signal_for_finished_task_is_not_kept`, `test_jobcontrol_signal_for_unknown_job_is_kept_early` (replaces `..._unknown_job_fails`, which asserted the drop), and `test_early_signal_is_adopted_by_task_start`, `test_early_signal_on_pending_start_is_recorded`, `test_stale_early_signal_expires`, `test_early_signals_are_bounded`, `test_head_start_does_not_inherit_early_signal` (`test_pending_signal.c`; RED was a compile error, the functions did not exist).
|
||||
- Suites: `xce-build.sh all asan xce.build --incremental` on the merged branch (scheduler-framework @ edad8c73), 2026-10-10: all C suites Ok 158 / Fail 0, frontend Go unit tests (apiserver + xcecli) ok; apiserver `CGO_ENABLED=0 go test ./...` (all packages) ok
|
||||
- Live: n/a: needs a real multi-host `xcecd` (systemd/dbus), not available in this container. Follow-up: `/xce-test-multihost` topology, kill (and stop) a plain multi-host job right after dispatch; the remote host should log "signal 9 is kept" then "a signal arrived before its tasks" and the job reach a final state.
|
||||
|
||||
## References
|
||||
- Fix commit: `d3da3140` (on `scheduler-framework`; worker-branch original in the merge)
|
||||
- Fix commits: `d3da3140`, `81e7187d` (worktree original `a66227a5`)
|
||||
- Review: `.kiro/reviews/project-gap-review-2026-10-10.md`
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue