myxce/.wolf/bugs/bug-1955.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	gitea_issue	gitea_synced_status	gitea_synced_hash
bug-1955	fixed	2026-10-10		worker	wrong-result	2026-10-10T08:46:10Z	2026-10-10T08:46:10Z	1	src/worker/src/controller_connector.c	worker job-control parallel signal	bug-1942 bug-1908	d3da3140 81e7187d	386	fixed	b2623497eb54

bug-1955: xcecd drops a signal for a remote task of a multi-host job that is still starting

Description

Observed: (source reading, 2026-10-10; found while fixing bug-1942) Only the head host's job start opens the pending-signal window (pending_signal_start(d->jobid), src/worker/src/controller_connector.c:1062, bug-1908). The runtasks handler for remote tasks (handle_runtasks(), :1147) never does, so a signal that arrives before the remote task's worker entry is registered is logged and dropped:

XCECD_LOG(XCE_LOG_ERROR, "Failed to find worker for job ID: %s", jobidStr);
FREEUP(task_ctx);
return -1;                               /* controller_connector.c:1606 */

Since bug-1942 every host of a multi-host job receives job control, so this window is now reachable.

Expected: like the head host (bug-1908), a signal for a remote task that is starting is kept and delivered once the task runs. Impact: wrong result: a kill or stop sent right after a multi-host job starts can miss some of its remote tasks, which keep running. Found by: review of the bug-1940/1942 fixes, 2026-10-10.

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (xce.build)
• Code: scheduler-framework @ 8568c8ab-dirty
• Deployment: real xcecd worker (systemd/dbus); multi-host for the remote-task case

Reproduction

1. Submit a multi-host job (e.g. /xce-test-multihost topology) and kill it immediately after dispatch.
2. A remote host logs "Failed to find worker for job ID" and its task keeps running.

• Frequency: deterministic by code reading (not yet run)
• Evidence: the code quoted in Description; no logs kept.

Root cause

Only the head-host start path opened the pending-signal window (pending_signal_start, controller_connector.c, bug-1908); handle_runtasks() queued remote tasks without it, so a signal arriving before the task's worker entry registered hit "Failed to find worker" and was dropped.

UPDATE (2026-10-10): the remaining window: job control reaches a remote host from the controller (job_ctrl(), src/controller/job.c, bug-1942) as soon as the job is dispatched, while that host's runtasks only arrives after the head xcecd starts and asks the controller (job_start_remote_task() → do_runtasks(), task.c, which keeps no per-job state). A control arriving before runtasks found neither a worker entry nor a pending start and was dropped.

Fix

handle_runtasks() calls pending_signal_start(jobid) before submit_fork_task() and takes it back if queuing fails; the existing RUNNING transition (xcecd_worker_utils.c, bug-1908) delivers the kept signal to the task.

UPDATE (2026-10-10, 81e7187d): worker-side "early" pending signals. pending_signal.c: pending_signal_record_early() keeps a signal for a job with no start on this host (or records it on a start already pending); pending_signal_start_task() (used by handle_runtasks()) adopts an unexpired early signal, which the existing RUNNING transition (bug-1908) delivers, so the task exits through the normal finish report (no hung job). Early entries are capped at 256 and expire after 60 s, swept on use under the table mutex (nothing in delaymgr). A head-host start (pending_signal_start) discards rather than inherits an early entry. handle_jobcontrol() keeps the control early unless the job already finished on this host (xcecd_finished_job_lookup). Accepted limits: a control arriving before a rerun's runtasks on the same host within the 60 s finished-job retention is still treated as late; a control for a host the job never starts on lingers up to 60 s.

Verification

• Test: test_jobcontrol_signal_for_queued_remote_task_is_kept (src/worker/testsuite/test_controller_connector.c): RED :1631: FAIL: Expected 0 Was -1. the kill must not be dropped, GREEN after. Second fix: test_jobcontrol_signal_before_remote_runtasks_is_applied RED test_controller_connector.c:1695:...:FAIL: Expected 0 Was -1. a kill arriving before runtasks must not be dropped, GREEN after; also test_jobcontrol_signal_for_finished_task_is_not_kept, test_jobcontrol_signal_for_unknown_job_is_kept_early (replaces ..._unknown_job_fails, which asserted the drop), and test_early_signal_is_adopted_by_task_start, test_early_signal_on_pending_start_is_recorded, test_stale_early_signal_expires, test_early_signals_are_bounded, test_head_start_does_not_inherit_early_signal (test_pending_signal.c; RED was a compile error, the functions did not exist).
• Suites: xce-build.sh all asan xce.build --incremental on the merged branch (scheduler-framework @ edad8c73), 2026-10-10: all C suites Ok 158 / Fail 0, frontend Go unit tests (apiserver + xcecli) ok; apiserver CGO_ENABLED=0 go test ./... (all packages) ok
• Live: n/a: needs a real multi-host xcecd (systemd/dbus), not available in this container. Follow-up: /xce-test-multihost topology, kill (and stop) a plain multi-host job right after dispatch; the remote host should log "signal 9 is kept" then "a signal arrived before its tasks" and the job reach a final state.

References

• Fix commits: d3da3140, 81e7187d (worktree original a66227a5)
• Review: .kiro/reviews/project-gap-review-2026-10-10.md