---
id: "bug-1589"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "worker"
severity: "unknown"
timestamp: "2026-08-19T03:20:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 2
files: ["src/worker/src/xcecd_dbus_manager.c (dbus_workers_update() -- THE fix)", "src/worker/src/main.c (confirms master_job_status_callback/ipc_master path is UNREACHABLE for xcecd_virtual -- and for real xcecd too)", "src/worker/testsuite/test_dbus_workers_update.c (new regression test)", "src/controller/job.c (on_job_start/dispatch_a_job_ -- read/traced", "confirmed innocent", "not touched)"]
tags: ["worker", "xcecd", "xcecd-virtual", "job-array", "concurrency", "job-id", "controller", "root-caused", "dbus-workers-update", "linked-list-traversal", "head-of-line-blocking"]
related: ["bug-1586"]
legacy: true
---

# bug-1589: User-reported, live 3-node deployment log analysis: a 5-element job array…

## Description
User-reported, live 3-node deployment log analysis: a 5-element job array (base_jobid=0-7-1787111880-0, array_size=5) was fully expanded and all 5 elements were dispatched by the controller (job.c's dispatch_a_job_ fires for [0]-[4], confirmed in logs/ctrl-1.log) and elected by the scheduler's limit plugin (logs/scheduler.log's limit_job_election fires for [0]-[4]), but only element [0] ever produced an on_job_start log line (job.c:1673) in either log file -- elements [1]-[4] were dispatched but never reported as started. User's own hypothesis: may be related to the worker running in 'virtual host mode' (xcecd_virtual) rather than a real xcecd worker.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
CONFIRMED via source trace + live reproduction (both hypotheses from the prior static-reading pass were REFUTED). (1) Code path: worker/meson.build shows xcecd_virtual == xcecd_srcs + main.c, built with -DVIRTUAL_CLUSTER -DDBUS_MANAGER -- and the REAL xcecd target uses the SAME xcecd_srcs + main.c with -DDBUS_MANAGER only. Because DBUS_MANAGER is defined for BOTH binaries, main.c's run loop (`#ifndef DBUS_MANAGER worker_process_update(...) #else dbus_workers_update(...) #endif`, main.c:603-616) ALWAYS takes the dbus_workers_update() branch for both -- process_worker_manager.c's worker_process_update() (the 'looks less obviously broken' alternative flagged in the original hypothesis) is dead code, never called, in either build. Separately, ipc_master_init() is `#ifndef VIRTUAL_CLUSTER`-gated (main.c:571-579) so for xcecd_virtual no IPC-master unix-socket listener is ever created; master_job_status_callback() (main.c:163) and the single global `static xcecd_job_t job` (main.c:32) are therefore UNREACHABLE for xcecd_virtual (no forked job-runner ever connects to report MSG_JOB_STATUS over that socket) -- the original hypothesis about a missing `job.job_id = job_status->job_id` assignment there was investigating dead code for this deployment shape; it may still be a latent issue for a REAL xcecd with genuinely concurrent forked job-runners, but is NOT bug-1589's cause and was not touched. (2) Real root cause: dbus_workers_update() (xcecd_dbus_manager.c:178, called from main.c's run loop every ~10ms) iterates xcecd_ctx.worker_list -- one xcecd_worker_t node per concurrently-dispatched job -- with `xcecd_worker_t* worker = *worker_list; while (worker) { ... worker = worker->next; }`. Several branches inside the loop body used a bare `continue` that re-tests the SAME `worker` pointer instead of advancing (the VIRTUAL_CLUSTER JOB_STARTING->JOB_RUNNING branch, xcecd_dbus_manager.c:201 pre-fix; the exit_code==0 and exit_code<0 VIRTUAL_CLUSTER completion-check branches, pre-fix lines 241/268; the real-mode 'job done but running not received' branch, pre-fix line 291) -- and worse, the WORKER_DONE/WORKER_EXITED early check (xcecd_dbus_manager.c:205-209 pre-fix) did `worker = worker->next; return true;`, which ABORTED THE ENTIRE FUNCTION the moment ANY worker in the list was already finished, discarding the scan of every worker queued behind it (also a latent bug: `return true` == `return 1` == XCECD_ERROR on a function whose only real caller only special-cases XCECD_DB_ERROR, so this wasn't even surfaced as an error). Net effect on a job array: once dbus_workers_update() started making progress on the HEAD of xcecd_ctx.worker_list (element [0], first added via add_worker()), every OTHER concurrently-dispatched element ([1]-[4]) was starved of update_worker_status() -- the function that fires xcecd_reply_startjob() i.e. the on_job_start report -- for as long as element [0] stayed un-reaped from the list (which for a long-running job is indefinite, matching the user's 'never, in either log file' observation for a job array of `sleep N`-style long jobs; for short/instant jobs the same bug still starves [1]-[4] but only for the ~10ms tick until cleanup_worker_list() reaps the finished head and the next element becomes head in turn). This is a WORKER-side bug only -- the controller's on_job_start()/record_job_start_fact() (job.c:1665-1663) and its job_state_is_backward() guard (job.c:43) were traced and confirmed to correctly key everything by jobid with no cross-job interference; they simply never received a report for [1]-[4] because the worker never sent one.

## Fix
FIXED. xcecd_dbus_manager.c's dbus_workers_update(): capture `xcecd_worker_t* next = worker->next;` once at the top of every loop iteration (right after `while (worker) {`), and changed every early-exit path in the loop body to `worker = next; continue;` instead of a bare `continue` (re-tests the same node forever) or the old `worker->worker_status == WORKER_DONE/WORKER_EXITED` branch's `return true` (aborted the whole scan on the first finished worker). This fixes traversal for BOTH VIRTUAL_CLUSTER and real (non-VIRTUAL_CLUSTER) branches of the function -- the real-mode `kill(pid,0)!=0` / 'job done but running not received' branch had the identical bare-continue bug and would starve a real xcecd's worker_list the same way under concurrent dispatch. No change to the state-machine logic itself (update_worker_status(), check_worker_sleep_time_elapsed(), the systemd dbus_is_worker_active()/get_systemd_job_exit_code() calls) -- purely a traversal/control-flow fix, matching the 'minimal, correct, no symptom-papering' guidance; considered and rejected converting worker_list to a per-jobid array (process_manager.c's processes[] precedent) since the linked list itself was never the problem, only its traversal. TDD: new src/worker/testsuite/test_dbus_workers_update.c (2 tests, white-box #includes xcecd_dbus_manager.c with -DVIRTUAL_CLUSTER -DDBUS_MANAGER matching xcecd_virtual's real build flags, stubs update_worker_status()/is_systemd_job_active()/get_systemd_job_exit_code()/stop_systemd_job()/make_job_sd_unit_name()/start_systemd_job()/get_systemd_job_pid()/check_worker_sleep_time_elapsed() to isolate the loop's own control flow) wired into src/worker/testsuite/meson.build. RED-verified live: `git stash push -- src/worker/src/xcecd_dbus_manager.c`, rebuild, both tests FAIL ('Expected 0 Was 1' -- the started-call-seen assertions correctly detect elements [1]-[4] never receiving their start transition, and a DONE head worker correctly blocks the scan); `git stash pop`, rebuild, both tests PASS (GREEN). Live end-to-end proof (not just the unit test): deployed a fresh 3-controller + scheduler + xcecd_virtual cluster (/xce-deploy-ha pattern, dedicated dir /tmp/xce-deploy-bug1589), submitted `jobsub -u rshen -q normal -c "sleep 300" -a 5` (5-element array, rshen used specifically to avoid limits.yaml's 'limit1' 3-slot cap on all-users-except-rshen, matching the user's original unconstrained-concurrency scenario) -- ctrl-1.log showed `dispatch_a_job_` AND `on_job_start` for ALL FIVE elements [0]-[4] (previously only [0] would ever show on_job_start), scheduler.log showed `limit_job_election` for all 5 (as it already did pre-fix), a follow-up `joblist` showed all 5 elements in state RUN, and xcecd-worker-1.log showed the `Job ID <id> running - sending status to controller` line (the exact xcecd_reply_startjob trigger) for 5 DISTINCT job ids within milliseconds of each other. Cluster torn down after capturing evidence (self-started, self-stopped, per boundaries). Note: this repro session also observed pre-existing, unrelated 'Failed to connect to system bus' ERROR spam from dbus_is_worker_active()/get_systemd_job_exit_code() polling every ~10ms per still-running virtual job (expected -- no real dbus session in this sandbox, and VIRTUAL_CLUSTER never creates a real systemd unit to poll in the first place) -- cosmetic log noise, not a correctness bug, pre-existing before this fix and unrelated to bug-1589's actual defect; flagged here for visibility, NOT fixed (out of this bug's scope). Also flagged, NOT fixed (out of scope, no buglog entry created per explicit task boundary): dbus_worker_clean() (xcecd_dbus_manager.c) has an apparently-inverted `#ifndef VIRTUAL_CLUSTER return true;` guard -- as written, the REAL (non-virtual) xcecd never actually runs its own systemd-unit cleanup body, while VIRTUAL_CLUSTER mode does (but finds nothing to clean, since no real unit ever existed) -- looks backwards from the evident intent, worth a separate look. Full suite after fix: worker 4/4 GREEN (resource_sync, xcecd_startup_order, dbus_workers_update, controller_connector). Controller 1101/1104 (3 unrelated FAILs in test_config_repl.c -- config version trimming/tag lookup/marker supersession, a file this fix never touches and that was confirmed clean/unmodified by this session; controller/ was not touched at all by this fix). Scheduler: bug-1560's known symlink-escape failure (expected, unrelated) plus 3 NEW failures in test_sim_verify.c/test_repl_breakpoints_inspection.c -- traced via `git status` to src/scheduler/src/driver.c and src/scheduler/src/recording_session.c being concurrently modified, uncommitted, by ANOTHER session's in-flight work in this shared worktree (unrelated 'bug-1591' job.c/driver.c changes observed mid-flight during this session's builds) -- outside this bug's boundary (scheduler/core files are explicitly off-limits for this task) and not caused by this worker-only fix; not investigated further.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1586