From 77eb88c6c99e02fec512f4ae8b43e7f692e2483a Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Wed, 19 Aug 2026 04:07:05 +0000 Subject: [PATCH] chore(wolf): checkpoint buglog.json (970 bugs) --- .wolf/buglog.json | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 90db73ed..680f6766 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -17457,6 +17457,30 @@ "related_bugs": [], "occurrences": 2, "last_seen": "2026-08-19T03:10:00Z" + }, + { + "id": "bug-1589", + "timestamp": "2026-08-19T03:20:00Z", + "error_message": "User-reported, live 3-node deployment log analysis: a 5-element job array (base_jobid=0-7-1787111880-0, array_size=5) was fully expanded and all 5 elements were dispatched by the controller (job.c's dispatch_a_job_ fires for [0]-[4], confirmed in logs/ctrl-1.log) and elected by the scheduler's limit plugin (logs/scheduler.log's limit_job_election fires for [0]-[4]), but only element [0] ever produced an on_job_start log line (job.c:1673) in either log file -- elements [1]-[4] were dispatched but never reported as started. User's own hypothesis: may be related to the worker running in 'virtual host mode' (xcecd_virtual) rather than a real xcecd worker.", + "file": "src/worker/src/main.c (master_job_status_callback, single global `static xcecd_job_t job`), src/worker/src/process_worker_manager.c (worker_process_update), src/controller/job.c (dispatch_a_job_, on_job_start) -- exact worker-side break point not yet confirmed live", + "root_cause": "NOT YET CONFIRMED -- strong leading hypothesis from static code reading, needs live reproduction to verify. src/worker/src/main.c declares a SINGLE file-scope global `static xcecd_job_t job` (main.c:32) used to track 'the current job' -- there is no per-jobid array/table backing it. `master_job_status_callback()` (main.c:163, the ipc_master status callback fired when a forked job-runner IPC-reports MSG_JOB_STATUS) writes `job.job_status`/`job.job_pid`/`job.exit_code`/`job.exdata` from the incoming `job_status_t*` but CRITICALLY NEVER assigns `job.job_id = job_status->job_id` anywhere in that function -- `job.job_id` is only ever set once, at the static initializer (main.c:32, value 0), so every downstream consumer of this shared `job` struct (`update_worker_status(...)` calls in both `master_job_status_callback` and `process_worker_manager.c`'s `worker_process_update()`) may see a stale or zero job_id regardless of which of several concurrently-dispatched jobs the status update was actually for. Since a 5-element array gets dispatched to the same worker within ~250ms (per the timestamps in the report), if a real (non-virtual) worker forks 5 concurrent job-runner children reporting status via 5 independent IPC messages into ONE shared mutable global, only whichever job's id happened to be the one `job.job_id` was left holding (apparently job [0]'s, if something else sets it once on first dispatch and never again) would ever be correctly attributed and reported onward to the controller as a START -- the other 4's status updates would either be dropped, misattributed to job [0], or never trigger the controller-side on_job_start path at all. HOWEVER this specific `ipc_master`/`master_job_status_callback`/`thread_pool`+`fork_task_t` code path is `#ifndef VIRTUAL_CLUSTER`-gated for `ipc_master_init()` (main.c ~571) -- NOT confirmed whether `xcecd_virtual` (this deployment's actual worker binary, VIRTUAL_CLUSTER build) takes this exact path at all, a materially different path (e.g. `process_worker_manager.c`'s direct fork+waitpid model, which DOES iterate multiple exited processes correctly per-pid in a loop and looks less obviously broken), or some third virtual-mode-specific job-launch mechanism not yet located. Needs live reproduction: submit a real job array to a real (or virtual) worker in this sandbox, trace which functions actually fire (log instrumentation or gdb breakpoints on `master_job_status_callback`/`worker_process_update`/`update_worker_status`/whatever xcecd_virtual's actual dispatch entry point is), and confirm whether `job.job_id` is genuinely stuck/wrong for elements [1]-[4], or whether the real break is somewhere else entirely (e.g. the worker never even attempts to launch [1]-[4]'s processes because of a virtual-mode job-launch limitation, matching the user's own hypothesis).", + "fix": "NOT YET FIXED -- root cause not yet confirmed. If the single-shared-`job`-struct / missing `job.job_id` assignment hypothesis is confirmed, the fix is almost certainly to key `update_worker_status()`'s job tracking by `job_status->job_id` (e.g. look up or construct a per-job_id struct at report time) instead of relying on one process-wide mutable `xcecd_job_t`, mirroring how `process_manager.c`'s `process_info_t manager->processes[]` array already tracks multiple concurrent processes correctly by index/pid. If it turns out xcecd_virtual doesn't use this code path at all, investigation needs to restart from whatever virtual-mode's real job-launch entry point is.", + "tags": [ + "worker", + "xcecd", + "xcecd-virtual", + "job-array", + "concurrency", + "job-id", + "controller", + "open", + "not-yet-root-caused" + ], + "related_bugs": [ + "bug-1586" + ], + "occurrences": 1, + "last_seen": "2026-08-19T03:20:00Z" } ] }