chore(wolf): checkpoint buglog.json (970 bugs)

.wolf/buglog.json

@@ -17457,6 +17457,30 @@
       "related_bugs": [],
       "occurrences": 2,
       "last_seen": "2026-08-19T03:10:00Z"
+    },
+    {
+      "id": "bug-1589",
+      "timestamp": "2026-08-19T03:20:00Z",
+      "error_message": "User-reported, live 3-node deployment log analysis: a 5-element job array (base_jobid=0-7-1787111880-0, array_size=5) was fully expanded and all 5 elements were dispatched by the controller (job.c's dispatch_a_job_ fires for [0]-[4], confirmed in logs/ctrl-1.log) and elected by the scheduler's limit plugin (logs/scheduler.log's limit_job_election fires for [0]-[4]), but only element [0] ever produced an on_job_start log line (job.c:1673) in either log file -- elements [1]-[4] were dispatched but never reported as started. User's own hypothesis: may be related to the worker running in 'virtual host mode' (xcecd_virtual) rather than a real xcecd worker.",
+      "file": "src/worker/src/main.c (master_job_status_callback, single global `static xcecd_job_t job`), src/worker/src/process_worker_manager.c (worker_process_update), src/controller/job.c (dispatch_a_job_, on_job_start) -- exact worker-side break point not yet confirmed live",
+      "root_cause": "NOT YET CONFIRMED -- strong leading hypothesis from static code reading, needs live reproduction to verify. src/worker/src/main.c declares a SINGLE file-scope global `static xcecd_job_t job` (main.c:32) used to track 'the current job' -- there is no per-jobid array/table backing it. `master_job_status_callback()` (main.c:163, the ipc_master status callback fired when a forked job-runner IPC-reports MSG_JOB_STATUS) writes `job.job_status`/`job.job_pid`/`job.exit_code`/`job.exdata` from the incoming `job_status_t*` but CRITICALLY NEVER assigns `job.job_id = job_status->job_id` anywhere in that function -- `job.job_id` is only ever set once, at the static initializer (main.c:32, value 0), so every downstream consumer of this shared `job` struct (`update_worker_status(...)` calls in both `master_job_status_callback` and `process_worker_manager.c`'s `worker_process_update()`) may see a stale or zero job_id regardless of which of several concurrently-dispatched jobs the status update was actually for. Since a 5-element array gets dispatched to the same worker within ~250ms (per the timestamps in the report), if a real (non-virtual) worker forks 5 concurrent job-runner children reporting status via 5 independent IPC messages into ONE shared mutable global, only whichever job's id happened to be the one `job.job_id` was left holding (apparently job [0]'s, if something else sets it once on first dispatch and never again) would ever be correctly attributed and reported onward to the controller as a START -- the other 4's status updates would either be dropped, misattributed to job [0], or never trigger the controller-side on_job_start path at all. HOWEVER this specific `ipc_master`/`master_job_status_callback`/`thread_pool`+`fork_task_t` code path is `#ifndef VIRTUAL_CLUSTER`-gated for `ipc_master_init()` (main.c ~571) -- NOT confirmed whether `xcecd_virtual` (this deployment's actual worker binary, VIRTUAL_CLUSTER build) takes this exact path at all, a materially different path (e.g. `process_worker_manager.c`'s direct fork+waitpid model, which DOES iterate multiple exited processes correctly per-pid in a loop and looks less obviously broken), or some third virtual-mode-specific job-launch mechanism not yet located. Needs live reproduction: submit a real job array to a real (or virtual) worker in this sandbox, trace which functions actually fire (log instrumentation or gdb breakpoints on `master_job_status_callback`/`worker_process_update`/`update_worker_status`/whatever xcecd_virtual's actual dispatch entry point is), and confirm whether `job.job_id` is genuinely stuck/wrong for elements [1]-[4], or whether the real break is somewhere else entirely (e.g. the worker never even attempts to launch [1]-[4]'s processes because of a virtual-mode job-launch limitation, matching the user's own hypothesis).",
+      "fix": "NOT YET FIXED -- root cause not yet confirmed. If the single-shared-`job`-struct / missing `job.job_id` assignment hypothesis is confirmed, the fix is almost certainly to key `update_worker_status()`'s job tracking by `job_status->job_id` (e.g. look up or construct a per-job_id struct at report time) instead of relying on one process-wide mutable `xcecd_job_t`, mirroring how `process_manager.c`'s `process_info_t manager->processes[]` array already tracks multiple concurrent processes correctly by index/pid. If it turns out xcecd_virtual doesn't use this code path at all, investigation needs to restart from whatever virtual-mode's real job-launch entry point is.",
+      "tags": [
+        "worker",
+        "xcecd",
+        "xcecd-virtual",
+        "job-array",
+        "concurrency",
+        "job-id",
+        "controller",
+        "open",
+        "not-yet-root-caused"
+      ],
+      "related_bugs": [
+        "bug-1586"
+      ],
+      "occurrences": 1,
+      "last_seen": "2026-08-19T03:20:00Z"
     }
   ]
 }