chore(wolf): checkpoint buglog.json (1029 bugs)

.wolf/buglog.json

@@ -18788,6 +18788,32 @@
       ],
       "occurrences": 1,
       "last_seen": "2026-08-25T17:30:00Z"
+    },
+    {
+      "id": "bug-1688",
+      "timestamp": "2026-08-25T17:40:00Z",
+      "error_message": "User-reported issue in the scheduler's interactive simulator REPL (`xce_scheduler -i`/`--sim`): `info worker all` and `info worker <name>` (e.g. `info worker pc13v`) initially print full per-resource detail (`[sim]   resource slots: value=22.00 used=0.00`, `resource mem: value=...`, etc -- ~19 resource lines including slots/mem/r1m/r15m/r15s/ut/pg/ls/it/swp/io/tmp/type/model/cpuf/maxcpus/maxmem/maxswap/maxtmp/ndisks). After stepping the simulation forward through several sessions (`step 5` repeated, crossing RESVALS_DELTA, JOB_PUSH (2 batches, 40+80 jobs), JOB_UPDATE, and WORKERS_PUSH events across sessions 2 through 30), the SAME commands later in the SAME REPL session regress to printing ONLY the header line: `[sim] worker pc13v: status=online` -- zero resource lines, even though the worker still exists and is still 'online' (not removed/dead). Binary under test: `./build.dbg/controller/xce_controller --version` reports `xce v0.0.1-bc3f76b4-5ff97622 built 2026-08-25T08:56:58Z` (a build.dbg build -- note the user ran the CONTROLLER binary's --version, though the symptom itself is in the SCHEDULER's sim REPL; worth confirming which binary actually serves `xce_scheduler -i`/`--sim` and its own version once investigated). A real replay file reproducing the sequence is preserved at /tmp/scheduler-replay-20260825-090251.replay (~1MB, captured 2026-08-25 09:24 UTC) for a future investigation to replay directly instead of needing to reproduce from scratch.",
+      "file": "src/scheduler/src/repl_info.c (repl_info_worker_one(), ~line 309-388, the `info worker` print path), src/scheduler/src/core_events.c (core_apply_worker_resvals(), ~line 478, the WORKERS_PUSH resval-merge path), src/lib/zarray.c (zarray_put_ex(), ~line 267-303, count-vs-slot bookkeeping)",
+      "root_cause": "PARTIALLY ROOT-CAUSED (2026-08-25), via live reproduction against the user-supplied /tmp/scheduler-replay-20260825-090251.replay (`build.dbg/scheduler/xce_scheduler --sim <file> -i`, driving the exact `step 5` sequence from the report) plus -vvv DEBUG3 TRACE log inspection -- TWO independent bugs found, neither fully pinned to its exact trigger line yet:\n(1) A count-doubling anomaly in core_apply_worker_resvals() (core_events.c): the first WORKERS_PUSH for worker 'pc13v' carries 20 resvals and correctly leaves worker->resvals.count=20. A SECOND WORKERS_PUSH for the SAME worker, carrying the SAME 20 resource names again, makes worker->resvals.count jump to 40 instead of staying at 20 (confirmed via TRACE: 'applied 20 resval(s)' then 'applied 40 resval(s)'). zarray_put_ex() (zarray.c:267-303) only increments array->count when the target slot's existing pointer is NULL ('if (ptr != NULL) { killer(ptr); } else { array->count++; }') -- a doubling means all 20 second-push writes landed on slots that read NULL at write time, implying something zeroed those 20 worker->resvals.data[] slots between the two pushes WITHOUT going through zarray's own removal path (which would have decremented count back down first). The exact clearing site was not isolated within the investigation's time budget -- core_reset_workers() (core_session.c:84-108) was checked and ruled out (only touches w->resInUse, not w->resvals); no resmgr rebuild/re-registration appears in the log either. Next step: add a one-line TRACE of each resource's zarray index (r->idx) per name across both pushes in core_apply_worker_resvals() and re-run this exact replay to see whether the same resource name maps to two different indices (an idx-resolution bug) or whether the backing array is being cleared by something else entirely.\n(2) A SEPARATE, independent display-layer bug in repl_info_worker_one() (repl_info.c:324): its resource-print loop uses `for (i = 0; i < w->resvals.count; i++)` -- treating `.count` (number of currently-populated slots) as a safe index upper bound for iterating the zarray by index. resource.c:228 shows the correct convention already established elsewhere in this codebase for the identical structure: `int num_res = (int)w->resvals.caps;` -- `.caps` (allocated capacity, monotonic, never shrinks) is the right loop bound for index-based zarray iteration, NOT `.count` (which can legitimately be less than the highest occupied index once entries are sparse/non-contiguous -- exactly the state bug (1)'s stale/over-reported count produces, but this display bug is real and reachable independently of bug (1) ever occurring, any time resvals becomes sparse for any reason). This is very likely the DIRECT cause of the user-visible symptom (all resource lines vanishing while status=online persists): once bug (1) leaves the array in a sparse state relative to a wrong/stale count, this loop bound mis-computes and silently under-iterates to zero visible lines.\nNo matching prior bug found in buglog.json (searched WORKERS_PUSH/resvals/'info worker'/resource-value-disappear) or cerebrum.md. Closest relative: bug-1568 (multiple WORKERS_PUSH on worker join dropping resvals via an add-only key_add()), already fixed via the core_upsert_worker() merge path -- but that very merge path (core_apply_worker_resvals()) is what this NEW count-doubling anomaly now implicates, so a full fix should re-examine bug-1568's fix for a second-push-specific edge case it may not have covered.",
+      "fix": "NOT FIXED -- root cause (1) (the count-doubling trigger site) still needs one more targeted trace to pin down exactly; root cause (2) (repl_info.c:324's `.count` vs `.caps` loop-bound bug) is independently understood and fixable on its own (mirror resource.c:228's `.caps` convention) but has NOT been implemented yet -- deferred as a follow-up task, not done in this session (this session's primary task was the unrelated bug-1687 controller/WAL fix). No code changes made for this bug yet.",
+      "tags": [
+        "scheduler",
+        "simulator",
+        "sim-repl",
+        "info-worker",
+        "resource-values",
+        "regression",
+        "partially-root-caused",
+        "needs-followup",
+        "zarray-count-vs-caps",
+        "resvals-merge",
+        "worker-push"
+      ],
+      "related_bugs": [
+        "bug-1568"
+      ],
+      "occurrences": 1,
+      "last_seen": "2026-08-25T18:05:00Z"
     }
   ]
 }