chore(wolf): checkpoint buglog.json (946 bugs)

.wolf/buglog.json

@@ -16864,25 +16864,27 @@
     {
       "id": "bug-1568",
       "timestamp": "2026-08-17T09:00:00Z",
-      "error_message": "xce_scheduler -i -s <replay>: 'info worker <name>' shows total=0.00 available=0.00 for every resource, even though the live cluster (xce_test resval/workerlist) reports real non-zero values for the same worker",
+      "error_message": "xce_scheduler -i -s <replay>: 'info worker <name>' shows total=0.00 available=0.00 for every resource, even though the live cluster (xce_test resval/workerlist) reports real non-zero values for the same worker -- CONFIRMED ROOT CAUSE (superseding this entry's earlier 'architecture gap' theory, which was wrong -- corrected after the user pointed to worker.c's real mark_host_res()/update_val_res() write path)",
       "file": "src/scheduler/src/repl_info.c, src/scheduler/src/snapshot_wire.c, src/scheduler/src/snapshot_load.c, src/controller/sched.c",
-      "root_cause": "Not fully root-caused -- open lead, lower confidence than bug-1567. Verified the snapshot SAVE path (build_wire_worker, snapshot_wire.c:200) and LOAD path (xce_sched_core_load_snapshot's worker loop, snapshot_load.c:58) both correctly move resource values through resvals_to_sched/w->resvals -- so if the scheduler's OWN live in-memory worker mirror had real values at 'record start' time, they would have been captured correctly into the .replay file. Whether it actually did depends on the controller->scheduler resource-value push path (src/controller/sched.c's sched_worker_change/xcesched_push_workerlist, fired on worker join/dead/update hooks) and whether xce_core_push_events's XCE_CORE_EVENT_WORKERS_PUSH handler (core_events.c:463-480) silently drops a later real-valued push: it calls sched_add_worker() but never checks its return value, and sched_add_worker() is a plain key_add() (external vendored rtti library, var-typed keyed container, src/scheduler/src/core.c:270) -- could not confirm from this repo's own sources whether key_add() re-adding an existing worker name update-in-places or fails/no-ops, since the concrete implementation lives in the vendored rtti-0.0.2 nix package, not this repo. Functional impact today is zero (bug-1567's finding (2): sim mode doesn't do real resource-matching allocation at all yet), but this is a real diagnostics-accuracy gap in 'info worker' worth fixing separately if/when allocation gets implemented.",
-      "fix": "INSTRUMENTED + ROOT-CAUSED via live deploy+record+replay reproduction (not a code fix -- per user request, this was scoped to diagnosis only). Added TRACE instrumentation at every hop of the pipeline: src/controller/sched.c (sched_worker_change's per-event push count, xcesched_push_workers' full-initial-sync count), src/scheduler/src/core_events.c (core_build_worker's carried-vs-applied resval counts + sched_add_worker's previously-unchecked return code), src/scheduler/src/snapshot_wire.c (build_wire_worker's captured count -- PLUS a genuinely missing WARN this exposed: an unresolvable resource idx was silently dropped here with zero diagnostic, unlike every analogous skip elsewhere in this codebase), src/scheduler/src/snapshot_load.c (restored count + sched_add_worker return check). Live-traced with a real 1-node deploy (xcecd_virtual worker): confirmed the controller->scheduler push carries 100% of what it collects, and the scheduler applies 100% of what it receives into worker->resvals, and (once a save-time idx-resolution race was avoided by using a freshly-restarted scheduler) build_wire_worker captures 100% (20/20) into the snapshot -- yet 'info worker' on the resulting .replay STILL showed all zeros, including 'mem' after waiting 25+s (past resource_sync's own 15s LOAD_INDICES_INTERVAL_DEFAULT reporting interval). CONCLUSION: this is NEITHER a controller->scheduler propagation bug NOR (primarily) a snapshot save/load bug -- the values are already zero at the true source. xce_test's own 'resval'/'workerlist' commands show real numbers (22 slots, 17GB mem, etc.) for the exact same worker because they read a DIFFERENT source -- almost certainly w->host.attrs (hostattrs_t: type/model/cpufactor/maxcpus/maxmem/maxswap/maxtmp/ndisks + derived 'slots'), populated once at JOIN from real host introspection -- while the ENTIRE controller->scheduler->snapshot pipeline instrumented above only ever carries w->host.resvals (the generic per-resource xce_val_t zarray, populated ONLY by resource_sync's periodic collect_resval()/collect_load_indices() push, which reports just 11 builtin 'load' metrics -- r15s/r1m/r15m/ut/pg/ls/it/swp/mem/tmp/io -- NONE of which include slots/maxcpus/maxmem/maxswap/maxtmp/type/model/cpuf/ndisks at all). Nothing in this codebase appears to ever copy hostattrs_t's real values INTO the resvals zarray, so anything downstream of collect_resval (which is everything the scheduler/snapshot/replay pipeline sees) is structurally blind to those 9 resources regardless of timing. Separately, even 'mem' (which genuinely IS one of the 11 periodically-synced metrics) stayed at 0.00 past the 15s interval in this xcecd_virtual test -- either the virtual worker's resource_sync/hardware_monitor getters return nothing meaningful in this sandboxed environment, or a second, distinct gap exists there; not fully isolated (lower priority, smaller blast radius than the hostattrs_t finding). NOT YET FIXED -- reported findings to user, awaiting direction on whether/how to wire hostattrs_t into the resvals zarray (a real, scoped feature gap, not a quick patch).",
+      "root_cause": "CONFIRMED via live value-level TRACE instrumentation (not just counts this time): xce_core_push_events()'s XCE_CORE_EVENT_WORKERS_PUSH handler (src/scheduler/src/core_events.c) calls sched_add_worker() UNCONDITIONALLY on every push -- and sched_add_worker() (src/scheduler/src/core.c:270) is a plain key_add() (ADD-ONLY semantics: fails -EEXIST if the worker name is already registered, never updates in place). A worker connecting triggers MULTIPLE WORKERS_PUSH events in quick succession (confirmed live: sched.c's collect_resval()/xcesched_push_workers() fire more than once during join sequencing -- e.g. an early push before src/controller/worker.c's apply_join_state()->add_hostattrs_resvals()/set_host_slots() has finished writing real hostattrs-derived values into w->host.resvals, carrying stale/zero data, followed by a later push carrying the real values). The FIRST push's sched_add_worker() call succeeds and registers the worker (confirmed: 'sched_add_worker(\"pc13v\") ok (idx=0)' at 10:02:34.848, immediately preceded by a TRACE showing it received maxmem=0.000000). The SECOND push's freshly-built xce_sched_worker_t -- this time carrying the REAL values (confirmed: 'resource maxmem received fvalue=33182089216.000000', matching xce_test's own real 30.9GB reading, at 10:02:36.849 -- silently fails sched_add_worker() with rc=-17 (-EEXIST) one line later in the SAME log, and its return value was unchecked until this investigation's own instrumentation added the WARN (core_events.c, added this session) that finally surfaced it. The correctly-valued worker object is discarded/leaked; core->workers['pc13v'] permanently keeps the FIRST, zero-valued object. This is not limited to the initial-join race -- because there is no 'update an existing worker in place' event type (unlike jobs, which have XCE_CORE_EVENT_JOB_STATE_CHANGE/JOB_UPDATE for in-place mutation of an already-added job), EVERY subsequent WORKERS_PUSH for an already-registered worker (sched_worker_change's ongoing per-event pushes throughout normal cluster operation, not just at join) hits the identical silent EEXIST-discard. The scheduler's view of a worker's resources freezes forever at whichever push happened to land first and win the race, and never updates again for the lifetime of the process.",
+      "fix": "NOT YET FIXED -- root cause confirmed live with reproducible TRACE evidence (this session's own instrumentation across src/controller/worker.c, dataex.c, sched.c and src/scheduler/src/core_events.c). Correct fix: XCE_CORE_EVENT_WORKERS_PUSH's handler (core_events.c, xce_core_push_events()) should look up an existing worker by name first (sched_get_worker()) and update it in place (status + resvals, mirroring how XCE_CORE_EVENT_JOB_STATE_CHANGE/JOB_UPDATE mutate an existing job rather than re-adding) instead of unconditionally building a fresh xce_sched_worker_t and calling sched_add_worker() (add-only, key_add()-backed) every time. Reported to user with full evidence chain; awaiting go-ahead to implement + TDD regression test.",
       "tags": [
         "scheduler",
+        "controller",
         "simulation-mode",
         "resource-values",
         "worker-resvals",
-        "needs-investigation",
         "root-caused",
         "live-verified",
-        "architecture-gap"
+        "confirmed-bug",
+        "sched_add_worker",
+        "key_add-eexist"
       ],
       "related_bugs": [
         "bug-1567"
       ],
-      "occurrences": 2,
-      "last_seen": "2026-08-17T09:40:00Z"
+      "occurrences": 3,
+      "last_seen": "2026-08-17T10:05:00Z"
     },
     {
       "id": "bug-945",