chore(wolf): checkpoint buglog.json (946 bugs)

.wolf/buglog.json

@@ -16845,7 +16845,7 @@
       "error_message": "xce_scheduler -i -s <replay>: WARN 'failed to resolve job picker plugin (rc=-2)' on load; all pre-existing jobs stay PEND forever, nothing ever dispatches in simulation/interactive mode",
       "file": "src/scheduler/src/sim.c, src/scheduler/src/sim_loop.c",
       "root_cause": "Two compounding, both explicitly documented 'Known scope boundary' gaps in the scheduler-core sim driver (task 11.1/11.2, .kiro/specs/scheduler-core): (1) sched_sim_driver_start() (sim.c) loads the initial snapshot via xce_sched_core_load_snapshot() BEFORE any picker plugin is registered into core->plugins -- only job_pick_plugin's NAME is threaded through from the ReplayHeader (sim.h:206-216's own doc comment). sched_resolve_job_picker() therefore returns -ENOENT (=-2), producing the WARN, and the picker-rebuild walk (on_queue_update/on_job_push) inside xce_sched_core_load_snapshot() is silently skipped for every job/queue in the initial snapshot. The real picker only gets registered moments later, in sim_register_default_picker() (sim_loop.c:381, called from sched_sim_driver_run() right after start() returns) -- but nothing re-pushes the already-loaded jobs/queues into it, so any job present in the replay file's INITIAL snapshot is permanently invisible to the fcfs picker's next_job(). (2) Independently, sim_picker_thread_fn() (sim_loop.c:448) has its own documented scope boundary: real resource-matching allocation (sched_make_allocation()) needs a live xce_scheduler_t* that the sim driver deliberately doesn't have, and no core-only equivalent exists yet (deferred to the separate, not-yet-designed .kiro/specs/scheduler-allocation spec) -- so EVERY candidate the picker selects is unconditionally turned into a PEND decision; dispatch is not implemented in simulation mode at all today, regardless of resource availability.",
-      "fix": "Not yet fixed -- diagnosis only, per user request ('Investigate following issue'). (1) is a scoped, tractable fix (re-run the picker-rebuild walk, or push the initial snapshot's jobs/queues into the picker, right after sim_register_default_picker() registers it in sched_sim_driver_run()). (2) is a much larger, intentionally out-of-scope gap (a whole not-yet-designed spec, scheduler-allocation) and should not be casually patched without a design discussion. Reported back to user with both findings; awaiting direction on whether/which to fix.",
+      "fix": "FIXED (TDD, RED->GREEN). New file src/scheduler/src/sim_picker.c: sim_bootstrap_picker() (replaces sim_loop.c's sim_register_default_picker(), moved out because sim_loop.c was already at its documented 800-line 'zero line-budget headroom' cap) registers the real builtin_fcfs picker AND then replays the picker-rebuild walk (on_queue_update per queue, on_job_push per pending job) that xce_sched_core_load_snapshot() had to skip at initial-snapshot-load time. Wired into scheduler_srcs, sched_sim_isolation_lib, and all 9 white-box test TUs that #include sim_loop.c directly (test_sim_isolation.c mechanically re-verified). New regression test: test_sim_event_loop.c's test_job_present_only_in_initial_snapshot_still_reaches_the_picker -- builds a fixture with a job baked into the INITIAL snapshot (not a ReplayEvent), asserts it gets a live PEND decision. RED confirmed by temporarily no-op'ing the rebuild walk (test failed as expected), GREEN with the real fix restored. Full scheduler suite green (41/41, only pre-existing unrelated bug-1560 skipped/excluded from that count). Note: finding (2) from the original diagnosis (sim mode never dispatches, only PENDs -- scheduler-allocation spec gap) is NOT fixed here, by design -- out of scope, a whole separate not-yet-designed spec.",
       "tags": [
         "scheduler",
         "simulation-mode",
@@ -16854,11 +16854,12 @@
         "job-picker",
         "scheduler-core",
         "known-scope-boundary",
-        "not-yet-fixed"
+        "fixed",
+        "tdd"
       ],
       "related_bugs": [],
-      "occurrences": 1,
+      "occurrences": 2,
-      "last_seen": "2026-08-17T09:00:00Z"
+      "last_seen": "2026-08-17T09:40:00Z"
     },
     {
       "id": "bug-1568",
@@ -16866,20 +16867,54 @@
       "error_message": "xce_scheduler -i -s <replay>: 'info worker <name>' shows total=0.00 available=0.00 for every resource, even though the live cluster (xce_test resval/workerlist) reports real non-zero values for the same worker",
       "file": "src/scheduler/src/repl_info.c, src/scheduler/src/snapshot_wire.c, src/scheduler/src/snapshot_load.c, src/controller/sched.c",
       "root_cause": "Not fully root-caused -- open lead, lower confidence than bug-1567. Verified the snapshot SAVE path (build_wire_worker, snapshot_wire.c:200) and LOAD path (xce_sched_core_load_snapshot's worker loop, snapshot_load.c:58) both correctly move resource values through resvals_to_sched/w->resvals -- so if the scheduler's OWN live in-memory worker mirror had real values at 'record start' time, they would have been captured correctly into the .replay file. Whether it actually did depends on the controller->scheduler resource-value push path (src/controller/sched.c's sched_worker_change/xcesched_push_workerlist, fired on worker join/dead/update hooks) and whether xce_core_push_events's XCE_CORE_EVENT_WORKERS_PUSH handler (core_events.c:463-480) silently drops a later real-valued push: it calls sched_add_worker() but never checks its return value, and sched_add_worker() is a plain key_add() (external vendored rtti library, var-typed keyed container, src/scheduler/src/core.c:270) -- could not confirm from this repo's own sources whether key_add() re-adding an existing worker name update-in-places or fails/no-ops, since the concrete implementation lives in the vendored rtti-0.0.2 nix package, not this repo. Functional impact today is zero (bug-1567's finding (2): sim mode doesn't do real resource-matching allocation at all yet), but this is a real diagnostics-accuracy gap in 'info worker' worth fixing separately if/when allocation gets implemented.",
-      "fix": "Not yet fixed -- open investigation lead, reported to user alongside bug-1567. Needs a live recording session trace (or reading the vendored rtti library's key_add source) to confirm whether this is a genuine capture/propagation bug or expected-at-record-time transient state.",
+      "fix": "INSTRUMENTED + ROOT-CAUSED via live deploy+record+replay reproduction (not a code fix -- per user request, this was scoped to diagnosis only). Added TRACE instrumentation at every hop of the pipeline: src/controller/sched.c (sched_worker_change's per-event push count, xcesched_push_workers' full-initial-sync count), src/scheduler/src/core_events.c (core_build_worker's carried-vs-applied resval counts + sched_add_worker's previously-unchecked return code), src/scheduler/src/snapshot_wire.c (build_wire_worker's captured count -- PLUS a genuinely missing WARN this exposed: an unresolvable resource idx was silently dropped here with zero diagnostic, unlike every analogous skip elsewhere in this codebase), src/scheduler/src/snapshot_load.c (restored count + sched_add_worker return check). Live-traced with a real 1-node deploy (xcecd_virtual worker): confirmed the controller->scheduler push carries 100% of what it collects, and the scheduler applies 100% of what it receives into worker->resvals, and (once a save-time idx-resolution race was avoided by using a freshly-restarted scheduler) build_wire_worker captures 100% (20/20) into the snapshot -- yet 'info worker' on the resulting .replay STILL showed all zeros, including 'mem' after waiting 25+s (past resource_sync's own 15s LOAD_INDICES_INTERVAL_DEFAULT reporting interval). CONCLUSION: this is NEITHER a controller->scheduler propagation bug NOR (primarily) a snapshot save/load bug -- the values are already zero at the true source. xce_test's own 'resval'/'workerlist' commands show real numbers (22 slots, 17GB mem, etc.) for the exact same worker because they read a DIFFERENT source -- almost certainly w->host.attrs (hostattrs_t: type/model/cpufactor/maxcpus/maxmem/maxswap/maxtmp/ndisks + derived 'slots'), populated once at JOIN from real host introspection -- while the ENTIRE controller->scheduler->snapshot pipeline instrumented above only ever carries w->host.resvals (the generic per-resource xce_val_t zarray, populated ONLY by resource_sync's periodic collect_resval()/collect_load_indices() push, which reports just 11 builtin 'load' metrics -- r15s/r1m/r15m/ut/pg/ls/it/swp/mem/tmp/io -- NONE of which include slots/maxcpus/maxmem/maxswap/maxtmp/type/model/cpuf/ndisks at all). Nothing in this codebase appears to ever copy hostattrs_t's real values INTO the resvals zarray, so anything downstream of collect_resval (which is everything the scheduler/snapshot/replay pipeline sees) is structurally blind to those 9 resources regardless of timing. Separately, even 'mem' (which genuinely IS one of the 11 periodically-synced metrics) stayed at 0.00 past the 15s interval in this xcecd_virtual test -- either the virtual worker's resource_sync/hardware_monitor getters return nothing meaningful in this sandboxed environment, or a second, distinct gap exists there; not fully isolated (lower priority, smaller blast radius than the hostattrs_t finding). NOT YET FIXED -- reported findings to user, awaiting direction on whether/how to wire hostattrs_t into the resvals zarray (a real, scoped feature gap, not a quick patch).",
       "tags": [
         "scheduler",
         "simulation-mode",
         "resource-values",
         "worker-resvals",
         "needs-investigation",
-        "not-yet-fixed"
+        "root-caused",
+        "live-verified",
+        "architecture-gap"
       ],
       "related_bugs": [
         "bug-1567"
       ],
+      "occurrences": 2,
+      "last_seen": "2026-08-17T09:40:00Z"
+    },
+    {
+      "id": "bug-945",
+      "timestamp": "2026-08-17T09:14:31.784Z",
+      "error_message": "Significant refactor of ",
+      "file": "src/scheduler/src/sim_loop.c",
+      "root_cause": "47 lines replaced/restructured",
+      "fix": "Rewrote 52\u219212 lines (47 removed)",
+      "tags": [
+        "auto-detected",
+        "refactor",
+        "c"
+      ],
+      "related_bugs": [],
       "occurrences": 1,
-      "last_seen": "2026-08-17T09:00:00Z"
+      "last_seen": "2026-08-17T09:14:31.784Z"
+    },
+    {
+      "id": "bug-946",
+      "timestamp": "2026-08-17T09:14:35.731Z",
+      "error_message": "Wrong reference: sim_register_default_picker should be sim_bootstrap_picker",
+      "file": "src/scheduler/src/sim_loop.c",
+      "root_cause": "Used \"sim_register_default_picker\" instead of \"sim_bootstrap_picker\"",
+      "fix": "Changed sim_register_default_picker \u2192 sim_bootstrap_picker",
+      "tags": [
+        "auto-detected",
+        "wrong-reference",
+        "c"
+      ],
+      "related_bugs": [],
+      "occurrences": 1,
+      "last_seen": "2026-08-17T09:14:35.731Z"
     }
   ]
 }