docs(buglog): bug-1923 invariant audit (scheduler dims 1/4) clean

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.wolf/buglog.json

@@ -17438,7 +17438,7 @@
       "error_message": "xce_scheduler --sim drops shared resource values from the snapshot (license shows value=(none))\n\n**Observed:** a shared (cluster-wide, `GLOBAL|SHARED|CONSUMABLE`) resource value that is present in a recording's `SnapshotEntry` is not loaded by `xce_scheduler --sim`. `info resource license`, after `step` or straight after load, shows:\n```\n[sim] resource license: desc= valtype=numeric ordtype=incr flags=GLOBAL,SHARED,CONSUMABLE interval=0\n[sim]   value=(none)\n[sim] DISPATCH jobid=0-0-0-101\n[sim] PEND jobid=0-0-0-102\n[sim] PEND jobid=0-0-0-103\n[sim] PEND jobid=0-0-0-104\n```\nWhen the same value arrives as a `RESVALS_FULL` replay event, it loads (`value=2.00 used=2.00`), and jobs 102 and 103 dispatch.\n\n**Expected:** every shared resource value in the snapshot is loaded into `core->sharedInst`, so the simulator starts with the same shared pool the recorder had.\n\n**Impact:** wrong result in the simulator. A replay whose shared resource values were set before recording started, and never re-sent as a `RESVALS_FULL` event during the recording, simulates with no license pool. Jobs needing the resource pend instead of dispatching, so a sim replay diverges from what live did. Nothing is logged when the value is dropped.\n\n**Found by:** spec task scheduler-sim-decision-trace 1.1, while building the trace-off REPL baseline fixture (`src/scheduler/testsuite/test_repl_trace_baseline.c`). That fixture now sends the license value as a `RESVALS_FULL` event to work around this bug.\n\nTwo separate defects were found (see Reproduction, variants A and C):\n- **Read side, confirmed with gdb and a decoder:** `snapshot_load.c:129` skips the entry when `rv->host != NULL`. After the capnp round trip, a host that was written as NULL decodes as `\"\"`, so every shared resval in a snapshot is skipped and line 140's `zarray_put` is never reached. The `RESVALS_FULL` path (`core_events.c:1124-1147`) has no such host check.\n- **Write side, Suspected (from code reading, matches the decoder output):** `snapshot.c:431` loops `for (i = 0; i < core->sharedInst.count; i++)`, but `zarray_put_ex` (`src/lib/zarray.c:291-330`) counts filled slots, not the highest index plus one. A sparse `sharedInst` therefore drops values at higher indices: with the license at idx 2 and `count == 1`, the snapshot carries `resvals count=0`. The same pattern may affect per-worker resvals at `snapshot_wire.c:242` (`w->resvals.count`); not checked.",
       "file": "src/scheduler/src/snapshot.c, src/scheduler/src/snapshot_load.c, src/scheduler/src/snapshot_wire.c",
       "root_cause": "Two separate defects on the SnapshotEntry save/load path, both pre-existing (origin unknown):\n- **Read side, `src/scheduler/src/snapshot_load.c:129`:** the shared-pool loop skipped any resval with `rv->host != NULL`. The writer stores shared values with `host = NULL`, but after the capnp round trip (`write_snapshot_entry()` in recording.c → `.replay` → `read_SnapshotEntry`/`decode_SnapshotEntry` in sim.c) a NULL host decodes as `\"\"`. So every shared value in a snapshot read from a file was silently skipped, and `core->sharedInst` started empty. The `RESVALS_FULL` event path (`core_events.c:1124-1147`) has no host check, which is why the workaround loaded. A non-empty host only appears on `RESVALS_DELTA` (routing a value to one worker).\n- **Write side, `src/scheduler/src/snapshot.c:431`:** the shared-pool loop was bounded by `core->sharedInst.count`. `zarray_put_ex` (`src/lib/zarray.c`) counts filled slots, not highest index + 1, so in a sparse `sharedInst` the values at higher indices were never written. In the baseline fixture, `license` (idx 2) and `limit=cluster_slots` were dropped.\n- **Same defect, per worker, `src/scheduler/src/snapshot_wire.c:242`:** the loop over `w->resvals` was bounded by `w->resvals.count`. A worker never holds a value at a global resource's idx, so its resvals always have holes, and `limit=host_slots` was dropped for every worker.\n- Scope checked: no other `.count`-bounded loop over a sparse resval zarray remains in src/scheduler or src/lib. `repl_info.c:524` and `resource.c:262` already use `.caps`; the other `.count` uses are dense lists or log lines.",
-      "fix": "- `snapshot_load.c`: a resval with a NULL **or empty** host is a shared-pool entry; only a named host is skipped.\n- `snapshot.c`: the shared-pool loop is bounded by `sharedInst.caps` (the project convention for sparse resval arrays). Empty slots were already skipped via `zarray_get` returning NULL.\n- `snapshot_wire.c`: the per-worker loop is bounded by `w->resvals.caps`.\n- These changes make the shared and per-worker values round-trip through both the write side and the load seam (persistence.md, snapshot field-completeness).\n- Compatibility: replay files written by the old writer lack the dropped values, so the reader fix alone can't recover them (`/home/claude/tmp/bug-1923/fixA` still shows `value=(none)`). Files written from now on carry them.\n- Fixture: `src/scheduler/testsuite/fixtures/repl_trace_baseline/set_output.replay` was regenerated. Only its SnapshotEntry changed (+304 bytes: `license`, `limit=cluster_slots` and two per-worker `limit=host_slots` values). The header differs only in the ignored wall-clock field, the transcript, `sim_output.replay` and verify output are unchanged, and messages 2-15 are byte-identical.\n\nVerification:\n- Test: new `src/scheduler/testsuite/test_snapshot_sparse_resvals.c` (\"scheduler snapshot sparse resvals test\"), which links the real scheduler sources. RED before the fix: 6 Tests 5 Failures (`test_save_keeps_sparse_shared_resval`, `test_save_keeps_sparse_worker_resval`, `test_save_load_round_trip_keeps_sparse_resvals`, `test_load_takes_shared_resval_with_empty_host`, and `test_replay_round_trip_keeps_sparse_resvals`, which goes through the real `.replay` writer and the `--sim` start-up reader). The guard test `test_load_skips_resval_with_named_host` passed. GREEN after: 6 Tests 0 Failures.\n- Suites: scheduler meson suite Ok 100 / Fail 0, asan (`src/xce.build` in the fix worktree); `scheduler repl trace baseline test` OK with the regenerated `set_output.replay`.\n- Live: `xce_scheduler --sim /home/claude/tmp/bug-1923/fixC/contention.replay --interactive` on the fixed asan build: `info resource license` shows `value=2.00 used=0.00`, then `value=2.00 used=2.00` after `step`, and jobs 102 and 103 DISPATCH (before the fix: `value=(none)`, and both PEND). A variant-A replay regenerated with the fixed writer also loads `value=2.00`. No HA soak was needed: the change is limited to the scheduler's snapshot save/load and does not touch the controller or WAL.",
+      "fix": "- `snapshot_load.c`: a resval with a NULL **or empty** host is a shared-pool entry; only a named host is skipped.\n- `snapshot.c`: the shared-pool loop is bounded by `sharedInst.caps` (the project convention for sparse resval arrays). Empty slots were already skipped via `zarray_get` returning NULL.\n- `snapshot_wire.c`: the per-worker loop is bounded by `w->resvals.caps`.\n- These changes make the shared and per-worker values round-trip through both the write side and the load seam (persistence.md, snapshot field-completeness).\n- Compatibility: replay files written by the old writer lack the dropped values, so the reader fix alone can't recover them (`/home/claude/tmp/bug-1923/fixA` still shows `value=(none)`). Files written from now on carry them.\n- Fixture: `src/scheduler/testsuite/fixtures/repl_trace_baseline/set_output.replay` was regenerated. Only its SnapshotEntry changed (+304 bytes: `license`, `limit=cluster_slots` and two per-worker `limit=host_slots` values). The header differs only in the ignored wall-clock field, the transcript, `sim_output.replay` and verify output are unchanged, and messages 2-15 are byte-identical.\n\nVerification:\n- Test: new `src/scheduler/testsuite/test_snapshot_sparse_resvals.c` (\"scheduler snapshot sparse resvals test\"), which links the real scheduler sources. RED before the fix: 6 Tests 5 Failures (`test_save_keeps_sparse_shared_resval`, `test_save_keeps_sparse_worker_resval`, `test_save_load_round_trip_keeps_sparse_resvals`, `test_load_takes_shared_resval_with_empty_host`, and `test_replay_round_trip_keeps_sparse_resvals`, which goes through the real `.replay` writer and the `--sim` start-up reader). The guard test `test_load_skips_resval_with_named_host` passed. GREEN after: 6 Tests 0 Failures.\n- Suites: scheduler meson suite Ok 100 / Fail 0, asan (`src/xce.build` in the fix worktree); `scheduler repl trace baseline test` OK with the regenerated `set_output.replay`.\n- Live: `xce_scheduler --sim /home/claude/tmp/bug-1923/fixC/contention.replay --interactive` on the fixed asan build: `info resource license` shows `value=2.00 used=0.00`, then `value=2.00 used=2.00` after `step`, and jobs 102 and 103 DISPATCH (before the fix: `value=(none)`, and both PEND). A variant-A replay regenerated with the fixed writer also loads `value=2.00`. No HA soak was needed: the change is limited to the scheduler's snapshot save/load and does not touch the controller or WAL.\n\nUPDATE (2026-10-08): Invariant audit (xce-audit-invariants scheduler, dimensions 1 and 4) run 2026-10-08 at dc624bfc: CLEAN. A temporary round-trip probe (24 resources, sparse shared slots 3..23, every value type, three workers with sparse/single/no per-worker values) went live core, sched_record_start, sched_sim_driver_start, then an in-memory save/load, and matched every slot by name, type and value; it failed when either half of the fix was reverted (save .caps -> r07 dropped; load empty-host -> r03 dropped). The probe source is kept at /home/claude/tmp/bug-1923/test_snapshot_resval_probe.c. The real-binary --sim-verify of a license-contention replay from a mid-run snapshot verifies rounds 3000-10000; the pre-fix fixture had extra mismatches in 6000-10000. Dimension 2 is covered by test_core_isolation and dimension 3 is untouched; the HA soak is controller-only and n/a.",
       "tags": [
         "sim",
         "snapshot",

.wolf/bugs/bug-1923.md

@@ -80,6 +80,8 @@ Two separate defects on the SnapshotEntry save/load path, both pre-existing (ori
 - Suites: scheduler meson suite Ok 100 / Fail 0, asan (`src/xce.build` in the fix worktree); `scheduler repl trace baseline test` OK with the regenerated `set_output.replay`.
 - Live: `xce_scheduler --sim /home/claude/tmp/bug-1923/fixC/contention.replay --interactive` on the fixed asan build: `info resource license` shows `value=2.00 used=0.00`, then `value=2.00 used=2.00` after `step`, and jobs 102 and 103 DISPATCH (before the fix: `value=(none)`, and both PEND). A variant-A replay regenerated with the fixed writer also loads `value=2.00`. No HA soak was needed: the change is limited to the scheduler's snapshot save/load and does not touch the controller or WAL.
 
+UPDATE (2026-10-08): Invariant audit (xce-audit-invariants scheduler, dimensions 1 and 4) run 2026-10-08 at dc624bfc: CLEAN. A temporary round-trip probe (24 resources, sparse shared slots 3..23, every value type, three workers with sparse/single/no per-worker values) went live core, sched_record_start, sched_sim_driver_start, then an in-memory save/load, and matched every slot by name, type and value; it failed when either half of the fix was reverted (save .caps -> r07 dropped; load empty-host -> r03 dropped). The probe source is kept at /home/claude/tmp/bug-1923/test_snapshot_resval_probe.c. The real-binary --sim-verify of a license-contention replay from a mid-run snapshot verifies rounds 3000-10000; the pre-fix fixture had extra mismatches in 6000-10000. Dimension 2 is covered by test_core_isolation and dimension 3 is untouched; the HA soak is controller-only and n/a.
+
 ## References
 - Related: bug-1568 (same `snapshot_wire.c` worker-resval loop; that bug added the missing-resource diagnostic)
 - Docs: .kiro/steering/persistence.md (Snapshot Save/Load Field-Completeness)