myxce/.wolf/bugs/bug-1923.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	gitea_issue	gitea_synced_status	gitea_synced_hash
bug-1923	fixed	2026-10-08		scheduler	wrong-result	2026-10-08T07:17:34Z	2026-10-08T07:17:34Z	1	src/scheduler/src/snapshot.c src/scheduler/src/snapshot_load.c src/scheduler/src/snapshot_wire.c	sim snapshot resvals replay persistence		16616d96	348	fixed	735fa9567f4e

bug-1923: xce_scheduler --sim drops shared resource values from the snapshot (license shows value=(none))

Description

Observed: a shared (cluster-wide, GLOBAL|SHARED|CONSUMABLE) resource value that is present in a recording's SnapshotEntry is not loaded by xce_scheduler --sim. info resource license, after step or straight after load, shows:

[sim] resource license: desc= valtype=numeric ordtype=incr flags=GLOBAL,SHARED,CONSUMABLE interval=0
[sim]   value=(none)
[sim] DISPATCH jobid=0-0-0-101
[sim] PEND jobid=0-0-0-102
[sim] PEND jobid=0-0-0-103
[sim] PEND jobid=0-0-0-104

When the same value arrives as a RESVALS_FULL replay event, it loads (value=2.00 used=2.00), and jobs 102 and 103 dispatch.

Expected: every shared resource value in the snapshot is loaded into core->sharedInst, so the simulator starts with the same shared pool the recorder had.

Impact: wrong result in the simulator. A replay whose shared resource values were set before recording started, and never re-sent as a RESVALS_FULL event during the recording, simulates with no license pool. Jobs needing the resource pend instead of dispatching, so a sim replay diverges from what live did. Nothing is logged when the value is dropped.

Found by: spec task scheduler-sim-decision-trace 1.1, while building the trace-off REPL baseline fixture (src/scheduler/testsuite/test_repl_trace_baseline.c). That fixture now sends the license value as a RESVALS_FULL event to work around this bug.

Two separate defects were found (see Reproduction, variants A and C):

• Read side, confirmed with gdb and a decoder: snapshot_load.c:129 skips the entry when rv->host != NULL. After the capnp round trip, a host that was written as NULL decodes as "", so every shared resval in a snapshot is skipped and line 140's zarray_put is never reached. The RESVALS_FULL path (core_events.c:1124-1147) has no such host check.
• Write side, Suspected (from code reading, matches the decoder output): snapshot.c:431 loops for (i = 0; i < core->sharedInst.count; i++), but zarray_put_ex (src/lib/zarray.c:291-330) counts filled slots, not the highest index plus one. A sparse sharedInst therefore drops values at higher indices: with the license at idx 2 and count == 1, the snapshot carries resvals count=0. The same pattern may affect per-worker resvals at snapshot_wire.c:242 (w->resvals.count); not checked.

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (xce.build)
• Code: scheduler-framework @ ed201686-dirty
• Deployment: none (offline xce_scheduler --sim <replay> --interactive; the replay is synthetic, written by a test generator with 2 virtual workers)
• Note: -dirty = only the uncommitted task 1.1 test files and unrelated .wolf/ bookkeeping; no production scheduler source was changed

Reproduction

Durable copies are in /home/claude/tmp/bug-1923/: fixA/contention.replay has the value in the snapshot (license idx 2); fixC/contention.replay is the same but the license is idx 0; fixB/contention.replay uses the RESVALS_FULL workaround.

1. Build the scheduler: bash src/tools/xce-build.sh scheduler (ASAN, src/xce.build).
2. cd "$(mktemp -d)" && cp /home/claude/tmp/bug-1923/fixA/contention.replay .
3. printf 'info resource license\nstep\ninfo resource license\nquit\n' | ASAN_OPTIONS=detect_leaks=0 TERM=dumb /home/rshen/projects/xce/scheduler-framework/src/xce.build/scheduler/xce_scheduler --sim contention.replay --interactive --logfile sched.log
4. Both info resource license calls print value=(none), and jobs 102 and 103 PEND. Repeat with fixC (same result: the read-side skip alone is enough) and fixB (the first call prints value=(none) because the event is applied in round 1; after step it prints value=2.00 used=2.00 and jobs 102 and 103 dispatch).

• How the replays are generated: /home/claude/tmp/bug-1923/gen.c is a copy of the test's write_contention_replay(), with -DSNAPSHOT_LICENSE (A) or -DLICENSE_FIRST (C). The value is put in with zarray_put(&sched.core.sharedInst, lic->idx, val) before sched_record_start(). build_and_run.sh [N] rebuilds and reruns all three; it needs the scratch object copies, which were not kept. decode.c dumps a replay's SnapshotEntry resvals: A shows count=0, C shows resval name=license host="" = 2.000000.
• Frequency: deterministic (A 20/20 and C 20/20 show value=(none); B 0/20; 2026-10-08)
• Evidence: /home/claude/tmp/bug-1923/ (replays, generator, decoder, probe script)

Root cause

Two separate defects on the SnapshotEntry save/load path, both pre-existing (origin unknown):

• Read side, src/scheduler/src/snapshot_load.c:129: the shared-pool loop skipped any resval with rv->host != NULL. The writer stores shared values with host = NULL, but after the capnp round trip (write_snapshot_entry() in recording.c → .replay → read_SnapshotEntry/decode_SnapshotEntry in sim.c) a NULL host decodes as "". So every shared value in a snapshot read from a file was silently skipped, and core->sharedInst started empty. The RESVALS_FULL event path (core_events.c:1124-1147) has no host check, which is why the workaround loaded. A non-empty host only appears on RESVALS_DELTA (routing a value to one worker).
• Write side, src/scheduler/src/snapshot.c:431: the shared-pool loop was bounded by core->sharedInst.count. zarray_put_ex (src/lib/zarray.c) counts filled slots, not highest index + 1, so in a sparse sharedInst the values at higher indices were never written. In the baseline fixture, license (idx 2) and limit=cluster_slots were dropped.
• Same defect, per worker, src/scheduler/src/snapshot_wire.c:242: the loop over w->resvals was bounded by w->resvals.count. A worker never holds a value at a global resource's idx, so its resvals always have holes, and limit=host_slots was dropped for every worker.
• Scope checked: no other .count-bounded loop over a sparse resval zarray remains in src/scheduler or src/lib. repl_info.c:524 and resource.c:262 already use .caps; the other .count uses are dense lists or log lines.

Fix

• snapshot_load.c: a resval with a NULL or empty host is a shared-pool entry; only a named host is skipped.
• snapshot.c: the shared-pool loop is bounded by sharedInst.caps (the project convention for sparse resval arrays). Empty slots were already skipped via zarray_get returning NULL.
• snapshot_wire.c: the per-worker loop is bounded by w->resvals.caps.
• These changes make the shared and per-worker values round-trip through both the write side and the load seam (persistence.md, snapshot field-completeness).
• Compatibility: replay files written by the old writer lack the dropped values, so the reader fix alone can't recover them (/home/claude/tmp/bug-1923/fixA still shows value=(none)). Files written from now on carry them.
• Fixture: src/scheduler/testsuite/fixtures/repl_trace_baseline/set_output.replay was regenerated. Only its SnapshotEntry changed (+304 bytes: license, limit=cluster_slots and two per-worker limit=host_slots values). The header differs only in the ignored wall-clock field, the transcript, sim_output.replay and verify output are unchanged, and messages 2-15 are byte-identical.

Verification

• Test: new src/scheduler/testsuite/test_snapshot_sparse_resvals.c ("scheduler snapshot sparse resvals test"), which links the real scheduler sources. RED before the fix: 6 Tests 5 Failures (test_save_keeps_sparse_shared_resval, test_save_keeps_sparse_worker_resval, test_save_load_round_trip_keeps_sparse_resvals, test_load_takes_shared_resval_with_empty_host, and test_replay_round_trip_keeps_sparse_resvals, which goes through the real .replay writer and the --sim start-up reader). The guard test test_load_skips_resval_with_named_host passed. GREEN after: 6 Tests 0 Failures.
• Suites: scheduler meson suite Ok 100 / Fail 0, asan (src/xce.build in the fix worktree); scheduler repl trace baseline test OK with the regenerated set_output.replay.
• Live: xce_scheduler --sim /home/claude/tmp/bug-1923/fixC/contention.replay --interactive on the fixed asan build: info resource license shows value=2.00 used=0.00, then value=2.00 used=2.00 after step, and jobs 102 and 103 DISPATCH (before the fix: value=(none), and both PEND). A variant-A replay regenerated with the fixed writer also loads value=2.00. No HA soak was needed: the change is limited to the scheduler's snapshot save/load and does not touch the controller or WAL.

References

• Related: bug-1568 (same snapshot_wire.c worker-resval loop; that bug added the missing-resource diagnostic)
• Docs: .kiro/steering/persistence.md (Snapshot Save/Load Field-Completeness)
• Found by: .kiro/specs/scheduler-sim-decision-trace task 1.1