86 lines
9.4 KiB
Markdown
86 lines
9.4 KiB
Markdown
---
|
|
id: "bug-1923"
|
|
status: "fixed"
|
|
status_updated: "2026-10-08"
|
|
status_note: ""
|
|
component: "scheduler"
|
|
severity: "wrong-result"
|
|
timestamp: "2026-10-08T07:17:34Z"
|
|
last_seen: "2026-10-08T07:17:34Z"
|
|
occurrences: 1
|
|
files: ["src/scheduler/src/snapshot.c", "src/scheduler/src/snapshot_load.c", "src/scheduler/src/snapshot_wire.c"]
|
|
tags: ["sim", "snapshot", "resvals", "replay", "persistence"]
|
|
related: []
|
|
fix_commits: ["16616d96"]
|
|
gitea_issue: 348
|
|
gitea_synced_status: "fixed"
|
|
gitea_synced_hash: "735fa9567f4e"
|
|
---
|
|
|
|
# bug-1923: xce_scheduler --sim drops shared resource values from the snapshot (license shows value=(none))
|
|
|
|
## Description
|
|
**Observed:** a shared (cluster-wide, `GLOBAL|SHARED|CONSUMABLE`) resource value that is present in a recording's `SnapshotEntry` is not loaded by `xce_scheduler --sim`. `info resource license`, after `step` or straight after load, shows:
|
|
```
|
|
[sim] resource license: desc= valtype=numeric ordtype=incr flags=GLOBAL,SHARED,CONSUMABLE interval=0
|
|
[sim] value=(none)
|
|
[sim] DISPATCH jobid=0-0-0-101
|
|
[sim] PEND jobid=0-0-0-102
|
|
[sim] PEND jobid=0-0-0-103
|
|
[sim] PEND jobid=0-0-0-104
|
|
```
|
|
When the same value arrives as a `RESVALS_FULL` replay event, it loads (`value=2.00 used=2.00`), and jobs 102 and 103 dispatch.
|
|
|
|
**Expected:** every shared resource value in the snapshot is loaded into `core->sharedInst`, so the simulator starts with the same shared pool the recorder had.
|
|
|
|
**Impact:** wrong result in the simulator. A replay whose shared resource values were set before recording started, and never re-sent as a `RESVALS_FULL` event during the recording, simulates with no license pool. Jobs needing the resource pend instead of dispatching, so a sim replay diverges from what live did. Nothing is logged when the value is dropped.
|
|
|
|
**Found by:** spec task scheduler-sim-decision-trace 1.1, while building the trace-off REPL baseline fixture (`src/scheduler/testsuite/test_repl_trace_baseline.c`). That fixture now sends the license value as a `RESVALS_FULL` event to work around this bug.
|
|
|
|
Two separate defects were found (see Reproduction, variants A and C):
|
|
- **Read side, confirmed with gdb and a decoder:** `snapshot_load.c:129` skips the entry when `rv->host != NULL`. After the capnp round trip, a host that was written as NULL decodes as `""`, so every shared resval in a snapshot is skipped and line 140's `zarray_put` is never reached. The `RESVALS_FULL` path (`core_events.c:1124-1147`) has no such host check.
|
|
- **Write side, Suspected (from code reading, matches the decoder output):** `snapshot.c:431` loops `for (i = 0; i < core->sharedInst.count; i++)`, but `zarray_put_ex` (`src/lib/zarray.c:291-330`) counts filled slots, not the highest index plus one. A sparse `sharedInst` therefore drops values at higher indices: with the license at idx 2 and `count == 1`, the snapshot carries `resvals count=0`. The same pattern may affect per-worker resvals at `snapshot_wire.c:242` (`w->resvals.count`); not checked.
|
|
|
|
## Environment
|
|
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
|
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: asan (xce.build)
|
|
- Code: scheduler-framework @ ed201686-dirty
|
|
- Deployment: none (offline `xce_scheduler --sim <replay> --interactive`; the replay is synthetic, written by a test generator with 2 virtual workers)
|
|
- Note: `-dirty` = only the uncommitted task 1.1 test files and unrelated .wolf/ bookkeeping; no production scheduler source was changed
|
|
|
|
## Reproduction
|
|
Durable copies are in `/home/claude/tmp/bug-1923/`: `fixA/contention.replay` has the value in the snapshot (license idx 2); `fixC/contention.replay` is the same but the license is idx 0; `fixB/contention.replay` uses the `RESVALS_FULL` workaround.
|
|
1. Build the scheduler: `bash src/tools/xce-build.sh scheduler` (ASAN, `src/xce.build`).
|
|
2. `cd "$(mktemp -d)" && cp /home/claude/tmp/bug-1923/fixA/contention.replay .`
|
|
3. `printf 'info resource license\nstep\ninfo resource license\nquit\n' | ASAN_OPTIONS=detect_leaks=0 TERM=dumb /home/rshen/projects/xce/scheduler-framework/src/xce.build/scheduler/xce_scheduler --sim contention.replay --interactive --logfile sched.log`
|
|
4. Both `info resource license` calls print `value=(none)`, and jobs 102 and 103 PEND. Repeat with `fixC` (same result: the read-side skip alone is enough) and `fixB` (the first call prints `value=(none)` because the event is applied in round 1; after `step` it prints `value=2.00 used=2.00` and jobs 102 and 103 dispatch).
|
|
- How the replays are generated: `/home/claude/tmp/bug-1923/gen.c` is a copy of the test's `write_contention_replay()`, with `-DSNAPSHOT_LICENSE` (A) or `-DLICENSE_FIRST` (C). The value is put in with `zarray_put(&sched.core.sharedInst, lic->idx, val)` before `sched_record_start()`. `build_and_run.sh [N]` rebuilds and reruns all three; it needs the scratch object copies, which were not kept. `decode.c` dumps a replay's `SnapshotEntry` resvals: A shows `count=0`, C shows `resval name=license host="" = 2.000000`.
|
|
- Frequency: deterministic (A 20/20 and C 20/20 show `value=(none)`; B 0/20; 2026-10-08)
|
|
- Evidence: `/home/claude/tmp/bug-1923/` (replays, generator, decoder, probe script)
|
|
|
|
## Root cause
|
|
Two separate defects on the SnapshotEntry save/load path, both pre-existing (origin unknown):
|
|
- **Read side, `src/scheduler/src/snapshot_load.c:129`:** the shared-pool loop skipped any resval with `rv->host != NULL`. The writer stores shared values with `host = NULL`, but after the capnp round trip (`write_snapshot_entry()` in recording.c → `.replay` → `read_SnapshotEntry`/`decode_SnapshotEntry` in sim.c) a NULL host decodes as `""`. So every shared value in a snapshot read from a file was silently skipped, and `core->sharedInst` started empty. The `RESVALS_FULL` event path (`core_events.c:1124-1147`) has no host check, which is why the workaround loaded. A non-empty host only appears on `RESVALS_DELTA` (routing a value to one worker).
|
|
- **Write side, `src/scheduler/src/snapshot.c:431`:** the shared-pool loop was bounded by `core->sharedInst.count`. `zarray_put_ex` (`src/lib/zarray.c`) counts filled slots, not highest index + 1, so in a sparse `sharedInst` the values at higher indices were never written. In the baseline fixture, `license` (idx 2) and `limit=cluster_slots` were dropped.
|
|
- **Same defect, per worker, `src/scheduler/src/snapshot_wire.c:242`:** the loop over `w->resvals` was bounded by `w->resvals.count`. A worker never holds a value at a global resource's idx, so its resvals always have holes, and `limit=host_slots` was dropped for every worker.
|
|
- Scope checked: no other `.count`-bounded loop over a sparse resval zarray remains in src/scheduler or src/lib. `repl_info.c:524` and `resource.c:262` already use `.caps`; the other `.count` uses are dense lists or log lines.
|
|
|
|
## Fix
|
|
- `snapshot_load.c`: a resval with a NULL **or empty** host is a shared-pool entry; only a named host is skipped.
|
|
- `snapshot.c`: the shared-pool loop is bounded by `sharedInst.caps` (the project convention for sparse resval arrays). Empty slots were already skipped via `zarray_get` returning NULL.
|
|
- `snapshot_wire.c`: the per-worker loop is bounded by `w->resvals.caps`.
|
|
- These changes make the shared and per-worker values round-trip through both the write side and the load seam (persistence.md, snapshot field-completeness).
|
|
- Compatibility: replay files written by the old writer lack the dropped values, so the reader fix alone can't recover them (`/home/claude/tmp/bug-1923/fixA` still shows `value=(none)`). Files written from now on carry them.
|
|
- Fixture: `src/scheduler/testsuite/fixtures/repl_trace_baseline/set_output.replay` was regenerated. Only its SnapshotEntry changed (+304 bytes: `license`, `limit=cluster_slots` and two per-worker `limit=host_slots` values). The header differs only in the ignored wall-clock field, the transcript, `sim_output.replay` and verify output are unchanged, and messages 2-15 are byte-identical.
|
|
|
|
## Verification
|
|
- Test: new `src/scheduler/testsuite/test_snapshot_sparse_resvals.c` ("scheduler snapshot sparse resvals test"), which links the real scheduler sources. RED before the fix: 6 Tests 5 Failures (`test_save_keeps_sparse_shared_resval`, `test_save_keeps_sparse_worker_resval`, `test_save_load_round_trip_keeps_sparse_resvals`, `test_load_takes_shared_resval_with_empty_host`, and `test_replay_round_trip_keeps_sparse_resvals`, which goes through the real `.replay` writer and the `--sim` start-up reader). The guard test `test_load_skips_resval_with_named_host` passed. GREEN after: 6 Tests 0 Failures.
|
|
- Suites: scheduler meson suite Ok 100 / Fail 0, asan (`src/xce.build` in the fix worktree); `scheduler repl trace baseline test` OK with the regenerated `set_output.replay`.
|
|
- Live: `xce_scheduler --sim /home/claude/tmp/bug-1923/fixC/contention.replay --interactive` on the fixed asan build: `info resource license` shows `value=2.00 used=0.00`, then `value=2.00 used=2.00` after `step`, and jobs 102 and 103 DISPATCH (before the fix: `value=(none)`, and both PEND). A variant-A replay regenerated with the fixed writer also loads `value=2.00`. No HA soak was needed: the change is limited to the scheduler's snapshot save/load and does not touch the controller or WAL.
|
|
|
|
## References
|
|
- Related: bug-1568 (same `snapshot_wire.c` worker-resval loop; that bug added the missing-resource diagnostic)
|
|
- Docs: .kiro/steering/persistence.md (Snapshot Save/Load Field-Completeness)
|
|
- Found by: .kiro/specs/scheduler-sim-decision-trace task 1.1
|