--- id: "bug-1880" status: "fixed" status_updated: "2026-10-02" status_note: "" component: "scheduler" severity: "wrong-result" timestamp: "2026-09-28T07:26:08Z" last_seen: "2026-09-28T07:26:08Z" occurrences: 1 files: ["src/scheduler/src/sim_verify.c"] tags: ["scheduler", "sim", "sim-verify", "replay", "array-job"] related: ["bug-1842", "bug-1844", "bug-1845"] fix_commits: ["077b1213"] gitea_issue: 331 gitea_synced_status: "fixed" gitea_synced_hash: "9f950f5346ec" --- # bug-1880: sim-verify flags every DISPATCH of array job 0-12-1790578108-0 as MISMATCH in a pc13v replay ## Description **Observed:** ``` [2026-09-28 06:50:11,909] [93231] [ ERROR] [ sim_log_mismatch] sim_log_mismatch decision verification MISMATCH at begin_round=1790578211909717418 decision_index=0 expected_ts=1790578211910119659 -- expected: DISPATCH jobid=0-12-1790578108-0[7] nhosts=1 -- actual: DISPATCH jobid=0-12-1790578108-0[7] nhosts=1 ... (one per dispatch, elements [7]..[49], through 07:00:21) ``` **Expected:** the simulator reproduces the recorded dispatches exactly, as for the other jobs in the capture. **Impact:** sim-verify reports every dispatch of this array job as divergent, so the replay cannot be used to check scheduler parity; either the simulator allocates differently from the live scheduler (a real parity bug) or the verifier compares a field that legitimately differs. **Found by:** user replaying a pc13v capture with sim-verify, 2026-09-28 ## Environment - OS: NixOS 26.05 (Yarara); NixOS kernel; kernel Linux 6.18.33 x86_64 - libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) - Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 - Build: debug (build.dbg) - Code: scheduler-framework @ 4c134127 (`xce_scheduler --version`: v0.0.0.1-4c134127-85e8fc00, built 2026-09-28T06:40:21Z) - Deployment: pc13v: xce_scheduler in simulation mode (pid 93231) replaying a capture of the live pc13v cluster's scheduler (scheduler-replay-20260928-064842.replay, captured from 06:48:42); exact command line and --sim-start-ts not recorded ## Reproduction 1. `xce_scheduler -s data/scheduler-replay-20260928-064842.replay --sim-verify --logfile ` with the default start (the file's only snapshot, taken at capture start 06:48:42, before the array's dispatches -- the earlier note that the default start skips them was wrong). 2. Current code (2026-10-01): the log shows, for every element of 0-12-1790578108-0 at snapshot load, `parse_resreq failed for jobid ...: Primary:\n Tasks: min = 3, max = 3\n Rusages: ...` and `has no scheduling object; it will not be scheduled`, then 4848 `MISMATCH ... expected: PEND|DISPATCH jobid=0-12-1790578108-0[k] ... -- actual: -- diff: no live decision` (4805 PEND, 43 DISPATCH). Mismatches go only to the log file, not the console. - Frequency: deterministic (2/2 runs, 2026-10-01) - Evidence: data/scheduler-replay-20260928-064842.replay; /home/claude/tmp/bug-1880/old-capture-parse-failure.txt ## Root cause The capture's snapshot was written by the pc13v scheduler at 4c134127, before bug-1888's fix: snapshot_wire.c saved each job's resreq as format_resreq()'s human-readable dump ("Primary:\n Tasks: min = 3, max = 3\n Rusages: res mem ..."), not the text the request was parsed from. On load, the scheduler of that time accepted the dump as a primary request with neither select nor rusage, so the simulator allocated this array (3 tasks, rusage[mem]) differently from the live scheduler -- same kind/jobid/nhosts, different per-host allocations: the original report. With today's loader the dump does not parse and bug-1897's guard leaves the job unscheduled, so the same capture now shows "no live decision" instead. Only jobs present in the snapshot are affected; jobs submitted after it arrive as events with their real resreq text. ## Fix 077b1213 (bug-1888, 2026-09-29): snapshots save the job's resreq text (`job->resreq_str`), so the loader rebuilds the same request. No further change: a capture written before 077b1213 cannot be repaired after the fact (its snapshot does not hold the request text) and has to be recorded again with a fixed scheduler. ## Verification - Test: `test_snapshot_save_load.c` resreq round trip (added by 077b1213); the old capture as control: 4848 MISMATCH and 50 parse failures with the current binary - Suites: scheduler meson suite green at 077b1213 and on the current branch (asan, src/build.jai) - Live: 2026-10-01, xce-deploy-ha + xcecli, scheduler started with --record-dir, `xceadmin scheduler record start` while a 20-element array (`-n 3,3 -R 'rusage[mem=400]'`) had 15 elements PEND and 5 RUN (so the jobs are in the snapshot), recording stopped after more dispatches; `xce_scheduler -vvv -s --sim-verify`: 402 simulated decisions, 0 MISMATCH, 0 parse_resreq failures (7956-line log; every simulated decision is compared with the recorded one at the same position and leftovers are reported). A second capture with the jobs submitted after the snapshot: 311 decisions, 0 MISMATCH. Captures: /home/claude/tmp/bug-1880/ ## References - Related: bug-1881 (the message hides the differing field), bug-1842, bug-1844, bug-1845 (earlier --sim-verify false positives); bug-1888 (the snapshot writer, the root cause), bug-1897 (the loader guard behind "no live decision")