chore(buglog): log bug-1880 (sim-verify mismatch on array job dispatches) and bug-1881 (mismatch message hides the diff)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-09-28 07:26:50 +00:00

3.3 KiB

Raw Blame History

status

status_updated

status_note

component

severity

timestamp

last_seen

occurrences

files

bug-1880: sim-verify flags every DISPATCH of array job 0-12-1790578108-0 as MISMATCH in a pc13v replay

Description

Observed:

[2026-09-28 06:50:11,909] [93231] [ ERROR] [      sim_log_mismatch] sim_log_mismatch decision verification MISMATCH at begin_round=1790578211909717418 decision_index=0 expected_ts=1790578211910119659 -- expected: DISPATCH jobid=0-12-1790578108-0[7] nhosts=1 -- actual: DISPATCH jobid=0-12-1790578108-0[7] nhosts=1
... (one per dispatch, elements [7]..[49], through 07:00:21)

Expected: the simulator reproduces the recorded dispatches exactly, as for the other jobs in the capture. Impact: sim-verify reports every dispatch of this array job as divergent, so the replay cannot be used to check scheduler parity; either the simulator allocates differently from the live scheduler (a real parity bug) or the verifier compares a field that legitimately differs. Found by: user replaying a pc13v capture with sim-verify, 2026-09-28

Environment

OS: NixOS 26.05 (Yarara); NixOS kernel; kernel Linux 6.18.33 x86_64
libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
Build: debug (build.dbg)
Code: scheduler-framework @ 4c134127 (xce_scheduler --version: v0.0.0.1-4c134127-85e8fc00, built 2026-09-28T06:40:21Z)
Deployment: pc13v: xce_scheduler in simulation mode (pid 93231) replaying a capture of the live pc13v cluster's scheduler (scheduler-replay-20260928-064842.replay, captured from 06:48:42); exact command line and --sim-start-ts not recorded

Reproduction

xce_scheduler -s data/scheduler-replay-20260928-064842.replay --sim-verify starting from a snapshot BEFORE the array job's first dispatch at 06:50:11 (the default start, the replay's most recent snapshot, is after them)

Frequency: 43/43 dispatches of array job 0-12-1790578108-0 (elements [7]..[49]) in the pc13v run; 0/3 here with the default start (those rounds are never replayed); an arbitrary --sim-start-ts (1790578000000000000) is rejected -- the right snapshot is not known yet
Evidence: data/scheduler-replay-20260928-064842.replay and data/xce_scheduler.log (ERROR lines only), copied from pc13v 2026-09-28

Root cause

Unconfirmed: the printed fields (kind, jobid, nhosts) are equal, so the difference is in what sim_decision_eq() (src/scheduler/src/sim_verify.c) compares beyond them -- per host, the allocation's resname, value, duration and reserve (sim_verify_alloc_eq/host_alloc_eq/job_alloc_eq). bug-1881 makes the message show which.

Fix

TODO: required for fixed/partial

Verification

Test: TODO: name + file; RED before the fix, GREEN after
Suites: TODO: suite results and build profile
Live: TODO: soak/deploy run, or n/a and why

References

Related: bug-1881 (the message hides the differing field), bug-1842, bug-1844, bug-1845 (earlier --sim-verify false positives)

3.3 KiB Raw Blame History