myxce/.wolf/bugs/bug-1881.md

3.5 KiB

id status status_updated status_note component severity timestamp last_seen occurrences files tags related
bug-1881 partial 2026-09-28 mismatch lines now carry a first-difference diff (24b05718); live default-speed replay proof waits on spec scheduler-sim-discrete-clock (task 5.3) scheduler wrong-result 2026-09-28T07:26:09Z 2026-09-28T07:26:09Z 1
src/scheduler/src/sim_verify.c
scheduler
sim
sim-verify
logging

bug-1881: sim_log_mismatch prints identical expected/actual, hiding the field that differs

Description

Observed:

... MISMATCH ... -- expected: DISPATCH jobid=0-12-1790578108-0[49] nhosts=1 -- actual: DISPATCH jobid=0-12-1790578108-0[49] nhosts=1

Expected: the message names the field(s) that differ, with both values. Impact: a mismatch cannot be diagnosed from the log (bug-1880: 43 such lines, none showing the difference). Found by: user reading the bug-1880 replay log, 2026-09-28

UPDATE (2026-09-28): Fix planned in spec libxce-alloc-preason-equality (Requirement 3, Requirement 5.3, task 5): the libxce equality functions gain an optional why/whylen buffer describing the first difference, and sim_log_mismatch appends it to the MISMATCH line.

Environment

  • OS: NixOS 26.05 (Yarara); NixOS kernel; kernel Linux 6.18.33 x86_64
  • libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
  • Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
  • Build: debug (build.dbg)
  • Code: scheduler-framework @ 4c134127 (xce_scheduler --version: v0.0.0.1-4c134127-85e8fc00, built 2026-09-28T06:40:21Z)
  • Deployment: pc13v: xce_scheduler in simulation mode (pid 93231) replaying a capture of the live pc13v cluster's scheduler (scheduler-replay-20260928-064842.replay, captured from 06:48:42); exact command line and --sim-start-ts not recorded

Reproduction

  1. Any --sim-verify mismatch whose decisions differ only below the jobid/nhosts level (bug-1880's replay)
  • Frequency: deterministic (43/43 lines in the pc13v run)
  • Evidence: data/xce_scheduler.log

Root cause

sim_log_mismatch() (src/scheduler/src/sim_verify.c) summarizes each decision as kind + jobid + nhosts, while sim_decision_eq() also compares every host's allocations (resname, value, duration, reserve), so a difference there prints two identical summaries.

Fix

Spec libxce-alloc-preason-equality (commits 8f25c1c3..dcf3acf3). sim_decision_eq() now compares DISPATCH decisions with libxce's public job_alloc_eq() and returns a caller-owned first-difference description (char** why) naming the field path and both values; sim_log_mismatch() appends it as -- diff: <why>, and the replay loop in sim_loop.c passes and frees it (fixed text "no live decision" for leftover recorded decisions) — 24b05718. job_alloc_eq() also compares from, so the scheduler now stamps every allocation's from with the deciding session's start (scheduler.c sched_make_primary_alloc_, c1960457) and the normal-mode recorder takes beginRound from the same reading (driver.c, 2d7ba660), making from reproducible on replay.

Remaining: the simulator's own session-begin clock read (sim_loop.c) still drifts at --sim-speed >= 1, so a live default-settings replay cannot yet be shown to report zero from mismatches. That is task 5.3, blocked on spec scheduler-sim-discrete-clock.

Verification

  • Test: TODO: name + file; RED before the fix, GREEN after
  • Suites: TODO: suite results and build profile
  • Live: TODO: soak/deploy run, or n/a and why

References

  • Related: bug-1880