Rongsong Shen
674b29c951
Specs: scheduler-allocator-runner (#67: 317-319), scheduler-sim-decision-trace (#68: 320, 324, 325), job-status-detail-display (#69: 321-323), scheduler-sim-discrete-clock (#70: 326-328). Bugs: #329-#333. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
60 lines
4.5 KiB
Markdown
60 lines
4.5 KiB
Markdown
---
|
|
id: "bug-1881"
|
|
status: "fixed"
|
|
status_updated: "2026-09-29"
|
|
status_note: ""
|
|
component: "scheduler"
|
|
severity: "wrong-result"
|
|
timestamp: "2026-09-28T07:26:09Z"
|
|
last_seen: "2026-09-28T07:26:09Z"
|
|
occurrences: 1
|
|
files: ["src/scheduler/src/sim_verify.c"]
|
|
tags: ["scheduler", "sim", "sim-verify", "logging"]
|
|
related: []
|
|
fix_commits: ["24b05718", "c1960457", "2d7ba660", "4e57bde6"]
|
|
gitea_issue: 332
|
|
gitea_synced_status: "fixed"
|
|
gitea_synced_hash: "e7258ca9149c"
|
|
---
|
|
|
|
# bug-1881: sim_log_mismatch prints identical expected/actual, hiding the field that differs
|
|
|
|
## Description
|
|
**Observed:**
|
|
```
|
|
... MISMATCH ... -- expected: DISPATCH jobid=0-12-1790578108-0[49] nhosts=1 -- actual: DISPATCH jobid=0-12-1790578108-0[49] nhosts=1
|
|
```
|
|
**Expected:** the message names the field(s) that differ, with both values.
|
|
**Impact:** a mismatch cannot be diagnosed from the log (bug-1880: 43 such lines, none showing the difference).
|
|
**Found by:** user reading the bug-1880 replay log, 2026-09-28
|
|
|
|
UPDATE (2026-09-28): Fix planned in spec libxce-alloc-preason-equality (Requirement 3, Requirement 5.3, task 5): the libxce equality functions gain an optional why/whylen buffer describing the first difference, and sim_log_mismatch appends it to the MISMATCH line.
|
|
|
|
## Environment
|
|
- OS: NixOS 26.05 (Yarara); NixOS kernel; kernel Linux 6.18.33 x86_64
|
|
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: debug (build.dbg)
|
|
- Code: scheduler-framework @ 4c134127 (`xce_scheduler --version`: v0.0.0.1-4c134127-85e8fc00, built 2026-09-28T06:40:21Z)
|
|
- Deployment: pc13v: xce_scheduler in simulation mode (pid 93231) replaying a capture of the live pc13v cluster's scheduler (scheduler-replay-20260928-064842.replay, captured from 06:48:42); exact command line and --sim-start-ts not recorded
|
|
|
|
## Reproduction
|
|
1. Any --sim-verify mismatch whose decisions differ only below the jobid/nhosts level (bug-1880's replay)
|
|
- Frequency: deterministic (43/43 lines in the pc13v run)
|
|
- Evidence: data/xce_scheduler.log
|
|
|
|
## Root cause
|
|
sim_log_mismatch() (src/scheduler/src/sim_verify.c) summarizes each decision as kind + jobid + nhosts, while sim_decision_eq() also compares every host's allocations (resname, value, duration, reserve), so a difference there prints two identical summaries.
|
|
|
|
## Fix
|
|
Spec libxce-alloc-preason-equality (commits 8f25c1c3..dcf3acf3). sim_decision_eq() now compares DISPATCH decisions with libxce's public job_alloc_eq() and returns a caller-owned first-difference description (`char** why`) naming the field path and both values; sim_log_mismatch() appends it as ` -- diff: <why>`, and the replay loop in sim_loop.c passes and frees it (fixed text "no live decision" for leftover recorded decisions) — 24b05718. job_alloc_eq() also compares `from`, so the scheduler now stamps every allocation's `from` with the deciding session's start (scheduler.c sched_make_primary_alloc_, c1960457) and the normal-mode recorder takes beginRound from the same reading (driver.c, 2d7ba660), making `from` reproducible on replay.
|
|
|
|
The remaining gap (the simulator's own session-begin clock read drifting at --sim-speed >= 1) was closed by spec scheduler-sim-discrete-clock (4f653d8f..d9bc36cd: the simulator clock is pinned to recorded timestamps, --sim-speed is a no-op), so a replayed session starts at exactly the recorded beginRound and every replayed `from` matches; task 5.3 adds the proof (4e57bde6).
|
|
|
|
## Verification
|
|
- Test: test_log_mismatch_appends_provided_description and test_verify_against_altered_copy_reports_exact_mismatch (asserts ` -- diff: main_reason: 99 != 0`) plus test_verify_leftover_recorded_decision_names_missing_side in src/scheduler/testsuite/test_sim_verify.c: RED before 24b05718 (no diff text on the MISMATCH line), GREEN after. test_sim_replay_reproduces_recorded_alloc_from in src/scheduler/testsuite/test_sim_live_allocation.c (4e57bde6): GREEN at default settings; a +0.5 s clock mutant at sim_loop.c's sim_advance_virtual_clock(sim, begin_round) makes it FAIL
|
|
- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 Unity :FAIL lines, 0 sanitizer reports (2026-09-29)
|
|
- Live: 2-round sequential HA soak (/home/claude/tmp/xce-alloceq-53, PASS, zero job loss) recorded 3 clean .replay files with 11 DISPATCH decisions; each replayed with `xce_scheduler --sim --sim-verify` and no --sim-speed (ASAN_OPTIONS=detect_leaks=0): rc=0, 0 MISMATCH lines
|
|
|
|
## References
|
|
- Related: bug-1880
|