chore: wrap-up -- spec phase, roadmap ticks, Gitea sync

- job-array-one-based-index spec.json phase -> implementation-complete
- roadmap: job-array-one-based-index entry updated (validated, bugs
  fixed); 7 implemented specs ticked (component-failover-adoption,
  config-upgrade-snapshot, ha-term-safety, job-array-single-wal-record,
  raft-sim-conformance, scheduler-sim-diagnostics, worker-rejoin-resync)
- Gitea #89 (bug-1856, wontfix), #331 (bug-1880), #335 (bug-1884) closed;
  records carry the synced status

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-10-02 08:57:39 +00:00

5.2 KiB

Raw Blame History

id status status_updated status_note component severity timestamp last_seen occurrences files tags related fix_commits gitea_issue gitea_synced_status gitea_synced_hash

bug-1880

fixed

2026-10-02

scheduler

wrong-result

2026-09-28T07:26:08Z

src/scheduler/src/sim_verify.c

scheduler

sim

sim-verify

replay

array-job

bug-1842

bug-1844

bug-1845

077b1213

331

fixed

9f950f5346ec

bug-1880: sim-verify flags every DISPATCH of array job 0-12-1790578108-0 as MISMATCH in a pc13v replay

Description

Observed:

[2026-09-28 06:50:11,909] [93231] [ ERROR] [      sim_log_mismatch] sim_log_mismatch decision verification MISMATCH at begin_round=1790578211909717418 decision_index=0 expected_ts=1790578211910119659 -- expected: DISPATCH jobid=0-12-1790578108-0[7] nhosts=1 -- actual: DISPATCH jobid=0-12-1790578108-0[7] nhosts=1
... (one per dispatch, elements [7]..[49], through 07:00:21)

Expected: the simulator reproduces the recorded dispatches exactly, as for the other jobs in the capture. Impact: sim-verify reports every dispatch of this array job as divergent, so the replay cannot be used to check scheduler parity; either the simulator allocates differently from the live scheduler (a real parity bug) or the verifier compares a field that legitimately differs. Found by: user replaying a pc13v capture with sim-verify, 2026-09-28

Environment

OS: NixOS 26.05 (Yarara); NixOS kernel; kernel Linux 6.18.33 x86_64
libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
Build: debug (build.dbg)
Code: scheduler-framework @ 4c134127 (xce_scheduler --version: v0.0.0.1-4c134127-85e8fc00, built 2026-09-28T06:40:21Z)
Deployment: pc13v: xce_scheduler in simulation mode (pid 93231) replaying a capture of the live pc13v cluster's scheduler (scheduler-replay-20260928-064842.replay, captured from 06:48:42); exact command line and --sim-start-ts not recorded

Reproduction

xce_scheduler -s data/scheduler-replay-20260928-064842.replay --sim-verify --logfile <log> with the default start (the file's only snapshot, taken at capture start 06:48:42, before the array's dispatches -- the earlier note that the default start skips them was wrong).
Current code (2026-10-01): the log shows, for every element of 0-12-1790578108-0 at snapshot load, parse_resreq failed for jobid ...: Primary:\n Tasks: min = 3, max = 3\n Rusages: ... and has no scheduling object; it will not be scheduled, then 4848 MISMATCH ... expected: PEND|DISPATCH jobid=0-12-1790578108-0[k] ... -- actual: <none> -- diff: no live decision (4805 PEND, 43 DISPATCH). Mismatches go only to the log file, not the console.

Frequency: deterministic (2/2 runs, 2026-10-01)
Evidence: data/scheduler-replay-20260928-064842.replay; /home/claude/tmp/bug-1880/old-capture-parse-failure.txt

Root cause

The capture's snapshot was written by the pc13v scheduler at 4c134127, before bug-1888's fix: snapshot_wire.c saved each job's resreq as format_resreq()'s human-readable dump ("Primary:\n Tasks: min = 3, max = 3\n Rusages: res mem ..."), not the text the request was parsed from. On load, the scheduler of that time accepted the dump as a primary request with neither select nor rusage, so the simulator allocated this array (3 tasks, rusage[mem]) differently from the live scheduler -- same kind/jobid/nhosts, different per-host allocations: the original report. With today's loader the dump does not parse and bug-1897's guard leaves the job unscheduled, so the same capture now shows "no live decision" instead. Only jobs present in the snapshot are affected; jobs submitted after it arrive as events with their real resreq text.

Fix

077b1213 (bug-1888, 2026-09-29): snapshots save the job's resreq text (job->resreq_str), so the loader rebuilds the same request. No further change: a capture written before 077b1213 cannot be repaired after the fact (its snapshot does not hold the request text) and has to be recorded again with a fixed scheduler.

Verification

Test: test_snapshot_save_load.c resreq round trip (added by 077b1213); the old capture as control: 4848 MISMATCH and 50 parse failures with the current binary
Suites: scheduler meson suite green at 077b1213 and on the current branch (asan, src/build.jai)
Live: 2026-10-01, xce-deploy-ha + xcecli, scheduler started with --record-dir, xceadmin scheduler record start while a 20-element array (-n 3,3 -R 'rusage[mem=400]') had 15 elements PEND and 5 RUN (so the jobs are in the snapshot), recording stopped after more dispatches; xce_scheduler -vvv -s <capture> --sim-verify: 402 simulated decisions, 0 MISMATCH, 0 parse_resreq failures (7956-line log; every simulated decision is compared with the recorded one at the same position and leftovers are reported). A second capture with the jobs submitted after the snapshot: 311 decisions, 0 MISMATCH. Captures: /home/claude/tmp/bug-1880/

References

Related: bug-1881 (the message hides the differing field), bug-1842, bug-1844, bug-1845 (earlier --sim-verify false positives); bug-1888 (the snapshot writer, the root cause), bug-1897 (the loader guard behind "no live decision")

5.2 KiB Raw Blame History