Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
41 lines
8.3 KiB
Markdown
41 lines
8.3 KiB
Markdown
---
|
|
id: "bug-1836"
|
|
status: "fixed"
|
|
status_updated: "2026-09-24"
|
|
status_note: ""
|
|
component: "scheduler"
|
|
severity: "unknown"
|
|
timestamp: "2026-09-20T22:00:00Z"
|
|
last_seen: "2026-09-21T05:46:00Z"
|
|
occurrences: 1
|
|
files: ["src/scheduler/src/sim_loop.c"]
|
|
tags: ["scheduler", "sim", "replay", "resource-accounting", "sim_reconcile_unaccounted_jobs", "scheduler-driver-equivalence", "over-commit", "slots", "capacity-check", "xce-audit-invariants", "user-reported", "found-live"]
|
|
related: ["bug-1611", "bug-1613", "bug-1799", "bug-1764", "bug-1759", "bug-1779"]
|
|
legacy: true
|
|
gitea_issue: 65
|
|
gitea_synced_status: "fixed"
|
|
---
|
|
|
|
# bug-1836: User-reported live over-dispatch in xce_scheduler -i -s <replay>: replaying a captured session…
|
|
|
|
## Description
|
|
User-reported live over-dispatch in xce_scheduler -i -s <replay>: replaying a captured session (JOB_PUSH of a 50-element array job) against a real single worker with 22 slots dispatched 44 concurrent single-slot jobs (`assigned jobs: 0-3-1789963203-0[0-43]`), exactly double the host's real capacity, while `info worker` simultaneously showed `resource slots: value=22.00 used=22.00` -- the accounting display looked self-consistent (used == value) even though double the job count the display's own 'used' figure implied was actually running. User confirmed normal mode (the live controller-connected driver) does not exhibit this under equivalent conditions, isolating the defect to simulation/replay mode specifically.
|
|
|
|
## Environment
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Reproduction
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Root cause
|
|
sim_reconcile_unaccounted_jobs() (the 'Recorded-Decision Reconciliation' mechanism, scheduler-sim-decision-parity/scheduler-sim-eager-reconciliation specs) unconditionally force-applied any recorded DISPATCH decision it found in decoded_session->decisions for a job with accounts==NULL -- setting job->state=JOB_STAT_RUN and job->allocs/job->accounts straight from the recording -- with ZERO call to sched_check_host_satisfy()/sched_check_shared_satisfy() or any other admission/capacity check. This directly contradicted the scheduler-sim-decision-parity spec's own design.md assumption underpinning its Requirement 2 ('capacity awareness'): 'sched_check_host_satisfy_()/sched_check_shared_satisfy_() already read resInUse/sharedInUse unconditionally; once those fields are correctly populated in simulation mode, the existing (unmodified) capacity-check code path produces the same outcome normal mode would' -- an assumption that only holds for jobs decided via the LIVE picker/allocator path (sched_election.c's sched_allocator(), which DOES call these checks via sched_try_alloc_on_host_()), not for jobs captured via this SEPARATE reconciliation backstop, which bypasses that code path entirely. Confirmed via the round sequence in sched_sim_driver_run(): xce_core_push_events() creates all 50 freshly-pushed jobs (PEND, accounts==NULL) BEFORE sim_reconcile_unaccounted_jobs() runs for that same round, so ALL 50 are eligible reconciliation candidates the moment the function starts -- for however many of them (44, in the live report) the session's OWN originally-recorded decisions said DISPATCH, reconciliation force-applied all of them with no regard for the CURRENT sim snapshot's actual worker capacity (22 slots), before the live picker/allocator (which runs afterward, per the fixed round order) ever got a chance to independently admission-check any of them -- explaining both the 44 dispatches and the 6 correctly-PENDed remainder (jobs whose OWN recorded decision was PEND, never matched by reconciliation's ReplayDecision_body_dispatch-only filter). This is a distinct mechanism from bug-1611/bug-1613 (both already fixed/superseded by scheduler-sim-live-allocation and scheduler-sim-eager-reconciliation) -- those closed a TIMING gap in when reconciliation's job->state transition happens; this bug is a CAPACITY-VALIDATION gap in whether it happens at all, independent of timing. Uncovered via /xce-audit-invariants scheduler, dimension 1 (driver state-equivalence).
|
|
|
|
## Fix
|
|
FIXED (2026-09-21). Added sim_reconciled_alloc_fits_capacity() (sim_loop.c, static) -- re-validates a candidate recorded allocation's full resource footprint (every per-host allocation plus any shared/global allocation) against LIVE resInUse/sharedInUse using the SAME sched_check_host_satisfy()/sched_check_shared_satisfy() admission-check functions (resource.c) the live picker/allocator path already calls, resolving each allocation's resnames against core->resmgr fresh (the decoded allocation carries only resnames/values, not resolved indices). sim_reconcile_unaccounted_jobs()'s FOUND branch now calls this BEFORE taking ownership of a matched entry; a capacity miss is handled exactly like 'no matching recorded decision' (job left unaccounted this round, entry NOT consumed/freed, naturally retried later or correctly PENDed by the live picker once it runs). A newly-admitted candidate's usage is charged into the LIVE resInUse/sharedInUse immediately via sched_charge_account() (mirroring sched_commit_decision()'s own immediate charge for a live dispatch) so a LATER candidate in the SAME reconciliation pass correctly sees an EARLIER accepted candidate's usage -- essential for the exact multi-job-contention scenario this bug exercises. To make the capacity check see genuinely prior-round (already-running) jobs' usage too, added sim_reconcile_toggle_prior_charges() (sim_loop.c, static) -- seeds resInUse/sharedInUse from every already-RUN+accounted job at the start of sim_reconcile_unaccounted_jobs(), then fully undoes that seed plus every incremental charge made during the pass at the end, leaving the function's net effect on the real accounting arrays at exactly zero on return; the caller's own unchanged, immediately-following sched_account_jobs() call remains the sole, non-double-counting authoritative charge for the round (unchanged architecture/ordering). Deliberately NOT implemented via sched_account_jobs()/a new sched_unaccount_jobs() exported pair (tried first, then reverted): that exact by-name function is separately instrumented and call-count-asserted by test_sim_contention_parity.c's g_round_idx mechanism ('sched_account_jobs() called exactly once per round' is itself a protected invariant there) -- doubling its call count broke 2 of that file's tests (masked at first by a coincidental g_round_resinuse[8] array-capacity cap that hid the true +5/+10 call-count delta behind an apparent '+2' pattern; properly diagnosed by reading the array declaration, not guessed). sched_check_host_satisfy()/sched_check_shared_satisfy() were relocated from scheduler.h to core.h (pure declaration move, bodies unchanged in resource.c, following the exact sched_account_jobs()/sched_resolve_job_picker() precedent already established for this exact purpose) since sim_loop.c must never include scheduler.h (pulls in client.h/xce/cluster.h, forbidden by test_sim_isolation.c's Req 5 AC 17 enforcement). New regression test test_reconcile_respects_host_capacity_under_contention (test_sim_reconcile_unaccounted.c) -- two PEND jobs each recorded-DISPATCH to the same 1-slot-capacity host; asserts exactly one is admitted (not both, not neither), the rejected one's entry is not consumed, and post-round resInUse reads exactly 1.0 (never 2.0). Fixing this surfaced 12 pre-existing tests (9 in test_sim_reconcile_unaccounted.c, 3 in test_sim_verify.c) whose fixtures named a host in a recorded allocation without ever registering a real xce_sched_worker_t of that name -- harmless before this fix (nothing checked capacity), now correctly rejected by the new gate; fixed by registering a real (or, for test_sim_verify.c, which never white-box-includes resource.c, a bare link-satisfying) worker for each such fixture, generous capacity since none of those 12 tests are actually about capacity limits. Verified: full scheduler suite 85/85 Ok, 0 Fail (asan profile, xce-build.sh scheduler asan), independently re-confirmed by running sched_sim_reconcile_unaccounted_test (22/22) and sched_sim_verify_test (17/17) directly with ASAN_OPTIONS=detect_leaks=0 (this test-file family's own established convention, unrelated pre-existing leak reports otherwise printed by these binaries when run standalone outside meson's own suite-level ASAN_OPTIONS).
|
|
|
|
## Verification
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## References
|
|
- Related: bug-1611, bug-1613, bug-1799, bug-1764, bug-1759, bug-1779
|
|
- Gitea: #65
|