Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
12 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | gitea_issue | gitea_synced_status | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1789 | fixed | 2026-09-24 | scheduler | unknown | 2026-09-07T10:30:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true | 46 | fixed |
bug-1789: After bug-1784 through bug-1788 all fixed and live-verified (all 7 limits in limits.yaml now…
Description
After bug-1784 through bug-1788 all fixed and live-verified (all 7 limits in limits.yaml now correctly match and instantiate usage tracking for every job), the actual numeric cap still does not reject anyone: 10 single-slot jobs submitted as user1 on queue normal against ug1_normal_slots's configured value:8 cap all still reach RUN, none PEND.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
NOT YET ROOT-CAUSED TO A FIX -- traced to a specific, likely-defective computation, but the correct intended semantics were not established this session (deliberately stopped here rather than guess at production logic). Chain traced and CONFIRMED CORRECT up to one point: match_job_global_limit_ populates obj->glimits (fixed, bug-1784-1788) -> plugin.c's plugin_prepare_allocation calls sched_fill_limit_requests(sched, &(ctx->alloc_req.shared), &(obj->glimits)) when ctx->worker==NULL (the global/non-host-scoped case, correct for ug1_normal_slots) -> seqjob_allocator.c/parjob_allocator.c both correctly call sched_check_shared_satisfy(sched, uncomit, &(ctx->alloc_req.shared), &(ctx->unmeet)) and correctly reject (PREASON_NO_RESOURCE, rc=0) when a shared resource's wanted amount would exceed its tracked total (itself correctly set to limit->val.val=8 by update_limit_res_value(), which DOES run correctly now). The break is inside sched_fill_limit_requests() itself (resource.c:912-931): for each matched limit, it computes the job's "wanted" quantity of the limit's virtual resource via val = get_resval_by_id(vec, vres->res->idx) where vec is ctx->alloc_req.shared ITSELF -- i.e. it looks up how many units of the limit's UNDERLYING resource (e.g. "slots") the job already requests, inside the SAME shared-resource-request vector it is about to write the limit's own synthetic entry into. But "slots" is inherently a per-host resource, requested via ctx->alloc_req.HOST (task/slot count per host), not ctx->alloc_req.shared -- so this lookup always misses, get_resval_by_id's own not-found fallback returns 0.0, and every job is recorded as requesting ZERO units of ug1_normal_slots regardless of how many slots it actually asks for. sched_check_shared_satisfy then trivially "satisfies" a wanted=0 request against any total, no matter how many jobs are already running -- the cap can never bind. NOT CONFIRMED whether this is a straightforward bug (should read from ctx->alloc_req.host or from resreq directly) or reflects a genuine, unaddressed design gap in how a GLOBAL (non-host-scoped) limit is meant to interact with a per-host-scoped underlying resource like slots -- the fix requires understanding what "consumption count" SHOULD mean for this combination, not just correcting an obvious return-code mistake like bug-1784-1788 all were.
Fix
FIXED (2026-09-07, follow-up session). Root-caused via codegraph trace of plugin_prepare_allocation()'s full hook loop (not just the two call sites the prior session had traced): seqjob_prepare_allocation()/parjob_prepare_allocation() (the allocator plugins' own prepare_allocation_hook) ALWAYS stage the job's wanted slot count into ctx->alloc_req.host -- even during the ctx->worker==NULL/global pass -- because "slots" is deliberately excluded from calc_wanted_global_resvals()/calc_wanted_host_resvals() (it's per-host, not XCE_RESF_GLOBAL) and is staged by a separate, unbranched follow-up store. This confirmed the prior session's NEXT-STEPS item (1): the job's real wanted slot count (ctx->nslots, set earlier in that SAME prepare_allocation call) is already available and correct for both phases -- 1/seqjob or primary->maxTasks/parjob-global or the current bucket's count/parjob-per-host -- it just was never read from the right place. FIRST ATTEMPT (reverted): tried mirroring the ctx->worker branch into the seqjob/parjob "add slots" store itself (writing into ctx->alloc_req.shared during the global phase) -- this broke 6 previously-passing tests (seqjob/parjob dispatch tests going from DISPATCH to PEND) because ctx->alloc_req.shared is also read by sched_check_shared_satisfy() against sched->core.sharedInst, the GLOBAL capacity pool, which never tracks "slots" (a per-host-only resource) -- injecting a spurious "slots" want there made EVERY job's general satisfy check fail, not just limit-scoped ones. REAL FIX: read-side only, confined to limit evaluation. sched_fill_limit_requests() (resource.c) now takes an added xce_sched_ctx_t* ctx parameter; when a matched limit's underlying resource is "slots" (by name, matching this codebase's existing convention of hardcoding "slots" by name in seqjob/parjob's own key_find() calls), it uses (float)ctx->nslots directly as the wanted quantity instead of the always-missing get_resval_by_id(vec, ...) lookup; every other resource keeps the original vec-lookup behavior unchanged. plugin.c's two call sites updated to pass ctx; scheduler.h's extern declaration updated; all 23 testsuite files providing a link-satisfying no-op stub for the old 3-arg signature mechanically updated to the new 4-arg one (added (void)ctx; each, zero behavior change). TDD: 2 new focused unit tests in test_resource_accounting.c (white-box #include of resource.c) -- test_fill_limit_requests_global_slots_limit_uses_ctx_nslots (builds a bare xce_vres_t{res="slots", vres="limit=X"}, ctx.nslots=6, empty vec, asserts the limit's own vres entry gets stored with val=6.0) and test_fill_limit_requests_non_slots_resource_reads_from_vec (same shape for a "mem"-resname'd limit, vec pre-staged with 3.0, ctx.nslots=99 deliberately mismatched, asserts the mem path is unaffected and still reads 3.0 from vec, not ctx->nslots). Full scheduler suite: 76/77 (only pre-existing, unrelated bug-1762 failure remains -- test_job_rusage_reconciles_mem_reservation_against_usage, mem-reservation subsystem, untouched by this change). LIVE-VERIFIED: fresh single-node deploy (localhost:19997), 10 single-slot jobs submitted as user1 (ug1 member) on queue normal. hosts.yaml's default host template has max_slot: -1 (unlimited), ruling out raw host capacity as a factor. Scheduler trace confirms match_job_global_limit_ matching ug1_slots/ug1_normal_slots/limit1 (all three global, slots-resname'd) for every job, and resource.c's unsatisfy_() now logging a REAL non-zero wanted value for the limit's virtual resource on every charge (previously always 0.0, per this bug's original root_cause) -- e.g. 'unsatisfy_ resource 35 total 3.000000 inuse 3.000000 uncomit 0.000000 wanted 1.000000' for limit1 (cap 3.0, matches every non-rshen user including user1) once its 3rd job committed, correctly PENDing the 4th and all subsequent jobs with a genuine shortfall. limit1 (cap 3, broader scope: all users except rshen) bound before ug1_normal_slots (cap 8, narrower: ug1+normal only) could, since both match user1 identically and limit1's cap is lower -- expected overlapping-limit interaction (flagged in an earlier session's own observation), not a gap in this fix; both run through the identical, now-fixed code path. REVISED (2026-09-07, same session, user feedback): the name-based strcmp(vres->res->name, "slots") special-case above was flagged as unreliable (correctly -- it silently fails to generalize to any future per-host resource that isn't literally named "slots"). Replaced with a flag-based dispatch: sched_fill_limit_requests() now takes xce_sched_alloc_req_t* alloc_req instead of xce_sched_ctx_t* ctx (nothing else from ctx was ever used), and for each matched limit reads its underlying resource's want from whichever of alloc_req->shared/alloc_req->host that resource's own XCE_RESF_GLOBAL flag indicates -- the SAME flag calc_wanted_global_resvals()/calc_wanted_host_resvals() (resreq.c) themselves already dispatch on to decide where THEY write a resource's want, so this is not a new rule, just the existing one applied consistently. plugin.c's two call sites, scheduler.h, and all 23 testsuite stubs updated accordingly (mechanical param rename). Replaced the 2 name-based unit tests with 3 flag-based ones in test_resource_accounting.c: test_fill_limit_requests_global_limit_on_host_resource_reads_host (bug-1789's actual case: a GLOBAL limit's own write target is alloc_req->shared, but its per-host underlying resource's want must be read from alloc_req->host), test_fill_limit_requests_host_limit_on_host_resource_reads_host (already-working sibling case, unaffected), and test_fill_limit_requests_host_limit_on_global_resource_reads_shared (the new symmetric case this refactor adds real coverage for: a HOST-scoped limit's write target is alloc_req->host, but a genuinely-XCE_RESF_GLOBAL-flagged underlying resource's want must be read from alloc_req->shared instead -- proves the dispatch is a real bidirectional rule, not a one-off special case). Full scheduler suite re-run: 76/77 (same pre-existing bug-1762 failure, no new failures). Live re-verified end to end with a fully fresh redeploy (not just a scheduler restart against old job state, to keep the signal clean): identical result to the first live verification, byte-identical scheduler trace line ('unsatisfy_ resource 35 total 3.000000 inuse 3.000000 uncomit 0.000000 wanted 1.000000'), confirming the refactor is behavior-preserving for the current config. KNOWN RESIDUAL GAP (explicitly out of scope, not silently solved): a hypothetical NEW global limit on some per-host resource OTHER than "slots" (e.g. an absolute-value global mem cap, as opposed to the existing percent-based host-scoped mem limits) would still hit an analogous bug -- calc_wanted_global_resvals() itself skips any non-XCE_RESF_GLOBAL-flagged resource entirely, so such a resource's want is never computed anywhere during the global phase; "slots" only works because seqjob/parjob_prepare_allocation() stage it via a separate, always-run step outside the normal rusages loop, which no other resource has. Fixing that would mean teaching calc_wanted_global_resvals() (or an equivalent) to also compute a non-global resource's want when a currently-matched limit needs it -- a materially larger change, not needed by any limit in the current limits.yaml (the two mem-based limits, night_mem_pct/limits2, are both host-scoped, unaffected either way) and deliberately not attempted here. FURTHER SIMPLIFIED (2026-09-07, same session, user request): sched_fill_limit_requests()'s 1st parameter (xce_scheduler_t* sched) was entirely unused in the function body (only alloc_req and limits are referenced) -- a leftover from an earlier signature shape, not flagged by the prior same-day refactor that removed the old vec parameter. Dropped it: signature is now sched_fill_limit_requests(xce_sched_alloc_req_t* alloc_req, xlist_t* limits). Updated scheduler.h's extern declaration, both call sites in plugin.c's plugin_prepare_allocation() (the sched local var stays -- still used elsewhere in that function for plugin iteration), all 23 link-satisfying testsuite stubs (mechanical Python replace, byte-identical body across all 23), and the 3 real call sites in test_resource_accounting.c (the local sched var in each test stays too -- still needed for xce_sched_core_init()/makeResource() resource setup, just no longer passed to this call). Full scheduler suite re-verified after this change: 76/77 (same pre-existing, unrelated bug-1762 failure, no new failures).
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1784, bug-1785, bug-1786, bug-1787, bug-1788, bug-1790
- Gitea: #46