myxce/.wolf/bugs/bug-1790.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	legacy
bug-1790	fixed	2026-09-24		lib	crash	2026-09-07T00:00:00Z	2026-09-19T09:00:00Z	1	src/lib/resreq.c	scheduler resource-accounting limits global-limit pre-existing severity-low live-verified	bug-1789	true

bug-1790: filed for its own tracking per user request

Description

OPEN, NOT A CRASH -- known residual gap explicitly carved out of bug-1789's fix, filed for its own tracking per user request. A hypothetical NEW global limit defined on a per-host resource OTHER than "slots" (e.g. a future absolute-value global mem cap, as opposed to the existing percent-based host-scoped mem limits in limits.yaml) would silently under-enforce the same way bug-1789's slots limits did, for a different reason than bug-1789's root cause.

Environment

Not recorded (migrated from buglog.json).

Reproduction

Not recorded (migrated from buglog.json).

Root cause

bug-1789 fixed sched_fill_limit_requests()'s READ side: it now picks whichever of alloc_req->shared/alloc_req->host to read a limit's underlying resource's want from by checking that resource's own XCE_RESF_GLOBAL flag on vres->res (and separately picks the WRITE destination for the limit's own vres entry by checking vres->vres's flag) -- see resource.c:928-955. That fix only works because alloc_req->host already has SOME value for "slots" to read, staged by seqjob_prepare_allocation()/parjob_prepare_allocation() via a separate, unconditional step outside the normal rusages loop, in EVERY phase including the global one. No other per-host resource gets this special treatment. The general population path -- calc_wanted_global_resvals() (src/lib/resreq.c:734) and calc_wanted_host_resvals() (src/lib/resreq.c:793), both called from seqjob_allocator.c/parjob_allocator.c's prepare_allocation hooks -- each unconditionally skip any resource whose XCE_RESF_GLOBAL flag doesn't match their own phase (calc_wanted_global_resvals() line 763: if ((r->flags & XCE_RESF_GLOBAL) == 0) continue;; calc_wanted_host_resvals() line 822: if ((r->flags & XCE_RESF_GLOBAL) != 0) continue;), and only calc_wanted_global_resvals() runs during the ctx->worker==NULL global phase. So for any per-host resource other than "slots", alloc_req->host is simply never populated with a want during the global phase -- sched_fill_limit_requests()'s now-correct read-side dispatch would read a genuine zero from an empty vector, not a population bug on its own side, but the net effect (a global limit on a per-host resource never seeing a nonzero want) is the same silent under-enforcement bug-1789 fixed for slots.

Fix

FIXED (2026-09-07, follow-up session, user-driven design). Corrected an earlier, wrong candidate fix (calling calc_wanted_host_resvals() during the global phase) identified during design discussion as invalid: that function's PER_HOST formula returns one host's own share (v->value, no multiplier) and, for parjob specifically, its nslots argument is reset PER BUCKET (parjob_allocator.c's parjob_try_plan_(), ctx->nslots = plan->bucket_slots[b], set right before each candidate host is tried) -- neither matches what a GLOBAL limit needs, which is the job-WIDE total, not any one host's/bucket's share.

REAL FIX: added calc_wanted_host_resval_job_totals() (src/lib/resreq.c), a new function for resources NOT flagged XCE_RESF_GLOBAL that computes the JOB-WIDE TOTAL want using the SAME formulas calc_wanted_global_resvals() already uses for GLOBAL resources (PER_TASK: valuenslots, PER_JOB: value, PER_HOST: valuenhosts -- nslots/nhosts being the whole job's totals, set during the global phase, not any bucket's share). Both functions now share a common static helper (calc_wanted_job_total_resvals_()) parameterized only by which flag direction to match, avoiding duplicating the ~30-line loop body. seqjob_prepare_allocation()/parjob_prepare_allocation()'s global-phase branch (ctx->worker == NULL) now calls this new function into ctx->alloc_req.host immediately alongside the existing calc_wanted_global_resvals() call into .shared.

SAFETY, traced precisely rather than assumed: plugin_prepare_allocation() (plugin.c:607-613) resets ONLY .shared at the top of a global-phase call and ONLY .host at the top of a per-host call -- never the other -- so .host is NOT reset when this new write happens. sched_fill_limit_requests() for obj->glimits runs later in that SAME plugin_prepare_allocation() invocation (plugin.c:644-647), reading and transferring the value into the limit's own permanent .shared record BEFORE any later per-host call ever resets .host. The value is therefore fully consumed and irrelevant by the time .host gets correctly wiped and repopulated with real per-host/per-bucket data for the actual capacity check -- two different consumers at two different times, not a conflict, confirmed by reading parjob_allocator.c's own per-bucket reset code directly (not inferred).

TDD: two new regression tests calling seqjob_prepare_allocation()/parjob_prepare_allocation() directly with ctx.worker == NULL and a "mem"-named (non-"slots", non-global) rusage entry -- test_seqjob_prepare_allocation_global_phase_stores_host_resource_job_total (test_seqjob_allocator.c) and test_parjob_prepare_allocation_global_phase_stores_host_resource_job_total (test_parjob_combinatorial.c, maxTasks=3 specifically to prove the formula uses the job's WHOLE nslots=3, giving 2.0*3=6.0, not any bucket's share). Both assert .shared stays empty (mem correctly excluded, not globally flagged) and .host carries both the new mem entry and the pre-existing unconditional "slots" entry. Both PASS standalone with full Unity PASS lines confirmed (not just suite-level OK).

Full xce-build.sh all asan --incremental: 92/94 Ok. The 2 failures are both pre-existing and unrelated, confirmed by cross-referencing this exact buglog: test_job_rusage_reconciles_mem_reservation_against_usage (bug-1762) and libxce test's 456-byte/29-allocation leak in test_xcealloc.c's make_alloc()/make_host_alloc() calls (bug-1733's fix note already confirms this exact leak signature -- 456 bytes, 29 allocations -- present byte-for-byte identically via an independent git-stash A/B rebuild in an earlier, unrelated session). LIVE-VERIFIED (2026-09-08, follow-up session). Added ug1_global_mem to limits.yaml -- a permanent, real GLOBAL limit on "mem" (a per-host resource, not "slots"), capping ug1's total cluster-wide mem usage at 5,000,000,000 -- exactly the previously-hypothetical scenario this bug was about, now real example config instead of untested. Fresh single-node deploy, 3 jobs submitted as user1 (ug1 member) each requesting rusage[mem=2000000000] (2GB) on queue normal: scheduler trace confirms ug1_global_mem matching (rc=1) and, for the first time, correctly computing a nonzero job-wide-total wanted value via calc_wanted_host_resval_job_totals() -- 'unsatisfy_ resource 36 total 5000000000.000000 inuse 4000000000.000000 uncomit 0.000000 wanted 2000000000.000000' / 'unsatisfy_ unmet resource 36 (limit=ug1_global_mem) shortfall 1000000000.000000' -- correctly detecting the 3rd job (4GB inuse + 2GB wanted = 6GB > 5GB cap) and PENDing it until an earlier job's mem freed up. Deploy config validated cleanly with zero new warnings. This closes the residual live-verification gap entirely -- bug-1790 is now proven correct for a resource other than "slots", not just unit-tested. REFACTORED (2026-09-08, same follow-up session, user-driven design review): the fix above worked and was live-verified, but put limit-plugin-specific logic (computing a job-total for a per-host resource, needed only when some GLOBAL limit references it) inside the ALLOCATOR plugins (seqjob_prepare_allocation()/parjob_prepare_allocation()) -- the wrong owner: those plugins should only ever compute what the job itself wants for capacity-satisfy purposes, and the duplication across two allocator plugins (a third would need it too) plus unconditionally computing totals for every per-host resource regardless of whether any limit needs one were both symptoms of that. Moved ownership to sched_fill_limit_requests() (resource.c) itself, called once centrally from plugin_prepare_allocation() regardless of which allocator ran: removed calc_wanted_host_resval_job_totals() and its calls from both allocator plugins entirely; added calc_wanted_job_total_resval() (resreq.c) -- a single-resource on-demand lookup (by name, no resmgr/key_find needed since the caller already has the resolved xce_resource_t), computing the SAME job-wide-total formula only for the one (scope, underlying-resource) combination that needs it (a GLOBAL limit on a per-host resource) -- the other three combinations are unchanged, still reading straight out of alloc_req. sched_fill_limit_requests() gained back rusages/nhosts/nslots parameters (undoing part of this session's earlier signature-trimming, but for a genuine need this time, not a re-added unused one); plugin.c's two call sites pass obj->u.primary->resreq->rusages (guarded on obj->tag == XCE_RESREQ_PRIMARY -- NULL for COMPOUND/ALTER, matching this codebase's existing scope convention for those tags) and ctx->nhosts/ctx->nslots (already correctly set by whichever allocator just ran, same invocation). All 23 testsuite link-satisfying stubs mechanically updated to the new 5-arg signature; the two allocator-level regression tests added by the first pass (test_seqjob_allocator.c/test_parjob_combinatorial.c) removed -- the behavior they proved no longer exists by design -- and test_resource_accounting.c's test_fill_limit_requests_global_limit_on_host_resource_reads_host rewritten to exercise the new on-demand path directly (job rusages instead of pre-staged .host, nslots=3 to prove the job-WIDE total is used, matching what the removed parjob-specific test proved). Full xce-build.sh all asan --incremental: 94/94 Ok, zero regressions. Live re-verified against the SAME ug1_global_mem deploy/scenario: byte-identical trace ('unsatisfy_ resource 36 total 5000000000.000000 inuse 4000000000.000000 uncomit 0.000000 wanted 2000000000.000000' / 'shortfall 1000000000.000000') to the pre-refactor result, confirming behavior-preserving. REGRESSION CAUGHT AND FIXED (2026-09-08, same follow-up session, user prompted "should we regression the test of gitea#46" after the ownership refactor above). calc_wanted_job_total_resval() scans rusages by name -- but "slots" is NEVER a declared rusage entry in this codebase: seqjob_prepare_allocation()/parjob_prepare_allocation() derive its want directly from the job's own task count (1.0 for seqjob, always maxTasks==1; (float)ctx->nslots for parjob), via their own unconditional key_find("slots", ...) + store, entirely outside the normal rusages loop -- confirmed by the user ("all resources except slots should be included in the rusage[] for request"). The refactor's on-demand lookup therefore always missed for "slots" and silently left val at its 0.0 default. Live-confirmed against a fresh deploy of the ORIGINAL gitea#46 scenario (9 plain sleep jobs, user1, queue normal, no -R flag -- exactly the issue's own repro): all 3 slots-based limits (limit1/ug1_slots/ug1_normal_slots) matched correctly (rc=1), but ZERO unsatisfy_ lines appeared anywhere and all 9 jobs dispatched with no cap enforcement at all -- gitea#46's exact original symptom, reintroduced by the refactor. The refactor's own new unit test had masked this: it artificially added a "slots" entry to its test rusages, which doesn't reflect production reality. FIX: sched_fill_limit_requests() (resource.c) now falls back to nslots directly (the SAME value both allocators already store as "slots"'s own per-host want) whenever the rusages lookup misses AND the resource is named "slots" -- the one documented exception to the rusages-lookup rule, matching this codebase's own existing "slots" special-casing convention (both allocators already key_find it by literal name, outside their normal rusages-driven computation). New test test_fill_limit_requests_global_limit_on_slots_uses_nslots (test_resource_accounting.c) proves the fallback with NO "slots" entry in rusages at all, matching the real shape; the original test renamed/reworked to use "mem" instead, so it now genuinely exercises the general on-demand path rather than accidentally exercising (and masking a gap in) the special case. Full xce-build.sh all asan --incremental: 94/94 Ok. Live re-verified against BOTH the original gitea#46 scenario (byte-identical to the pre-refactor trace: 'unsatisfy_ resource 35 total 3.000000 inuse 3.000000 uncomit 0.000000 wanted 1.000000' / 'shortfall 1.000000' for limit1) and the ug1_global_mem scenario from the refactor's own prior verification -- both correct. SIMPLIFIED (2026-09-08, same follow-up session, user insight: "alloc_req should have enough information? slots should be added into it?"). The nslots-fallback fix above worked but was an unnecessary name-based special case: alloc_req->host ALREADY holds exactly "slots"'s job-wide total during the global phase -- both allocators' own unconditional "add slots" step stages it there regardless of phase, and plugin_prepare_allocation() (plugin.c) never resets .host during the global phase (confirmed: sched_ctx_init(), schedobj.c, gives every job a fresh, empty alloc_req, so by the time sched_fill_limit_requests() runs -- right after the allocator's own hook, same invocation -- .host contains exactly "slots"'s total and nothing else). Replaced the strcmp(vres->res->name, "slots") check with reading alloc_req->host FIRST (get_resval_by_id()) and falling back to the rusages-based on-demand computation only when nothing is staged there (val stays the function's 0.0 default, get_resval_by_id()'s own "not found" signal -- safe here since "slots" is never legitimately 0: seqjob only runs for maxTasks==1, parjob only for maxTasks>1, so nslots is always >=1). This is data-driven, not name-based -- matching the same design philosophy bug-1789's own earlier flag-based-dispatch revision established -- and generalizes for free to any future resource an allocator plugin similarly pre-stages outside its normal rusages loop, with no further sched_fill_limit_requests() changes needed. Test renamed test_fill_limit_requests_global_limit_on_slots_reads_host, now pre-staging alloc_req->host exactly as the allocators do (rather than relying on the removed nslots fallback) with rusages left empty, matching production exactly. Full xce-build.sh all asan --incremental: 94/94 Ok. Live re-verified against BOTH the original gitea#46 scenario and the ug1_global_mem scenario: byte-identical scheduler traces to every prior verification round, confirming this simplification is behavior-preserving.

Verification

Not recorded (migrated from buglog.json).

References

• Related: bug-1789