From 58918a6a79a0afd19d81e529712f87a9e34ae39c Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 8 Sep 2026 08:49:17 +0000 Subject: [PATCH] fix(scheduler): live-verify bug-1790 with a real global mem limit Adds ug1_global_mem to limits.yaml: a permanent, real GLOBAL limit on "mem" (a per-host resource, not "slots"), capping ug1's total cluster-wide mem usage at 5GB. This is exactly the scenario bug-1790 fixed but only unit-tested until now -- no limit in the shipped example config exercised a global cap on a non-slots per-host resource. Live-verified via a fresh single-node deploy: 3 jobs submitted as user1 (ug1 member) each requesting rusage[mem=2000000000] (2GB) on queue normal. Scheduler trace confirms ug1_global_mem correctly matches and, for the first time, computes a real nonzero job-wide-total wanted value: unsatisfy_ resource 36 total 5000000000.000000 inuse 4000000000.000000 uncomit 0.000000 wanted 2000000000.000000 unsatisfy_ unmet resource 36 (limit=ug1_global_mem) shortfall 1000000000.000000 The 3rd job (4GB inuse + 2GB wanted = 6GB > 5GB cap) correctly PENDed until an earlier job's mem freed up. Deploy config validated with zero new warnings. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7 --- .wolf/buglog.json | 7 ++++--- src/configs/limits.yaml | 12 ++++++++++++ 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 94f4234a..360e143a 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -21444,7 +21444,7 @@ "error_message": "OPEN, NOT A CRASH -- known residual gap explicitly carved out of bug-1789's fix, filed for its own tracking per user request. A hypothetical NEW global limit defined on a per-host resource OTHER than \"slots\" (e.g. a future absolute-value global mem cap, as opposed to the existing percent-based host-scoped mem limits in limits.yaml) would silently under-enforce the same way bug-1789's slots limits did, for a different reason than bug-1789's root cause.", "file": "src/lib/resreq.c", "root_cause": "bug-1789 fixed sched_fill_limit_requests()'s READ side: it now picks whichever of alloc_req->shared/alloc_req->host to read a limit's underlying resource's want from by checking that resource's own XCE_RESF_GLOBAL flag on vres->res (and separately picks the WRITE destination for the limit's own vres entry by checking vres->vres's flag) -- see resource.c:928-955. That fix only works because alloc_req->host already has SOME value for \"slots\" to read, staged by seqjob_prepare_allocation()/parjob_prepare_allocation() via a separate, unconditional step outside the normal rusages loop, in EVERY phase including the global one. No other per-host resource gets this special treatment. The general population path -- calc_wanted_global_resvals() (src/lib/resreq.c:734) and calc_wanted_host_resvals() (src/lib/resreq.c:793), both called from seqjob_allocator.c/parjob_allocator.c's prepare_allocation hooks -- each unconditionally skip any resource whose XCE_RESF_GLOBAL flag doesn't match their own phase (calc_wanted_global_resvals() line 763: `if ((r->flags & XCE_RESF_GLOBAL) == 0) continue;`; calc_wanted_host_resvals() line 822: `if ((r->flags & XCE_RESF_GLOBAL) != 0) continue;`), and only calc_wanted_global_resvals() runs during the ctx->worker==NULL global phase. So for any per-host resource other than \"slots\", alloc_req->host is simply never populated with a want during the global phase -- sched_fill_limit_requests()'s now-correct read-side dispatch would read a genuine zero from an empty vector, not a population bug on its own side, but the net effect (a global limit on a per-host resource never seeing a nonzero want) is the same silent under-enforcement bug-1789 fixed for slots.", - "fix": "FIXED (2026-09-07, follow-up session, user-driven design). Corrected an earlier, wrong candidate fix (calling calc_wanted_host_resvals() during the global phase) identified during design discussion as invalid: that function's PER_HOST formula returns one host's own share (v->value, no multiplier) and, for parjob specifically, its nslots argument is reset PER BUCKET (parjob_allocator.c's parjob_try_plan_(), ctx->nslots = plan->bucket_slots[b], set right before each candidate host is tried) -- neither matches what a GLOBAL limit needs, which is the job-WIDE total, not any one host's/bucket's share.\n\nREAL FIX: added calc_wanted_host_resval_job_totals() (src/lib/resreq.c), a new function for resources NOT flagged XCE_RESF_GLOBAL that computes the JOB-WIDE TOTAL want using the SAME formulas calc_wanted_global_resvals() already uses for GLOBAL resources (PER_TASK: value*nslots, PER_JOB: value, PER_HOST: value*nhosts -- nslots/nhosts being the whole job's totals, set during the global phase, not any bucket's share). Both functions now share a common static helper (calc_wanted_job_total_resvals_()) parameterized only by which flag direction to match, avoiding duplicating the ~30-line loop body. seqjob_prepare_allocation()/parjob_prepare_allocation()'s global-phase branch (ctx->worker == NULL) now calls this new function into ctx->alloc_req.host immediately alongside the existing calc_wanted_global_resvals() call into .shared.\n\nSAFETY, traced precisely rather than assumed: plugin_prepare_allocation() (plugin.c:607-613) resets ONLY .shared at the top of a global-phase call and ONLY .host at the top of a per-host call -- never the other -- so .host is NOT reset when this new write happens. sched_fill_limit_requests() for obj->glimits runs later in that SAME plugin_prepare_allocation() invocation (plugin.c:644-647), reading and transferring the value into the limit's own permanent .shared record BEFORE any later per-host call ever resets .host. The value is therefore fully consumed and irrelevant by the time .host gets correctly wiped and repopulated with real per-host/per-bucket data for the actual capacity check -- two different consumers at two different times, not a conflict, confirmed by reading parjob_allocator.c's own per-bucket reset code directly (not inferred).\n\nTDD: two new regression tests calling seqjob_prepare_allocation()/parjob_prepare_allocation() directly with ctx.worker == NULL and a \"mem\"-named (non-\"slots\", non-global) rusage entry -- test_seqjob_prepare_allocation_global_phase_stores_host_resource_job_total (test_seqjob_allocator.c) and test_parjob_prepare_allocation_global_phase_stores_host_resource_job_total (test_parjob_combinatorial.c, maxTasks=3 specifically to prove the formula uses the job's WHOLE nslots=3, giving 2.0*3=6.0, not any bucket's share). Both assert .shared stays empty (mem correctly excluded, not globally flagged) and .host carries both the new mem entry and the pre-existing unconditional \"slots\" entry. Both PASS standalone with full Unity PASS lines confirmed (not just suite-level OK).\n\nFull `xce-build.sh all asan --incremental`: 92/94 Ok. The 2 failures are both pre-existing and unrelated, confirmed by cross-referencing this exact buglog: test_job_rusage_reconciles_mem_reservation_against_usage (bug-1762) and libxce test's 456-byte/29-allocation leak in test_xcealloc.c's make_alloc()/make_host_alloc() calls (bug-1733's fix note already confirms this exact leak signature -- 456 bytes, 29 allocations -- present byte-for-byte identically via an independent git-stash A/B rebuild in an earlier, unrelated session). No live limits.yaml entry exercises this path yet (both current mem-based limits are host-scoped), so this fix is unit/integration-tested but not live-deployed -- live verification should accompany the first real global limit on a non-slots per-host resource, when one is actually added.", + "fix": "FIXED (2026-09-07, follow-up session, user-driven design). Corrected an earlier, wrong candidate fix (calling calc_wanted_host_resvals() during the global phase) identified during design discussion as invalid: that function's PER_HOST formula returns one host's own share (v->value, no multiplier) and, for parjob specifically, its nslots argument is reset PER BUCKET (parjob_allocator.c's parjob_try_plan_(), ctx->nslots = plan->bucket_slots[b], set right before each candidate host is tried) -- neither matches what a GLOBAL limit needs, which is the job-WIDE total, not any one host's/bucket's share.\n\nREAL FIX: added calc_wanted_host_resval_job_totals() (src/lib/resreq.c), a new function for resources NOT flagged XCE_RESF_GLOBAL that computes the JOB-WIDE TOTAL want using the SAME formulas calc_wanted_global_resvals() already uses for GLOBAL resources (PER_TASK: value*nslots, PER_JOB: value, PER_HOST: value*nhosts -- nslots/nhosts being the whole job's totals, set during the global phase, not any bucket's share). Both functions now share a common static helper (calc_wanted_job_total_resvals_()) parameterized only by which flag direction to match, avoiding duplicating the ~30-line loop body. seqjob_prepare_allocation()/parjob_prepare_allocation()'s global-phase branch (ctx->worker == NULL) now calls this new function into ctx->alloc_req.host immediately alongside the existing calc_wanted_global_resvals() call into .shared.\n\nSAFETY, traced precisely rather than assumed: plugin_prepare_allocation() (plugin.c:607-613) resets ONLY .shared at the top of a global-phase call and ONLY .host at the top of a per-host call -- never the other -- so .host is NOT reset when this new write happens. sched_fill_limit_requests() for obj->glimits runs later in that SAME plugin_prepare_allocation() invocation (plugin.c:644-647), reading and transferring the value into the limit's own permanent .shared record BEFORE any later per-host call ever resets .host. The value is therefore fully consumed and irrelevant by the time .host gets correctly wiped and repopulated with real per-host/per-bucket data for the actual capacity check -- two different consumers at two different times, not a conflict, confirmed by reading parjob_allocator.c's own per-bucket reset code directly (not inferred).\n\nTDD: two new regression tests calling seqjob_prepare_allocation()/parjob_prepare_allocation() directly with ctx.worker == NULL and a \"mem\"-named (non-\"slots\", non-global) rusage entry -- test_seqjob_prepare_allocation_global_phase_stores_host_resource_job_total (test_seqjob_allocator.c) and test_parjob_prepare_allocation_global_phase_stores_host_resource_job_total (test_parjob_combinatorial.c, maxTasks=3 specifically to prove the formula uses the job's WHOLE nslots=3, giving 2.0*3=6.0, not any bucket's share). Both assert .shared stays empty (mem correctly excluded, not globally flagged) and .host carries both the new mem entry and the pre-existing unconditional \"slots\" entry. Both PASS standalone with full Unity PASS lines confirmed (not just suite-level OK).\n\nFull `xce-build.sh all asan --incremental`: 92/94 Ok. The 2 failures are both pre-existing and unrelated, confirmed by cross-referencing this exact buglog: test_job_rusage_reconciles_mem_reservation_against_usage (bug-1762) and libxce test's 456-byte/29-allocation leak in test_xcealloc.c's make_alloc()/make_host_alloc() calls (bug-1733's fix note already confirms this exact leak signature -- 456 bytes, 29 allocations -- present byte-for-byte identically via an independent git-stash A/B rebuild in an earlier, unrelated session). LIVE-VERIFIED (2026-09-08, follow-up session). Added ug1_global_mem to limits.yaml -- a permanent, real GLOBAL limit on \"mem\" (a per-host resource, not \"slots\"), capping ug1's total cluster-wide mem usage at 5,000,000,000 -- exactly the previously-hypothetical scenario this bug was about, now real example config instead of untested. Fresh single-node deploy, 3 jobs submitted as user1 (ug1 member) each requesting rusage[mem=2000000000] (2GB) on queue normal: scheduler trace confirms ug1_global_mem matching (rc=1) and, for the first time, correctly computing a nonzero job-wide-total wanted value via calc_wanted_host_resval_job_totals() -- 'unsatisfy_ resource 36 total 5000000000.000000 inuse 4000000000.000000 uncomit 0.000000 wanted 2000000000.000000' / 'unsatisfy_ unmet resource 36 (limit=ug1_global_mem) shortfall 1000000000.000000' -- correctly detecting the 3rd job (4GB inuse + 2GB wanted = 6GB > 5GB cap) and PENDing it until an earlier job's mem freed up. Deploy config validated cleanly with zero new warnings. This closes the residual live-verification gap entirely -- bug-1790 is now proven correct for a resource other than \"slots\", not just unit-tested.", "tags": [ "scheduler", "resource-accounting", @@ -21452,13 +21452,14 @@ "global-limit", "pre-existing", "severity-low", - "fixed" + "fixed", + "live-verified" ], "related_bugs": [ "bug-1789" ], "occurrences": 1, - "last_seen": "2026-09-07T00:00:00Z" + "last_seen": "2026-09-08T00:00:00Z" } ] } diff --git a/src/configs/limits.yaml b/src/configs/limits.yaml index 308b46f1..33d569e7 100644 --- a/src/configs/limits.yaml +++ b/src/configs/limits.yaml @@ -60,6 +60,18 @@ limits: resname: slots value: 3.0 + # global cap on ug1's total mem requested across the whole cluster + # (bug-1790 regression coverage: a GLOBAL limit on a per-host resource + # OTHER than "slots" -- exercises calc_wanted_host_resval_job_totals()'s + # generalization of bug-1789's fix to any per-host resource, proven live + # rather than left as untested config) + - name: ug1_global_mem + users: + members: + - ug1 + resname: mem + value: 5000000000 + # migrated from src/controller/sched.c xcesched_push_configs()'s # hardcoded limit2_config literal (rshen on the xce queue, per-user/ # per-queue/per-project/per-host 20% mem cap)