lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

refactor(scheduler): move job-total limit computation to the limit plugin (bug-1790)

Browse source 
bug-1790's fix worked but put limit-plugin-specific logic in the wrong
place: seqjob_prepare_allocation()/parjob_prepare_allocation() (the
allocator plugins) unconditionally pre-computed a job-wide total for
every per-host resource in the job's rusages, "just in case" some
global limit needed one. That's the allocator plugins' concern leaking
into limit evaluation -- duplicated across both allocators (a third
would need it too), and wasteful (computed for resources no limit
actually references).

Moved ownership to sched_fill_limit_requests() (resource.c), called
once centrally from plugin_prepare_allocation() regardless of which
allocator ran:

- Removed calc_wanted_host_resval_job_totals() and its calls from both
  allocator plugins entirely.
- Added calc_wanted_job_total_resval() (resreq.c): an on-demand,
  single-resource lookup by name (no resmgr/key_find needed -- the
  caller already has the resolved xce_resource_t), computing the same
  job-wide-total formula only for the one (scope, underlying-resource)
  combination that needs it -- a GLOBAL limit on a per-host resource.
  The other three combinations are unchanged, still reading straight
  out of alloc_req.
- sched_fill_limit_requests() gains rusages/nhosts/nslots parameters
  back (a genuine need this time, not a re-added unused one).
  plugin.c's two call sites pass obj->u.primary->resreq->rusages
  (NULL for COMPOUND/ALTER, matching this codebase's existing scope
  convention for those tags) and ctx->nhosts/ctx->nslots.

All 23 testsuite link-satisfying stubs mechanically updated. Removed
the two allocator-level regression tests from the first pass (the
behavior they proved no longer exists by design); rewrote
test_fill_limit_requests_global_limit_on_host_resource_reads_host to
exercise the new on-demand path directly, nslots=3 to prove the job's
WHOLE total is used.

Full xce-build.sh all asan --incremental: 94/94 Ok, zero regressions.
Live re-verified against the same ug1_global_mem deploy/scenario:
byte-identical scheduler trace to the pre-refactor result, confirming
behavior-preserving.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
This commit is contained in:
Rongsong Shen 2026-09-08 10:44:07 +00:00
parent b612f80bd6
commit fa34b038f7
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
33 changed files with 308 additions and 350 deletions
Download patch file Download diff file Expand all files Collapse all files

77
src/scheduler/testsuite/test_resource_accounting.c
View file

@ -817,28 +817,29 @@ static void test_ctx_uncommit_host_reverses_provisional_debit(void)
}
/*
* bug-1789: sched_fill_limit_requests() (resource.c) computes, for each
* matched limit, how many units of the limit's own virtual resource
* (vres->vres) a job wants -- by reading how many units of the limit's
* UNDERLYING resource (vres->res) the job already requested. Both the
* READ source and the WRITE destination are one of alloc_req->shared/
* .host, each independently chosen by that resource's own
* XCE_RESF_GLOBAL flag: vres->res's flag decides the read source, the
* SAME flag calc_wanted_global_resvals()/calc_wanted_host_resvals()
* (resreq.c) dispatch on to decide where THEY write it; vres->vres's own
* flag (set by instantiate_limit()->add_vres() from the same
* is_host_based_limit() call that put this limit in obj->glimits vs
* obj->hlimits in the first place) decides the write destination -- no
* caller-supplied vector needed for either side. These two flags are
* independent: "slots" is per-host (not XCE_RESF_GLOBAL) even when the
* limit built on it is GLOBAL (limits.yaml's ug1_normal_slots, summing
* slots across the whole cluster) -- seqjob_prepare_allocation()/
* parjob_prepare_allocation() only ever stage a per-host resource's want
* into alloc_req->host, never alloc_req->shared, regardless of phase.
* This test proves the GLOBAL-limit/per-host-resource combination
* (bug-1789's actual reported case): the limit's own vres is
* GLOBAL-flagged (write target alloc_req->shared), but its underlying
* resource is not (its want must be read from alloc_req->host).
* bug-1789/1790, limit-plugin-ownership refactor: sched_fill_limit_requests()
* (resource.c) computes, for each matched limit, how many units of the
* limit's own virtual resource (vres->vres) a job wants -- by reading how
* many units of the limit's UNDERLYING resource (vres->res) the job already
* requested. The write destination is one of alloc_req->shared/.host,
* chosen by vres->vres's own XCE_RESF_GLOBAL flag (set by instantiate_
* limit()->add_vres() from the same is_host_based_limit() call that put
* this limit in obj->glimits vs obj->hlimits in the first place).
*
* The read source depends on BOTH flags together: three of the four
* (scope, underlying-resource) combinations read straight out of alloc_req
* (already populated by whichever allocator plugin ran). This test proves
* the fourth, different-in-kind combination (bug-1789's actual reported
* case): a GLOBAL limit (write target alloc_req->shared) on a per-host
* underlying resource like "slots" (limits.yaml's ug1_normal_slots,
* summing slots across the whole cluster) -- alloc_req->host never holds
* a job-WIDE total for this (only per-host/per-bucket-specific data for
* the real per-host capacity check), so the want is computed directly
* from the job's own rusages instead, via calc_wanted_job_total_resval()
* (resreq.c) -- PER_TASK: value * nslots, using nslots=3 here specifically
* to prove the job's WHOLE total is used, not any one host's/bucket's
* share (this is the limit-evaluation layer, allocator-agnostic -- no
* seqjob/parjob-specific test needed for this any more).
*/
static void
test_fill_limit_requests_global_limit_on_host_resource_reads_host(void)
@ -851,6 +852,8 @@ test_fill_limit_requests_global_limit_on_host_resource_reads_host(void)
xce_sched_alloc_req_t alloc_req;
xlist_t limits;
xce_vres_t vres;
xlist_t rusages;
rusage_value_t rv;
int rc;
memset(&sched, 0, sizeof(sched));
@ -880,26 +883,34 @@ test_fill_limit_requests_global_limit_on_host_resource_reads_host(void)
rc = xlist_store(&limits, &vres);
TEST_ASSERT_TRUE(rc >= 0);
/* the job's own declared "slots" rusage -- 2.0/task, 3 tasks total
* (nslots below), for an expected job-wide total of 6.0. */
rc = xlist_init(&rusages, 4);
TEST_ASSERT_EQUAL_INT(0, rc);
memset(&rv, 0, sizeof(rv));
rv.res = strdup("slots");
rv.value = 2.0f;
rv.per_flag = PER_TASK;
rc = xlist_store(&rusages, &rv);
TEST_ASSERT_TRUE(rc >= 0);
rc = xvec_idval_init(&(alloc_req.shared), 8);
TEST_ASSERT_EQUAL_INT(0, rc);
rc = xvec_idval_init(&(alloc_req.host), 8);
TEST_ASSERT_EQUAL_INT(0, rc);
/* pre-staged into .host, as seqjob/parjob_prepare_allocation() always
* do for "slots" regardless of phase -- .shared (this limit's own
* write target, derived from limit_res's GLOBAL flag) starts empty. */
rc = xvec_idval_store(&(alloc_req.host), slots_res->idx, 6.0f);
TEST_ASSERT_TRUE(rc >= 0);
rc = sched_fill_limit_requests(&alloc_req, &limits);
rc = sched_fill_limit_requests(&alloc_req, &limits, &rusages, 1, 3);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(1, alloc_req.shared.count);
TEST_ASSERT_EQUAL_INT(limit_res->idx, ((int*)alloc_req.shared.idxs.data)[0]);
TEST_ASSERT_EQUAL_FLOAT(6.0f, ((float*)alloc_req.shared.vals.data)[0]);
/* .host stays exactly as pre-staged -- the limit's own vres entry
* went to .shared, nothing new was written to .host. */
TEST_ASSERT_EQUAL_INT(1, alloc_req.host.count);
/* .host is untouched -- the limit's own vres entry went to .shared,
* and this combination never reads/writes .host at all. */
TEST_ASSERT_EQUAL_INT(0, alloc_req.host.count);
xlist_fini(&rusages);
FREEUP(rv.res);
xlist_fini(&limits);
}
@ -956,7 +967,7 @@ test_fill_limit_requests_host_limit_on_host_resource_reads_host(void)
rc = xvec_idval_store(&(alloc_req.host), mem_res->idx, 3.0f);
TEST_ASSERT_TRUE(rc >= 0);
rc = sched_fill_limit_requests(&alloc_req, &limits);
rc = sched_fill_limit_requests(&alloc_req, &limits, NULL, 0, 0);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(2, alloc_req.host.count);
@ -1022,7 +1033,7 @@ test_fill_limit_requests_host_limit_on_global_resource_reads_shared(void)
rc = xvec_idval_store(&(alloc_req.shared), license_res->idx, 5.0f);
TEST_ASSERT_TRUE(rc >= 0);
rc = sched_fill_limit_requests(&alloc_req, &limits);
rc = sched_fill_limit_requests(&alloc_req, &limits, NULL, 0, 0);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(1, alloc_req.host.count);