lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

refactor(scheduler): move job-total limit computation to the limit plugin (bug-1790)

Browse source 
bug-1790's fix worked but put limit-plugin-specific logic in the wrong
place: seqjob_prepare_allocation()/parjob_prepare_allocation() (the
allocator plugins) unconditionally pre-computed a job-wide total for
every per-host resource in the job's rusages, "just in case" some
global limit needed one. That's the allocator plugins' concern leaking
into limit evaluation -- duplicated across both allocators (a third
would need it too), and wasteful (computed for resources no limit
actually references).

Moved ownership to sched_fill_limit_requests() (resource.c), called
once centrally from plugin_prepare_allocation() regardless of which
allocator ran:

- Removed calc_wanted_host_resval_job_totals() and its calls from both
  allocator plugins entirely.
- Added calc_wanted_job_total_resval() (resreq.c): an on-demand,
  single-resource lookup by name (no resmgr/key_find needed -- the
  caller already has the resolved xce_resource_t), computing the same
  job-wide-total formula only for the one (scope, underlying-resource)
  combination that needs it -- a GLOBAL limit on a per-host resource.
  The other three combinations are unchanged, still reading straight
  out of alloc_req.
- sched_fill_limit_requests() gains rusages/nhosts/nslots parameters
  back (a genuine need this time, not a re-added unused one).
  plugin.c's two call sites pass obj->u.primary->resreq->rusages
  (NULL for COMPOUND/ALTER, matching this codebase's existing scope
  convention for those tags) and ctx->nhosts/ctx->nslots.

All 23 testsuite link-satisfying stubs mechanically updated. Removed
the two allocator-level regression tests from the first pass (the
behavior they proved no longer exists by design); rewrote
test_fill_limit_requests_global_limit_on_host_resource_reads_host to
exercise the new on-demand path directly, nslots=3 to prove the job's
WHOLE total is used.

Full xce-build.sh all asan --incremental: 94/94 Ok, zero regressions.
Live re-verified against the same ug1_global_mem deploy/scenario:
byte-identical scheduler trace to the pre-refactor result, confirming
behavior-preserving.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
This commit is contained in:
Rongsong Shen 2026-09-08 10:44:07 +00:00
parent b612f80bd6
commit fa34b038f7
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
33 changed files with 308 additions and 350 deletions
Download patch file Download diff file Expand all files Collapse all files

130
src/lib/resreq.c
View file

@ -731,21 +731,9 @@ int format_resreq(char** buf, xce_resreq_t* resreq)
return rc;
}
/*
* calc_wanted_job_total_resvals_ - shared body for calc_wanted_global_resvals()
* and calc_wanted_host_resval_job_totals(): computes each matching rusage
* entry's JOB-WIDE TOTAL want (PER_TASK: value*nslots, PER_JOB: value,
* PER_HOST: value*nhosts -- nslots/nhosts are the whole job's totals, not
* any one host's share). `match_global` selects which resources are
* "matching": nonzero processes only XCE_RESF_GLOBAL resources (the
* original calc_wanted_global_resvals() behavior), zero processes only
* non-GLOBAL (per-host) ones instead.
*/
static int calc_wanted_job_total_resvals_(xvec_idval_t* resvals, var resmgr,
xlist_t* rusages,
rusage_per_type_t perflag,
int nhosts, int nslots,
int match_global)
int calc_wanted_global_resvals(xvec_idval_t* resvals, var resmgr,
xlist_t* rusages, rusage_per_type_t perflag,
int nhosts, int nslots)
{
int i;
@ -772,14 +760,8 @@ static int calc_wanted_job_total_resvals_(xvec_idval_t* resvals, var resmgr,
continue;
}
if (match_global) {
if ((r->flags & XCE_RESF_GLOBAL) == 0) {
continue;
}
} else {
if ((r->flags & XCE_RESF_GLOBAL) != 0) {
continue;
}
if ((r->flags & XCE_RESF_GLOBAL) == 0) {
continue;
}
perflag1 = v->per_flag;
@ -808,45 +790,77 @@ static int calc_wanted_job_total_resvals_(xvec_idval_t* resvals, var resmgr,
return 0;
}
int calc_wanted_global_resvals(xvec_idval_t* resvals, var resmgr,
xlist_t* rusages, rusage_per_type_t perflag,
int nhosts, int nslots)
{
return calc_wanted_job_total_resvals_(resvals, resmgr, rusages, perflag,
nhosts, nslots, 1);
}
/*
* calc_wanted_host_resval_job_totals - bug-1790: like
* calc_wanted_global_resvals(), but for resources NOT flagged
* XCE_RESF_GLOBAL (per-host resources) -- computes each such resource's
* JOB-WIDE TOTAL want, not any one host's own share (calc_wanted_host_
* resvals(), below, computes that instead, for the real per-host capacity
* check). Exists so a GLOBAL limit built on top of a per-host resource
* (e.g. a future absolute global mem cap, "slots" already being the one
* bug-1789 fixed) can read a real aggregate instead of an always-missing
* value.
* calc_wanted_job_total_resval - bug-1790, limit-plugin-ownership refactor:
* looks up a SINGLE resource by name in `rusages` and returns its JOB-WIDE
* TOTAL want (PER_TASK: value*nslots, PER_JOB: value, PER_HOST:
* value*nhosts -- nslots/nhosts are the whole job's totals, not any one
* host's share) via *out. Returns 1 if `resname` was found (*out set), 0
* if not (*out left untouched).
*
* seqjob_prepare_allocation()/parjob_prepare_allocation()'s global-phase
* branch (ctx->worker == NULL) calls this into ctx->alloc_req.host
* immediately alongside its existing calc_wanted_global_resvals() call.
* This is safe despite .host normally being per-host/per-bucket-varying
* data (see parjob_allocator.c's own per-bucket reset): plugin_prepare_
* allocation() (plugin.c) only resets .host at the START of a PER-HOST
* call (ctx->worker != NULL), never during the global phase, so the value
* written here survives long enough for sched_fill_limit_requests()
* (resource.c) to read it -- called later in the SAME global-phase
* invocation, for obj->glimits -- before any later per-host call wipes
* and correctly repopulates .host with that specific host's own share via
* calc_wanted_host_resvals().
* Same job-wide-total formula as calc_wanted_global_resvals() above, but
* targeted at ONE resource by name instead of iterating every rusage entry
* into a destination vector -- for sched_fill_limit_requests() (resource.c)
* to call on demand, only for the specific per-host resources a matched
* GLOBAL limit actually references (e.g. a future absolute global mem cap,
* "slots" being the one bug-1789 fixed). This replaces an earlier version
* of this fix that instead had seqjob_prepare_allocation()/parjob_
* prepare_allocation() unconditionally pre-compute totals for every
* per-host resource in the job's rusages "just in case some limit wants
* one" -- the wrong owner: those allocator plugins should only ever
* compute what the JOB itself wants for capacity-satisfy purposes, not
* what some unrelated limit might need to see. No resmgr/key_find needed
* here (unlike the two functions above) since the caller already has the
* limit's own resolved xce_resource_t and just needs the matching rusage
* entry's value/per_flag by name.
*/
int calc_wanted_host_resval_job_totals(xvec_idval_t* resvals, var resmgr,
xlist_t* rusages,
rusage_per_type_t perflag, int nhosts,
int nslots)
int calc_wanted_job_total_resval(xlist_t* rusages, rusage_per_type_t perflag,
int nhosts, int nslots, const char* resname,
float* out)
{
return calc_wanted_job_total_resvals_(resvals, resmgr, rusages, perflag,
nhosts, nslots, 0);
int i;
if ((rusages == NULL) || (resname == NULL) || (out == NULL)) {
return 0;
}
for (i = 0; i < rusages->count; i++) {
rusage_value_t* v = (rusage_value_t*)xlist_get(rusages, i);
rusage_per_type_t perflag1;
if ((v == NULL) || (v->res == NULL)) {
continue;
}
if (strcmp(v->res, resname) != 0) {
continue;
}
perflag1 = v->per_flag;
if (perflag1 == PER_DEFAULT) {
perflag1 = perflag;
}
switch (perflag1) {
case PER_TASK:
*out = v->value * nslots;
break;
case PER_JOB:
*out = v->value;
break;
case PER_HOST:
*out = v->value * nhosts;
break;
default:
/* something wrong */
*out = 0.0;
break;
}
return 1;
}
return 0;
}
int calc_wanted_host_resvals(xvec_idval_t* resvals, var resmgr,