lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(scheduler): reconcile mem reservation against real usage

Browse source 
Adds reservation-vs-actual-usage reconciliation for real (gauge-style)
resources, using mem as the concrete case -- documented in the new
resource-accounting.md steering doc.

Policy: charged = max(reserved, actual). A job's rusage[mem=...]
request is an estimate made before it has run, and the real usage the
worker later reports may land above or below it; charging a flat
reservation for the job's whole life either starves other jobs of
memory it never uses (when actual stays below the estimate) or lets
the host be driven into genuine oversubscription (when actual exceeds
it). Available memory for scheduling is then the host's OS-reported
available figure minus, per job, only the UNUSED SLACK of the
reservation (max(0, reserved - used)) -- never the job's actual usage
on top of that, since the OS's own figure already reflects it.

The existing xce_res_account_t{used, reserved, allocated} fields and
sched_update_res_accounts()'s used+reserved charge formula already
implement this exactly, once `used` is set to the OVERAGE beyond the
reservation (max(0, actual - reserved)) rather than left at its
permanent 0.0 default -- nothing previously computed it.

New (resource.c): sched_reconcile_host_resource_usage() sets that
overage on one xce_res_account_t and applies just the DELTA to the
affected worker's resInUse (not a full uncharge/recharge of the whole
job_account, which would needlessly re-touch sibling resources like
"slots" charged in the same account). sched_reconcile_job_resource_usage()
is the wrapper driven from a raw rusage report: maps the worker's real
wire resname ("memory_rss", same worker-resname-vs-scheduler-resource-
name split bug-1735 already established for the controller side) to
the scheduler's "mem" resource, and is a no-op when the job spans more
than one host (the wire report carries no per-host attribution today --
a separate, undocumented-until-now gap, out of this fix's scope) or
never requested rusage[mem=...] in the first place (reserved=0, nothing
to reconcile).

Wired into sched_apply_job_rusage() (schedobj.c), which already
receives every job rusage tick but previously only stored it for
display/plugin hooks with no resource-accounting effect at all.

Compatible with sched_account_jobs()'s existing per-round resInUse
reset+re-derive cycle (bug-1624): the reconciled `used` value is
persisted on the job's own account record, which that re-derivation
already reads from, not just a transient pool-level delta that a
round boundary would wipe.

Added link-satisfying stubs to the three white-box test files that
#include schedobj.c directly (test_job_update.c,
test_picker_config_fields.c, test_picker_wiring.c) -- none of them
exercise real job->accounts, so the new call is unreachable there, but
still needs to link. 10 new regression tests in
test_resource_accounting.c cover the core formula (below/at/above the
reservation, delta-only pool updates, sibling-resource isolation) and
the wrapper's scoping rules (resname mapping, multi-host skip,
no-request no-op).

Full scheduler suite: 55/55 passing, zero regressions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
This commit is contained in:
Rongsong Shen 2026-08-31 07:02:32 +00:00
parent dc430ae112
commit 87a2a46b94
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
9 changed files with 691 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

12
src/scheduler/testsuite/test_job_update.c
View file

@ -52,6 +52,18 @@ int plugin_job_rusage_update(void* s, xce_sched_job_t* j, xce_job_rusages_t* r)
}
xce_job_account_t* make_job_account(xce_job_alloc_t* a) { (void)a; return NULL; }
void free_job_account(xce_job_account_t* a) { (void)a; }
/* resource-accounting.md's reconciliation (resource.c, not linked here --
* this file only #includes schedobj.c). Never actually reached: make_job_
* account() above always returns NULL, so every test job's job->accounts
* stays NULL and sched_apply_job_rusage()'s own NULL-guard skips the call
* entirely -- link-satisfying stub only, same rationale as every other
* stub in this block. */
int sched_reconcile_job_resource_usage(xce_sched_core_t* c,
xce_job_account_t* a,
xce_job_rusages_t* r)
{
(void)c; (void)a; (void)r; return 0;
}
int sched_add_user(xce_sched_core_t* s, xce_sched_user_t* u)
{
(void)s;

8
src/scheduler/testsuite/test_picker_config_fields.c
View file

@ -72,6 +72,14 @@ int plugin_job_rusage_update(void* s, xce_sched_job_t* j, xce_job_rusages_t* r)
}
xce_job_account_t* make_job_account(xce_job_alloc_t* a) { (void)a; return NULL; }
void free_job_account(xce_job_account_t* a) { (void)a; }
/* resource-accounting.md's reconciliation (resource.c, not linked here) --
* link-satisfying stub only, same rationale as every other stub above. */
int sched_reconcile_job_resource_usage(xce_sched_core_t* c,
xce_job_account_t* a,
xce_job_rusages_t* r)
{
(void)c; (void)a; (void)r; return 0;
}
void sched_worker_release_host_ctx(xce_sched_worker_t* w) { (void)w; }
int plugin_prepare_allocation(void* s, xce_sched_ctx_t* c) { (void)s; (void)c; return 0; }
void plugin_unregister(void* arg) { (void)arg; }

8
src/scheduler/testsuite/test_picker_wiring.c
View file

@ -103,6 +103,14 @@ xce_job_account_t* make_job_account(xce_job_alloc_t* a)
return NULL;
}
void free_job_account(xce_job_account_t* a) { (void)a; }
/* resource-accounting.md's reconciliation (resource.c, not linked here) --
* link-satisfying stub only, same rationale as every other stub here. */
int sched_reconcile_job_resource_usage(xce_sched_core_t* c,
xce_job_account_t* a,
xce_job_rusages_t* r)
{
(void)c; (void)a; (void)r; return 0;
}
int sched_add_user(xce_sched_core_t* s, xce_sched_user_t* u)
{
(void)s;

320
src/scheduler/testsuite/test_resource_accounting.c
View file

@ -350,10 +350,330 @@ static void test_commit_job_alloc_identical_across_core_containers(void)
TEST_ASSERT_EQUAL_FLOAT(shared_sched[0], shared_sim[0]);
}
/*
* resource-accounting.md's reservation-vs-actual-usage reconciliation for
* a REAL (gauge-style) resource, e.g. "mem" -- sched_reconcile_host_
* resource_usage() (the single-account-entry primitive) and
* sched_reconcile_job_resource_usage() (the rusages-report wrapper it's
* built for). Both are defined in resource.c, already white-box #included
* above.
*/
#define MEM_RESIDX 2
#define MEM_RESERVED_MB 4.0
/* make_mem_account - a single-host xce_job_account_t with one "mem"
* xce_res_account_t entry (residx MEM_RESIDX, reserved MEM_RESERVED_MB,
* used 0.0 -- a job that has just been dispatched and never had a rusage
* tick applied yet). resname is real (strdup'd), unlike make_test_account's
* residx-only entries above, since sched_reconcile_job_resource_usage()
* looks accounts up BY NAME. */
static xce_job_account_t* make_mem_account(const char* worker_name)
{
xce_job_account_t* acct = (xce_job_account_t*)calloc(1, sizeof(*acct));
TEST_ASSERT_NOT_NULL(acct);
acct->nhosts = 1;
acct->accounts = (xce_host_account_t**)calloc(1, sizeof(*acct->accounts));
TEST_ASSERT_NOT_NULL(acct->accounts);
acct->accounts[0]
= (xce_host_account_t*)calloc(1, sizeof(xce_host_account_t));
TEST_ASSERT_NOT_NULL(acct->accounts[0]);
acct->accounts[0]->host = strdup(worker_name);
acct->accounts[0]->naccount = 1;
acct->accounts[0]->accounts
= (xce_res_account_t**)calloc(1, sizeof(*acct->accounts[0]->accounts));
TEST_ASSERT_NOT_NULL(acct->accounts[0]->accounts);
acct->accounts[0]->accounts[0]
= (xce_res_account_t*)calloc(1, sizeof(xce_res_account_t));
TEST_ASSERT_NOT_NULL(acct->accounts[0]->accounts[0]);
acct->accounts[0]->accounts[0]->resname = strdup("mem");
acct->accounts[0]->accounts[0]->residx = MEM_RESIDX;
acct->accounts[0]->accounts[0]->used = 0.0;
acct->accounts[0]->accounts[0]->reserved = MEM_RESERVED_MB;
acct->accounts[0]->accounts[0]->allocated = MEM_RESERVED_MB;
return acct;
}
static xce_job_rusages_t* make_mem_rusages(float memory_rss_mb)
{
xce_job_rusages_t* r = (xce_job_rusages_t*)calloc(1, sizeof(*r));
TEST_ASSERT_NOT_NULL(r);
r->count = 1;
r->usages = (xce_rusage_t**)calloc(1, sizeof(*r->usages));
TEST_ASSERT_NOT_NULL(r->usages);
r->usages[0] = (xce_rusage_t*)calloc(1, sizeof(xce_rusage_t));
TEST_ASSERT_NOT_NULL(r->usages[0]);
r->usages[0]->resname = strdup("memory_rss");
r->usages[0]->value = memory_rss_mb;
return r;
}
static float get_worker_resinuse(xce_sched_worker_t* w, int residx)
{
int idx[1] = { residx };
float val[1];
get_resvals(&(w->resInUse), 1, idx, val);
return val[0];
}
/*
* test_reconcile_host_resource_usage_below_reservation_keeps_reservation -
* resource-accounting.md case (a): actual usage below the reservation must
* still charge the FULL reservation (used stays 0, since overage is 0).
*/
static void
test_reconcile_host_resource_usage_below_reservation_keeps_reservation(void)
{
xce_scheduler_t sched;
xce_sched_worker_t* w;
xce_res_account_t acct_entry = { 0 };
int rc;
memset(&sched, 0, sizeof(sched));
setup_core_with_worker(&(sched.core), WORKER_NAME);
w = sched_get_worker(&(sched.core), WORKER_NAME);
TEST_ASSERT_NOT_NULL(w);
acct_entry.residx = MEM_RESIDX;
acct_entry.reserved = MEM_RESERVED_MB;
acct_entry.used = 0.0;
/* First charge: reservation only (mirrors sched_charge_account() at
* dispatch time -- used=0, reserved=4 -> pool charge 4). */
rc = sched_reconcile_host_resource_usage(&(sched.core), w, &acct_entry,
0.0);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_FLOAT(0.0, (float)acct_entry.used);
/* actual usage (1MB) is below the 4MB reservation -- used must stay 0,
* the pool must still show the full 4MB reservation charged, not the
* 1MB actually in use. */
rc = sched_reconcile_host_resource_usage(&(sched.core), w, &acct_entry,
1.0);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_FLOAT(0.0, (float)acct_entry.used);
}
/*
* test_reconcile_host_resource_usage_above_reservation_charges_actual -
* resource-accounting.md case (b): actual usage at/above the reservation
* must charge the REAL usage instead of the stale reservation.
*/
static void
test_reconcile_host_resource_usage_above_reservation_charges_actual(void)
{
xce_scheduler_t sched;
xce_sched_worker_t* w;
xce_res_account_t acct_entry = { 0 };
int rc;
int idx[1] = { MEM_RESIDX };
float val[1];
memset(&sched, 0, sizeof(sched));
setup_core_with_worker(&(sched.core), WORKER_NAME);
w = sched_get_worker(&(sched.core), WORKER_NAME);
TEST_ASSERT_NOT_NULL(w);
acct_entry.residx = MEM_RESIDX;
acct_entry.reserved = MEM_RESERVED_MB;
acct_entry.used = 0.0;
/* Seed the pool with the initial reservation charge, exactly as
* sched_charge_account() would at dispatch time. */
val[0] = (float)(acct_entry.used + acct_entry.reserved);
put_resvals(&(w->resInUse), 1, idx, val);
/* actual usage (6MB) EXCEEDS the 4MB reservation -- overage = 2MB, so
* used must become 2 and the pool must show 6 (reserved 4 + overage 2 =
* actual 6), not the stale 4MB reservation. */
rc = sched_reconcile_host_resource_usage(&(sched.core), w, &acct_entry,
6.0);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_FLOAT(2.0, (float)acct_entry.used);
TEST_ASSERT_EQUAL_FLOAT(6.0, get_worker_resinuse(w, MEM_RESIDX));
/* usage drops back to 3MB (still >= reserved? no: 3 < 4) -- back to
* charging the flat reservation, pool returns to 4. */
rc = sched_reconcile_host_resource_usage(&(sched.core), w, &acct_entry,
3.0);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_FLOAT(0.0, (float)acct_entry.used);
TEST_ASSERT_EQUAL_FLOAT(4.0, get_worker_resinuse(w, MEM_RESIDX));
}
/*
* test_reconcile_host_resource_usage_repeated_growth_applies_only_delta -
* confirms the pool is updated by DELTA each call, not a full recompute
* that would double-count a sibling resource's own charge in the same
* pool array (resource.c's own doc comment on this function).
*/
static void
test_reconcile_host_resource_usage_repeated_growth_applies_only_delta(void)
{
xce_scheduler_t sched;
xce_sched_worker_t* w;
xce_res_account_t acct_entry = { 0 };
int slots_idx[1] = { 0 }; /* a sibling resource */
float slots_val[1] = { 7.0f };
int rc;
memset(&sched, 0, sizeof(sched));
setup_core_with_worker(&(sched.core), WORKER_NAME);
w = sched_get_worker(&(sched.core), WORKER_NAME);
TEST_ASSERT_NOT_NULL(w);
/* A sibling resource ("slots", a different residx) already charged in
* the same pool array -- must be untouched by every call below. */
put_resvals(&(w->resInUse), 1, slots_idx, slots_val);
acct_entry.residx = MEM_RESIDX;
acct_entry.reserved = MEM_RESERVED_MB;
acct_entry.used = 0.0;
rc = sched_reconcile_host_resource_usage(&(sched.core), w, &acct_entry,
5.0); /* overage 1 */
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_FLOAT(1.0, get_worker_resinuse(w, MEM_RESIDX));
rc = sched_reconcile_host_resource_usage(&(sched.core), w, &acct_entry,
9.0); /* overage 5 -- delta +4 */
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_FLOAT(5.0, get_worker_resinuse(w, MEM_RESIDX));
/* Sibling resource never touched by any of the above. */
TEST_ASSERT_EQUAL_FLOAT(7.0, get_worker_resinuse(w, 0));
}
static void test_reconcile_host_resource_usage_null_args_return_einval(void)
{
xce_scheduler_t sched;
xce_res_account_t acct_entry = { 0 };
memset(&sched, 0, sizeof(sched));
setup_core_with_worker(&(sched.core), WORKER_NAME);
TEST_ASSERT_EQUAL_INT(
-EINVAL, sched_reconcile_host_resource_usage(NULL, NULL, NULL, 0.0));
TEST_ASSERT_EQUAL_INT(-EINVAL,
sched_reconcile_host_resource_usage(
&(sched.core), NULL, &acct_entry, 0.0));
}
/*
* test_reconcile_job_resource_usage_* - the rusages-report wrapper: resname
* mapping ("memory_rss" on the wire -> "mem" in the account), the single-
* host scope, and the no-request no-op.
*/
static void
test_reconcile_job_resource_usage_maps_memory_rss_to_mem_account(void)
{
xce_scheduler_t sched;
xce_sched_worker_t* w;
xce_job_account_t* jacct;
xce_job_rusages_t* rusages;
int rc;
memset(&sched, 0, sizeof(sched));
setup_core_with_worker(&(sched.core), WORKER_NAME);
w = sched_get_worker(&(sched.core), WORKER_NAME);
TEST_ASSERT_NOT_NULL(w);
jacct = make_mem_account(WORKER_NAME);
rusages = make_mem_rusages(7.0f); /* > 4MB reserved -> overage 3 */
/* Seed the pool with the initial dispatch-time reservation charge
* (mirrors what sched_charge_account()/sched_commit_job_alloc() would
* already have done before any rusage tick ever arrives -- the
* function under test only ever applies a DELTA on top of this, per
* its own doc comment, never a from-scratch total). */
{
int idx[1] = { MEM_RESIDX };
float val[1] = { (float)MEM_RESERVED_MB };
put_resvals(&(w->resInUse), 1, idx, val);
}
rc = sched_reconcile_job_resource_usage(&(sched.core), jacct, rusages);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_FLOAT(3.0,
(float)jacct->accounts[0]->accounts[0]->used);
TEST_ASSERT_EQUAL_FLOAT(7.0, get_worker_resinuse(w, MEM_RESIDX));
}
static void test_reconcile_job_resource_usage_skips_multihost_jobs(void)
{
xce_scheduler_t sched;
xce_job_account_t jacct = { 0 };
xce_job_rusages_t* rusages = make_mem_rusages(99.0f);
int rc;
memset(&sched, 0, sizeof(sched));
setup_core_with_worker(&(sched.core), WORKER_NAME);
jacct.nhosts = 2; /* multi-host: no per-report host attribution possible
* (see sched_reconcile_job_resource_usage()'s own
* doc comment) -- must be a documented no-op. */
rc = sched_reconcile_job_resource_usage(&(sched.core), &jacct, rusages);
TEST_ASSERT_EQUAL_INT(0, rc);
}
static void
test_reconcile_job_resource_usage_skips_when_no_mem_request(void)
{
xce_scheduler_t sched;
xce_job_account_t* jacct;
xce_job_rusages_t* rusages = make_mem_rusages(99.0f);
int rc;
memset(&sched, 0, sizeof(sched));
setup_core_with_worker(&(sched.core), WORKER_NAME);
/* A single-host account with NO "mem" entry at all -- resource-
* accounting.md's "no rusage[mem=...] means reserved=0, nothing to
* reconcile" case. */
jacct = (xce_job_account_t*)calloc(1, sizeof(*jacct));
jacct->nhosts = 1;
jacct->accounts = (xce_host_account_t**)calloc(1, sizeof(*jacct->accounts));
jacct->accounts[0] = (xce_host_account_t*)calloc(1, sizeof(xce_host_account_t));
jacct->accounts[0]->host = strdup(WORKER_NAME);
jacct->accounts[0]->naccount = 0;
rc = sched_reconcile_job_resource_usage(&(sched.core), jacct, rusages);
TEST_ASSERT_EQUAL_INT(0, rc);
}
static void test_reconcile_job_resource_usage_null_args_return_einval(void)
{
xce_scheduler_t sched;
memset(&sched, 0, sizeof(sched));
setup_core_with_worker(&(sched.core), WORKER_NAME);
TEST_ASSERT_EQUAL_INT(-EINVAL,
sched_reconcile_job_resource_usage(NULL, NULL, NULL));
}
int main(void)
{
UNITY_BEGIN();
RUN_TEST(test_charge_account_identical_across_core_containers);
RUN_TEST(test_commit_job_alloc_identical_across_core_containers);
RUN_TEST(
test_reconcile_host_resource_usage_below_reservation_keeps_reservation);
RUN_TEST(
test_reconcile_host_resource_usage_above_reservation_charges_actual);
RUN_TEST(
test_reconcile_host_resource_usage_repeated_growth_applies_only_delta);
RUN_TEST(test_reconcile_host_resource_usage_null_args_return_einval);
RUN_TEST(test_reconcile_job_resource_usage_maps_memory_rss_to_mem_account);
RUN_TEST(test_reconcile_job_resource_usage_skips_multihost_jobs);
RUN_TEST(test_reconcile_job_resource_usage_skips_when_no_mem_request);
RUN_TEST(test_reconcile_job_resource_usage_null_args_return_einval);
return UNITY_END();
}