myxce/src/scheduler/testsuite/test_picker_wiring.c
Rongsong Shen 87a2a46b94
feat(scheduler): reconcile mem reservation against real usage
Adds reservation-vs-actual-usage reconciliation for real (gauge-style)
resources, using mem as the concrete case -- documented in the new
resource-accounting.md steering doc.

Policy: charged = max(reserved, actual). A job's rusage[mem=...]
request is an estimate made before it has run, and the real usage the
worker later reports may land above or below it; charging a flat
reservation for the job's whole life either starves other jobs of
memory it never uses (when actual stays below the estimate) or lets
the host be driven into genuine oversubscription (when actual exceeds
it). Available memory for scheduling is then the host's OS-reported
available figure minus, per job, only the UNUSED SLACK of the
reservation (max(0, reserved - used)) -- never the job's actual usage
on top of that, since the OS's own figure already reflects it.

The existing xce_res_account_t{used, reserved, allocated} fields and
sched_update_res_accounts()'s used+reserved charge formula already
implement this exactly, once `used` is set to the OVERAGE beyond the
reservation (max(0, actual - reserved)) rather than left at its
permanent 0.0 default -- nothing previously computed it.

New (resource.c): sched_reconcile_host_resource_usage() sets that
overage on one xce_res_account_t and applies just the DELTA to the
affected worker's resInUse (not a full uncharge/recharge of the whole
job_account, which would needlessly re-touch sibling resources like
"slots" charged in the same account). sched_reconcile_job_resource_usage()
is the wrapper driven from a raw rusage report: maps the worker's real
wire resname ("memory_rss", same worker-resname-vs-scheduler-resource-
name split bug-1735 already established for the controller side) to
the scheduler's "mem" resource, and is a no-op when the job spans more
than one host (the wire report carries no per-host attribution today --
a separate, undocumented-until-now gap, out of this fix's scope) or
never requested rusage[mem=...] in the first place (reserved=0, nothing
to reconcile).

Wired into sched_apply_job_rusage() (schedobj.c), which already
receives every job rusage tick but previously only stored it for
display/plugin hooks with no resource-accounting effect at all.

Compatible with sched_account_jobs()'s existing per-round resInUse
reset+re-derive cycle (bug-1624): the reconciled `used` value is
persisted on the job's own account record, which that re-derivation
already reads from, not just a transient pool-level delta that a
round boundary would wipe.

Added link-satisfying stubs to the three white-box test files that
#include schedobj.c directly (test_job_update.c,
test_picker_config_fields.c, test_picker_wiring.c) -- none of them
exercise real job->accounts, so the new call is unreachable there, but
still needs to link. 10 new regression tests in
test_resource_accounting.c cover the core formula (below/at/above the
reservation, delta-only pool updates, sibling-resource isolation) and
the wrapper's scoping rules (resname mapping, multi-host skip,
no-request no-op).

Full scheduler suite: 55/55 passing, zero regressions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
2026-08-31 07:02:32 +00:00

505 lines
20 KiB
C

/*
* test_picker_wiring.c - white-box tests for wiring builtin_fcfs in as the
* registered default picker and removing the hardcoded picker string
* (scheduler-core spec, task 2.4, Req 1 AC 6, Req 2 AC 5, Req 3 AC 1).
*
* RED: before task 2.4, register_builtin_fcfs() did not exist (declared in
* plugin.h but no body in builtin_fcfs.c) -- this file failed to link.
* sched_resolve_job_picker() (schedobj.c, task 2.4) existed but nothing
* had ever registered "builtin_fcfs" for it to find, so the "default
* resolves to builtin_fcfs" half of this file's coverage could not pass;
* sched_job_picker() (scheduler.c) still hardcoded "xce_sched_fcfs".
* GREEN: register_builtin_fcfs() registers builtin_fcfs's native
* xce_job_picker_iface_t under "builtin_fcfs" via the real plugin_register()
* (job_elect_fn stays NULL -- no legacy-wrapper substitution);
* sched_set_default_job_pick_plugin() + sched_resolve_job_picker() together
* resolve an unset job_pick_plugin to the registered builtin_fcfs plugin; an
* unregistered configured name fails resolution with -ENOENT (distinct from
* -EINVAL for "registered but not a valid picker" and "unset name") rather
* than silently falling back to anything.
*
* Scope note (this file does NOT drive sched_job_picker() itself): that
* function is `static` in scheduler.c, which has a much larger external
* dependency surface (xcecli_ctx_*, sel_register_alloc_thread, resource.c's
* allocation path, decision.c's decision constructors, ...) than any other
* white-box test in this directory pulls in -- no existing test #includes
* scheduler.c directly (test.c only pulls in resource.c), and doing so here
* would mean stubbing on the order of twenty external symbols this file's
* tests would never actually exercise, just to satisfy the linker. Per the
* task's own guidance to prefer a tractable direct test over spinning real
* threads, this file instead directly tests the exact condition
* sched_job_picker() branches on
* (`job_picker->job_elect_fn != NULL` selects the legacy delegation path,
* `== NULL` selects the generic picker_iface loop): resolving a
* legacy-registered plugin (job_elect_fn set, no picker_iface of its own)
* comes back with job_elect_fn non-NULL and picker_iface pointing at
* plugin.c's legacy_picker_wrapper_iface (visible in this TU -- see below);
* resolving builtin_fcfs comes back with job_elect_fn NULL and a fully
* populated native picker_iface. That is the entire input the branch reads.
*
* #include's the real plugin.c, schedobj.c, and plugins/builtin_fcfs.c
* (white-box, same convention as test_plugin_picker_registration.c /
* test_picker_config_fields.c / test_picker_fcfs.c) so this test exercises
* the real plugin_register(), the real sched_set_default_job_pick_plugin()/
* sched_resolve_job_picker(), and the real register_builtin_fcfs() -- not a
* re-implementation of any of them. No symbol name collisions exist between
* the three files (verified: distinct static helper names throughout).
*/
#include <errno.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <tlog.h>
#include <unity/unity.h>
#include "kvmgr.h"
#include "scheduler.h"
#include "xstring.h"
/* ---------------------------------------------------------------------- */
/* Stubs for externs referenced by plugin.c/schedobj.c/builtin_fcfs.c that
* are defined in other .c files not linked into this whole-file-#include
* TU. None of this file's tests exercise these paths -- present only to
* satisfy the linker, same stub set (minus plugin_job_update/
* plugin_queue_update/plugin_job_rusage_update/plugin_prepare_allocation,
* which the REAL plugin.c now provides in this TU) as
* test_picker_config_fields.c uses for its own schedobj.c-only include.
* ---------------------------------------------------------------------- */
/* sched_get_user/sched_get_project/sched_get_queue/sched_get_job/
* sched_add_user/sched_add_project relocated to core.h (scheduler-core
* spec, task 4.1) -- now operate on xce_sched_core_t*, not
* xce_scheduler_t*. Kept as stubs here (not pulling in core.c's real
* implementations): this file never exercises make_sched_job/queue
* lookup, only picker registration/resolution. */
xce_sched_user_t* sched_get_user(xce_sched_core_t* s, char* n)
{
(void)s;
(void)n;
return NULL;
}
xce_sched_project_t* sched_get_project(xce_sched_core_t* s, char* n)
{
(void)s;
(void)n;
return NULL;
}
xce_sched_queue_t* sched_get_queue(xce_sched_core_t* s, char* n)
{
(void)s;
(void)n;
return NULL;
}
xce_sched_job_t* sched_get_job(xce_sched_core_t* s, uint64_t jobid)
{
(void)s;
(void)jobid;
return NULL;
}
xce_job_account_t* make_job_account(xce_job_alloc_t* a)
{
(void)a;
return NULL;
}
void free_job_account(xce_job_account_t* a) { (void)a; }
/* resource-accounting.md's reconciliation (resource.c, not linked here) --
* link-satisfying stub only, same rationale as every other stub here. */
int sched_reconcile_job_resource_usage(xce_sched_core_t* c,
xce_job_account_t* a,
xce_job_rusages_t* r)
{
(void)c; (void)a; (void)r; return 0;
}
int sched_add_user(xce_sched_core_t* s, xce_sched_user_t* u)
{
(void)s;
(void)u;
return 0;
}
int sched_add_project(xce_sched_core_t* s, xce_sched_project_t* p)
{
(void)s;
(void)p;
return 0;
}
void sched_worker_release_host_ctx(xce_sched_worker_t* w) { (void)w; }
/* plugin.c's plugin_prepare_allocation() (a category unrelated to picker
* resolution) calls this -- stub matching test_plugin_picker_registration.c's
* own. */
int sched_fill_limit_requests(xce_scheduler_t* sched, xvec_idval_t* req,
xlist_t* limits)
{
(void)sched;
(void)req;
(void)limits;
return 0;
}
/* builtin_fcfs.c's next_job() (compiled into this TU even though this
* file's tests never call it) calls these -- always-ready stub, same
* convention as test_picker_fcfs.c. */
int sched_queue_is_ready(xce_sched_queue_t* q)
{
(void)q;
return 1;
}
int sched_job_is_ready(xce_sched_job_t* j)
{
(void)j;
return 1;
}
/* Pull in the real implementations under test. */
#include "../src/plugin.c"
#include "../src/plugins/builtin_fcfs.c"
#include "../src/schedobj.c"
void setUp(void) { return; }
void tearDown(void) { return; }
/* ---------------------------------------------------------------------- */
/* Fixture: a real sched->core.plugins kvmgr (same setup scheduler.c's
* sched_plugin_init() uses) plus the xstring/CSE intern tables
* sched_set_default_job_pick_plugin()'s makeString() call needs. Nothing is
* torn down between tests (suite convention, ASAN_OPTIONS=detect_leaks=0). */
/* ---------------------------------------------------------------------- */
static void fixture_init(xce_scheduler_t* sched)
{
kvmgr_params_t params = { .key_type = KVMGR_KEY_STRING,
.nslots = 16,
.delta = 8,
.datakiller = plugin_unregister };
memset(sched, 0, sizeof(*sched));
resreq_init(1);
xstring_table_init(1024 * 16);
sched->core.plugins = makeInstance(&xkvmgr_type, &params);
TEST_ASSERT_NOT_NULL(sched->core.plugins);
}
/* Minimal legacy (job_elect_fn-only) plugin, mirroring
* test_plugin_picker_registration.c's legacy_plugin_init -- a standalone
* stand-in for plugins/fcfs.c's actual shape, not a dependency on fcfs.c
* itself. */
static void* legacy_job_elect_fn_stub(void* arg)
{
(void)arg;
return NULL;
}
static xce_sched_plugin_t* make_legacy_plugin(char* name)
{
xce_sched_plugin_t* plugin
= (xce_sched_plugin_t*)calloc(1, sizeof(*plugin));
TEST_ASSERT_NOT_NULL(plugin);
plugin->name = name;
plugin->job_elect_fn = (plugin_job_election_fn_t)legacy_job_elect_fn_stub;
return plugin;
}
/* ---------------------------------------------------------------------- */
/* Req 3 AC 1: register_builtin_fcfs() */
/* ---------------------------------------------------------------------- */
static void test_register_builtin_fcfs_registers_under_name_builtin_fcfs(void)
{
xce_scheduler_t sched;
xce_sched_plugin_t* found = NULL;
int rc;
fixture_init(&sched);
rc = register_builtin_fcfs(&sched);
TEST_ASSERT_GREATER_OR_EQUAL(0, rc);
lock(sched.core.plugins, RTTI_LOCK_READ);
rc = key_find(sched.core.plugins, "builtin_fcfs", (void**)&found);
unlock(sched.core.plugins);
TEST_ASSERT_GREATER_THAN(0, rc);
TEST_ASSERT_NOT_NULL(found);
TEST_ASSERT_EQUAL_STRING("builtin_fcfs", found->name);
}
/* builtin_fcfs registers as a NATIVE picker: picker_iface non-NULL,
* job_elect_fn NULL -- this is the exact input sched_job_picker()'s branch
* condition reads (see this file's own header comment). Also confirms
* every MUST hook from tasks 2.1-2.3 is wired, and on_config/on_job_rusage
* are NULL per Req 3 AC 2's table. */
static void test_register_builtin_fcfs_is_native_not_legacy(void)
{
xce_scheduler_t sched;
xce_sched_plugin_t* found = NULL;
fixture_init(&sched);
TEST_ASSERT_GREATER_OR_EQUAL(0, register_builtin_fcfs(&sched));
lock(sched.core.plugins, RTTI_LOCK_READ);
key_find(sched.core.plugins, "builtin_fcfs", (void**)&found);
unlock(sched.core.plugins);
TEST_ASSERT_NOT_NULL(found);
TEST_ASSERT_NULL(found->job_elect_fn);
TEST_ASSERT_NOT_NULL(found->picker_iface);
TEST_ASSERT_EQUAL_PTR(builtin_fcfs_next_job, found->picker_iface->next_job);
TEST_ASSERT_EQUAL_PTR(builtin_fcfs_begin_session,
found->picker_iface->begin_session);
TEST_ASSERT_EQUAL_PTR(builtin_fcfs_end_session,
found->picker_iface->end_session);
TEST_ASSERT_EQUAL_PTR(builtin_fcfs_on_dispatch,
found->picker_iface->on_dispatch);
TEST_ASSERT_EQUAL_PTR(builtin_fcfs_on_pend, found->picker_iface->on_pend);
TEST_ASSERT_EQUAL_PTR(builtin_fcfs_on_queue_update,
found->picker_iface->on_queue_update);
TEST_ASSERT_EQUAL_PTR(builtin_fcfs_on_job_push,
found->picker_iface->on_job_push);
TEST_ASSERT_EQUAL_PTR(builtin_fcfs_on_job_state_change,
found->picker_iface->on_job_state_change);
TEST_ASSERT_EQUAL_PTR(builtin_fcfs_on_job_update,
found->picker_iface->on_job_update);
TEST_ASSERT_NULL(found->picker_iface->on_config);
TEST_ASSERT_NULL(found->picker_iface->on_job_rusage);
}
/* register_builtin_fcfs() wires init_fn/fini_fn adapters into the EXISTING
* plugin_register()/plugin_unregister() call sites (see builtin_fcfs.c's
* own comment on why) -- confirms init actually ran: plugin->data is
* non-NULL immediately after registration, not left NULL until some other
* call. */
static void test_register_builtin_fcfs_runs_init_via_init_fn_adapter(void)
{
xce_scheduler_t sched;
xce_sched_plugin_t* found = NULL;
fixture_init(&sched);
TEST_ASSERT_GREATER_OR_EQUAL(0, register_builtin_fcfs(&sched));
lock(sched.core.plugins, RTTI_LOCK_READ);
key_find(sched.core.plugins, "builtin_fcfs", (void**)&found);
unlock(sched.core.plugins);
TEST_ASSERT_NOT_NULL(found);
TEST_ASSERT_NOT_NULL(found->data);
}
/* ---------------------------------------------------------------------- */
/* Req 1 AC 6 / Req 2 AC 1 / Req 2 AC 5: default resolution + fail-loud */
/* ---------------------------------------------------------------------- */
/* tasks.md Observable, half 1: "starting the scheduler with no picker
* config uses builtin_fcfs automatically." */
static void test_unset_job_pick_plugin_resolves_to_builtin_fcfs_by_default(void)
{
xce_scheduler_t sched;
xce_sched_plugin_t* resolved = NULL;
int rc;
fixture_init(&sched);
TEST_ASSERT_GREATER_OR_EQUAL(0, register_builtin_fcfs(&sched));
TEST_ASSERT_NULL(sched.core.job_pick_plugin);
sched_set_default_job_pick_plugin(&sched);
TEST_ASSERT_EQUAL_STRING("builtin_fcfs", sched.core.job_pick_plugin);
rc = sched_resolve_job_picker(&sched.core, &resolved);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_NOT_NULL(resolved);
TEST_ASSERT_EQUAL_STRING("builtin_fcfs", resolved->name);
/* This is the exact condition sched_job_picker() (scheduler.c) branches
* on: NULL job_elect_fn selects the generic picker_iface loop. */
TEST_ASSERT_NULL(resolved->job_elect_fn);
TEST_ASSERT_NOT_NULL(resolved->picker_iface);
}
/* tasks.md Observable, half 2: "starting it with an unregistered picker
* name fails startup with a clear error instead of silently falling
* back." -ENOENT specifically (distinct from -EINVAL below) --
* sched_resolve_job_picker's documented contract (core.h). */
static void
test_unregistered_configured_name_fails_with_enoent_not_silent_fallback(void)
{
xce_scheduler_t sched;
xce_sched_plugin_t* resolved = (xce_sched_plugin_t*)0x1; /* sentinel */
int rc;
fixture_init(&sched);
TEST_ASSERT_GREATER_OR_EQUAL(0, register_builtin_fcfs(&sched));
sched.core.job_pick_plugin = "totally_unregistered_picker_name";
rc = sched_resolve_job_picker(&sched.core, &resolved);
TEST_ASSERT_EQUAL_INT(-ENOENT, rc);
/* Not a silent fallback to builtin_fcfs (still registered) or to
* whatever the sentinel was -- *out is reset to NULL on any failure
* path (schedobj.c's sched_resolve_job_picker, `*out = NULL;` up
* front). */
TEST_ASSERT_NULL(resolved);
}
/* A DIFFERENT failure mode than "not registered at all": the name IS
* registered, but the registered plugin is neither a native picker nor a
* legacy job_elect_fn plugin (e.g. today's limit/allocator categories) --
* -EINVAL, not -ENOENT. Confirms the two failure modes are genuinely
* distinguishable, not the same generic "no" collapsed into one code. */
static void test_registered_but_not_a_picker_fails_with_einval_not_enoent(void)
{
xce_scheduler_t sched;
xce_sched_plugin_t* non_picker = NULL;
xce_sched_plugin_t* resolved = NULL;
int rc;
fixture_init(&sched);
non_picker = (xce_sched_plugin_t*)calloc(1, sizeof(*non_picker));
TEST_ASSERT_NOT_NULL(non_picker);
non_picker->name = "not_a_picker";
/* job_elect_fn and picker_iface both left NULL -- today's
* allocator/limit plugin shape (register_fcfs's own sibling
* categories), which plugin_register() leaves untouched (task 1.2). */
TEST_ASSERT_GREATER_OR_EQUAL(0, plugin_register(&sched, non_picker));
sched.core.job_pick_plugin = "not_a_picker";
rc = sched_resolve_job_picker(&sched.core, &resolved);
TEST_ASSERT_EQUAL_INT(-EINVAL, rc);
TEST_ASSERT_NULL(resolved);
}
/* Unset job_pick_plugin, and the caller never called
* sched_set_default_job_pick_plugin() either -- -EINVAL (scheduler.h's
* documented contract), distinct from both cases above. */
static void
test_unset_job_pick_plugin_without_defaulting_fails_with_einval(void)
{
xce_scheduler_t sched;
xce_sched_plugin_t* resolved = NULL;
int rc;
fixture_init(&sched);
TEST_ASSERT_NULL(sched.core.job_pick_plugin);
rc = sched_resolve_job_picker(&sched.core, &resolved);
TEST_ASSERT_EQUAL_INT(-EINVAL, rc);
TEST_ASSERT_NULL(resolved);
}
/* ---------------------------------------------------------------------- */
/* sched_job_picker()'s legacy-vs-native branch condition (scheduler.c) --
* see this file's header comment for why this is tested at the resolved-
* plugin level rather than by driving the picker thread itself.
* ---------------------------------------------------------------------- */
/* Legacy path input: a job_elect_fn-only plugin, once registered and
* resolved, comes back with job_elect_fn set AND picker_iface pointing at
* plugin.c's legacy_picker_wrapper_iface (visible in this TU -- same
* assertion test_plugin_picker_registration.c makes for plugin_register()
* itself). This is exactly the condition
* `if (job_picker->job_elect_fn != NULL) return
* job_picker->job_elect_fn(job_picker);` in sched_job_picker() reads to
* choose the "delegate to fcfs_elect_job-style forever loop" path. */
static void
test_resolved_legacy_plugin_has_job_elect_fn_set_selecting_legacy_branch(void)
{
xce_scheduler_t sched;
xce_sched_plugin_t* legacy = NULL;
xce_sched_plugin_t* resolved = NULL;
int rc;
fixture_init(&sched);
legacy = make_legacy_plugin("test_legacy_picker_for_wiring");
TEST_ASSERT_GREATER_OR_EQUAL(0, plugin_register(&sched, legacy));
sched.core.job_pick_plugin = "test_legacy_picker_for_wiring";
rc = sched_resolve_job_picker(&sched.core, &resolved);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_NOT_NULL(resolved);
TEST_ASSERT_NOT_NULL(resolved->job_elect_fn);
TEST_ASSERT_EQUAL_PTR(&legacy_picker_wrapper_iface, resolved->picker_iface);
}
/* Native path input: builtin_fcfs, once registered and resolved, comes
* back with job_elect_fn NULL -- the condition that selects the generic
* picker_iface session loop instead. Distinct test from the "default"
* test above: this one asserts on an EXPLICITLY configured name, not the
* unset-defaults-to-builtin_fcfs path, isolating the branch condition
* itself from the defaulting behavior. */
static void
test_resolved_builtin_fcfs_has_job_elect_fn_null_selecting_native_branch(void)
{
xce_scheduler_t sched;
xce_sched_plugin_t* resolved = NULL;
int rc;
fixture_init(&sched);
TEST_ASSERT_GREATER_OR_EQUAL(0, register_builtin_fcfs(&sched));
sched.core.job_pick_plugin = "builtin_fcfs"; /* explicit, not defaulted */
rc = sched_resolve_job_picker(&sched.core, &resolved);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_NOT_NULL(resolved);
TEST_ASSERT_NULL(resolved->job_elect_fn);
TEST_ASSERT_NOT_NULL(resolved->picker_iface);
TEST_ASSERT_EQUAL_PTR(builtin_fcfs_next_job,
resolved->picker_iface->next_job);
}
/* Both builtin_fcfs and a legacy plugin can be registered simultaneously
* (Req 3 AC 1's "wire builtin_fcfs as the registered default" does not
* mean "the only" -- tasks.md's own builtin_fcfs.c header comment: fcfs.c
* "stays registered ... reachable only by explicit config"). Confirms
* choosing one by name does not disturb the other's own registration or
* resolution. */
static void
test_legacy_and_native_pickers_coexist_and_resolve_independently(void)
{
xce_scheduler_t sched;
xce_sched_plugin_t* legacy = NULL;
xce_sched_plugin_t* resolved = NULL;
int rc;
fixture_init(&sched);
legacy = make_legacy_plugin("xce_sched_fcfs_stub");
TEST_ASSERT_GREATER_OR_EQUAL(0, plugin_register(&sched, legacy));
TEST_ASSERT_GREATER_OR_EQUAL(0, register_builtin_fcfs(&sched));
sched.core.job_pick_plugin = "xce_sched_fcfs_stub";
rc = sched_resolve_job_picker(&sched.core, &resolved);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_NOT_NULL(resolved->job_elect_fn);
sched.core.job_pick_plugin = "builtin_fcfs";
rc = sched_resolve_job_picker(&sched.core, &resolved);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_NULL(resolved->job_elect_fn);
}
int main(void)
{
UNITY_BEGIN();
RUN_TEST(test_register_builtin_fcfs_registers_under_name_builtin_fcfs);
RUN_TEST(test_register_builtin_fcfs_is_native_not_legacy);
RUN_TEST(test_register_builtin_fcfs_runs_init_via_init_fn_adapter);
RUN_TEST(test_unset_job_pick_plugin_resolves_to_builtin_fcfs_by_default);
RUN_TEST(
test_unregistered_configured_name_fails_with_enoent_not_silent_fallback);
RUN_TEST(test_registered_but_not_a_picker_fails_with_einval_not_enoent);
RUN_TEST(test_unset_job_pick_plugin_without_defaulting_fails_with_einval);
RUN_TEST(
test_resolved_legacy_plugin_has_job_elect_fn_set_selecting_legacy_branch);
RUN_TEST(
test_resolved_builtin_fcfs_has_job_elect_fn_null_selecting_native_branch);
RUN_TEST(test_legacy_and_native_pickers_coexist_and_resolve_independently);
return UNITY_END();
}