diff --git a/.kiro/specs/scheduler-sim-decision-trace/design.md b/.kiro/specs/scheduler-sim-decision-trace/design.md index 25e9be23..5d8b9a29 100644 --- a/.kiro/specs/scheduler-sim-decision-trace/design.md +++ b/.kiro/specs/scheduler-sim-decision-trace/design.md @@ -551,7 +551,7 @@ Each hook is one `SCHED_TRACE(core, ...)` statement placed after the value it re - `; no plan tried: all generated plans discarded` (4.9) when g > 0 and d = g; - `; no plan could be placed` (4.6) when rc = 0, t > 0 and no plan ended in an internal error. -When rc > 0 or rc < 0 there is no statement; the commit or internal-error entries already say what happened. Like every other entry it counts against the line budget. It also bumps the matching decision-wide counter (requirements with no plan generated / all discarded / none placed), under the same conditions, which appear in the fixed-size totals block after truncation (9.3). | 4.6, 4.7, 4.9 | +When rc > 0 or rc < 0 there is no statement; the commit or internal-error entries already say what happened. A declined requirement and a generator that cannot start end the runner before the search end: their `no_plan` entry is the requirement's "no plan generated" line and bumps that counter itself, once; a search end after a `no_plan` entry in the same requirement makes no statement and counts nothing. Like every other entry it counts against the line budget. It also bumps the matching decision-wide counter (requirements with no plan generated / all discarded / none placed), under the same conditions, which appear in the fixed-size totals block after truncation (9.3). | 4.6, 4.7, 4.9 | | Plan loop | malformed / reserve failure / try / result | `plan_discarded`; `plan_not_tried(err)`; `plan_begin`; `plan_end(rc, failed_pos)` | 4.3, 4.4, 4.6, 4.9 | | Position | `sched_claim_bucket_` | `pos_begin`; iterator init failure → `pos_end(SCAN_ERROR)`; snapshot failure after acceptance (`alloc_runner.c:237-243`) → `pos_end(INTERNAL)`; else `pos_end(FILLED/UNFILLED/SCAN_ERROR, host)` | 4.4, 5.9 | | Candidate | claimed skip / per-host result | `host_begin` + `host_verdict(CLAIMED)`; `host_begin`, call, `host_verdict(rc>0 ? ACCEPTED : rc==0 ? from rows : INTERNAL, err)` | 5.1–5.7 | diff --git a/.kiro/specs/scheduler-sim-decision-trace/tasks.md b/.kiro/specs/scheduler-sim-decision-trace/tasks.md index 4109cabb..f8670700 100644 --- a/.kiro/specs/scheduler-sim-decision-trace/tasks.md +++ b/.kiro/specs/scheduler-sim-decision-trace/tasks.md @@ -123,7 +123,7 @@ - _Requirements: 11.2, 11.4_ - _Depends: 2.3_ -- [ ] 3. Allocator-thread hooks at the decision seams +- [x] 3. Allocator-thread hooks at the decision seams - [x] 3.1 Hook the allocator loop: position, record lifecycle, selection and no-decision paths (P) - **Position:** when session settings are present, every real job pop increments it. - **Record start:** a wanted job starts a record with its selection facts; its cluster-wide limits are added after election. @@ -163,7 +163,7 @@ - _Boundary: TraceHooks (host iterator, satisfaction check)_ - _Depends: 2.2_ -- [ ] 3.4 Hook the allocator runner: cluster-wide check, plans, positions, candidates and finalize +- [x] 3.4 Hook the allocator runner: cluster-wide check, plans, positions, candidates and finalize - The runner records: - an allocator decline and a generator start failure, as no-plan reasons; - the cluster-wide check scope and its end, with "no plans tried" on failure; @@ -363,3 +363,4 @@ - 3.1: allocator-loop hooks live in `sched_allocator()` (sched_election.c), with helpers `sched_trace_push_no_decision()` and `sched_trace_attach()`. Selection is captured at job lookup, before election, so NO_SCHEDOBJ still has its selection; cluster-wide limits are captured after election. The position counter (`ctl->position`) is not SCHED_TRACE-gated (it creates the record) and counts only real pops. `sched_sim_isolation_lib` stays without the trace sources (unresolved references in a static archive are harmless); 5.1 decides whether to add them. `core->decisions` holds 5 items, so trace-only decisions take FIFO slots and cannot reorder real ones. - 3.2: requirement-branch hooks are in scheduler.c (`sched_make_allocation`, `sched_make_alter_allocation`, `sched_make_compound_alloc_`). The alternatives after the chosen one are not stored as entries; 4.1 renders "alternatives i+1..n not tried" from the alt_result (i,n). A merge failure stores an ATTEMPT_REASONS entry with ncodes==0. A failing single requirement records no attempt reasons; its pend outcome carries them. A single job with a NULL resreq stores its begin as a container with 0 members before returning -EINVAL, and 4.1 must tolerate that. - 3.3: the pre-filter (`sched_host_iter_next`) opens a host scope (`sched_trace_host_begin`) for every candidate and closes it with a verdict on each rejection: SELECT (pre-filter), INTERNAL(rc) for an uncommitted-fetch or check error, from-rows for a resource rejection (unreachable from the runner). A host the iterator returns is left in an open scope with no entry, so 3.4 must end or replace it for the accepted, already-claimed, snapshot-failure and per-host-error paths, or later rows attach to a stale host. A scan stopped with rc<0 calls `pos_scan_error`; exhaustion does not. `unsatisfy_` hands every row to `sched_trace_check_rows`. The sim thread's reconciliation reaches that hook but sees trace_rec NULL. +- 3.4: runner hooks are in alloc_runner.c. The 1.5 OPEN item is settled: `sched_trace_no_plan` counts `reqs_none_generated` once and sets `scope_no_plan`, after which `search_end` makes no statement and `resreq_begin`/`search_end` reset the flag. So a declined or generator-failed requirement has a NO_PLAN entry and no SEARCH_END line, and 4.1 must render the NO_PLAN line as "no plan generated: ". The 3.3 open scope is closed: every returned candidate gets a fresh host scope and one verdict, and check_begin drops any stale host rows. When the iterator can't start, the position ends as SCAN_ERROR with err 0, so 4.1 must render a scan error with no code. trace.h is at 790 lines; split it before adding to it. diff --git a/src/scheduler/include/sched/trace.h b/src/scheduler/include/sched/trace.h index 3be93b53..6eac461e 100644 --- a/src/scheduler/include/sched/trace.h +++ b/src/scheduler/include/sched/trace.h @@ -515,6 +515,7 @@ struct xce_sched_trace_rec_s { int scope_discarded; int scope_not_tried; bool scope_plan_err; /* a tried plan ended in an internal error */ + bool scope_no_plan; /* a no-plan reason already counted the search */ /* summary level: the current position's host tallies. The limit * tallies live in builder memory (malloc'd, not the arena) until the * position ends, so an omitted position allocates nothing in the @@ -616,13 +617,17 @@ sched_trace_attempt_reasons(xce_sched_trace_rec_t* rec, int ncodes, const char* const* hosts, int nhosts, const char* const* res, int nres); -/* Cluster-wide check scope. check_begin stores no entry; check_end stores - * the result, which on failure says no placement plans were tried (4.8). */ +/* Cluster-wide check scope. check_begin stores no entry and drops any + * host scope left open; check_end stores the result, which on failure + * says no placement plans were tried (4.8). */ extern void sched_trace_check_begin(xce_sched_trace_rec_t* rec, const xce_sched_obj_t* obj); extern void sched_trace_check_end(xce_sched_trace_rec_t* rec, int rc, bool plans_follow); +/* No plan generated for the current requirement, and why (4.7). Counts + * the requirement under "no plan generated" (9.3), once: a search end + * later in the same requirement makes no statement and counts nothing. */ extern void sched_trace_no_plan(xce_sched_trace_rec_t* rec, xce_sched_trace_noplan_t why); @@ -696,6 +701,7 @@ extern void sched_trace_finalize_failed(xce_sched_trace_rec_t* rec, int err); * generated (g == 0), all discarded (g > 0, d == g), or none placed (t > 0 * and no tried plan ended in an internal error). The matching decision- * wide counter is bumped under the same conditions, stored or not (9.3). + * After a no_plan entry in the requirement, there is no statement. */ extern void sched_trace_search_end(xce_sched_trace_rec_t* rec, int rc); diff --git a/src/scheduler/src/alloc_runner.c b/src/scheduler/src/alloc_runner.c index 518490b6..da013dad 100644 --- a/src/scheduler/src/alloc_runner.c +++ b/src/scheduler/src/alloc_runner.c @@ -29,6 +29,7 @@ #include "preason.h" #include "resobj.h" #include "sched/alloc_iter.h" +#include "sched/trace.h" #include "sched/types.h" #include "scheduler.h" #include "utils.h" @@ -212,9 +213,14 @@ static int sched_claim_bucket_(xce_sched_core_t* core, xce_sched_ctx_t* ctx, xce_host_iter_t it; xce_sched_worker_t* cand; int rc = 0; + int hrc; sched_alloc_clear_host_req_(ctx); + /* Decision trace (4.4): the position, numbered from 1 */ + SCHED_TRACE(core, sched_trace_pos_begin(core->trace_rec, b + 1, + plan->bucket_slots[b])); + it.init = sched_host_iter_init; it.next = sched_host_iter_next; it.fini = sched_host_iter_fini; @@ -222,18 +228,41 @@ static int sched_claim_bucket_(xce_sched_core_t* core, xce_sched_ctx_t* ctx, /* An iterator that cannot start offers no candidate: an * infrastructure failure, not a resource verdict (alloc_iter.h). */ if (it.init(&it, plan, core->workers) != 0) { + SCHED_TRACE(core, sched_trace_pos_end(core->trace_rec, + XCE_TRACE_POS_SCAN_ERROR, NULL)); return 0; } + /* Decision trace: every candidate the iterator returns arrives in the + * pre-filter's open host scope (host_iter.c); each path below replaces + * it with its own scope and ends that with the candidate's verdict. */ while ((cand = it.next(&it)) != NULL) { if (sched_alloc_is_claimed_(claims, *n, cand)) { + /* Decision trace (5.6) */ + SCHED_TRACE(core, (sched_trace_host_begin(core->trace_rec, + ctx->schedobj, cand), + sched_trace_host_verdict( + core->trace_rec, XCE_TRACE_V_CLAIMED, 0))); continue; } ctx->nslots = plan->bucket_slots[b]; ctx->nhosts = 1; - if (sched_try_alloc_on_host_(cand, core, ctx) > 0) { + /* Decision trace (5.1-5.7): the per-host check's own scope, so its + * unmet rows (resource.c) belong to this host; a rejection takes + * its verdict from those rows, or is a per-host selection mismatch + * when there are none */ + SCHED_TRACE( + core, sched_trace_host_begin(core->trace_rec, ctx->schedobj, cand)); + hrc = sched_try_alloc_on_host_(cand, core, ctx); + SCHED_TRACE( + core, sched_trace_host_verdict(core->trace_rec, + (hrc > 0) ? XCE_TRACE_V_ACCEPTED + : (hrc == 0) ? 0u + : XCE_TRACE_V_INTERNAL, + (hrc < 0) ? hrc : 0)); + if (hrc > 0) { rc = sched_snapshot_host_req_(&(claims[*n].host_req), &(ctx->alloc_req.host)); if (rc < 0) { @@ -255,6 +284,15 @@ static int sched_claim_bucket_(xce_sched_core_t* core, xce_sched_ctx_t* ctx, it.fini(&it); + /* Decision trace (4.4): filled with the claimed host, unfilled (a scan + * error when host_iter.c marked the position), or an internal error + * after the candidate was accepted */ + SCHED_TRACE(core, sched_trace_pos_end(core->trace_rec, + (rc > 0) ? XCE_TRACE_POS_FILLED + : (rc == 0) ? XCE_TRACE_POS_UNFILLED + : XCE_TRACE_POS_INTERNAL, + (rc != 0) ? cand->name : NULL)); + return rc; } @@ -273,14 +311,26 @@ static int sched_try_plan_(xce_sched_core_t* core, xce_sched_ctx_t* ctx, { int n = 0; int b; + int k; int rc; plan->ctx = ctx; *nclaimed = 0; + /* Decision trace (4.3): the plan, in the order tried */ + SCHED_TRACE(core, sched_trace_plan_begin(core->trace_rec, plan)); + for (b = 0; b < plan->nbuckets; b++) { rc = sched_claim_bucket_(core, ctx, plan, b, claims, &n); if (rc <= 0) { + /* Decision trace (5.8): the earlier positions' hosts released, + * one line; then the plan's end at the failed position (4.4) */ + SCHED_TRACE( + core, for (k = 0; k < n; k++) { + sched_trace_rollback_host(core->trace_rec, + claims[k].w->name); + }); + SCHED_TRACE(core, sched_trace_plan_end(core->trace_rec, rc, b + 1)); sched_rollback_claims_(ctx, claims, n); return rc; } @@ -288,6 +338,8 @@ static int sched_try_plan_(xce_sched_core_t* core, xce_sched_ctx_t* ctx, *nclaimed = n; + SCHED_TRACE(core, sched_trace_plan_end(core->trace_rec, 1, 0)); + return 1; } @@ -425,16 +477,27 @@ static int sched_alloc_search_(const xce_sched_allocator_iface_t* iface, /* A generator that cannot start is not a resource verdict: the attempt * pends, with nothing for plan_fini to release (Req 3.4). */ if (iface->plan_init(&iter, ctx->primary, core) != 0) { + /* Decision trace (4.7): counted as "no plan generated" here; no + * search end follows */ + SCHED_TRACE(core, + sched_trace_no_plan(core->trace_rec, + XCE_TRACE_NOPLAN_GENERATOR_FAILED)); return 0; } while ((plan = iface->plan_next(&iter)) != NULL) { if (sched_plan_malformed_(plan)) { + /* Decision trace (4.9) */ + SCHED_TRACE(core, + sched_trace_plan_discarded(core->trace_rec, plan)); continue; } rc = sched_alloc_claims_reserve_(&claims, &cap, plan->nbuckets); if (rc < 0) { + /* Decision trace (4.4) */ + SCHED_TRACE(core, + sched_trace_plan_not_tried(core->trace_rec, plan, rc)); break; } @@ -446,11 +509,18 @@ static int sched_alloc_search_(const xce_sched_allocator_iface_t* iface, if (rc > 0) { rc = sched_finalize_job_alloc_(sched, ctx, claims, n); + /* Decision trace (4.5) */ + SCHED_TRACE( + core, (rc > 0) ? sched_trace_commit(core->trace_rec) + : sched_trace_finalize_failed(core->trace_rec, rc)); } iface->plan_fini(&iter); free(claims); + /* Decision trace (4.6, 4.7, 4.9): the requirement's plan counts */ + SCHED_TRACE(core, sched_trace_search_end(core->trace_rec, rc)); + return rc; } @@ -472,6 +542,10 @@ int sched_allocator_run(void* plugin_arg, xce_sched_ctx_t* ctx) } if ((iface->accepts != NULL) && !iface->accepts(ctx->primary)) { + /* Decision trace (4.7): counted as "no plan generated" here */ + SCHED_TRACE(plugin->core, + sched_trace_no_plan(plugin->core->trace_rec, + XCE_TRACE_NOPLAN_DECLINED)); return 0; } @@ -480,7 +554,13 @@ int sched_allocator_run(void* plugin_arg, xce_sched_ctx_t* ctx) return rc; } + /* Decision trace (4.1, 4.2, 4.8): the cluster-wide check's scope; its + * rows come from resource.c, its end says whether plans follow */ + SCHED_TRACE(plugin->core, sched_trace_check_begin(plugin->core->trace_rec, + ctx->schedobj)); rc = sched_alloc_prepare_shared_(plugin->core, ctx); + SCHED_TRACE(plugin->core, + sched_trace_check_end(plugin->core->trace_rec, rc, rc > 0)); if (rc <= 0) { return rc; } diff --git a/src/scheduler/src/sched_trace.c b/src/scheduler/src/sched_trace.c index d2367c12..d5e00628 100644 --- a/src/scheduler/src/sched_trace.c +++ b/src/scheduler/src/sched_trace.c @@ -426,6 +426,7 @@ void sched_trace_resreq_begin(xce_sched_trace_rec_t* rec, rec->scope_discarded = 0; rec->scope_not_tried = 0; rec->scope_plan_err = false; + rec->scope_no_plan = false; e = sti_scope_begin(rec, SCHED_TRACE_E_RESREQ_BEGIN); if (e == NULL) { @@ -550,8 +551,10 @@ void sched_trace_check_begin(xce_sched_trace_rec_t* rec, const xce_sched_obj_t* obj) { sti_rollback_run_end(rec); - rec->scope_obj = obj; - rec->scope_w = NULL; + rec->scope_obj = obj; + rec->scope_w = NULL; + rec->host_row_verdicts = 0; + sti_host_rows_clear(rec); /* never a stale host's rows (task 3.4) */ rec->check_open = true; } @@ -574,8 +577,17 @@ void sched_trace_check_end(xce_sched_trace_rec_t* rec, int rc, void sched_trace_no_plan(xce_sched_trace_rec_t* rec, xce_sched_trace_noplan_t why) { - sched_trace_entry_t* e = sti_entry_add(rec, SCHED_TRACE_E_NO_PLAN); + sched_trace_entry_t* e; + /* counted here, once: the runner returns before any search end on a + * decline or a generator start failure, and a search end after this + * entry must not report "no plan generated" again (4.7, 9.3) */ + if (!rec->scope_no_plan) { + rec->scope_no_plan = true; + rec->counters.reqs_none_generated++; + } + + e = sti_entry_add(rec, SCHED_TRACE_E_NO_PLAN); if (e == NULL) { return; } diff --git a/src/scheduler/src/sched_trace_scan.c b/src/scheduler/src/sched_trace_scan.c index cb4196c8..8385b238 100644 --- a/src/scheduler/src/sched_trace_scan.c +++ b/src/scheduler/src/sched_trace_scan.c @@ -459,7 +459,7 @@ void sched_trace_search_end(xce_sched_trace_rec_t* rec, int rc) int x = rec->scope_not_tried; int g = t + d + x; - if (rc == 0) { + if ((rc == 0) && !rec->scope_no_plan) { if (g == 0) { stmt = SCHED_TRACE_SEARCH_NONE_GENERATED; rec->counters.reqs_none_generated++; @@ -477,6 +477,7 @@ void sched_trace_search_end(xce_sched_trace_rec_t* rec, int rc) rec->scope_discarded = 0; rec->scope_not_tried = 0; rec->scope_plan_err = false; + rec->scope_no_plan = false; e = sti_entry_add(rec, SCHED_TRACE_E_SEARCH_END); if (e == NULL) { diff --git a/src/scheduler/testsuite/meson.build b/src/scheduler/testsuite/meson.build index 037dba7e..5a24ab3c 100644 --- a/src/scheduler/testsuite/meson.build +++ b/src/scheduler/testsuite/meson.build @@ -2696,3 +2696,23 @@ sched_trace_host_hooks_test = executable( test('scheduler trace host hooks test', sched_trace_host_hooks_test, suite: 'scheduler', env: ['ASAN_OPTIONS=detect_leaks=0']) + +# scheduler-sim-decision-trace task 3.4: the allocator runner's trace hooks +# (design.md "TraceHooks": Runner entry, Cluster-wide check, Generator start +# failure, Search end, Plan loop, Position, Candidate, Rollback, Finalize; +# Requirements 4.1, 4.3-4.9, 5.1, 5.2, 5.6-5.8) on the real +# sched_allocator_run(). Same white-box #include set as +# sched_alloc_runner_search_test; xvec_dup (claim snapshot) and +# make_job_alloc (finalize) are wrapped to fail on demand. +sched_trace_hooks_runner_test = executable( + 'sched_trace_hooks_runner_test', + ['test_trace_hooks_runner.c'], + include_directories:[incdir], + dependencies: [scheduler_test_deps], + link_args: ['-Wl,--wrap=xvec_dup', + '-Wl,--wrap=make_job_alloc'], + install: false) + +test('scheduler trace hooks runner test', + sched_trace_hooks_runner_test, suite: 'scheduler', + env: ['ASAN_OPTIONS=detect_leaks=0']) diff --git a/src/scheduler/testsuite/test_trace_hooks_runner.c b/src/scheduler/testsuite/test_trace_hooks_runner.c new file mode 100644 index 00000000..e12f7a75 --- /dev/null +++ b/src/scheduler/testsuite/test_trace_hooks_runner.c @@ -0,0 +1,1015 @@ +/* + * test_trace_hooks_runner.c - decision-trace hooks in the allocator runner + * (scheduler-sim-decision-trace task 3.4; design.md "TraceHooks": Runner + * entry, Cluster-wide check, Generator start failure, Search end, Plan + * loop, Position, Candidate, Rollback, Finalize; Requirements 4.1, 4.3-4.9, + * 5.1, 5.2, 5.6-5.8). + * + * Drives the real sched_allocator_run() over the white-box stack of + * test_alloc_runner_search.c (alloc_runner.c, core.c, host_iter.c, + * resource.c, schedobj.c and the trace sources, all #included) with a + * scripted plan generator and a full-level trace record set on the core. + * Every scenario runs twice, untraced and traced, from the same script and + * faults; the runner's return value, pend reasons, result hosts and + * per-host debits must be identical (Requirement 10), and the traced run's + * entries must tell what the runner did: + * - an allocator decline and a generator start failure are no-plan + * reasons, counted once as "no plan generated"; + * - the cluster-wide check ends with its result, "no plans tried" on a + * failure; + * - plans are tried, discarded or not tried, and each tried plan ends + * with its result and the failed position; + * - positions begin and end filled, unfilled, as a scan error (iterator + * start failure or a scan stopped by 3.3's mark) or as an internal + * error after acceptance; + * - every candidate the iterator returns gets one verdict: accepted, + * already claimed, rejected with the verdict of the rows the per-host + * check recorded (or a per-host selection mismatch when it recorded + * none), or an internal error; + * - a failed plan with claims releases them; the search ends with a + * commit or a finalize failure and its search-end line. + * + * Faults: realloc() and sched_host_iter_init() are renamed inside + * alloc_runner.c; iter_next() and the pre-filter's selection call inside + * host_iter.c; xvec_dup() and make_job_alloc() are link-wrapped. + */ +#include +#include +#include +#include + +#include + +#include "preason.h" +#include "resobj.h" +#include "resreq.h" +#include "sched/alloc_iter.h" +#include "sched/trace.h" +#include "sched/types.h" +#include "scheduler.h" +#include "utils.h" +#include "xcealloc.h" +#include "xstring.h" +#include "xvec.h" + +/* ---- fault seams renamed inside the sources under test ----------------- */ + +static void* test_realloc(void* p, size_t n); +static int test_host_iter_init(xce_host_iter_t* iter, xce_alloc_plan_t* plan, + var workers); +#define realloc test_realloc +#define sched_host_iter_init test_host_iter_init +#include "../src/alloc_runner.c" +#undef realloc +#undef sched_host_iter_init + +#include "../src/core.c" + +static int test_iter_next(iterator_t* it, void** item); +static int test_prefilter_select(xce_sched_core_t* core, xce_sched_worker_t* w, + xce_select_t* select); +#define iter_next test_iter_next +#define sched_meet_host_selection test_prefilter_select +#include "../src/host_iter.c" +#undef iter_next +#undef sched_meet_host_selection + +#include "../src/resource.c" +#include "../src/sched_trace.c" +#include "../src/sched_trace_capture.c" +#include "../src/sched_trace_outcome.c" +#include "../src/sched_trace_scan.c" +#include "../src/schedobj.c" + +/* ---- link-satisfying stubs (as test_alloc_runner_search.c) ------------- */ + +int plugin_register(void* s, void* p) { return (void)s, (void)p, 0; } +void plugin_unregister(void* arg) { (void)arg; } + +int plugin_job_update(void* s, xce_sched_job_t* j) +{ + return (void)s, (void)j, 0; +} +int plugin_queue_update(void* s, xce_sched_queue_t* q) +{ + return (void)s, (void)q, 0; +} +int plugin_job_rusage_update(void* s, xce_sched_job_t* j, xce_job_rusages_t* r) +{ + return (void)s, (void)j, (void)r, 0; +} + +xce_alloc_t** sched_make_allocs(xce_scheduler_t* sched, char* reserve, int nres, + int* idxs, float* vals, int* duration) +{ + xce_alloc_t** allocs = (xce_alloc_t**)calloc(nres, sizeof(*allocs)); + int i; + + (void)duration; + TEST_ASSERT_NOT_NULL(allocs); + for (i = 0; i < nres; i++) { + xce_resource_t* r = NULL; + + vget(sched->core.resmgr, idxs[i], (void**)&r); + TEST_ASSERT_NOT_NULL(r); + allocs[i] = make_alloc(r->name, reserve, 0, vals[i], -1); + TEST_ASSERT_NOT_NULL(allocs[i]); + } + return allocs; +} + +/* ---- faults ------------------------------------------------------------ */ + +static int g_realloc_calls, g_realloc_fail_on; +static int g_itinit_calls, g_itinit_fail_on; +static int g_iter_calls, g_iter_fail_on; +static int g_dup_calls, g_dup_fail_on; +static int g_jalloc_fail; +static const char* g_prefilter_pass; /* host the pre-filter lets through */ +static const char* g_prep_fail_host; /* per-host prepare fails on it */ + +static void* test_realloc(void* p, size_t n) +{ + if (++g_realloc_calls == g_realloc_fail_on) { + errno = ENOMEM; + return NULL; + } + return realloc(p, n); +} + +static int test_host_iter_init(xce_host_iter_t* iter, xce_alloc_plan_t* plan, + var workers) +{ + if (++g_itinit_calls == g_itinit_fail_on) { + return -ENOMEM; + } + return sched_host_iter_init(iter, plan, workers); +} + +static int test_iter_next(iterator_t* it, void** item) +{ + if (++g_iter_calls == g_iter_fail_on) { + return -EIO; + } + return iter_next(it, item); +} + +/* The pre-filter's selection: the real evaluator, except for the one host + * let through so the per-host check's own selection branch rejects it. */ +static int test_prefilter_select(xce_sched_core_t* core, xce_sched_worker_t* w, + xce_select_t* select) +{ + if ((g_prefilter_pass != NULL) + && (strcmp(w->name, g_prefilter_pass) == 0)) { + return 1; + } + return sched_meet_host_selection(core, w, select); +} + +int __real_xvec_dup(xvec_t** dst, xvec_t* src); + +int __wrap_xvec_dup(xvec_t** dst, xvec_t* src) +{ + if ((g_dup_fail_on > 0) && (++g_dup_calls == g_dup_fail_on)) { + *dst = NULL; + return -ENOMEM; + } + return __real_xvec_dup(dst, src); +} + +xce_job_alloc_t* __real_make_job_alloc(uint64_t jobid, int nhost, + xce_host_alloc_t** hallocs); + +xce_job_alloc_t* __wrap_make_job_alloc(uint64_t jobid, int nhost, + xce_host_alloc_t** hallocs) +{ + if (g_jalloc_fail) { + errno = ENOMEM; + return NULL; + } + return __real_make_job_alloc(jobid, nhost, hallocs); +} + +/* ---- per-host prepare stub (plugin.c + parjob's hook) ------------------ */ + +static int g_slots_idx = -1; +static int g_lic_idx = -1; +static int g_rack_idx = -1; +static float g_shared_want; + +int plugin_prepare_allocation(xce_sched_core_t* sched, xce_sched_ctx_t* ctx) +{ + (void)sched; + if (ctx->worker == NULL) { + xvec_idval_fini(&(ctx->alloc_req.shared)); + xvec_idval_init(&(ctx->alloc_req.shared), 8); + xvec_idval_fini(&(ctx->alloc_req.host)); + xvec_idval_init(&(ctx->alloc_req.host), 8); + if (g_shared_want > 0.0f) { + TEST_ASSERT_TRUE(xvec_idval_store(&(ctx->alloc_req.shared), + g_lic_idx, g_shared_want) + > 0); + } + return 0; + } + + if ((g_prep_fail_host != NULL) + && (strcmp(ctx->worker->name, g_prep_fail_host) == 0)) { + return -ENOMEM; + } + + xvec_idval_fini(&(ctx->alloc_req.host)); + xvec_idval_init(&(ctx->alloc_req.host), 8); + TEST_ASSERT_TRUE(xvec_idval_store(&(ctx->alloc_req.host), g_slots_idx, + (float)ctx->nslots) + > 0); + return 0; +} + +/* ---- scripted plan generator -------------------------------------------- */ + +#define MAX_PLANS 6 +#define MAX_BUCKETS 4 + +static int g_plan_nb[MAX_PLANS]; +static int g_plan_slots[MAX_PLANS][MAX_BUCKETS]; +static int g_nplans; +static int g_next_calls; +static int g_init_result; +static int g_accepts_result; +static int g_cur_slots[MAX_BUCKETS]; +static xce_alloc_plan_t g_cur_plan; + +static int fake_accepts(const xce_primary_resreq_t* primary) +{ + (void)primary; + return g_accepts_result; +} + +static int fake_plan_init(xce_alloc_plan_iter_t* iter, + xce_primary_resreq_t* resreq, xce_sched_core_t* core) +{ + (void)resreq, (void)core; + iter->private = NULL; + return g_init_result; +} + +static xce_alloc_plan_t* fake_plan_next(xce_alloc_plan_iter_t* iter) +{ + int p = g_next_calls++; + + (void)iter; + if (p >= g_nplans) { + return NULL; + } + memset(&g_cur_plan, 0, sizeof(g_cur_plan)); + memcpy(g_cur_slots, g_plan_slots[p], sizeof(g_cur_slots)); + g_cur_plan.nbuckets = g_plan_nb[p]; + g_cur_plan.bucket_slots = g_cur_slots; + return &g_cur_plan; +} + +static void fake_plan_fini(xce_alloc_plan_iter_t* iter) { (void)iter; } + +static xce_sched_allocator_iface_t g_iface = { + .accepts = fake_accepts, + .plan_init = fake_plan_init, + .plan_next = fake_plan_next, + .plan_fini = fake_plan_fini, +}; + +/* One plan of `nb` buckets (nb == 0: malformed) */ +static void script_plan(int nb, int s0, int s1) +{ + TEST_ASSERT_TRUE(g_nplans < MAX_PLANS); + g_plan_nb[g_nplans] = nb; + g_plan_slots[g_nplans][0] = s0; + g_plan_slots[g_nplans][1] = s1; + g_nplans++; +} + +/* Run-local state: the generator's progress and the armed faults' counts. + * The script and the faults themselves stay, so both runs see them. */ +static void reset_run_state(void) +{ + g_next_calls = 0; + g_realloc_calls = 0; + g_itinit_calls = 0; + g_iter_calls = 0; + g_dup_calls = 0; +} + +void setUp(void) +{ + reset_run_state(); + g_realloc_fail_on = 0; + g_itinit_fail_on = 0; + g_iter_fail_on = 0; + g_dup_fail_on = 0; + g_jalloc_fail = 0; + g_prefilter_pass = NULL; + g_prep_fail_host = NULL; + g_shared_want = 0.0f; + g_nplans = 0; + g_init_result = 0; + g_accepts_result = 1; + memset(g_plan_slots, 0, sizeof(g_plan_slots)); +} + +void tearDown(void) { } + +/* ---- fixture ------------------------------------------------------------ */ + +#define MAX_HOSTS 6 + +typedef struct { + xce_scheduler_t sched; + xce_resource_t* slots; + xce_resource_t* lic; /* shared, 10 in total */ + xce_resource_t* rack; + xce_sched_plugin_t plugin; + xce_primary_resreq_t req; + xce_sched_obj_t obj; + xce_primary_schedobj_t primary_obj; + xce_sched_ctx_t ctx; + xce_select_node_t sel_res, sel_num, sel_eq; + xce_select_t select; + xce_sched_trace_ctl_t ctl; + xce_sched_trace_rec_t* rec; + xce_sched_worker_t* hosts[MAX_HOSTS]; + int nhosts; +} rt_fixture_t; + +/* A host spec: name, free slots, rack */ +typedef struct { + const char* name; + float slots; + float rack; +} rt_host_t; + +typedef struct { + rt_host_t hosts[MAX_HOSTS]; + int nhosts; + int max_tasks; + int select_rack; /* 0: no select expression */ +} rt_scenario_t; + +static void rt_add_host(rt_fixture_t* f, const rt_host_t* h) +{ + xce_sched_worker_t* w; + int idx[2] = { f->slots->idx, f->rack->idx }; + float total[2] = { h->slots, h->rack }; + float inuse[1] = { 0.0f }; + + w = (xce_sched_worker_t*)calloc(1, sizeof(*w)); + TEST_ASSERT_NOT_NULL(w); + w->name = makeString((char*)h->name); + TEST_ASSERT_NOT_NULL(w->name); + zarray_init(&(w->resvals), 8, (zarray_item_killer_t)kill_sched_resval); + zarray_init(&(w->resInUse), 8, (zarray_item_killer_t)kill_sched_resval); + put_resvals(&(w->resvals), 2, idx, total); + put_resvals(&(w->resInUse), 1, idx, inuse); + TEST_ASSERT_TRUE(sched_add_worker(&(f->sched.core), w) >= 0); + f->hosts[f->nhosts++] = w; +} + +static void rt_init(rt_fixture_t* f, const rt_scenario_t* sc, bool traced) +{ + int i; + int idx[1]; + float total[1] = { 10.0f }; + + memset(f, 0, sizeof(*f)); + TEST_ASSERT_EQUAL_INT(0, xce_sched_core_init(&(f->sched.core))); + for (i = 0; i < 2; i++) { + f->sched.core.pend_updates[i] = (dataq_t*)calloc(1, sizeof(dataq_t)); + TEST_ASSERT_NOT_NULL(f->sched.core.pend_updates[i]); + dataq_init(f->sched.core.pend_updates[i], NULL); + } + + f->slots = makeResource(f->sched.core.resmgr, + &(xce_resource_t) { .name = (char*)"slots" }); + f->lic = makeResource( + f->sched.core.resmgr, + &(xce_resource_t) { .name = (char*)"lic", .flags = XCE_RESF_GLOBAL }); + f->rack = makeResource(f->sched.core.resmgr, + &(xce_resource_t) { .name = (char*)"rack" }); + TEST_ASSERT_NOT_NULL(f->slots); + TEST_ASSERT_NOT_NULL(f->lic); + TEST_ASSERT_NOT_NULL(f->rack); + g_slots_idx = f->slots->idx; + g_lic_idx = f->lic->idx; + g_rack_idx = f->rack->idx; + idx[0] = f->lic->idx; + put_resvals(&(f->sched.core.sharedInst), 1, idx, total); + + f->plugin.name = (char*)"fake_allocator"; + f->plugin.sched = &(f->sched); + f->plugin.core = &(f->sched.core); + f->plugin.allocator_iface = &g_iface; + + for (i = 0; i < sc->nhosts; i++) { + rt_add_host(f, &(sc->hosts[i])); + } + + f->req.minTasks = 1; + f->req.maxTasks = sc->max_tasks; + if (sc->select_rack > 0) { + /* select `rack == select_rack` */ + f->sel_res.type = SEL_RESNAME; + f->sel_res.u.resname.res_idx = g_rack_idx; + f->sel_res.u.resname.res_type = RES_TYPE_NUMERIC; + f->sel_num.type = SEL_NUM; + f->sel_num.u.numval = sc->select_rack; + f->sel_eq.type = SEL_EQ; + f->sel_eq.u.binop.left = &(f->sel_res); + f->sel_eq.u.binop.right = &(f->sel_num); + f->select.root = &(f->sel_eq); + f->req.select = &(f->select); + } + f->primary_obj.resreq = &(f->req); + f->obj.tag = XCE_RESREQ_PRIMARY; + f->obj.u.primary = &(f->primary_obj); + TEST_ASSERT_EQUAL_INT( + 0, sched_ctx_init(&(f->ctx), &(f->sched.core), &(f->obj))); + sched_ctx_set_primary(&(f->ctx), &(f->req)); + + if (traced) { + f->ctl.all = true; + f->ctl.level = XCE_TRACE_LEVEL_FULL; + f->ctl.line_bound = XCE_TRACE_LINE_BOUND_DEFAULT; + f->ctl.position = 1; + f->rec = sched_trace_rec_new(&(f->ctl), 7, XCE_TRACE_REC_DECISION); + TEST_ASSERT_NOT_NULL(f->rec); + f->sched.core.trace_rec = f->rec; + } +} + +static void rt_fini(rt_fixture_t* f) +{ + f->sched.core.trace_rec = NULL; + sched_trace_rec_free(f->rec); + free_JobAlloc_ptr(&(f->ctx.alloc)); + free_PendReason_ptr(&(f->ctx.reason)); + sched_ctx_fini(&(f->ctx)); + xce_sched_core_fini(&(f->sched.core)); + xstring_table_fini(); +} + +/* What the runner did, as text: return value, pend reasons, result hosts + * and each host's slot debit. */ +static void rt_digest(rt_fixture_t* f, int rc, char* buf, size_t len) +{ + size_t n = 0; + int i; + +#define PUT(...) \ + n += (size_t)snprintf(buf + n, (n < len) ? len - n : 0, __VA_ARGS__) + PUT("rc=%d;", rc); + if (f->ctx.reason != NULL) { + const xce_job_preason_t* r = f->ctx.reason->reasons; + + PUT("main=%d;", r->main_reason); + for (i = 0; i < r->nhost_reasons; i++) { + PUT("h:%s/%d;", r->host_based_reasons[i]->host, + r->host_based_reasons[i]->reason); + } + for (i = 0; i < r->nres_reasons; i++) { + const xce_res_reason_t* rr = r->res_based_reasons[i]; + + PUT("r:%s/%s/%d;", (rr->host != NULL) ? rr->host : "-", + (rr->resource != NULL) ? rr->resource : "-", rr->reason); + } + } + if (f->ctx.alloc != NULL) { + for (i = 0; i < f->ctx.alloc->nhosts; i++) { + PUT("a:%s;", f->ctx.alloc->allocs[i]->host); + } + PUT("shared=%d;", f->ctx.alloc->nres_shared); + } + for (i = 0; i < f->nhosts; i++) { + xvec_t* v + = (xvec_t*)zarray_get(&(f->ctx.uncomitted.hosts), f->hosts[i]->idx); + + PUT("d:%s=%g;", f->hosts[i]->name, + ((v == NULL) || (v->count <= g_slots_idx)) + ? 0.0 + : (double)((float*)v->data)[g_slots_idx]); + } +#undef PUT + TEST_ASSERT_TRUE(n < len); +} + +/* + * Runs the scenario untraced, then traced into `t` (left initialised for + * the caller's entry checks; the caller rt_fini()s it). Both runs must + * agree on everything the runner produced. Returns the traced rc. + */ +static int rt_pair(const rt_scenario_t* sc, rt_fixture_t* t) +{ + rt_fixture_t u; + char du[2048], dt[2048]; + int rcu, rct; + + reset_run_state(); + rt_init(&u, sc, false); + rcu = sched_allocator_run(&(u.plugin), &(u.ctx)); + rt_digest(&u, rcu, du, sizeof(du)); + rt_fini(&u); + + reset_run_state(); + rt_init(t, sc, true); + rct = sched_allocator_run(&(t->plugin), &(t->ctx)); + rt_digest(t, rct, dt, sizeof(dt)); + + TEST_ASSERT_EQUAL_STRING(du, dt); + /* the runner leaves no host scope open behind it */ + TEST_ASSERT_NULL(t->rec->scope_w); + TEST_ASSERT_EQUAL_INT(0, t->rec->host_nrows); + TEST_ASSERT_FALSE(t->rec->check_open); + return rct; +} + +/* ---- entry checks ------------------------------------------------------- */ + +/* The record's entry types, in order, must be exactly `types`. */ +static void assert_types(const xce_sched_trace_rec_t* rec, const int* types, + int n) +{ + const sched_trace_entry_t* e = rec->entries; + int k; + char msg[64]; + + for (k = 0; k < n; k++) { + snprintf(msg, sizeof(msg), "entry %d", k); + TEST_ASSERT_NOT_NULL_MESSAGE(e, msg); + TEST_ASSERT_EQUAL_INT_MESSAGE(types[k], e->type, msg); + e = e->next; + } + TEST_ASSERT_NULL_MESSAGE(e, "more entries than expected"); +} + +static const sched_trace_entry_t* entry_at(const xce_sched_trace_rec_t* rec, + int k) +{ + const sched_trace_entry_t* e = rec->entries; + + while ((e != NULL) && (k-- > 0)) { + e = e->next; + } + TEST_ASSERT_NOT_NULL(e); + return e; +} + +static void assert_host(const sched_trace_entry_t* e, const char* host, + unsigned verdicts, sched_trace_phase_t phase, int err) +{ + TEST_ASSERT_EQUAL_INT(SCHED_TRACE_E_HOST, e->type); + TEST_ASSERT_EQUAL_STRING(host, e->u.host.host); + TEST_ASSERT_EQUAL_UINT(verdicts, e->u.host.verdicts); + TEST_ASSERT_EQUAL_INT(phase, e->u.host.phase); + TEST_ASSERT_EQUAL_INT(err, e->u.host.err); +} + +static void assert_pos_end(const sched_trace_entry_t* e, int pos, + xce_sched_trace_pos_end_t how, const char* host, + int err) +{ + TEST_ASSERT_EQUAL_INT(SCHED_TRACE_E_POS_END, e->type); + TEST_ASSERT_EQUAL_INT(pos, e->u.pos_end.pos); + TEST_ASSERT_EQUAL_INT(how, e->u.pos_end.how); + if (host == NULL) { + TEST_ASSERT_NULL(e->u.pos_end.host); + } else { + TEST_ASSERT_EQUAL_STRING(host, e->u.pos_end.host); + } + TEST_ASSERT_EQUAL_INT(err, e->u.pos_end.err); +} + +static void assert_plan_end(const sched_trace_entry_t* e, int plan_no, int rc, + int failed_pos) +{ + TEST_ASSERT_EQUAL_INT(SCHED_TRACE_E_PLAN_END, e->type); + TEST_ASSERT_EQUAL_INT(plan_no, e->u.plan_end.plan_no); + TEST_ASSERT_EQUAL_INT(rc, e->u.plan_end.rc); + TEST_ASSERT_EQUAL_INT(failed_pos, e->u.plan_end.failed_pos); +} + +static void assert_search_end(const sched_trace_entry_t* e, int g, int t, int d, + int x, int rc, sched_trace_search_stmt_t stmt) +{ + TEST_ASSERT_EQUAL_INT(SCHED_TRACE_E_SEARCH_END, e->type); + TEST_ASSERT_EQUAL_INT_MESSAGE(g, e->u.search_end.generated, "generated"); + TEST_ASSERT_EQUAL_INT_MESSAGE(t, e->u.search_end.tried, "tried"); + TEST_ASSERT_EQUAL_INT_MESSAGE(d, e->u.search_end.discarded, "discarded"); + TEST_ASSERT_EQUAL_INT_MESSAGE(x, e->u.search_end.not_tried, "not tried"); + TEST_ASSERT_EQUAL_INT(rc, e->u.search_end.rc); + TEST_ASSERT_EQUAL_INT_MESSAGE(stmt, e->u.search_end.stmt, "statement"); +} + +static void assert_check_end(const sched_trace_entry_t* e, int rc, + bool plans_follow) +{ + TEST_ASSERT_EQUAL_INT(SCHED_TRACE_E_CHECK_END, e->type); + TEST_ASSERT_EQUAL_INT(rc, e->u.check_end.rc); + TEST_ASSERT_EQUAL(plans_follow, e->u.check_end.plans_follow); +} + +/* verdict bit -> host_evals index */ +static int vk(unsigned bit) +{ + int k = 0; + + while ((bit >> k) != 1u) { + k++; + } + return k; +} + +#define E(x) SCHED_TRACE_E_##x +#define NTYPES(a) ((int)(sizeof(a) / sizeof((a)[0]))) + +/* ---- no plan (4.7) and the cluster-wide check (4.1, 4.8) ---------------- */ + +/* The allocator declines: one "no plan generated" reason, counted once, + * no check and no search. */ +static void test_declined_is_a_no_plan_reason(void) +{ + rt_scenario_t sc + = { .hosts = { { "rt-a", 2, 1 } }, .nhosts = 1, .max_tasks = 2 }; + rt_fixture_t t; + static const int types[] = { E(NO_PLAN) }; + + g_accepts_result = 0; + script_plan(1, 2, 0); + TEST_ASSERT_EQUAL_INT(0, rt_pair(&sc, &t)); + assert_types(t.rec, types, NTYPES(types)); + TEST_ASSERT_EQUAL_INT(XCE_TRACE_NOPLAN_DECLINED, + t.rec->entries->u.no_plan.why); + TEST_ASSERT_EQUAL_INT(1, t.rec->counters.reqs_none_generated); + rt_fini(&t); +} + +/* The generator cannot start: after the passed check, one "no plan + * generated" reason, counted once, and no search-end line repeating it. */ +static void test_generator_failure_is_a_no_plan_reason(void) +{ + rt_scenario_t sc + = { .hosts = { { "rt-a", 2, 1 } }, .nhosts = 1, .max_tasks = 2 }; + rt_fixture_t t; + static const int types[] = { E(CHECK_END), E(NO_PLAN) }; + + g_init_result = -ENOMEM; + script_plan(1, 2, 0); + TEST_ASSERT_EQUAL_INT(0, rt_pair(&sc, &t)); + assert_types(t.rec, types, NTYPES(types)); + assert_check_end(entry_at(t.rec, 0), 1, true); + TEST_ASSERT_EQUAL_INT(XCE_TRACE_NOPLAN_GENERATOR_FAILED, + entry_at(t.rec, 1)->u.no_plan.why); + TEST_ASSERT_EQUAL_INT(1, t.rec->counters.reqs_none_generated); + rt_fini(&t); +} + +/* The cluster-wide check fails on the shared license: its row, then its + * end saying no plans were tried; the generator is never started. */ +static void test_failed_check_tries_no_plans(void) +{ + rt_scenario_t sc + = { .hosts = { { "rt-a", 2, 1 } }, .nhosts = 1, .max_tasks = 2 }; + rt_fixture_t t; + static const int types[] = { E(CHECK_ROW), E(CHECK_END) }; + const sched_trace_entry_t* e; + + g_shared_want = 20.0f; + script_plan(1, 2, 0); + TEST_ASSERT_EQUAL_INT(0, rt_pair(&sc, &t)); + assert_types(t.rec, types, NTYPES(types)); + e = entry_at(t.rec, 0); + TEST_ASSERT_EQUAL_STRING("lic", e->u.row.res); + TEST_ASSERT_EQUAL_FLOAT(10.0f, e->u.row.total); + TEST_ASSERT_EQUAL_FLOAT(20.0f, e->u.row.requested); + TEST_ASSERT_FALSE(e->u.row.met); + assert_check_end(entry_at(t.rec, 1), 0, false); + TEST_ASSERT_EQUAL_INT(0, g_next_calls); + TEST_ASSERT_EQUAL_INT(0, t.rec->counters.reqs_none_generated); + rt_fini(&t); +} + +/* ---- search-end statements (4.6, 4.7, 4.9) ------------------------------ */ + +static void test_no_plan_generated_search_end(void) +{ + rt_scenario_t sc + = { .hosts = { { "rt-a", 2, 1 } }, .nhosts = 1, .max_tasks = 2 }; + rt_fixture_t t; + static const int types[] = { E(CHECK_END), E(SEARCH_END) }; + + TEST_ASSERT_EQUAL_INT(0, rt_pair(&sc, &t)); + assert_types(t.rec, types, NTYPES(types)); + assert_search_end(entry_at(t.rec, 1), 0, 0, 0, 0, 0, + SCHED_TRACE_SEARCH_NONE_GENERATED); + TEST_ASSERT_EQUAL_INT(1, t.rec->counters.reqs_none_generated); + rt_fini(&t); +} + +static void test_all_plans_discarded(void) +{ + rt_scenario_t sc + = { .hosts = { { "rt-a", 2, 1 } }, .nhosts = 1, .max_tasks = 2 }; + rt_fixture_t t; + static const int types[] + = { E(CHECK_END), E(PLAN_DISCARDED), E(PLAN_DISCARDED), E(SEARCH_END) }; + + script_plan(0, 0, 0); + script_plan(1, 0, 0); /* a bucket of no slot */ + TEST_ASSERT_EQUAL_INT(0, rt_pair(&sc, &t)); + assert_types(t.rec, types, NTYPES(types)); + TEST_ASSERT_EQUAL_INT(1, entry_at(t.rec, 1)->u.plan.plan_no); + TEST_ASSERT_EQUAL_INT(2, entry_at(t.rec, 2)->u.plan.plan_no); + assert_search_end(entry_at(t.rec, 3), 2, 0, 2, 0, 0, + SCHED_TRACE_SEARCH_ALL_DISCARDED); + TEST_ASSERT_EQUAL_INT(2, t.rec->counters.plans_discarded); + TEST_ASSERT_EQUAL_INT(1, t.rec->counters.reqs_all_discarded); + rt_fini(&t); +} + +/* ---- positions, candidates and rollback (4.3, 4.4, 4.6, 5.1-5.8) -------- */ + +/* + * A discarded plan, then [2,2] over A (2 slots) and B (1 slot): position 1 + * accepts A; position 2 skips A (already claimed), rejects B for its slot + * shortfall (with the row under it) and is unfilled; A is released, the + * plan ends failed at position 2, and no plan could be placed. + */ +static void test_unfilled_position_rolls_back_claims(void) +{ + rt_scenario_t sc = { + .hosts = { { "rt-a", 2, 1 }, { "rt-b", 1, 1 } }, + .nhosts = 2, + .max_tasks = 4 + }; + rt_fixture_t t; + static const int types[] + = { E(CHECK_END), E(PLAN_DISCARDED), E(PLAN_BEGIN), E(POS_BEGIN), + E(HOST), E(POS_END), E(POS_BEGIN), E(HOST), + E(HOST), E(HOST_ROW), E(POS_END), E(ROLLBACK), + E(PLAN_END), E(SEARCH_END) }; + const sched_trace_entry_t* e; + + script_plan(0, 0, 0); + script_plan(2, 2, 2); + TEST_ASSERT_EQUAL_INT(0, rt_pair(&sc, &t)); + assert_types(t.rec, types, NTYPES(types)); + + e = entry_at(t.rec, 2); + TEST_ASSERT_EQUAL_INT(2, e->u.plan.plan_no); + TEST_ASSERT_EQUAL_INT(2, e->u.plan.nbuckets); + TEST_ASSERT_EQUAL_INT(2, e->u.plan.slots[0]); + TEST_ASSERT_EQUAL_INT(2, e->u.plan.slots[1]); + e = entry_at(t.rec, 3); + TEST_ASSERT_EQUAL_INT(1, e->u.pos_begin.pos); + TEST_ASSERT_EQUAL_INT(2, e->u.pos_begin.slots); + assert_host(entry_at(t.rec, 4), "rt-a", XCE_TRACE_V_ACCEPTED, + SCHED_TRACE_PHASE_NONE, 0); + assert_pos_end(entry_at(t.rec, 5), 1, XCE_TRACE_POS_FILLED, "rt-a", 0); + TEST_ASSERT_EQUAL_INT(2, entry_at(t.rec, 6)->u.pos_begin.pos); + assert_host(entry_at(t.rec, 7), "rt-a", XCE_TRACE_V_CLAIMED, + SCHED_TRACE_PHASE_NONE, 0); + assert_host(entry_at(t.rec, 8), "rt-b", XCE_TRACE_V_SHORTFALL, + SCHED_TRACE_PHASE_PERHOST, 0); + e = entry_at(t.rec, 9); + TEST_ASSERT_EQUAL_STRING("slots", e->u.row.res); + TEST_ASSERT_EQUAL_FLOAT(1.0f, e->u.row.total); + TEST_ASSERT_EQUAL_FLOAT(2.0f, e->u.row.requested); + TEST_ASSERT_FALSE(e->u.row.met); + assert_pos_end(entry_at(t.rec, 10), 2, XCE_TRACE_POS_UNFILLED, NULL, 0); + e = entry_at(t.rec, 11); + TEST_ASSERT_EQUAL_INT(1, e->u.rollback.nhosts); + TEST_ASSERT_EQUAL_STRING("rt-a", e->u.rollback.hosts[0]); + assert_plan_end(entry_at(t.rec, 12), 2, 0, 2); + assert_search_end(entry_at(t.rec, 13), 2, 1, 1, 0, 0, + SCHED_TRACE_SEARCH_NONE_PLACED); + + TEST_ASSERT_EQUAL_INT(1, t.rec->counters.plans_tried); + TEST_ASSERT_EQUAL_INT(1, t.rec->counters.plans_discarded); + TEST_ASSERT_EQUAL_INT(1, t.rec->counters.reqs_none_placed); + TEST_ASSERT_EQUAL_INT(1, + t.rec->counters.host_evals[vk(XCE_TRACE_V_ACCEPTED)]); + TEST_ASSERT_EQUAL_INT(1, + t.rec->counters.host_evals[vk(XCE_TRACE_V_CLAIMED)]); + TEST_ASSERT_EQUAL_INT( + 1, t.rec->counters.host_evals[vk(XCE_TRACE_V_SHORTFALL)]); + rt_fini(&t); +} + +/* + * select `rack == 1`, plan [1] over P (rack 2), Q (rack 2, let through the + * pre-filter), R (rack 1, its per-host prepare fails) and S (rack 1): + * P is a pre-filter selection mismatch, Q a per-host one (no rows), R an + * internal error with its code, S accepted; the plan is committed. + */ +static void test_candidate_verdicts_and_commit(void) +{ + rt_scenario_t sc = { + .hosts = { { "rt-p", 2, 2 }, + { "rt-q", 2, 2 }, + { "rt-r", 2, 1 }, + { "rt-s", 2, 1 } }, + .nhosts = 4, + .max_tasks = 1, + .select_rack = 1 + }; + rt_fixture_t t; + static const int types[] + = { E(CHECK_END), E(PLAN_BEGIN), E(POS_BEGIN), E(HOST), + E(HOST), E(HOST), E(HOST), E(POS_END), + E(PLAN_END), E(COMMIT), E(SEARCH_END) }; + + g_prefilter_pass = "rt-q"; + g_prep_fail_host = "rt-r"; + script_plan(1, 1, 0); + TEST_ASSERT_EQUAL_INT(1, rt_pair(&sc, &t)); + assert_types(t.rec, types, NTYPES(types)); + assert_host(entry_at(t.rec, 3), "rt-p", XCE_TRACE_V_SELECT, + SCHED_TRACE_PHASE_PREFILTER, 0); + assert_host(entry_at(t.rec, 4), "rt-q", XCE_TRACE_V_SELECT, + SCHED_TRACE_PHASE_PERHOST, 0); + assert_host(entry_at(t.rec, 5), "rt-r", XCE_TRACE_V_INTERNAL, + SCHED_TRACE_PHASE_NONE, -ENOMEM); + assert_host(entry_at(t.rec, 6), "rt-s", XCE_TRACE_V_ACCEPTED, + SCHED_TRACE_PHASE_NONE, 0); + assert_pos_end(entry_at(t.rec, 7), 1, XCE_TRACE_POS_FILLED, "rt-s", 0); + assert_plan_end(entry_at(t.rec, 8), 1, 1, 0); + TEST_ASSERT_EQUAL_INT(1, entry_at(t.rec, 9)->u.commit.plan_no); + assert_search_end(entry_at(t.rec, 10), 1, 1, 0, 0, 1, + SCHED_TRACE_SEARCH_NO_STMT); + TEST_ASSERT_EQUAL_INT(2, + t.rec->counters.host_evals[vk(XCE_TRACE_V_SELECT)]); + TEST_ASSERT_EQUAL_INT(1, + t.rec->counters.host_evals[vk(XCE_TRACE_V_INTERNAL)]); + rt_fini(&t); +} + +/* + * The host scan stops on an error during plan [4]'s position after A was + * rejected: the position ends as a scan error with the error code; the + * next plan [1] is still tried and committed on A (4.4). + */ +static void test_scan_error_then_next_plan(void) +{ + rt_scenario_t sc = { + .hosts = { { "rt-a", 1, 1 }, { "rt-b", 4, 1 } }, + .nhosts = 2, + .max_tasks = 4 + }; + rt_fixture_t t; + static const int types[] + = { E(CHECK_END), E(PLAN_BEGIN), E(POS_BEGIN), E(HOST), + E(HOST_ROW), E(POS_END), E(PLAN_END), E(PLAN_BEGIN), + E(POS_BEGIN), E(HOST), E(POS_END), E(PLAN_END), + E(COMMIT), E(SEARCH_END) }; + + g_iter_fail_on = 2; /* A comes first; the step to B fails */ + script_plan(1, 4, 0); + script_plan(1, 1, 0); + TEST_ASSERT_EQUAL_INT(1, rt_pair(&sc, &t)); + assert_types(t.rec, types, NTYPES(types)); + assert_host(entry_at(t.rec, 3), "rt-a", XCE_TRACE_V_SHORTFALL, + SCHED_TRACE_PHASE_PERHOST, 0); + assert_pos_end(entry_at(t.rec, 5), 1, XCE_TRACE_POS_SCAN_ERROR, NULL, -EIO); + assert_plan_end(entry_at(t.rec, 6), 1, 0, 1); + assert_host(entry_at(t.rec, 9), "rt-a", XCE_TRACE_V_ACCEPTED, + SCHED_TRACE_PHASE_NONE, 0); + assert_pos_end(entry_at(t.rec, 10), 1, XCE_TRACE_POS_FILLED, "rt-a", 0); + assert_plan_end(entry_at(t.rec, 11), 2, 1, 0); + TEST_ASSERT_EQUAL_INT(2, entry_at(t.rec, 12)->u.commit.plan_no); + assert_search_end(entry_at(t.rec, 13), 2, 2, 0, 0, 1, + SCHED_TRACE_SEARCH_NO_STMT); + rt_fini(&t); +} + +/* The host iterator cannot start: the position is a scan error with no + * host; the next plan is tried (4.4). */ +static void test_iterator_start_failure_is_a_scan_error(void) +{ + rt_scenario_t sc + = { .hosts = { { "rt-a", 2, 1 } }, .nhosts = 1, .max_tasks = 2 }; + rt_fixture_t t; + static const int types[] + = { E(CHECK_END), E(PLAN_BEGIN), E(POS_BEGIN), E(POS_END), + E(PLAN_END), E(PLAN_BEGIN), E(POS_BEGIN), E(HOST), + E(POS_END), E(PLAN_END), E(COMMIT), E(SEARCH_END) }; + + g_itinit_fail_on = 1; + script_plan(1, 2, 0); + script_plan(1, 1, 0); + TEST_ASSERT_EQUAL_INT(1, rt_pair(&sc, &t)); + assert_types(t.rec, types, NTYPES(types)); + assert_pos_end(entry_at(t.rec, 3), 1, XCE_TRACE_POS_SCAN_ERROR, NULL, 0); + assert_plan_end(entry_at(t.rec, 4), 1, 0, 1); + assert_pos_end(entry_at(t.rec, 8), 1, XCE_TRACE_POS_FILLED, "rt-a", 0); + rt_fini(&t); +} + +/* + * [2,2] over A and B; B's claim snapshot fails after B was accepted: the + * position ends in an internal error naming B, A is released, the plan + * ends in the error and the search makes no statement (4.4, 5.8). + */ +static void test_snapshot_failure_ends_position_internal(void) +{ + rt_scenario_t sc = { + .hosts = { { "rt-a", 2, 1 }, { "rt-b", 2, 1 } }, + .nhosts = 2, + .max_tasks = 4 + }; + rt_fixture_t t; + static const int types[] + = { E(CHECK_END), E(PLAN_BEGIN), E(POS_BEGIN), E(HOST), + E(POS_END), E(POS_BEGIN), E(HOST), E(HOST), + E(POS_END), E(ROLLBACK), E(PLAN_END), E(SEARCH_END) }; + + g_dup_fail_on = 3; /* A's snapshot is calls 1-2; B's first copy fails */ + script_plan(2, 2, 2); + script_plan(1, 4, 0); /* never offered */ + TEST_ASSERT_EQUAL_INT(-ENOMEM, rt_pair(&sc, &t)); + assert_types(t.rec, types, NTYPES(types)); + assert_host(entry_at(t.rec, 6), "rt-a", XCE_TRACE_V_CLAIMED, + SCHED_TRACE_PHASE_NONE, 0); + assert_host(entry_at(t.rec, 7), "rt-b", XCE_TRACE_V_ACCEPTED, + SCHED_TRACE_PHASE_NONE, 0); + assert_pos_end(entry_at(t.rec, 8), 2, XCE_TRACE_POS_INTERNAL, "rt-b", 0); + TEST_ASSERT_EQUAL_STRING("rt-a", entry_at(t.rec, 9)->u.rollback.hosts[0]); + assert_plan_end(entry_at(t.rec, 10), 1, -ENOMEM, 2); + assert_search_end(entry_at(t.rec, 11), 1, 1, 0, 0, -ENOMEM, + SCHED_TRACE_SEARCH_NO_STMT); + TEST_ASSERT_EQUAL_INT(0, t.rec->counters.reqs_none_placed); + rt_fini(&t); +} + +/* The claims buffer cannot grow: the plan is not tried, with the error, + * and the search stops there (4.4). */ +static void test_reserve_failure_plan_not_tried(void) +{ + rt_scenario_t sc + = { .hosts = { { "rt-a", 2, 1 } }, .nhosts = 1, .max_tasks = 2 }; + rt_fixture_t t; + static const int types[] + = { E(CHECK_END), E(PLAN_NOT_TRIED), E(SEARCH_END) }; + const sched_trace_entry_t* e; + + g_realloc_fail_on = 1; + script_plan(1, 2, 0); + script_plan(1, 1, 0); + TEST_ASSERT_EQUAL_INT(-ENOMEM, rt_pair(&sc, &t)); + assert_types(t.rec, types, NTYPES(types)); + e = entry_at(t.rec, 1); + TEST_ASSERT_EQUAL_INT(1, e->u.plan.plan_no); + TEST_ASSERT_EQUAL_INT(-ENOMEM, e->u.plan.err); + assert_search_end(entry_at(t.rec, 2), 1, 0, 0, 1, -ENOMEM, + SCHED_TRACE_SEARCH_NO_STMT); + rt_fini(&t); +} + +/* Every position filled, but the allocation cannot be built: the plan + * ends satisfied and the finalize failure follows, with no commit (4.5). */ +static void test_finalize_failure(void) +{ + rt_scenario_t sc + = { .hosts = { { "rt-a", 2, 1 } }, .nhosts = 1, .max_tasks = 2 }; + rt_fixture_t t; + static const int types[] + = { E(CHECK_END), E(PLAN_BEGIN), E(POS_BEGIN), E(HOST), + E(POS_END), E(PLAN_END), E(FINALIZE_FAILED), E(SEARCH_END) }; + const sched_trace_entry_t* e; + + g_jalloc_fail = 1; + script_plan(1, 2, 0); + TEST_ASSERT_EQUAL_INT(-ENOMEM, rt_pair(&sc, &t)); + assert_types(t.rec, types, NTYPES(types)); + assert_plan_end(entry_at(t.rec, 5), 1, 1, 0); + e = entry_at(t.rec, 6); + TEST_ASSERT_EQUAL_INT(1, e->u.commit.plan_no); + TEST_ASSERT_EQUAL_INT(-ENOMEM, e->u.commit.err); + assert_search_end(entry_at(t.rec, 7), 1, 1, 0, 0, -ENOMEM, + SCHED_TRACE_SEARCH_NO_STMT); + rt_fini(&t); +} + +int main(void) +{ + UNITY_BEGIN(); + RUN_TEST(test_declined_is_a_no_plan_reason); + RUN_TEST(test_generator_failure_is_a_no_plan_reason); + RUN_TEST(test_failed_check_tries_no_plans); + RUN_TEST(test_no_plan_generated_search_end); + RUN_TEST(test_all_plans_discarded); + RUN_TEST(test_unfilled_position_rolls_back_claims); + RUN_TEST(test_candidate_verdicts_and_commit); + RUN_TEST(test_scan_error_then_next_plan); + RUN_TEST(test_iterator_start_failure_is_a_scan_error); + RUN_TEST(test_snapshot_failure_ends_position_internal); + RUN_TEST(test_reserve_failure_plan_not_tried); + RUN_TEST(test_finalize_failure); + return UNITY_END(); +} diff --git a/src/scheduler/testsuite/test_trace_model.c b/src/scheduler/testsuite/test_trace_model.c index 303dcaa7..8f3529e5 100644 --- a/src/scheduler/testsuite/test_trace_model.c +++ b/src/scheduler/testsuite/test_trace_model.c @@ -1334,8 +1334,7 @@ static void test_search_end_none_generated(void) fixture_primary_t fp; TEST_ASSERT_NOT_NULL(rec); - open_single(rec, &fp); - sched_trace_no_plan(rec, XCE_TRACE_NOPLAN_NONE_GENERATED); + open_single(rec, &fp); /* the search states it: no no_plan entry */ sched_trace_search_end(rec, 0); assert_search_end(rec->entries_tail, 0, 0, 0, 0, 0, SCHED_TRACE_SEARCH_NONE_GENERATED); @@ -1481,8 +1480,7 @@ static void test_search_end_nonzero_rc_has_no_statement(void) SCHED_TRACE_SEARCH_NO_STMT); close_single(rec, 1); - open_single(rec, &fp); /* generator failed: no plan, rc < 0 */ - sched_trace_no_plan(rec, XCE_TRACE_NOPLAN_GENERATOR_FAILED); + open_single(rec, &fp); /* nothing generated, but rc < 0 */ sched_trace_search_end(rec, -12); assert_search_end(rec->entries_tail, 0, 0, 0, 0, -12, SCHED_TRACE_SEARCH_NO_STMT); @@ -1492,6 +1490,44 @@ static void test_search_end_nonzero_rc_has_no_statement(void) sched_trace_rec_free(rec); } +/* A declined requirement or a generator that cannot start (the runner + * returns 0 there, without a search end) is counted as "no plan generated" + * by its no_plan entry, once; a search end after it in the same + * requirement neither repeats the statement nor counts it again (4.7, + * 9.3). The next requirement starts over. */ +static void test_no_plan_counts_none_generated_once(void) +{ + xce_sched_trace_rec_t* rec = new_rec(XCE_TRACE_LINE_BOUND_DEFAULT); + fixture_primary_t fp; + + TEST_ASSERT_NOT_NULL(rec); + make_primary(&fp, 1, 1); + sched_trace_path_push(rec, XCE_TRACE_PATH_SINGLE, 1, 1); + sched_trace_resreq_begin(rec, &fp.p, 0); + sched_trace_no_plan(rec, XCE_TRACE_NOPLAN_DECLINED); + close_single(rec, 0); + assert_req_counters(rec, 1, 0, 0); + + open_single(rec, &fp); + sched_trace_no_plan(rec, XCE_TRACE_NOPLAN_GENERATOR_FAILED); + TEST_ASSERT_EQUAL_INT(SCHED_TRACE_E_NO_PLAN, rec->entries_tail->type); + assert_req_counters(rec, 2, 0, 0); + sched_trace_search_end(rec, 0); + assert_search_end(rec->entries_tail, 0, 0, 0, 0, 0, + SCHED_TRACE_SEARCH_NO_STMT); + close_single(rec, 0); + assert_req_counters(rec, 2, 0, 0); + + /* a later requirement with no no_plan entry states it itself */ + open_single(rec, &fp); + sched_trace_search_end(rec, 0); + assert_search_end(rec->entries_tail, 0, 0, 0, 0, 0, + SCHED_TRACE_SEARCH_NONE_GENERATED); + close_single(rec, 0); + assert_req_counters(rec, 3, 0, 0); + sched_trace_rec_free(rec); +} + /* Counts belong to the requirement the search ran for: they restart with * each member, while the decision-wide counters add up. */ static void test_search_end_counts_per_requirement(void) @@ -1539,6 +1575,35 @@ static void test_search_end_counts_per_requirement(void) sched_trace_rec_free(rec); } +/* A cluster-wide check never runs in a host scope left open: check_begin + * drops the stale host, its held rows and its row verdicts, so the check's + * rows are cluster-wide rows (task 3.4). */ +static void test_check_begin_drops_a_stale_host_scope(void) +{ + xce_sched_trace_rec_t* rec = new_rec(XCE_TRACE_LINE_BOUND_DEFAULT); + char name[8] = "n01"; + xce_sched_worker_t w = make_worker(name, 0); + sched_trace_row_t row; + + TEST_ASSERT_NOT_NULL(rec); + memset(&row, 0, sizeof(row)); + row.res = "mem"; + sched_trace_host_begin(rec, NULL, &w); + sched_trace_row(rec, &row); /* held for the host's verdict */ + rec->host_row_verdicts |= XCE_TRACE_V_SHORTFALL; + TEST_ASSERT_EQUAL_INT(1, rec->host_nrows); + + sched_trace_check_begin(rec, NULL); + TEST_ASSERT_NULL(rec->scope_w); + TEST_ASSERT_EQUAL_INT(0, rec->host_nrows); + TEST_ASSERT_EQUAL_UINT(0, rec->host_row_verdicts); + sched_trace_row(rec, &row); + TEST_ASSERT_EQUAL_INT(SCHED_TRACE_E_CHECK_ROW, rec->entries_tail->type); + sched_trace_check_end(rec, 1, true); + TEST_ASSERT_EQUAL_INT(2, rec->nentries); + sched_trace_rec_free(rec); +} + /* ---- counters under truncation (9.3) ---------------------------------- */ /* @@ -2008,6 +2073,8 @@ int main(void) RUN_TEST(test_search_end_plan_internal_error_has_no_statement); RUN_TEST(test_search_end_not_tried_after_discarded); RUN_TEST(test_search_end_nonzero_rc_has_no_statement); + RUN_TEST(test_no_plan_counts_none_generated_once); + RUN_TEST(test_check_begin_drops_a_stale_host_scope); RUN_TEST(test_search_end_counts_per_requirement); RUN_TEST(test_counters_exact_after_truncation); RUN_TEST(test_summary_position_tallies);