feat(scheduler-sim-decision-trace): hook the allocator runner

Task 3.4: the runner records no-plan reasons, the cluster-wide check,
plans (tried, discarded, not tried) and their ends, positions, claimed
skips and per-host candidate verdicts, rollbacks, commit or finalize
failure, and search end. A declined or generator-failed requirement now
counts once as 'no plan generated'. Runner results are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-08 12:04:16 +00:00
parent d4bee4b5d0
commit 16f40fa9dc
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
9 changed files with 1216 additions and 14 deletions

View file

@ -551,7 +551,7 @@ Each hook is one `SCHED_TRACE(core, ...)` statement placed after the value it re
- `; no plan tried: all generated plans discarded` (4.9) when g > 0 and d = g;
- `; no plan could be placed` (4.6) when rc = 0, t > 0 and no plan ended in an internal error.
When rc > 0 or rc < 0 there is no statement; the commit or internal-error entries already say what happened. Like every other entry it counts against the line budget. It also bumps the matching decision-wide counter (requirements with no plan generated / all discarded / none placed), under the same conditions, which appear in the fixed-size totals block after truncation (9.3). | 4.6, 4.7, 4.9 |
When rc > 0 or rc < 0 there is no statement; the commit or internal-error entries already say what happened. A declined requirement and a generator that cannot start end the runner before the search end: their `no_plan` entry is the requirement's "no plan generated" line and bumps that counter itself, once; a search end after a `no_plan` entry in the same requirement makes no statement and counts nothing. Like every other entry it counts against the line budget. It also bumps the matching decision-wide counter (requirements with no plan generated / all discarded / none placed), under the same conditions, which appear in the fixed-size totals block after truncation (9.3). | 4.6, 4.7, 4.9 |
| Plan loop | malformed / reserve failure / try / result | `plan_discarded`; `plan_not_tried(err)`; `plan_begin`; `plan_end(rc, failed_pos)` | 4.3, 4.4, 4.6, 4.9 |
| Position | `sched_claim_bucket_` | `pos_begin`; iterator init failure → `pos_end(SCAN_ERROR)`; snapshot failure after acceptance (`alloc_runner.c:237-243`) → `pos_end(INTERNAL)`; else `pos_end(FILLED/UNFILLED/SCAN_ERROR, host)` | 4.4, 5.9 |
| Candidate | claimed skip / per-host result | `host_begin` + `host_verdict(CLAIMED)`; `host_begin`, call, `host_verdict(rc>0 ? ACCEPTED : rc==0 ? from rows : INTERNAL, err)` | 5.1–5.7 |

View file

@ -123,7 +123,7 @@
- _Requirements: 11.2, 11.4_
- _Depends: 2.3_
- [ ] 3. Allocator-thread hooks at the decision seams
- [x] 3. Allocator-thread hooks at the decision seams
- [x] 3.1 Hook the allocator loop: position, record lifecycle, selection and no-decision paths (P)
- **Position:** when session settings are present, every real job pop increments it.
- **Record start:** a wanted job starts a record with its selection facts; its cluster-wide limits are added after election.
@ -163,7 +163,7 @@
- _Boundary: TraceHooks (host iterator, satisfaction check)_
- _Depends: 2.2_
- [ ] 3.4 Hook the allocator runner: cluster-wide check, plans, positions, candidates and finalize
- [x] 3.4 Hook the allocator runner: cluster-wide check, plans, positions, candidates and finalize
- The runner records:
- an allocator decline and a generator start failure, as no-plan reasons;
- the cluster-wide check scope and its end, with "no plans tried" on failure;
@ -363,3 +363,4 @@
- 3.1: allocator-loop hooks live in `sched_allocator()` (sched_election.c), with helpers `sched_trace_push_no_decision()` and `sched_trace_attach()`. Selection is captured at job lookup, before election, so NO_SCHEDOBJ still has its selection; cluster-wide limits are captured after election. The position counter (`ctl->position`) is not SCHED_TRACE-gated (it creates the record) and counts only real pops. `sched_sim_isolation_lib` stays without the trace sources (unresolved references in a static archive are harmless); 5.1 decides whether to add them. `core->decisions` holds 5 items, so trace-only decisions take FIFO slots and cannot reorder real ones.
- 3.2: requirement-branch hooks are in scheduler.c (`sched_make_allocation`, `sched_make_alter_allocation`, `sched_make_compound_alloc_`). The alternatives after the chosen one are not stored as entries; 4.1 renders "alternatives i+1..n not tried" from the alt_result (i,n). A merge failure stores an ATTEMPT_REASONS entry with ncodes==0. A failing single requirement records no attempt reasons; its pend outcome carries them. A single job with a NULL resreq stores its begin as a container with 0 members before returning -EINVAL, and 4.1 must tolerate that.
- 3.3: the pre-filter (`sched_host_iter_next`) opens a host scope (`sched_trace_host_begin`) for every candidate and closes it with a verdict on each rejection: SELECT (pre-filter), INTERNAL(rc) for an uncommitted-fetch or check error, from-rows for a resource rejection (unreachable from the runner). A host the iterator returns is left in an open scope with no entry, so 3.4 must end or replace it for the accepted, already-claimed, snapshot-failure and per-host-error paths, or later rows attach to a stale host. A scan stopped with rc<0 calls `pos_scan_error`; exhaustion does not. `unsatisfy_` hands every row to `sched_trace_check_rows`. The sim thread's reconciliation reaches that hook but sees trace_rec NULL.
- 3.4: runner hooks are in alloc_runner.c. The 1.5 OPEN item is settled: `sched_trace_no_plan` counts `reqs_none_generated` once and sets `scope_no_plan`, after which `search_end` makes no statement and `resreq_begin`/`search_end` reset the flag. So a declined or generator-failed requirement has a NO_PLAN entry and no SEARCH_END line, and 4.1 must render the NO_PLAN line as "no plan generated: <reason>". The 3.3 open scope is closed: every returned candidate gets a fresh host scope and one verdict, and check_begin drops any stale host rows. When the iterator can't start, the position ends as SCAN_ERROR with err 0, so 4.1 must render a scan error with no code. trace.h is at 790 lines; split it before adding to it.

View file

@ -515,6 +515,7 @@ struct xce_sched_trace_rec_s {
int scope_discarded;
int scope_not_tried;
bool scope_plan_err; /* a tried plan ended in an internal error */
bool scope_no_plan; /* a no-plan reason already counted the search */
/* summary level: the current position's host tallies. The limit
* tallies live in builder memory (malloc'd, not the arena) until the
* position ends, so an omitted position allocates nothing in the
@ -616,13 +617,17 @@ sched_trace_attempt_reasons(xce_sched_trace_rec_t* rec,
int ncodes, const char* const* hosts, int nhosts,
const char* const* res, int nres);
/* Cluster-wide check scope. check_begin stores no entry; check_end stores
* the result, which on failure says no placement plans were tried (4.8). */
/* Cluster-wide check scope. check_begin stores no entry and drops any
* host scope left open; check_end stores the result, which on failure
* says no placement plans were tried (4.8). */
extern void sched_trace_check_begin(xce_sched_trace_rec_t* rec,
const xce_sched_obj_t* obj);
extern void sched_trace_check_end(xce_sched_trace_rec_t* rec, int rc,
bool plans_follow);
/* No plan generated for the current requirement, and why (4.7). Counts
* the requirement under "no plan generated" (9.3), once: a search end
* later in the same requirement makes no statement and counts nothing. */
extern void sched_trace_no_plan(xce_sched_trace_rec_t* rec,
xce_sched_trace_noplan_t why);
@ -696,6 +701,7 @@ extern void sched_trace_finalize_failed(xce_sched_trace_rec_t* rec, int err);
* generated (g == 0), all discarded (g > 0, d == g), or none placed (t > 0
* and no tried plan ended in an internal error). The matching decision-
* wide counter is bumped under the same conditions, stored or not (9.3).
* After a no_plan entry in the requirement, there is no statement.
*/
extern void sched_trace_search_end(xce_sched_trace_rec_t* rec, int rc);

View file

@ -29,6 +29,7 @@
#include "preason.h"
#include "resobj.h"
#include "sched/alloc_iter.h"
#include "sched/trace.h"
#include "sched/types.h"
#include "scheduler.h"
#include "utils.h"
@ -212,9 +213,14 @@ static int sched_claim_bucket_(xce_sched_core_t* core, xce_sched_ctx_t* ctx,
xce_host_iter_t it;
xce_sched_worker_t* cand;
int rc = 0;
int hrc;
sched_alloc_clear_host_req_(ctx);
/* Decision trace (4.4): the position, numbered from 1 */
SCHED_TRACE(core, sched_trace_pos_begin(core->trace_rec, b + 1,
plan->bucket_slots[b]));
it.init = sched_host_iter_init;
it.next = sched_host_iter_next;
it.fini = sched_host_iter_fini;
@ -222,18 +228,41 @@ static int sched_claim_bucket_(xce_sched_core_t* core, xce_sched_ctx_t* ctx,
/* An iterator that cannot start offers no candidate: an
* infrastructure failure, not a resource verdict (alloc_iter.h). */
if (it.init(&it, plan, core->workers) != 0) {
SCHED_TRACE(core, sched_trace_pos_end(core->trace_rec,
XCE_TRACE_POS_SCAN_ERROR, NULL));
return 0;
}
/* Decision trace: every candidate the iterator returns arrives in the
* pre-filter's open host scope (host_iter.c); each path below replaces
* it with its own scope and ends that with the candidate's verdict. */
while ((cand = it.next(&it)) != NULL) {
if (sched_alloc_is_claimed_(claims, *n, cand)) {
/* Decision trace (5.6) */
SCHED_TRACE(core, (sched_trace_host_begin(core->trace_rec,
ctx->schedobj, cand),
sched_trace_host_verdict(
core->trace_rec, XCE_TRACE_V_CLAIMED, 0)));
continue;
}
ctx->nslots = plan->bucket_slots[b];
ctx->nhosts = 1;
if (sched_try_alloc_on_host_(cand, core, ctx) > 0) {
/* Decision trace (5.1-5.7): the per-host check's own scope, so its
* unmet rows (resource.c) belong to this host; a rejection takes
* its verdict from those rows, or is a per-host selection mismatch
* when there are none */
SCHED_TRACE(
core, sched_trace_host_begin(core->trace_rec, ctx->schedobj, cand));
hrc = sched_try_alloc_on_host_(cand, core, ctx);
SCHED_TRACE(
core, sched_trace_host_verdict(core->trace_rec,
(hrc > 0) ? XCE_TRACE_V_ACCEPTED
: (hrc == 0) ? 0u
: XCE_TRACE_V_INTERNAL,
(hrc < 0) ? hrc : 0));
if (hrc > 0) {
rc = sched_snapshot_host_req_(&(claims[*n].host_req),
&(ctx->alloc_req.host));
if (rc < 0) {
@ -255,6 +284,15 @@ static int sched_claim_bucket_(xce_sched_core_t* core, xce_sched_ctx_t* ctx,
it.fini(&it);
/* Decision trace (4.4): filled with the claimed host, unfilled (a scan
* error when host_iter.c marked the position), or an internal error
* after the candidate was accepted */
SCHED_TRACE(core, sched_trace_pos_end(core->trace_rec,
(rc > 0) ? XCE_TRACE_POS_FILLED
: (rc == 0) ? XCE_TRACE_POS_UNFILLED
: XCE_TRACE_POS_INTERNAL,
(rc != 0) ? cand->name : NULL));
return rc;
}
@ -273,14 +311,26 @@ static int sched_try_plan_(xce_sched_core_t* core, xce_sched_ctx_t* ctx,
{
int n = 0;
int b;
int k;
int rc;
plan->ctx = ctx;
*nclaimed = 0;
/* Decision trace (4.3): the plan, in the order tried */
SCHED_TRACE(core, sched_trace_plan_begin(core->trace_rec, plan));
for (b = 0; b < plan->nbuckets; b++) {
rc = sched_claim_bucket_(core, ctx, plan, b, claims, &n);
if (rc <= 0) {
/* Decision trace (5.8): the earlier positions' hosts released,
* one line; then the plan's end at the failed position (4.4) */
SCHED_TRACE(
core, for (k = 0; k < n; k++) {
sched_trace_rollback_host(core->trace_rec,
claims[k].w->name);
});
SCHED_TRACE(core, sched_trace_plan_end(core->trace_rec, rc, b + 1));
sched_rollback_claims_(ctx, claims, n);
return rc;
}
@ -288,6 +338,8 @@ static int sched_try_plan_(xce_sched_core_t* core, xce_sched_ctx_t* ctx,
*nclaimed = n;
SCHED_TRACE(core, sched_trace_plan_end(core->trace_rec, 1, 0));
return 1;
}
@ -425,16 +477,27 @@ static int sched_alloc_search_(const xce_sched_allocator_iface_t* iface,
/* A generator that cannot start is not a resource verdict: the attempt
* pends, with nothing for plan_fini to release (Req 3.4). */
if (iface->plan_init(&iter, ctx->primary, core) != 0) {
/* Decision trace (4.7): counted as "no plan generated" here; no
* search end follows */
SCHED_TRACE(core,
sched_trace_no_plan(core->trace_rec,
XCE_TRACE_NOPLAN_GENERATOR_FAILED));
return 0;
}
while ((plan = iface->plan_next(&iter)) != NULL) {
if (sched_plan_malformed_(plan)) {
/* Decision trace (4.9) */
SCHED_TRACE(core,
sched_trace_plan_discarded(core->trace_rec, plan));
continue;
}
rc = sched_alloc_claims_reserve_(&claims, &cap, plan->nbuckets);
if (rc < 0) {
/* Decision trace (4.4) */
SCHED_TRACE(core,
sched_trace_plan_not_tried(core->trace_rec, plan, rc));
break;
}
@ -446,11 +509,18 @@ static int sched_alloc_search_(const xce_sched_allocator_iface_t* iface,
if (rc > 0) {
rc = sched_finalize_job_alloc_(sched, ctx, claims, n);
/* Decision trace (4.5) */
SCHED_TRACE(
core, (rc > 0) ? sched_trace_commit(core->trace_rec)
: sched_trace_finalize_failed(core->trace_rec, rc));
}
iface->plan_fini(&iter);
free(claims);
/* Decision trace (4.6, 4.7, 4.9): the requirement's plan counts */
SCHED_TRACE(core, sched_trace_search_end(core->trace_rec, rc));
return rc;
}
@ -472,6 +542,10 @@ int sched_allocator_run(void* plugin_arg, xce_sched_ctx_t* ctx)
}
if ((iface->accepts != NULL) && !iface->accepts(ctx->primary)) {
/* Decision trace (4.7): counted as "no plan generated" here */
SCHED_TRACE(plugin->core,
sched_trace_no_plan(plugin->core->trace_rec,
XCE_TRACE_NOPLAN_DECLINED));
return 0;
}
@ -480,7 +554,13 @@ int sched_allocator_run(void* plugin_arg, xce_sched_ctx_t* ctx)
return rc;
}
/* Decision trace (4.1, 4.2, 4.8): the cluster-wide check's scope; its
* rows come from resource.c, its end says whether plans follow */
SCHED_TRACE(plugin->core, sched_trace_check_begin(plugin->core->trace_rec,
ctx->schedobj));
rc = sched_alloc_prepare_shared_(plugin->core, ctx);
SCHED_TRACE(plugin->core,
sched_trace_check_end(plugin->core->trace_rec, rc, rc > 0));
if (rc <= 0) {
return rc;
}

View file

@ -426,6 +426,7 @@ void sched_trace_resreq_begin(xce_sched_trace_rec_t* rec,
rec->scope_discarded = 0;
rec->scope_not_tried = 0;
rec->scope_plan_err = false;
rec->scope_no_plan = false;
e = sti_scope_begin(rec, SCHED_TRACE_E_RESREQ_BEGIN);
if (e == NULL) {
@ -550,8 +551,10 @@ void sched_trace_check_begin(xce_sched_trace_rec_t* rec,
const xce_sched_obj_t* obj)
{
sti_rollback_run_end(rec);
rec->scope_obj = obj;
rec->scope_w = NULL;
rec->scope_obj = obj;
rec->scope_w = NULL;
rec->host_row_verdicts = 0;
sti_host_rows_clear(rec); /* never a stale host's rows (task 3.4) */
rec->check_open = true;
}
@ -574,8 +577,17 @@ void sched_trace_check_end(xce_sched_trace_rec_t* rec, int rc,
void sched_trace_no_plan(xce_sched_trace_rec_t* rec,
xce_sched_trace_noplan_t why)
{
sched_trace_entry_t* e = sti_entry_add(rec, SCHED_TRACE_E_NO_PLAN);
sched_trace_entry_t* e;
/* counted here, once: the runner returns before any search end on a
* decline or a generator start failure, and a search end after this
* entry must not report "no plan generated" again (4.7, 9.3) */
if (!rec->scope_no_plan) {
rec->scope_no_plan = true;
rec->counters.reqs_none_generated++;
}
e = sti_entry_add(rec, SCHED_TRACE_E_NO_PLAN);
if (e == NULL) {
return;
}

View file

@ -459,7 +459,7 @@ void sched_trace_search_end(xce_sched_trace_rec_t* rec, int rc)
int x = rec->scope_not_tried;
int g = t + d + x;
if (rc == 0) {
if ((rc == 0) && !rec->scope_no_plan) {
if (g == 0) {
stmt = SCHED_TRACE_SEARCH_NONE_GENERATED;
rec->counters.reqs_none_generated++;
@ -477,6 +477,7 @@ void sched_trace_search_end(xce_sched_trace_rec_t* rec, int rc)
rec->scope_discarded = 0;
rec->scope_not_tried = 0;
rec->scope_plan_err = false;
rec->scope_no_plan = false;
e = sti_entry_add(rec, SCHED_TRACE_E_SEARCH_END);
if (e == NULL) {

View file

@ -2696,3 +2696,23 @@ sched_trace_host_hooks_test = executable(
test('scheduler trace host hooks test',
sched_trace_host_hooks_test, suite: 'scheduler',
env: ['ASAN_OPTIONS=detect_leaks=0'])
# scheduler-sim-decision-trace task 3.4: the allocator runner's trace hooks
# (design.md "TraceHooks": Runner entry, Cluster-wide check, Generator start
# failure, Search end, Plan loop, Position, Candidate, Rollback, Finalize;
# Requirements 4.1, 4.3-4.9, 5.1, 5.2, 5.6-5.8) on the real
# sched_allocator_run(). Same white-box #include set as
# sched_alloc_runner_search_test; xvec_dup (claim snapshot) and
# make_job_alloc (finalize) are wrapped to fail on demand.
sched_trace_hooks_runner_test = executable(
'sched_trace_hooks_runner_test',
['test_trace_hooks_runner.c'],
include_directories:[incdir],
dependencies: [scheduler_test_deps],
link_args: ['-Wl,--wrap=xvec_dup',
'-Wl,--wrap=make_job_alloc'],
install: false)
test('scheduler trace hooks runner test',
sched_trace_hooks_runner_test, suite: 'scheduler',
env: ['ASAN_OPTIONS=detect_leaks=0'])

File diff suppressed because it is too large Load diff

View file

@ -1334,8 +1334,7 @@ static void test_search_end_none_generated(void)
fixture_primary_t fp;
TEST_ASSERT_NOT_NULL(rec);
open_single(rec, &fp);
sched_trace_no_plan(rec, XCE_TRACE_NOPLAN_NONE_GENERATED);
open_single(rec, &fp); /* the search states it: no no_plan entry */
sched_trace_search_end(rec, 0);
assert_search_end(rec->entries_tail, 0, 0, 0, 0, 0,
SCHED_TRACE_SEARCH_NONE_GENERATED);
@ -1481,8 +1480,7 @@ static void test_search_end_nonzero_rc_has_no_statement(void)
SCHED_TRACE_SEARCH_NO_STMT);
close_single(rec, 1);
open_single(rec, &fp); /* generator failed: no plan, rc < 0 */
sched_trace_no_plan(rec, XCE_TRACE_NOPLAN_GENERATOR_FAILED);
open_single(rec, &fp); /* nothing generated, but rc < 0 */
sched_trace_search_end(rec, -12);
assert_search_end(rec->entries_tail, 0, 0, 0, 0, -12,
SCHED_TRACE_SEARCH_NO_STMT);
@ -1492,6 +1490,44 @@ static void test_search_end_nonzero_rc_has_no_statement(void)
sched_trace_rec_free(rec);
}
/* A declined requirement or a generator that cannot start (the runner
* returns 0 there, without a search end) is counted as "no plan generated"
* by its no_plan entry, once; a search end after it in the same
* requirement neither repeats the statement nor counts it again (4.7,
* 9.3). The next requirement starts over. */
static void test_no_plan_counts_none_generated_once(void)
{
xce_sched_trace_rec_t* rec = new_rec(XCE_TRACE_LINE_BOUND_DEFAULT);
fixture_primary_t fp;
TEST_ASSERT_NOT_NULL(rec);
make_primary(&fp, 1, 1);
sched_trace_path_push(rec, XCE_TRACE_PATH_SINGLE, 1, 1);
sched_trace_resreq_begin(rec, &fp.p, 0);
sched_trace_no_plan(rec, XCE_TRACE_NOPLAN_DECLINED);
close_single(rec, 0);
assert_req_counters(rec, 1, 0, 0);
open_single(rec, &fp);
sched_trace_no_plan(rec, XCE_TRACE_NOPLAN_GENERATOR_FAILED);
TEST_ASSERT_EQUAL_INT(SCHED_TRACE_E_NO_PLAN, rec->entries_tail->type);
assert_req_counters(rec, 2, 0, 0);
sched_trace_search_end(rec, 0);
assert_search_end(rec->entries_tail, 0, 0, 0, 0, 0,
SCHED_TRACE_SEARCH_NO_STMT);
close_single(rec, 0);
assert_req_counters(rec, 2, 0, 0);
/* a later requirement with no no_plan entry states it itself */
open_single(rec, &fp);
sched_trace_search_end(rec, 0);
assert_search_end(rec->entries_tail, 0, 0, 0, 0, 0,
SCHED_TRACE_SEARCH_NONE_GENERATED);
close_single(rec, 0);
assert_req_counters(rec, 3, 0, 0);
sched_trace_rec_free(rec);
}
/* Counts belong to the requirement the search ran for: they restart with
* each member, while the decision-wide counters add up. */
static void test_search_end_counts_per_requirement(void)
@ -1539,6 +1575,35 @@ static void test_search_end_counts_per_requirement(void)
sched_trace_rec_free(rec);
}
/* A cluster-wide check never runs in a host scope left open: check_begin
* drops the stale host, its held rows and its row verdicts, so the check's
* rows are cluster-wide rows (task 3.4). */
static void test_check_begin_drops_a_stale_host_scope(void)
{
xce_sched_trace_rec_t* rec = new_rec(XCE_TRACE_LINE_BOUND_DEFAULT);
char name[8] = "n01";
xce_sched_worker_t w = make_worker(name, 0);
sched_trace_row_t row;
TEST_ASSERT_NOT_NULL(rec);
memset(&row, 0, sizeof(row));
row.res = "mem";
sched_trace_host_begin(rec, NULL, &w);
sched_trace_row(rec, &row); /* held for the host's verdict */
rec->host_row_verdicts |= XCE_TRACE_V_SHORTFALL;
TEST_ASSERT_EQUAL_INT(1, rec->host_nrows);
sched_trace_check_begin(rec, NULL);
TEST_ASSERT_NULL(rec->scope_w);
TEST_ASSERT_EQUAL_INT(0, rec->host_nrows);
TEST_ASSERT_EQUAL_UINT(0, rec->host_row_verdicts);
sched_trace_row(rec, &row);
TEST_ASSERT_EQUAL_INT(SCHED_TRACE_E_CHECK_ROW, rec->entries_tail->type);
sched_trace_check_end(rec, 1, true);
TEST_ASSERT_EQUAL_INT(2, rec->nentries);
sched_trace_rec_free(rec);
}
/* ---- counters under truncation (9.3) ---------------------------------- */
/*
@ -2008,6 +2073,8 @@ int main(void)
RUN_TEST(test_search_end_plan_internal_error_has_no_statement);
RUN_TEST(test_search_end_not_tried_after_discarded);
RUN_TEST(test_search_end_nonzero_rc_has_no_statement);
RUN_TEST(test_no_plan_counts_none_generated_once);
RUN_TEST(test_check_begin_drops_a_stale_host_scope);
RUN_TEST(test_search_end_counts_per_requirement);
RUN_TEST(test_counters_exact_after_truncation);
RUN_TEST(test_summary_position_tallies);