lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(scheduler-sim-decision-trace): render trace records to text

Browse source 
Task 4.1: hierarchy, summary-level position summaries, truncation marker
with the fixed totals block, 50-host outcome cap with aggregates and
separate limit charges, no-decision and reconstructed records, amounts
in the REPL units mode, identical bytes to one or two sinks.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-08 13:30:06 +00:00
parent 16f40fa9dc
commit fe445e9453
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
8 changed files with 2479 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

3
.kiro/specs/scheduler-sim-decision-trace/tasks.md
View file

@ -180,7 +180,7 @@
- _Depends: 2.2, 3.3_
- [ ] 4. Simulator REPL trace surface
- [ ] 4.1 Render trace records to text, test first (P)
- [x] 4.1 Render trace records to text, test first (P)
- **Hierarchy:** header, selection, requirements, cluster-wide checks and plans, positions and hosts, then the outcome. Every budgeted entry renders as exactly one line, in the order shown in the design's example.
- **Summary level:** renders position summaries with per-limit names.
- **Truncated records:** render a marker with the omitted line count, followed by the fixed-size totals block.
@ -364,3 +364,4 @@
- 3.2: requirement-branch hooks are in scheduler.c (`sched_make_allocation`, `sched_make_alter_allocation`, `sched_make_compound_alloc_`). The alternatives after the chosen one are not stored as entries; 4.1 renders "alternatives i+1..n not tried" from the alt_result (i,n). A merge failure stores an ATTEMPT_REASONS entry with ncodes==0. A failing single requirement records no attempt reasons; its pend outcome carries them. A single job with a NULL resreq stores its begin as a container with 0 members before returning -EINVAL, and 4.1 must tolerate that.
- 3.3: the pre-filter (`sched_host_iter_next`) opens a host scope (`sched_trace_host_begin`) for every candidate and closes it with a verdict on each rejection: SELECT (pre-filter), INTERNAL(rc) for an uncommitted-fetch or check error, from-rows for a resource rejection (unreachable from the runner). A host the iterator returns is left in an open scope with no entry, so 3.4 must end or replace it for the accepted, already-claimed, snapshot-failure and per-host-error paths, or later rows attach to a stale host. A scan stopped with rc<0 calls `pos_scan_error`; exhaustion does not. `unsatisfy_` hands every row to `sched_trace_check_rows`. The sim thread's reconciliation reaches that hook but sees trace_rec NULL.
- 3.4: runner hooks are in alloc_runner.c. The 1.5 OPEN item is settled: `sched_trace_no_plan` counts `reqs_none_generated` once and sets `scope_no_plan`, after which `search_end` makes no statement and `resreq_begin`/`search_end` reset the flag. So a declined or generator-failed requirement has a NO_PLAN entry and no SEARCH_END line, and 4.1 must render the NO_PLAN line as "no plan generated: <reason>". The 3.3 open scope is closed: every returned candidate gets a fresh host scope and one verdict, and check_begin drops any stale host rows. When the iterator can't start, the position ends as SCAN_ERROR with err 0, so 4.1 must render a scan error with no code. trace.h is at 790 lines; split it before adding to it.
- 4.1: the renderer is `repl_trace_render(rec, units, out1, out2)`, declared in repl_internal.h, in repl_trace_render.c plus repl_trace_render_outcome.c, with a private repl_trace_render.h. It renders once per sink, so the bytes are identical. Amounts use `repl_format_numeric_resource_value` ("8.00 GB" in human mode) and times use `repl_format_ts_human`. Wording where the design is silent is fixed by the golden tests in test_trace_render.c, so 4.2/4.6/6.1 must reuse the renderer, never re-format. Pend-reason arrays may contain NULL entries (capture keeps them); every walk must skip them. The renderer is not yet in sched_sim_isolation_lib or the sim isolation scan list (5.1).

2
src/scheduler/meson.build
View file

@ -29,6 +29,8 @@ scheduler_srcs = [
'src/repl_break.c',
'src/repl_info.c',
'src/repl_params.c',
'src/repl_trace_render.c',
'src/repl_trace_render_outcome.c',
'src/schedobj.c',
'src/plugin.c',
'src/resource.c',

15
src/scheduler/src/repl_internal.h
View file

@ -523,4 +523,19 @@ extern int repl_count_jobs(xce_sched_core_t* core);
*/
extern void repl_round_rollup_process(xce_sched_sim_driver_t* sim);
/*
* repl_trace_render (repl_trace_render.c, scheduler-sim-decision-trace
* spec, task 4.1, design.md "TraceRenderer") - writes one decision trace
* record as text to `out1` and, when non-NULL, the same bytes to `out2`:
* the header, the selection block, the requirement/check/plan/host entries
* (one line per stored entry, two-space indent per level), the truncation
* marker and totals, then the outcome or the no-decision line. Amounts use
* `units` (7.7). The record is only read. NULL `rec` or `out1`: nothing.
* The record type stays opaque here (sched/trace.h has the layout).
*/
struct xce_sched_trace_rec_s;
extern void repl_trace_render(const struct xce_sched_trace_rec_s* rec,
xce_repl_units_mode_t units, FILE* out1,
FILE* out2);
#endif

773
src/scheduler/src/repl_trace_render.c Normal file
View file

@ -0,0 +1,773 @@
/*
* repl_trace_render.c - decision trace record to text
* (scheduler-sim-decision-trace spec, task 4.1, design.md "TraceRenderer").
*
* One record renders as:
*
* [trace] job <jobid> session <n> position <p> level <level>
* selection: ... (2.2; or "unavailable", 2.4)
* cluster-wide limits: ... (2.3; or "unavailable", 2.4)
* <one line per stored entry> (3.x, 4.x, 5.x)
* ... <k> trace lines omitted (line bound N) + 3 totals lines (9.2, 9.3)
* ... trace incomplete (out of memory) (design "Error Strategy")
* outcome: ... (6.x, 9.4)
*
* Every stored entry is exactly one line, indented two spaces per nesting
* level below the header (7.6), so the requirement, cluster-wide check,
* plan and host sections never print more lines than the record's line
* bound (design "Line contract"). A no-decision record (2.4) ends with one
* line naming its kind and error in place of the outcome; a reconstructed
* record (2.5) has a notice in place of the selection and entries.
*
* Amounts are formatted by the resource they measure (a limit amount by the
* resource the limit limits) with the units mode passed in, through
* repl_fmt.c's repl_format_numeric_resource_value() (7.7); job ids use
* fmt_jobid(). The same bytes go to each sink: the record is rendered once
* per sink by the same deterministic code.
*
* REPL side (design "Allowed Dependencies"): reads the record only (layout
* in sched/trace.h), touches no scheduler state and calls no scheduler
* API. The outcome section and the helpers both files use (indent,
* amounts, error text, limit names, pend-reason names) live in
* repl_trace_render_outcome.c, declared in repl_trace_render.h.
*/
#include <stdio.h>
#include "repl_trace_render.h"
#include "utils.h"
/* Verdict kind names, indexed by the kind's bit position (5.10). */
static const char* const KIND_NAMES[XCE_TRACE_V_NKINDS] = {
"accepted", "selection mismatch", "resource shortfall",
"limit exceeded", "already claimed", "not evaluated (internal error)"
};
/* ---- requirement labels (3.1) ------------------------------------------- */
static void put_level(FILE* f, xce_sched_trace_path_kind_t kind, int i, int n)
{
switch (kind) {
case XCE_TRACE_PATH_SINGLE:
fputs("single", f);
break;
case XCE_TRACE_PATH_COMPOUND:
fputs("compound", f);
break;
case XCE_TRACE_PATH_ALT:
fprintf(f, "alternative %d/%d", i, n);
break;
case XCE_TRACE_PATH_MEMBER:
default:
fprintf(f, "member %d/%d", i, n);
break;
}
}
/* "requirement <level> > <level>": the full nesting path. */
static void put_requirement(FILE* f, const sched_trace_path_t* p)
{
int k;
fputs("requirement", f);
for (k = 0; k < p->depth; k++) {
fputs((k == 0) ? " " : " > ", f);
put_level(f, p->lvl[k].kind, p->lvl[k].i, p->lvl[k].n);
}
}
/* The innermost level as the subject of an outcome line. */
static void put_subject(FILE* f, const sched_trace_path_t* p)
{
if (p->depth <= 0) {
fputs("requirement", f);
return;
}
switch (p->lvl[p->depth - 1].kind) {
case XCE_TRACE_PATH_SINGLE:
fputs("single requirement", f);
break;
case XCE_TRACE_PATH_COMPOUND:
fputs("compound requirement", f);
break;
default:
put_level(f, p->lvl[p->depth - 1].kind, p->lvl[p->depth - 1].i,
p->lvl[p->depth - 1].n);
break;
}
}
/* "; member 3/3 not tried" or "; members 2..3 not tried": the `count`
* siblings after sibling `i` of `n` (3.3, 3.4). */
static void put_not_tried(FILE* f, const char* word, int i, int n, int count)
{
int first = i + 1;
int last = i + count;
if (count <= 0) {
return;
}
if (first == last) {
fprintf(f, "; %s %d/%d not tried", word, first, n);
} else {
fprintf(f, "; %ss %d..%d not tried", word, first, last);
}
}
static void put_outcome_word(FILE* f, int rc, int err)
{
char eb[RTR_BUFSZ];
if (rc > 0) {
fputs(" satisfied", f);
} else if (rc == 0) {
fputs(" not satisfied", f);
} else {
fprintf(f, " ended in an internal error%s",
rtr_err(err, eb, sizeof(eb)));
}
}
static const char* member_suffix(int n)
{
return (n == 1) ? "" : "s";
}
/* ---- entry lines --------------------------------------------------------- */
static void put_resreq_begin(FILE* f, const sched_trace_entry_t* e,
xce_repl_units_mode_t units)
{
const sched_trace_path_t* p = &e->path;
char buf[RTR_BUFSZ];
int n = e->u.resreq_begin.nmembers;
int k;
put_requirement(f, p);
if (e->u.resreq_begin.container) {
xce_sched_trace_path_kind_t kind
= (p->depth > 0) ? p->lvl[p->depth - 1].kind
: XCE_TRACE_PATH_SINGLE;
if (kind == XCE_TRACE_PATH_ALT) {
fprintf(f, " (compound, %d member%s)", n, member_suffix(n));
} else if ((kind == XCE_TRACE_PATH_COMPOUND) || (n > 0)) {
fprintf(f, " (%d member%s)", n, member_suffix(n));
} else {
/* a single job without a requirement (task 3.2) */
fputs(" (no requirement content)", f);
}
return;
}
fprintf(f, " tasks %d..%d requests", e->u.resreq_begin.min_tasks,
e->u.resreq_begin.max_tasks);
if ((e->u.resreq_begin.req == NULL) || (e->u.resreq_begin.nreq <= 0)) {
fputs(" none", f);
return;
}
for (k = 0; k < e->u.resreq_begin.nreq; k++) {
const sched_trace_req_t* r = &e->u.resreq_begin.req[k];
/* a NULL rusage left a NULL name (task 1.4) */
fprintf(f, " %s=%s", (r->name != NULL) ? r->name : "?",
rtr_value(units, r->name, r->value, buf, sizeof(buf)));
if (r->per == PER_JOB) {
fputs("/job", f);
} else if (r->per == PER_HOST) {
fputs("/host", f);
}
}
}
static void put_resreq_end(FILE* f, const sched_trace_entry_t* e)
{
const sched_trace_path_t* p = &e->path;
put_subject(f, p);
put_outcome_word(f, e->u.resreq_end.rc, e->u.resreq_end.rc);
if ((p->depth > 0)
&& (p->lvl[p->depth - 1].kind == XCE_TRACE_PATH_MEMBER)) {
put_not_tried(f, "member", p->lvl[p->depth - 1].i,
p->lvl[p->depth - 1].n, e->u.resreq_end.not_tried_after);
}
}
static void put_alt_result(FILE* f, int i, int n)
{
if (i <= 0) {
fprintf(f, "no alternative chosen: none of the %d alternatives was "
"satisfied",
n);
return;
}
fprintf(f, "alternative %d/%d chosen", i, n);
if (i >= n) {
fputs("; no alternatives after it", f);
} else {
put_not_tried(f, "alternative", i, n, n - i);
}
}
static void put_reasons(FILE* f, const sched_trace_entry_t* e)
{
char buf[RTR_BUFSZ];
int nh = (e->u.reasons.hosts != NULL) ? e->u.reasons.nhosts : 0;
int nr = (e->u.reasons.res != NULL) ? e->u.reasons.nres : 0;
int k;
fputs("failure reasons: ", f);
if ((e->u.reasons.codes == NULL) || (e->u.reasons.ncodes <= 0)) {
/* a merge failure or an empty reason set (tasks 2.1, 3.2) */
fputs("none recorded", f);
}
for (k = 0; (e->u.reasons.codes != NULL) && (k < e->u.reasons.ncodes);
k++) {
fprintf(f, "%s%s x%d", (k > 0) ? ", " : "",
rtr_reason_name(e->u.reasons.codes[k].code, buf, sizeof(buf)),
e->u.reasons.codes[k].count);
}
if ((nh == 0) && (nr == 0)) {
return;
}
fputs(" (", f);
for (k = 0; k < nh; k++) {
fprintf(f, "%s%s", (k == 0) ? "hosts: " : ", ",
(e->u.reasons.hosts[k] != NULL) ? e->u.reasons.hosts[k] : "?");
}
for (k = 0; k < nr; k++) {
const char* sep = (k > 0) ? ", "
: (nh > 0) ? "; resources: "
: "resources: ";
fprintf(f, "%s%s", sep,
(e->u.reasons.res[k] != NULL) ? e->u.reasons.res[k] : "?");
}
fputc(')', f);
}
/* A satisfaction-check row: a resource row (4.1, 5.4) or a limit row (4.2,
* 5.5), never both. A limit row's amounts use the limited resource, or
* the raw number when it is unknown (task 2.2); a row whose resource index
* is unknown is named by that index (task 2.2). */
static void put_row(FILE* f, const sched_trace_row_t* r, bool cluster,
xce_repl_units_mode_t units)
{
char b1[RTR_BUFSZ];
char b2[RTR_BUFSZ];
char b3[RTR_BUFSZ];
char b4[RTR_BUFSZ];
const char* by;
if ((r->flags & XCE_RESF_LIMIT) != 0) {
by = r->limited_res;
fputs(cluster ? "cluster-wide limit " : "limit ", f);
rtr_limit_ref(f, (r->limit != NULL) ? r->limit : r->res, r->scope);
if (by != NULL) {
fprintf(f, " on %s", by);
}
fprintf(f, ": value %s usage %s reserved %s requested %s",
rtr_value(units, by, r->total, b1, sizeof(b1)),
rtr_value(units, by, r->inuse, b2, sizeof(b2)),
rtr_value(units, by, r->reserved, b3, sizeof(b3)),
rtr_value(units, by, r->requested, b4, sizeof(b4)));
} else {
by = r->res;
if (cluster) {
fputs("cluster-wide: ", f);
}
if (by != NULL) {
fputs(by, f);
} else {
fprintf(f, "resource #%d", r->idx);
}
fprintf(f, " total %s in-use %s reserved %s requested %s",
rtr_value(units, by, r->total, b1, sizeof(b1)),
rtr_value(units, by, r->inuse, b2, sizeof(b2)),
rtr_value(units, by, r->reserved, b3, sizeof(b3)),
rtr_value(units, by, r->requested, b4, sizeof(b4)));
}
if (cluster) {
fputs(r->met ? " passed" : " failed", f);
}
}
static void put_check_end(FILE* f, int rc)
{
char eb[RTR_BUFSZ];
if (rc > 0) {
fputs("cluster-wide check passed", f);
} else if (rc == 0) {
fputs("cluster-wide check failed; no placement plans tried", f);
} else {
fprintf(f, "cluster-wide check ended in an internal error%s; no "
"placement plans tried",
rtr_err(rc, eb, sizeof(eb)));
}
}
static void put_no_plan(FILE* f, xce_sched_trace_noplan_t why)
{
fputs("no plan generated: ", f);
switch (why) {
case XCE_TRACE_NOPLAN_DECLINED:
fputs("allocator declined the requirement", f);
break;
case XCE_TRACE_NOPLAN_GENERATOR_FAILED:
fputs("plan generator failed to start", f);
break;
case XCE_TRACE_NOPLAN_NONE_GENERATED:
default:
fputs("generator produced no plan", f);
break;
}
}
/* "plan <n> hosts <h> tasks per position [a, b]" (4.3). */
static void put_plan(FILE* f, const sched_trace_entry_t* e)
{
int k;
fprintf(f, "plan %d hosts %d tasks per position [", e->u.plan.plan_no,
e->u.plan.nbuckets);
for (k = 0; (e->u.plan.slots != NULL) && (k < e->u.plan.nbuckets); k++) {
fprintf(f, "%s%d", (k > 0) ? ", " : "", e->u.plan.slots[k]);
}
fputc(']', f);
}
static void put_plan_end(FILE* f, const sched_trace_entry_t* e)
{
char eb[RTR_BUFSZ];
int rc = e->u.plan_end.rc;
fprintf(f, "plan %d", e->u.plan_end.plan_no);
if (rc > 0) {
fputs(" succeeded", f);
} else if (rc == 0) {
fputs(" failed", f);
if (e->u.plan_end.failed_pos > 0) {
fprintf(f, ": position %d could not be filled",
e->u.plan_end.failed_pos);
}
} else {
fprintf(f, " ended in an internal error%s; no further plans tried",
rtr_err(rc, eb, sizeof(eb)));
}
}
static void put_pos_end(FILE* f, const sched_trace_entry_t* e)
{
char eb[RTR_BUFSZ];
const char* host = e->u.pos_end.host;
fprintf(f, "position %d ", e->u.pos_end.pos);
switch (e->u.pos_end.how) {
case XCE_TRACE_POS_FILLED:
fprintf(f, "filled by %s", (host != NULL) ? host : "?");
break;
case XCE_TRACE_POS_UNFILLED:
fputs("not filled: no host accepted", f);
break;
case XCE_TRACE_POS_SCAN_ERROR:
/* err 0: the scan could not start (task 3.4) */
fprintf(f, "not filled: internal error in the host scan%s",
rtr_err(e->u.pos_end.err, eb, sizeof(eb)));
break;
case XCE_TRACE_POS_INTERNAL:
default:
fprintf(f, "not filled: internal error after accepting %s%s",
(host != NULL) ? host : "?",
rtr_err(e->u.pos_end.err, eb, sizeof(eb)));
break;
}
}
/* Summary level (5.9): the position's end with its host tallies. */
static void put_pos_summary(FILE* f, const sched_trace_entry_t* e)
{
char eb[RTR_BUFSZ];
const char* host = e->u.pos_summary.host;
const int* c = e->u.pos_summary.counts;
int k;
switch (e->u.pos_summary.how) {
case XCE_TRACE_POS_FILLED:
fprintf(f, "accepted %s", (host != NULL) ? host : "?");
break;
case XCE_TRACE_POS_SCAN_ERROR:
/* the scan error survives into the summary (task 1.5) */
fprintf(f, "accepted none; position not filled: internal error in "
"the host scan%s",
rtr_err(e->u.pos_summary.err, eb, sizeof(eb)));
break;
case XCE_TRACE_POS_INTERNAL:
fprintf(f, "accepted %s; position not filled: internal error after "
"accepting it%s",
(host != NULL) ? host : "?",
rtr_err(e->u.pos_summary.err, eb, sizeof(eb)));
break;
case XCE_TRACE_POS_UNFILLED:
default:
fputs("accepted none", f);
break;
}
fprintf(f, "; rejected: %s %d, %s %d, %s %d", KIND_NAMES[1], c[1],
KIND_NAMES[2], c[2], KIND_NAMES[3], c[3]);
if ((e->u.pos_summary.limits != NULL) && (e->u.pos_summary.nlimits > 0)) {
fputs(" (", f);
for (k = 0; k < e->u.pos_summary.nlimits; k++) {
const sched_trace_limit_tally_t* t = &e->u.pos_summary.limits[k];
if (k > 0) {
fputs(", ", f);
}
rtr_limit_ref(f, t->limit, t->scope);
fprintf(f, " %d", t->count);
}
fputc(')', f);
}
fprintf(f, ", %s %d, %s %d", KIND_NAMES[4], c[4], KIND_NAMES[5], c[5]);
}
/* Full level: one host and its verdict (5.1-5.7, 5.10). */
static void put_host(FILE* f, const sched_trace_entry_t* e)
{
char eb[RTR_BUFSZ];
unsigned v = e->u.host.verdicts;
int n = 0;
int k;
fprintf(f, "host %s ",
(e->u.host.host != NULL) ? e->u.host.host : "?");
if ((v & XCE_TRACE_V_ACCEPTED) != 0) {
fputs("accepted", f);
return;
}
if ((v & XCE_TRACE_V_CLAIMED) != 0) {
fputs("skipped: already claimed by an earlier position", f);
return;
}
if ((v & XCE_TRACE_V_INTERNAL) != 0) {
fprintf(f, "not evaluated: internal error%s",
rtr_err(e->u.host.err, eb, sizeof(eb)));
return;
}
fputs("rejected", f);
if (e->u.host.phase == SCHED_TRACE_PHASE_PREFILTER) {
fputs(" (pre-filter)", f);
} else if (e->u.host.phase == SCHED_TRACE_PHASE_PERHOST) {
fputs(" (per-host check)", f);
}
fputs(": ", f);
for (k = 1; k <= 3; k++) { /* selection, shortfall, limit */
if ((v & (1u << k)) != 0) {
fprintf(f, "%s%s", (n++ > 0) ? ", " : "", KIND_NAMES[k]);
}
}
if (n == 0) {
fputs("no reason recorded", f);
}
if (e->u.host.phase == SCHED_TRACE_PHASE_PREFILTER) {
fputs("; per-host check not performed", f);
}
}
static void put_search_end(FILE* f, const sched_trace_entry_t* e)
{
fprintf(f, "plans: %d generated, %d tried, %d discarded, %d not tried",
e->u.search_end.generated, e->u.search_end.tried,
e->u.search_end.discarded, e->u.search_end.not_tried);
switch (e->u.search_end.stmt) {
case SCHED_TRACE_SEARCH_NONE_GENERATED:
fputs("; no plan generated", f);
break;
case SCHED_TRACE_SEARCH_ALL_DISCARDED:
fputs("; no plan tried: all generated plans were discarded", f);
break;
case SCHED_TRACE_SEARCH_NONE_PLACED:
fputs("; no plan could be placed", f);
break;
case SCHED_TRACE_SEARCH_NO_STMT:
default:
break;
}
}
/* One stored entry, one line. */
static void put_entry(FILE* f, const sched_trace_entry_t* e,
xce_repl_units_mode_t units)
{
char eb[RTR_BUFSZ];
int k;
rtr_indent(f, e->depth + 1);
switch (e->type) {
case SCHED_TRACE_E_RESREQ_BEGIN:
put_resreq_begin(f, e, units);
break;
case SCHED_TRACE_E_RESREQ_END:
put_resreq_end(f, e);
break;
case SCHED_TRACE_E_ALT_SKIPPED:
fprintf(f, "alternative %d/%d skipped: no requirement content",
e->u.alt.i, e->u.alt.n);
break;
case SCHED_TRACE_E_ALT_RESULT:
put_alt_result(f, e->u.alt.i, e->u.alt.n);
break;
case SCHED_TRACE_E_MEMBER_FAILED:
fprintf(f, "member %d/%d: merging its allocation ended in an "
"internal error%s",
e->u.member_failed.j, e->u.member_failed.m,
rtr_err(e->u.member_failed.err, eb, sizeof(eb)));
put_not_tried(f, "member", e->u.member_failed.j, e->u.member_failed.m,
e->u.member_failed.m - e->u.member_failed.j);
break;
case SCHED_TRACE_E_ATTEMPT_REASONS:
put_reasons(f, e);
break;
case SCHED_TRACE_E_CHECK_END:
put_check_end(f, e->u.check_end.rc);
break;
case SCHED_TRACE_E_NO_PLAN:
put_no_plan(f, e->u.no_plan.why);
break;
case SCHED_TRACE_E_PLAN_BEGIN:
put_plan(f, e);
break;
case SCHED_TRACE_E_PLAN_DISCARDED:
put_plan(f, e);
fputs(" discarded", f);
break;
case SCHED_TRACE_E_PLAN_NOT_TRIED:
put_plan(f, e);
fprintf(f, " not tried: internal error%s; no further plans tried",
rtr_err(e->u.plan.err, eb, sizeof(eb)));
break;
case SCHED_TRACE_E_PLAN_END:
put_plan_end(f, e);
break;
case SCHED_TRACE_E_POS_BEGIN:
fprintf(f, "position %d (%d task%s)", e->u.pos_begin.pos,
e->u.pos_begin.slots, (e->u.pos_begin.slots == 1) ? "" : "s");
break;
case SCHED_TRACE_E_POS_END:
put_pos_end(f, e);
break;
case SCHED_TRACE_E_POS_SUMMARY:
put_pos_summary(f, e);
break;
case SCHED_TRACE_E_HOST:
put_host(f, e);
break;
case SCHED_TRACE_E_COMMIT:
fprintf(f, "plan %d committed; no further plans tried",
e->u.commit.plan_no);
break;
case SCHED_TRACE_E_FINALIZE_FAILED:
fprintf(f, "plan %d ended in an internal error building the "
"allocation%s; not committed, no further plans tried",
e->u.commit.plan_no, rtr_err(e->u.commit.err, eb, sizeof(eb)));
break;
case SCHED_TRACE_E_SEARCH_END:
put_search_end(f, e);
break;
case SCHED_TRACE_E_ROLLBACK:
fputs("released hosts: ", f);
for (k = 0; (e->u.rollback.hosts != NULL) && (k < e->u.rollback.nhosts);
k++) {
fprintf(f, "%s%s", (k > 0) ? ", " : "",
(e->u.rollback.hosts[k] != NULL) ? e->u.rollback.hosts[k]
: "?");
}
break;
case SCHED_TRACE_E_CHECK_ROW:
put_row(f, &e->u.row, true, units);
break;
case SCHED_TRACE_E_HOST_ROW:
/* stored one level below its host line (task 2.2) */
put_row(f, &e->u.row, false, units);
break;
case SCHED_TRACE_E_NTYPES:
default:
fprintf(f, "unknown trace entry %d", (int)e->type);
break;
}
fputc('\n', f);
}
/* ---- header, selection, totals ------------------------------------------ */
static void put_header(FILE* f, const xce_sched_trace_rec_t* rec)
{
char job[RTR_BUFSZ];
if (fmt_jobid(job, sizeof(job), rec->jobid) < 0) {
snprintf(job, sizeof(job), "%llu", (unsigned long long)rec->jobid);
}
fprintf(f, "[trace] job %s session %llu position %d level %s\n", job,
(unsigned long long)rec->session_no, rec->position,
(rec->level == XCE_TRACE_LEVEL_FULL) ? "full" : "summary");
}
/* The selection line (2.2) and the cluster-wide limits line (2.3); either
* says "unavailable" when it was never captured (2.4, task 2.1). */
static void put_selection(FILE* f, const xce_sched_trace_rec_t* rec)
{
const char* policy = (rec->policy_name != NULL) ? rec->policy_name
: "unknown";
char ts[64];
int k;
rtr_indent(f, 1);
if (!rec->selection_set) {
fprintf(f, "selection: policy %s; ordering keys unavailable\n",
policy);
} else {
repl_format_ts_human((uint64_t)rec->queue_time * 1000000000ULL, ts,
sizeof(ts));
fprintf(f, "selection: policy %s queue ", policy);
if (rec->queue_name != NULL) {
fprintf(f, "%s (priority %d)", rec->queue_name,
(int)rec->queue_priority);
} else {
fputs("none", f);
}
fprintf(f, " job priority %d queued %s\n", (int)rec->job_priority,
ts);
}
rtr_indent(f, 1);
if (!rec->glimits_set) {
fprintf(f, "cluster-wide limits: unavailable%s\n",
(rec->kind == XCE_TRACE_REC_NO_SCHEDOBJ)
? " (no scheduling object)"
: "");
return;
}
fputs("cluster-wide limits: ", f);
if ((rec->glimits == NULL) || (rec->nglimits <= 0)) {
fputs("none", f);
}
for (k = 0; (rec->glimits != NULL) && (k < rec->nglimits); k++) {
if (k > 0) {
fputs(", ", f);
}
rtr_limit_ref(f, rec->glimits[k].limit, rec->glimits[k].scope);
}
fputs("; host-scoped limits are reported with each host's verdict\n", f);
}
/* After truncation: the marker and the fixed-size totals (9.2, 9.3). */
static void put_truncation(FILE* f, const xce_sched_trace_rec_t* rec)
{
const sched_trace_counters_t* c = &rec->counters;
int k;
if (c->omitted_lines <= 0) {
return;
}
rtr_indent(f, 1);
fprintf(f, "... %d trace lines omitted (line bound %d)\n",
c->omitted_lines, rec->line_bound);
rtr_indent(f, 1);
fprintf(f, "totals: plans tried %d, plans discarded %d\n", c->plans_tried,
c->plans_discarded);
rtr_indent(f, 1);
fprintf(f, "totals: requirements with no plan generated %d, all plans "
"discarded %d, no plan placed %d\n",
c->reqs_none_generated, c->reqs_all_discarded,
c->reqs_none_placed);
rtr_indent(f, 1);
fputs("totals: host evaluations: ", f);
for (k = 0; k < XCE_TRACE_V_NKINDS; k++) {
fprintf(f, "%s%s %d", (k > 0) ? ", " : "", KIND_NAMES[k],
c->host_evals[k]);
}
fputc('\n', f);
}
/* The line a no-decision record ends with, in place of the outcome (2.4). */
static void put_no_decision(FILE* f, const xce_sched_trace_rec_t* rec)
{
char eb[RTR_BUFSZ];
const char* what;
switch (rec->kind) {
case XCE_TRACE_REC_NOT_FOUND:
what = "job not found by the allocator";
break;
case XCE_TRACE_REC_NO_SCHEDOBJ:
what = "job has no scheduling object (its resource requirement "
"could not be parsed)";
break;
case XCE_TRACE_REC_INTERNAL_ERROR:
what = "allocation attempt ended in an internal error";
break;
case XCE_TRACE_REC_PEND_NO_REASON:
default:
what = "job pended without any pend reason";
break;
}
rtr_indent(f, 1);
fprintf(f, "no decision: %s%s\n", what, rtr_err(rec->err, eb, sizeof(eb)));
}
static void render_one(const xce_sched_trace_rec_t* rec,
xce_repl_units_mode_t units, FILE* f)
{
const sched_trace_entry_t* e;
put_header(f, rec);
if (rec->kind == XCE_TRACE_REC_RECONSTRUCTED) {
/* 2.5: no selection, requirement, plan or host sections */
rtr_indent(f, 1);
fputs("dispatch reconstructed from the recorded allocation; "
"selection, requirement, plan and host sections do not "
"apply\n",
f);
rtr_outcome(f, rec, units);
return;
}
if (rec->kind == XCE_TRACE_REC_NOT_FOUND) {
rtr_indent(f, 1);
fputs("selection: ordering keys and limits unavailable (job not "
"found by the allocator)\n",
f);
} else {
put_selection(f, rec);
}
for (e = rec->entries; e != NULL; e = e->next) {
put_entry(f, e, units);
}
put_truncation(f, rec);
if (rec->incomplete) {
rtr_indent(f, 1);
fputs("... trace incomplete (out of memory)\n", f);
}
if (rec->kind == XCE_TRACE_REC_DECISION) {
rtr_outcome(f, rec, units);
} else {
put_no_decision(f, rec);
}
}
void repl_trace_render(const xce_sched_trace_rec_t* rec,
xce_repl_units_mode_t units, FILE* out1, FILE* out2)
{
if ((rec == NULL) || (out1 == NULL)) {
return;
}
render_one(rec, units, out1);
if ((out2 != NULL) && (out2 != out1)) {
render_one(rec, units, out2);
}
}

53
src/scheduler/src/repl_trace_render.h Normal file
View file

@ -0,0 +1,53 @@
#if !defined(_REPL_TRACE_RENDER_H_)
#define _REPL_TRACE_RENDER_H_ 1
/*
* repl_trace_render.h - helpers shared by the two decision trace renderer
* sources (scheduler-sim-decision-trace spec, task 4.1, design.md
* "TraceRenderer"). NOT installed and NOT included outside
* src/scheduler/src/repl_trace_render*.c; the entry point,
* repl_trace_render(), is declared in repl_internal.h.
*
* - repl_trace_render.c: the entry point, header and selection block,
* the entry hierarchy, the truncation marker and totals, and the
* no-decision and reconstructed lines.
* - repl_trace_render_outcome.c: the dispatch and pend outcome, and the
* helpers below.
*
* REPL side: stdio sinks only, records read and never changed, no
* scheduler state (design "Allowed Dependencies").
*/
#include <stddef.h>
#include <stdio.h>
#include "repl_internal.h"
#include "sched/trace.h"
/* Big enough for any one formatted amount, error text or job id. */
#define RTR_BUFSZ 128
/* Two spaces per level (7.6). */
extern void rtr_indent(FILE* f, int level);
/* An amount formatted by the resource it measures, in `units` (7.7); a
* NULL resource name formats the raw number. Returns `buf`. */
extern const char* rtr_value(xce_repl_units_mode_t units, const char* res,
double value, char* buf, size_t bufsz);
/* "" for err == 0, else " (err <n>: <strerror>)". Returns `buf`. */
extern const char* rtr_err(int err, char* buf, size_t bufsz);
/* A limit and its scope instance: "<limit> (<scope>)", or "<limit>" when
* the scope is empty. A NULL limit name prints as "?". */
extern void rtr_limit_ref(FILE* f, const char* limit, const char* scope);
/* A pend-reason code's name (preason.h), or "code <n>". Returns a static
* string or `buf`. */
extern const char* rtr_reason_name(int code, char* buf, size_t bufsz);
/* The outcome section (6.1-6.4, 9.4), at indent level 1. */
extern void rtr_outcome(FILE* f, const xce_sched_trace_rec_t* rec,
xce_repl_units_mode_t units);
#endif /* _REPL_TRACE_RENDER_H_ */

326
src/scheduler/src/repl_trace_render_outcome.c Normal file
View file

@ -0,0 +1,326 @@
/*
* repl_trace_render_outcome.c - the outcome section of a rendered decision
* trace (scheduler-sim-decision-trace spec, task 4.1, design.md
* "TraceRenderer", Outcome), split from repl_trace_render.c to keep each
* file under the 800-line cap (repl_trace_render.h).
*
* A dispatch lists each host kept in detail (at most
* XCE_TRACE_OUTCOME_HOSTS) with its allocated resources and, separately,
* its host-scoped limit charges, then the job's shared resources and
* cluster-wide limit charges, then, when hosts were omitted, their totals
* per resource and per limit and a marker counting them (6.1-6.3, 9.4). A
* pend lists the main reason, the entries that name no host, then each
* host's entries, then the omitted hosts' counts per code and their marker
* (6.4, 9.4). TraceCapture already applied the host cap; this file only
* prints what the record holds.
*
* It also holds the small formatting helpers both renderer files share
* (repl_trace_render.h): indentation, amounts in the units mode passed in
* (repl_fmt.c, 7.7), error text, limit names and pend-reason names.
*/
#include <stdio.h>
#include <string.h>
#include "preason.h"
#include "repl_trace_render.h"
/* ---- shared helpers (repl_trace_render.h) ------------------------------- */
void rtr_indent(FILE* f, int level)
{
int k;
for (k = 0; k < level; k++) {
fputs(" ", f);
}
}
const char* rtr_value(xce_repl_units_mode_t units, const char* res,
double value, char* buf, size_t bufsz)
{
repl_format_numeric_resource_value(units, res, value, buf, bufsz);
return buf;
}
const char* rtr_err(int err, char* buf, size_t bufsz)
{
if (err == 0) {
buf[0] = '\0';
} else {
snprintf(buf, bufsz, " (err %d: %s)", err,
strerror((err < 0) ? -err : err));
}
return buf;
}
void rtr_limit_ref(FILE* f, const char* limit, const char* scope)
{
fputs((limit != NULL) ? limit : "?", f);
if ((scope != NULL) && (scope[0] != '\0')) {
fprintf(f, " (%s)", scope);
}
}
const char* rtr_reason_name(int code, char* buf, size_t bufsz)
{
switch (code) {
case PREASON_BAD_RESREQ:
return "BAD_RESREQ";
case PREASON_NO_RESOURCE:
return "NO_RESOURCE";
case PREASON_MATCH_FAIL:
return "MATCH_FAIL";
default:
snprintf(buf, bufsz, "code %d", code);
return buf;
}
}
/* "name=value ..." for allocated resources; "none" when there are none. */
static void put_res_list(FILE* f, const sched_trace_alloc_row_t* rows, int n,
xce_repl_units_mode_t units)
{
char buf[RTR_BUFSZ];
int k;
if ((rows == NULL) || (n <= 0)) {
fputs("none", f);
return;
}
for (k = 0; k < n; k++) {
const char* res = rows[k].res;
fprintf(f, "%s%s=%s", (k > 0) ? " " : "",
(res != NULL) ? res : "?",
rtr_value(units, res, rows[k].value, buf, sizeof(buf)));
}
}
/* " limit charges: <limit> (<scope>) <resource> <amount>, ..." -- the
* amount formatted by the limited resource, or raw when it is unknown
* (7.7). Nothing when there are no charges (6.3: never a resource row). */
static void put_charges(FILE* f, const sched_trace_alloc_row_t* rows, int n,
xce_repl_units_mode_t units)
{
char buf[RTR_BUFSZ];
int k;
if ((rows == NULL) || (n <= 0)) {
return;
}
fputs(" limit charges: ", f);
for (k = 0; k < n; k++) {
const sched_trace_alloc_row_t* c = &rows[k];
if (k > 0) {
fputs(", ", f);
}
rtr_limit_ref(f, (c->limit != NULL) ? c->limit : c->res, c->scope);
if (c->limited_res != NULL) {
fprintf(f, " %s", c->limited_res);
}
fprintf(f, " %s",
rtr_value(units, c->limited_res, c->value, buf, sizeof(buf)));
}
}
static void put_dispatch(FILE* f, const sched_trace_dispatch_out_t* d,
xce_repl_units_mode_t units)
{
int k;
for (k = 0; (d->hosts != NULL) && (k < d->nhosts); k++) {
const sched_trace_host_out_t* h = &d->hosts[k];
rtr_indent(f, 2);
fprintf(f, "host %s: ", (h->host != NULL) ? h->host : "?");
put_res_list(f, h->res, h->nres, units);
put_charges(f, h->charges, h->ncharges, units);
fputc('\n', f);
}
rtr_indent(f, 2);
fputs("shared: ", f);
put_res_list(f, d->shared, d->nshared, units);
put_charges(f, d->shared_charges, d->nshared_charges, units);
fputc('\n', f);
if (d->omitted_hosts <= 0) {
return;
}
/* 9.4: the omitted hosts' totals, limits totalled per limit */
rtr_indent(f, 2);
fputs("omitted hosts total: ", f);
put_res_list(f, d->agg_res, d->nagg_res, units);
put_charges(f, d->agg_charges, d->nagg_charges, units);
fputc('\n', f);
rtr_indent(f, 2);
fprintf(f, "... %d hosts omitted\n", d->omitted_hosts);
}
static bool hostless(const char* host)
{
return (host == NULL) || (host[0] == '\0');
}
static bool same_host(const char* a, const char* b)
{
return !hostless(a) && !hostless(b) && (strcmp(a, b) == 0);
}
/* The host named by entry `k` of the combined list (host-based entries
* first, then resource-based ones), or NULL when that entry is NULL: the
* outcome copy keeps NULL entries (sched_trace_outcome.c), so every walk
* over these lists goes through here and never dereferences one. */
static const char* entry_host(const xce_job_preason_t* r, int k)
{
if (k < r->nhost_reasons) {
const xce_host_reason_t* h = (r->host_based_reasons != NULL)
? r->host_based_reasons[k]
: NULL;
return (h != NULL) ? h->host : NULL;
} else {
const xce_res_reason_t* s
= (r->res_based_reasons != NULL)
? r->res_based_reasons[k - r->nhost_reasons]
: NULL;
return (s != NULL) ? s->host : NULL;
}
}
/* True when an entry before `idx` of the combined list names `host`. */
static bool host_seen(const xce_job_preason_t* r, int idx, const char* host)
{
int k;
for (k = 0; k < idx; k++) {
if (same_host(entry_host(r, k), host)) {
return true;
}
}
return false;
}
/* One line per host: its host-based reasons, then its resource reasons. */
static void put_pend_host(FILE* f, const xce_job_preason_t* r,
const char* host)
{
char buf[RTR_BUFSZ];
int n = 0;
int k;
rtr_indent(f, 2);
fprintf(f, "host %s: ", host);
for (k = 0; (r->host_based_reasons != NULL) && (k < r->nhost_reasons);
k++) {
const xce_host_reason_t* h = r->host_based_reasons[k];
if ((h != NULL) && same_host(h->host, host)) {
fprintf(f, "%s%s", (n++ > 0) ? ", " : "",
rtr_reason_name(h->reason, buf, sizeof(buf)));
}
}
for (k = 0; (r->res_based_reasons != NULL) && (k < r->nres_reasons);
k++) {
const xce_res_reason_t* s = r->res_based_reasons[k];
if ((s != NULL) && same_host(s->host, host)) {
fprintf(f, "%sresource %s: %s", (n++ > 0) ? ", " : "",
(s->resource != NULL) ? s->resource : "?",
rtr_reason_name(s->reason, buf, sizeof(buf)));
}
}
fputc('\n', f);
}
static void put_pend(FILE* f, const sched_trace_pend_out_t* p)
{
const xce_job_preason_t* r = p->reasons;
char buf[RTR_BUFSZ];
int total;
int k;
if (r == NULL) {
rtr_indent(f, 2);
fputs("pend reasons: none\n", f);
return;
}
rtr_indent(f, 2);
fprintf(f, "main reason: %s\n",
rtr_reason_name(r->main_reason, buf, sizeof(buf)));
/* entries tied to no host (a shared resource carries host "") */
for (k = 0; (r->host_based_reasons != NULL) && (k < r->nhost_reasons);
k++) {
const xce_host_reason_t* h = r->host_based_reasons[k];
if ((h != NULL) && hostless(h->host)) {
rtr_indent(f, 2);
fprintf(f, "reason: %s\n",
rtr_reason_name(h->reason, buf, sizeof(buf)));
}
}
for (k = 0; (r->res_based_reasons != NULL) && (k < r->nres_reasons);
k++) {
const xce_res_reason_t* s = r->res_based_reasons[k];
if ((s != NULL) && hostless(s->host)) {
rtr_indent(f, 2);
fprintf(f, "resource %s: %s\n",
(s->resource != NULL) ? s->resource : "?",
rtr_reason_name(s->reason, buf, sizeof(buf)));
}
}
/* each host once, in the order first named */
total = r->nhost_reasons + r->nres_reasons;
for (k = 0; k < total; k++) {
const char* host = entry_host(r, k); /* NULL entry: hostless */
if (hostless(host) || host_seen(r, k, host)) {
continue;
}
put_pend_host(f, r, host);
}
if (p->omitted_hosts <= 0) {
return;
}
/* 9.4: the omitted hosts' entries counted per code */
rtr_indent(f, 2);
fputs("omitted hosts' pend reasons: ", f);
if ((p->omitted_codes == NULL) || (p->ncodes <= 0)) {
fputs("none", f);
}
for (k = 0; (p->omitted_codes != NULL) && (k < p->ncodes); k++) {
fprintf(f, "%s%s x%d", (k > 0) ? ", " : "",
rtr_reason_name(p->omitted_codes[k].code, buf, sizeof(buf)),
p->omitted_codes[k].count);
}
fputc('\n', f);
rtr_indent(f, 2);
fprintf(f, "... %d hosts omitted\n", p->omitted_hosts);
}
void rtr_outcome(FILE* f, const xce_sched_trace_rec_t* rec,
xce_repl_units_mode_t units)
{
rtr_indent(f, 1);
switch (rec->outcome) {
case SCHED_TRACE_OUTCOME_DISPATCH:
fputs("outcome: DISPATCH\n", f);
put_dispatch(f, &rec->dispatch, units);
break;
case SCHED_TRACE_OUTCOME_PEND:
fputs("outcome: PEND\n", f);
put_pend(f, &rec->pend);
break;
case SCHED_TRACE_OUTCOME_NONE:
default:
fputs("outcome: none recorded\n", f);
break;
}
}

19
src/scheduler/testsuite/meson.build
View file

@ -2716,3 +2716,22 @@ sched_trace_hooks_runner_test = executable(
test('scheduler trace hooks runner test',
sched_trace_hooks_runner_test, suite: 'scheduler',
env: ['ASAN_OPTIONS=detect_leaks=0'])
# scheduler-sim-decision-trace task 4.1: the trace renderer (design.md
# "TraceRenderer"; Requirements 2.4, 2.5, 3.x, 4.x, 5.x, 6.x, 7.6, 7.7,
# 9.2-9.4). Golden text for full and summary levels, every units mode,
# truncation, the outcome cap, the no-decision kinds and reconstructed
# records. Links only the record model, the renderer and the shared REPL
# formatting module; records are built without the capture sources.
sched_trace_render_test = executable(
'sched_trace_render_test',
['test_trace_render.c', '../src/sched_trace.c',
'../src/sched_trace_scan.c', '../src/repl_trace_render.c',
'../src/repl_trace_render_outcome.c', '../src/repl_fmt.c'],
include_directories:[incdir],
dependencies: [scheduler_test_deps],
install: false)
test('scheduler trace render test',
sched_trace_render_test, suite: 'scheduler',
env: ['ASAN_OPTIONS=detect_leaks=0'])

1289
src/scheduler/testsuite/test_trace_render.c Normal file
View file

File diff suppressed because it is too large Load diff