diff --git a/.kiro/specs/scheduler-core/tasks.md b/.kiro/specs/scheduler-core/tasks.md index 41f294c2..041e9905 100644 --- a/.kiro/specs/scheduler-core/tasks.md +++ b/.kiro/specs/scheduler-core/tasks.md @@ -330,11 +330,12 @@ - _Depends: 11.1_ - _Verified: 2026-08-06 — new `test_sim_isolation.c` (548 lines) enforces AC 17 across `sim.h`/`sim.c`/`sim_loop.c` in three layers, extending `test_core_isolation.c`'s two-layer pattern rather than duplicating it: (1) source-text scan for forbidden `#include`s and forbidden-symbol call sites, reusing `test_core_isolation.c`'s exact `FORBIDDEN_SYMBOLS` list; (2) a structural check `test_core_isolation.c` has no equivalent of — extracts only the `typedef struct {...} xce_sched_sim_driver_t;` body text (via a comment-stripping pass first, since `sim.h`'s own boundary doc comment legitimately names `xcecli_ctx_t`/`xce_cluster_config_t` in prose) and scans that narrower region for forbidden field types; (3) the literal build/link-time proof — `sim.c` and `sim_loop.c` compiled together into a new isolated `sched_sim_isolation_lib` static_library (`meson.build`), inspected via `nm(1)`'s undefined-symbol table to prove what the compiled object actually calls, independent of transitive header declarations. Deliberately excludes the wall-clock ban (`FORBIDDEN_WALLCLOCK`) — out of this task's Req 5 AC 17 scope, and `sim.c`/`sim_loop.c` legitimately call `clock_gettime()` to seed/advance the core's virtual clock from the driver side, already documented at those call sites. 5/5 new sub-tests pass; 29/29 full scheduler suite passes; whole-project build clean. Independently reviewed and approved. -- [ ] 11.4 Implement decision verification +- [x] 11.4 Implement decision verification - Compare each core-emitted decision against the corresponding recorded decision at the same logical position when verification is enabled; report any mismatch with the logical time and decision detail - Observable: replaying a fixture file with verification enabled against its own previously-recorded output reports zero mismatches; replaying against a deliberately-altered copy reports the exact mismatch - _Requirements: 5.15_ - _Depends: 11.2_ + - _Verified: 2026-08-06 — new `sim_verify.c` (`sim_decision_eq()` + `sim_log_mismatch()`) and changes to `sim_loop.c`'s `ReplaySession` branch implement AC 15/18: `decode_ReplaySession_ptr()` decodes the session's recorded `decisions` list (a deep-copied `replay_session_t*`, generated capnp typemap code already present from task 7.x, its first real consumer) before `capn_free()`, kept alive across the whole per-session drain loop and freed via `free_ReplaySession_ptr()` on every exit path. `sim_decision_eq()` is a genuine field-by-field structural comparison (`xce_alloc_t`/`xce_host_alloc_t`/`xce_job_alloc_t` for DISPATCH, `xce_job_preason_t` including nested host/resource reason lists for PEND; PREEMPT is defensively false-with-log, matching `recording_session.c`'s own documented "not producible on any live path today" scope), not a jobid-only shortcut. Comparison runs positionally (index `di`) only when `sim->verify` is true (confirmed already fully wired end-to-end from `--sim-verify` through `main.c`→`sched_sim_args_t`→`sim.c`, no gap); both live-longer-than-recorded and recorded-longer-than-live length mismatches are surfaced, not silently dropped. New `test_sim_verify.c` (14 tests: 11 unit-level direct calls on `sim_decision_eq`/`sim_log_mismatch` + 3 integration-level through a real `sched_sim_driver_run()` fixture) covers both Observables: verify-against-self reports zero mismatches, verify-against-a-deliberately-altered copy reports the exact mismatch, verify=false reports nothing even when altered. RED phase independently reproduced by review (neutered `sim_decision_eq` to always-match, 8/14 tests failed including the altered-copy integration test, reverted, confirmed GREEN). 30/30 scheduler suite (was 29), whole-project build clean. Review APPROVED with two non-blocking FYI notes (a stale test-count label in a doc comment; `sim_log_mismatch()`'s signature intentionally drops AC 18 pseudocode's illustrative `sim` parameter and skips a wasteful allocate-then-discard `from_replay_decision()` conversion step, documented in-file, no behavioral gap against AC 15's actual contract). - [ ] 11.5 Implement what-if branching and simulation output recording - Support loading the snapshot from a base replay file while replaying events from a separate alternate-events file diff --git a/src/scheduler/meson.build b/src/scheduler/meson.build index cd946a57..4a6a92e8 100644 --- a/src/scheduler/meson.build +++ b/src/scheduler/meson.build @@ -16,6 +16,7 @@ scheduler_srcs = [ 'src/recording_session.c', 'src/sim.c', 'src/sim_loop.c', + 'src/sim_verify.c', 'src/schedobj.c', 'src/plugin.c', 'src/resource.c', diff --git a/src/scheduler/src/sim_loop.c b/src/scheduler/src/sim_loop.c index 731e8a79..6b63e89c 100644 --- a/src/scheduler/src/sim_loop.c +++ b/src/scheduler/src/sim_loop.c @@ -14,18 +14,21 @@ /* * sim_loop.c - simulation-mode driver event loop (scheduler-core spec, task - * 11.2, Req 5 AC 14/18). Completes Req 5 AC 13's five-step startup - * sequence step 5 (sim.c/task 11.1 implements steps 1-4) and hosts - * sched_sim_driver_run() itself -- the design.md-literal SimDriver entry - * point sim.h declares. + * 11.2, Req 5 AC 14/18; decision verification added by task 11.4, Req 5 AC + * 15/18). Completes Req 5 AC 13's five-step startup sequence step 5 + * (sim.c/task 11.1 implements steps 1-4) and hosts sched_sim_driver_run() + * itself -- the design.md-literal SimDriver entry point sim.h declares. * - * Scope boundary vs. sibling tasks 11.4/11.5 (both depend on this task): - * task 11.4 owns comparing each drained decision against the recorded - * ReplaySession.decisions when --sim-verify is set; task 11.5 owns writing - * decisions to a --sim-output file and loading --sim-alt-events. Neither - * branch is built here -- sim_print_decision() below is this task's own, - * narrower Req 5 AC 19 default ("if --sim-output is not specified, - * decisions are printed to stdout only"). + * Scope boundary vs. sibling task 11.5 (depends on this task): task 11.5 + * owns writing decisions to a --sim-output file and loading + * --sim-alt-events. That branch is not built here -- sim_print_decision() + * below is this task's own, narrower Req 5 AC 19 default ("if --sim-output + * is not specified, decisions are printed to stdout only"). Task 11.4 (this + * revision) owns comparing each drained decision against the recorded + * ReplaySession.decisions when --sim-verify is set -- see the ReplaySession + * branch in sched_sim_driver_run() below, and sim_verify.c (sibling file, + * same task) for the sim_decision_eq()/sim_log_mismatch() comparison + * logic itself. * * Picker-driving thread (sim_picker_thread_fn, below): Req 5 AC 18's * pseudocode calls `core_go()` then blocks draining `core->decisions` until @@ -58,6 +61,20 @@ extern int msg_root_has_shape(struct capn* c, capn_ptr* out, size_t want_words, size_t want_ptrs); +/* + * Shared with sim_verify.c (task 11.4, Req 5 AC 15/18) -- same "extern + * declared where consumed" idiom as msg_root_has_shape() above. Declared + * locally here rather than in sim.h, which stays capnp-type-free (main.c + * and other non-capnp-aware callers include it) -- see sim.h's own file + * header comment for that boundary, and sim_verify.c's own file header + * comment for the full rationale. + */ +extern int sim_decision_eq(const replay_decision_t* expected, + const xce_decision_t* actual); +extern void sim_log_mismatch(uint64_t begin_round, int decision_index, + const replay_decision_t* expected, + const xce_decision_t* actual); + /* * sim_advance_virtual_clock - Req 5 AC 14 (task 11.2): re-anchors * sim->core's virtual clock so xce_core_now(&sim->core) reports exactly @@ -505,13 +522,37 @@ int sched_sim_driver_run(xce_sched_sim_driver_t* sim, ReplaySession_pointer_count)) { ReplaySession_ptr sp; uint64_t begin_round; + replay_session_t* decoded_session = NULL; xce_decision_t* d; int push_rc; + int di; sp.p = root; begin_round = ReplaySession_get_beginRound(sp); + + /* Req 5 AC 15/18 (task 11.4): decode the FULL session -- + * including its `decisions` list -- before the capnp arena is + * freed. decode_ReplaySession_ptr() deep-copies every payload + * out of the arena into plain heap structures first (same + * "safe to capn_free() immediately after decode" contract + * sim_process_replay_event_message()'s own doc comment already + * establishes for decode_ReplayEvent(), verified against the + * generated decode_ReplaySession()/decode_ReplayDecision()/ + * decode_DispatchDecision()/decode_PendDecision() chain in + * controller.capnp.c -- every one calloc()s fresh storage and + * copies field values out, never aliasing capn arena memory), + * so `decoded_session` stays valid across the whole drain loop + * below, independent of `c`'s lifetime. */ + decode_ReplaySession_ptr(&decoded_session, sp); capn_free(&c); + if (decoded_session == NULL) { + ERROR("%s out of memory decoding ReplaySession at " + "begin_round=%llu -- decision verification skipped " + "for this round", + __func__, (unsigned long long)begin_round); + } + /* Req 5 AC 18 pseudocode: push the accumulated batch first, * THEN (mirroring requirements.md's own "beginRound = * xce_core_now(core) on core_go()" prose) jump the clock to @@ -529,13 +570,47 @@ int sched_sim_driver_run(xce_sched_sim_driver_t* sim, ERROR("%s xce_core_go failed -- skipping this round's " "decision drain", __func__); + free_ReplaySession_ptr(&decoded_session); continue; } + /* Req 5 AC 15 (task 11.4): when --sim-verify is set, walk the + * decoded `decisions` array positionally (di) alongside the + * live drained decisions, exactly matching AC 18's pseudocode + * `di < msg.session.decisions.len` bound check -- never compare + * past the end of the recorded array. A live decision with no + * corresponding recorded entry is itself reported as a + * mismatch (expected=NULL) rather than silently skipped; the + * inverse case (recorded decisions left over after the live + * stream ends) is reported the same way just below the loop. */ + di = 0; while ((cbuf_get(&(sim->core.decisions), &d) == 0) && (d != NULL)) { + if (sim->verify && (decoded_session != NULL)) { + const replay_decision_t* expected + = (di < decoded_session->n_decisions) + ? decoded_session->decisions[di] + : NULL; + + if (!sim_decision_eq(expected, d)) { + sim_log_mismatch(begin_round, di, expected, d); + } + di++; + } sim_print_decision(d); } + if (sim->verify && (decoded_session != NULL) + && (di < decoded_session->n_decisions)) { + int j; + + for (j = di; j < decoded_session->n_decisions; j++) { + sim_log_mismatch(begin_round, j, + decoded_session->decisions[j], NULL); + } + } + + free_ReplaySession_ptr(&decoded_session); + xce_core_end_session(&(sim->core)); continue; } diff --git a/src/scheduler/src/sim_verify.c b/src/scheduler/src/sim_verify.c new file mode 100644 index 00000000..f9603a30 --- /dev/null +++ b/src/scheduler/src/sim_verify.c @@ -0,0 +1,431 @@ +#include +#include + +#include "controller.capnp.h" +#include "decision.h" +#include "utils.h" +#include "xce/types.h" + +/* + * sim_verify.c - deterministic replay verification (scheduler-core spec, + * task 11.4, Req 5 AC 15/18: "In simulation mode with --sim-verify, the + * driver shall compare each core-emitted decision against the recorded + * decisions in the replay file. Any mismatch shall be reported as a + * regression with exact logical-time and decision-detail context."). + * + * Split out of sim_loop.c (task 11.2's own file) as a sibling rather than + * appended there directly -- the exact same 800-line-cap precedent task 4.3 + * set splitting core_events.c out of core.c (tasks.md's "Implementation + * Notes": "Task 4.3 split its implementation into a NEW file core_events.c + * ... because implementing xce_core_push_events() directly in core.c would + * have pushed it to 977 lines, over this project's 800-line hard cap"), + * later repeated for core_session.c (task 4.4) and core_clock.c (task 4.5), + * and again for sim_loop.c itself (task 11.2, sibling of sim.c). This + * codebase's own per-function doc-comment convention (every function in + * sim.c/sim_loop.c carries a substantial rationale comment) made the same + * call here: the full structural-equality + mismatch-reporting logic below, + * documented to the same standard, would not comfortably fit inside + * sim_loop.c's own 800-line budget alongside its existing content. + * + * Owns exactly two entry points, both consumed by sim_loop.c's + * sched_sim_driver_run() via a local `extern` declaration -- the same + * "extern declared where consumed" idiom sim.c/sim_loop.c already use to + * share msg_root_has_shape(), and recording.c/recording_session.c use to + * share write_capnp_message() -- rather than adding a capnp-typed + * declaration to sim.h, which stays deliberately capnp-type-free (main.c + * and other non-capnp-aware callers include it; see sim.h's own file + * header comment for that boundary): + * + * sim_decision_eq() -- Req 5 AC 18 pseudocode's `decision_eq()`: true + * iff a decoded recorded ReplayDecision and a live + * core-emitted xce_decision_t represent the same + * scheduling outcome. + * sim_log_mismatch() -- Req 5 AC 18 pseudocode's `log_mismatch()`: + * reports one mismatch (logical time + decision + * detail) via this codebase's ERROR() logging + * convention (utils.h) -- the same macro every + * anomaly path in sim.c/sim_loop.c already uses, + * not a new mechanism. Deliberately takes + * (begin_round, decision_index, expected, actual) + * directly rather than AC 18's literal + * `log_mismatch(sim, &expected, d)` signature -- + * nothing here reads xce_sched_sim_driver_t itself, + * so threading it through would add a needless + * coupling for zero behavioral difference. + * + * Comparison strategy (Req 5 AC 15: "compare each core-emitted decision + * against the corresponding recorded decision"): structural, not pointer, + * equality. A live xce_decision_t's xce_job_alloc_t* or xce_pendreason_t* + * payload is ALWAYS a distinct heap allocation from the recorded + * ReplayDecision's decoded replay_decision_t payload -- one came from the + * picker/allocator thread's own make_dispatch_decision()/ + * make_pend_decision() calls (decision.c), the other from + * decode_ReplaySession_ptr()'s deep-copy-from-capnp-arena walk + * (controller.capnp.c). Pointer or memcmp equality would therefore always + * report a false mismatch; every comparator below instead walks both trees + * field-by-field, matching the exact xce_job_alloc_t/xce_job_preason_t + * shapes recording_session.c's own encode side (the inverse direction) + * already documents. + */ + +/* + * sim_verify_str_eq - NULL-safe string equality. Both trees under + * comparison route every string through STRING_DUP()/makeString()-style + * heap copies on their own independent path (decode_* on one side, + * make_pend_decision()/the picker's own construction on the other), so a + * NULL/non-NULL string field is itself real signal, not noise -- (a==b== + * NULL) counts as equal, exactly one of the two being NULL never does. + */ +static int sim_verify_str_eq(const char* a, const char* b) +{ + if (a == b) { + return 1; + } + if ((a == NULL) || (b == NULL)) { + return 0; + } + return strcmp(a, b) == 0; +} + +/* + * sim_verify_alloc_eq - xce_alloc_t (one resource allocation record, e.g. + * one GPU/memory grant) field-by-field equality. `value` is compared with + * plain `==`: both sides are exact copies (decode/construction, no + * arithmetic in between) of the same double, so bit-exact comparison is + * correct here, not the usual floating-point-tolerance concern. + */ +static int sim_verify_alloc_eq(const xce_alloc_t* a, const xce_alloc_t* b) +{ + if (a == b) { + return 1; + } + if ((a == NULL) || (b == NULL)) { + return 0; + } + + return (a->instid == b->instid) && (a->value == b->value) + && (a->from == b->from) && (a->duration == b->duration) + && sim_verify_str_eq(a->resname, b->resname) + && sim_verify_str_eq(a->reserve, b->reserve); +} + +/* + * sim_verify_host_alloc_eq - xce_host_alloc_t equality: same host name, + * same count and (in-order) content of per-resource xce_alloc_t entries. + * Order-sensitive comparison is correct here: both the recorded and the + * live tree are built by the SAME deterministic scheduling round replaying + * against the SAME core state (Req 5's whole simulation-mode premise), so + * the host/resource allocation arrays are expected to come out in the same + * order, not just the same multiset. + */ +static int sim_verify_host_alloc_eq(const xce_host_alloc_t* a, + const xce_host_alloc_t* b) +{ + int i; + + if (a == b) { + return 1; + } + if ((a == NULL) || (b == NULL)) { + return 0; + } + if (a->nallocs != b->nallocs) { + return 0; + } + if (!sim_verify_str_eq(a->host, b->host)) { + return 0; + } + + for (i = 0; i < a->nallocs; i++) { + if (!sim_verify_alloc_eq(a->allocs[i], b->allocs[i])) { + return 0; + } + } + + return 1; +} + +/* + * sim_verify_job_alloc_eq - xce_job_alloc_t (a DISPATCH decision's full + * payload) equality: jobid, per-host allocation list (xce_host_alloc_t, + * via sim_verify_host_alloc_eq above), and the shared-resource allocation + * list (xce_alloc_t, via sim_verify_alloc_eq). + */ +static int sim_verify_job_alloc_eq(const xce_job_alloc_t* a, + const xce_job_alloc_t* b) +{ + int i; + + if (a == b) { + return 1; + } + if ((a == NULL) || (b == NULL)) { + return 0; + } + if ((a->jobid != b->jobid) || (a->nhosts != b->nhosts) + || (a->nres_shared != b->nres_shared)) { + return 0; + } + + for (i = 0; i < a->nhosts; i++) { + if (!sim_verify_host_alloc_eq(a->allocs[i], b->allocs[i])) { + return 0; + } + } + + for (i = 0; i < a->nres_shared; i++) { + if (!sim_verify_alloc_eq(a->allocs_shared[i], b->allocs_shared[i])) { + return 0; + } + } + + return 1; +} + +/* + * sim_verify_job_preason_eq - xce_job_preason_t (a PEND decision's reason + * payload) equality: main_reason plus (as a deeper, not-strictly-mandatory + * but cheap-to-add check per this task's own "don't leave an obviously- + * wrong shallow check if the deep one is a few extra lines" guidance) the + * full host-based and resource-based reason lists. + */ +static int sim_verify_job_preason_eq(const xce_job_preason_t* a, + const xce_job_preason_t* b) +{ + int i; + + if (a == b) { + return 1; + } + if ((a == NULL) || (b == NULL)) { + return 0; + } + if ((a->main_reason != b->main_reason) + || (a->nhost_reasons != b->nhost_reasons) + || (a->nres_reasons != b->nres_reasons)) { + return 0; + } + + for (i = 0; i < a->nhost_reasons; i++) { + const xce_host_reason_t* ha = a->host_based_reasons[i]; + const xce_host_reason_t* hb = b->host_based_reasons[i]; + + if (ha == hb) { + continue; + } + if ((ha == NULL) || (hb == NULL)) { + return 0; + } + if ((ha->reason != hb->reason) + || !sim_verify_str_eq(ha->host, hb->host)) { + return 0; + } + } + + for (i = 0; i < a->nres_reasons; i++) { + const xce_res_reason_t* ra = a->res_based_reasons[i]; + const xce_res_reason_t* rb = b->res_based_reasons[i]; + + if (ra == rb) { + continue; + } + if ((ra == NULL) || (rb == NULL)) { + return 0; + } + if ((ra->reason != rb->reason) + || !sim_verify_str_eq(ra->resource, rb->resource) + || !sim_verify_str_eq(ra->host, rb->host)) { + return 0; + } + } + + return 1; +} + +/* + * sim_decision_eq - Req 5 AC 15/18's `decision_eq()`. `expected` is one + * decoded ReplayDecision from the recorded replay file (NULL means "no + * recorded entry at this position" -- always a mismatch); `actual` is one + * live decision the core just emitted (NULL is defensively treated the + * same way, though sim_loop.c's drain loop never calls this with a NULL + * actual today). Dispatches on `actual->op`, requiring `expected`'s own + * body_which to name the SAME decision kind before comparing payloads -- + * an op/body_which mismatch (e.g. recorded PEND, live DISPATCH) is itself + * reported as a mismatch, not silently ignored. + */ +int sim_decision_eq(const replay_decision_t* expected, + const xce_decision_t* actual) +{ + if ((expected == NULL) || (actual == NULL)) { + return 0; + } + + switch (actual->op) { + case XCE_DECISION_DISPATCH: + if (expected->body_which != ReplayDecision_body_dispatch) { + return 0; + } + return (expected->body.dispatch != NULL) + && sim_verify_job_alloc_eq(expected->body.dispatch->alloc, + actual->u.alloc); + + case XCE_DECISION_PEND: + if (expected->body_which != ReplayDecision_body_pend) { + return 0; + } + if ((expected->body.pend == NULL) || (actual->u.reason == NULL)) { + return 0; + } + return (expected->body.pend->jobid == actual->u.reason->jobid) + && sim_verify_job_preason_eq(expected->body.pend->reason, + actual->u.reason->reasons); + + case XCE_DECISION_PREEMPT: + /* recording_session.c's own documented reason (see its + * sched_record_session_add_decision(), XCE_DECISION_PREEMPT case): + * no live allocator path produces this decision type today + * (scheduler-preemption spec territory). A decision this driver + * could never emit for real cannot be meaningfully structurally + * verified either -- log loud and report a mismatch, rather than + * silently treating an unreachable-today comparison as a "pass". */ + ERROR("%s PREEMPT decision comparison is not supported " + "(scheduler-preemption spec territory -- no live path " + "produces this decision type today) -- jobid=%llu reported " + "as a mismatch", + __func__, + (unsigned long long)((actual->u.preempt != NULL) + ? actual->u.preempt->jobid + : 0ULL)); + return 0; + + default: + return 0; + } +} + +#define SIM_VERIFY_DESC_BUF_SZ (256) + +/* + * sim_verify_describe_replay_decision - formats one recorded ReplayDecision + * into `buf` for sim_log_mismatch()'s actionable detail. Never fails -- + * worst case (a malformed/NULL payload) produces a clearly-labeled + * placeholder string instead of crashing on a mismatch report itself. + */ +static void sim_verify_describe_replay_decision(const replay_decision_t* d, + char* buf, size_t buf_sz) +{ + if (d == NULL) { + snprintf(buf, buf_sz, ""); + return; + } + + switch (d->body_which) { + case ReplayDecision_body_dispatch: + if ((d->body.dispatch != NULL) && (d->body.dispatch->alloc != NULL)) { + snprintf(buf, buf_sz, "DISPATCH jobid=%llu nhosts=%d", + (unsigned long long)d->body.dispatch->alloc->jobid, + d->body.dispatch->alloc->nhosts); + } else { + snprintf(buf, buf_sz, "DISPATCH "); + } + break; + + case ReplayDecision_body_pend: + if (d->body.pend != NULL) { + snprintf(buf, buf_sz, "PEND jobid=%llu main_reason=%d", + (unsigned long long)d->body.pend->jobid, + (d->body.pend->reason != NULL) + ? d->body.pend->reason->main_reason + : -1); + } else { + snprintf(buf, buf_sz, "PEND "); + } + break; + + case ReplayDecision_body_preempt: + snprintf(buf, buf_sz, "PREEMPT jobid=%llu", + (d->body.preempt != NULL) + ? (unsigned long long)d->body.preempt->jobid + : 0ULL); + break; + + default: + snprintf(buf, buf_sz, "", d->body_which); + break; + } +} + +/* + * sim_verify_describe_decision - same job as + * sim_verify_describe_replay_decision() above, for the live xce_decision_t + * side. + */ +static void sim_verify_describe_decision(const xce_decision_t* d, char* buf, + size_t buf_sz) +{ + if (d == NULL) { + snprintf(buf, buf_sz, ""); + return; + } + + switch (d->op) { + case XCE_DECISION_DISPATCH: + snprintf(buf, buf_sz, "DISPATCH jobid=%llu nhosts=%d", + (d->u.alloc != NULL) ? (unsigned long long)d->u.alloc->jobid + : 0ULL, + (d->u.alloc != NULL) ? d->u.alloc->nhosts : 0); + break; + + case XCE_DECISION_PEND: + snprintf(buf, buf_sz, "PEND jobid=%llu main_reason=%d", + (d->u.reason != NULL) ? (unsigned long long)d->u.reason->jobid + : 0ULL, + ((d->u.reason != NULL) && (d->u.reason->reasons != NULL)) + ? d->u.reason->reasons->main_reason + : -1); + break; + + case XCE_DECISION_PREEMPT: + snprintf(buf, buf_sz, "PREEMPT jobid=%llu", + (d->u.preempt != NULL) + ? (unsigned long long)d->u.preempt->jobid + : 0ULL); + break; + + default: + snprintf(buf, buf_sz, "", d->op); + break; + } +} + +/* + * sim_log_mismatch - Req 5 AC 15/18's `log_mismatch()`: reports one + * decision-verification mismatch with the exact logical time + * (`begin_round` -- the enclosing ReplaySession's own timestamp, same + * value AC 18's pseudocode reads off `msg.session`) plus each side's + * decision detail, via ERROR() (utils.h) -- the same logging convention + * every other anomaly in sim.c/sim_loop.c already uses (e.g. + * `xce_core_push_events failed`, `xce_core_go failed`), not a new + * mechanism. `decision_index` is this decision's position within the + * session's decisions list (`di` in AC 18's pseudocode). Either `expected` + * or `actual` may be NULL (a length mismatch between the recorded and live + * decision counts for this session -- sim_loop.c's own drain loop reports + * that case through this same function rather than a separate one). + */ +void sim_log_mismatch(uint64_t begin_round, int decision_index, + const replay_decision_t* expected, + const xce_decision_t* actual) +{ + char expected_desc[SIM_VERIFY_DESC_BUF_SZ]; + char actual_desc[SIM_VERIFY_DESC_BUF_SZ]; + uint64_t expected_ts = (expected != NULL) ? expected->ts : 0ULL; + + sim_verify_describe_replay_decision(expected, expected_desc, + sizeof(expected_desc)); + sim_verify_describe_decision(actual, actual_desc, sizeof(actual_desc)); + + ERROR("%s decision verification MISMATCH at begin_round=%llu " + "decision_index=%d expected_ts=%llu -- expected: %s -- actual: %s", + __func__, (unsigned long long)begin_round, decision_index, + (unsigned long long)expected_ts, expected_desc, actual_desc); +} diff --git a/src/scheduler/testsuite/meson.build b/src/scheduler/testsuite/meson.build index c36545cd..74d1d5ae 100644 --- a/src/scheduler/testsuite/meson.build +++ b/src/scheduler/testsuite/meson.build @@ -535,6 +535,37 @@ sched_sim_isolation_test = executable( + '"'], install: false) +# Contract test for decision verification (scheduler-core spec, task 11.4, +# Req 5 AC 15/18): each core-emitted decision is compared against the +# corresponding recorded decision at the same logical position when +# --sim-verify is set, and any mismatch is reported with the exact logical +# time and decision detail. Unit-level: sim_decision_eq()/sim_log_mismatch() +# (new ../src/sim_verify.c, task 11.4's sibling file to sim_loop.c) driven +# directly against hand-built DISPATCH/PEND/PREEMPT decision pairs -- +# including DISPATCH, which sim_loop.c's own real picker thread can never +# produce live (Req 5 AC 17-adjacent scope boundary: real resource-matching +# allocation belongs to the not-yet-designed scheduler-allocation spec), so +# this is the only way to prove that comparator for real. Integration-level: +# a full sched_sim_driver_run() run against a real QUEUE_UPDATE+JOBS_PUSH +# fixture feeding the real builtin_fcfs picker to a genuine live PEND +# decision, --sim-verify both on (self-match reports zero mismatches, +# altered-copy reports the exact mismatch) and off (no mismatch reported +# even against an altered fixture) -- mismatch output observed via the same +# STDOUT_FILENO dup()/dup2() capture technique +# test_apply_correctness_verification.c's own +# test_apply_divergent_loud_fence_log_omits_false_peer_hwm_clause +# establishes for ERROR()'s tlog early-print path. Same white-box #include +# pattern as sched_sim_event_loop_test (core.c, core_clock.c, core_events.c, +# core_session.c, decision.c, plugin.c, plugins/builtin_fcfs.c, recording.c, +# sim.c, sim_loop.c, snapshot.c/snapshot_wire.c/snapshot_load.c) PLUS the +# new ../src/sim_verify.c under test. +sched_sim_verify_test = executable( + 'sched_sim_verify_test', + ['test_sim_verify.c'], + include_directories:[incdir], + dependencies: [scheduler_test_deps], + install: false) + # Register as meson tests so `meson test --suite scheduler` runs them all. # detect_leaks=0 matches the suite convention (fixtures are not torn down). test('scheduler test', scheduler_test, suite: 'scheduler', @@ -613,3 +644,6 @@ test('scheduler sim event loop test', test('scheduler sim isolation test', sched_sim_isolation_test, suite: 'scheduler', env: ['ASAN_OPTIONS=detect_leaks=0']) +test('scheduler sim verify test', + sched_sim_verify_test, suite: 'scheduler', + env: ['ASAN_OPTIONS=detect_leaks=0']) diff --git a/src/scheduler/testsuite/test_sim_event_loop.c b/src/scheduler/testsuite/test_sim_event_loop.c index b6f8c17f..a15ae03d 100644 --- a/src/scheduler/testsuite/test_sim_event_loop.c +++ b/src/scheduler/testsuite/test_sim_event_loop.c @@ -55,6 +55,12 @@ * same Req 5 AC 17 boundary sim.c/sim_loop.c themselves respect) PLUS * ../src/recording.c (fixture-building only, same as test_sim_startup.c) * PLUS ../src/sim.c/../src/sim_loop.c (the code under test) PLUS + * ../src/sim_verify.c (task 11.4's sibling file -- sim_loop.c's + * sched_sim_driver_run() now calls sim_decision_eq()/sim_log_mismatch() as + * extern functions defined there; this white-box TU must include it too or + * those calls are undefined references at link time, even though neither + * test in this file exercises --sim-verify itself -- that is + * test_sim_verify.c's own job) PLUS * ../src/snapshot.c/snapshot_wire.c/snapshot_load.c (task 8.1, needed by * both the fixture-building and sim.c's own snapshot load). */ @@ -85,6 +91,7 @@ #include "../src/recording.c" #include "../src/sim.c" #include "../src/sim_loop.c" +#include "../src/sim_verify.c" #include "../src/snapshot.c" #include "../src/snapshot_load.c" #include "../src/snapshot_wire.c" diff --git a/src/scheduler/testsuite/test_sim_verify.c b/src/scheduler/testsuite/test_sim_verify.c new file mode 100644 index 00000000..48e362bc --- /dev/null +++ b/src/scheduler/testsuite/test_sim_verify.c @@ -0,0 +1,908 @@ +/* + * test_sim_verify.c - contract test for decision verification + * (scheduler-core spec, task 11.4, Req 5 AC 15/18). + * + * Observable under test (task 11.4's own text): + * - replaying a fixture file with verification enabled against its own + * previously-recorded output reports zero mismatches. + * - replaying against a deliberately-altered copy reports the exact + * mismatch. + * + * Two granularities, same split test_sim_event_loop.c's own header comment + * already establishes for task 11.2: + * + * Unit-level (white-box): calls sim_decision_eq()/sim_log_mismatch() + * (sim_verify.c, task 11.4's own sibling file to sim_loop.c) directly + * against hand-built replay_decision_t/xce_decision_t pairs -- DISPATCH + * match/mismatch (jobid, host, resource value), PEND match/mismatch + * (jobid, main_reason), a body_which/op type mismatch, and PREEMPT's + * documented always-false path. This is the only way to exercise DISPATCH + * comparison at all: sim_loop.c's real picker thread + * (sim_picker_thread_fn) only ever emits PEND decisions (its own doc + * comment: "conservatively PENDs every candidate ... rather than + * fabricate a DISPATCH decision with no real worker/resource assignment + * behind it") -- real resource-matching allocation belongs to the + * not-yet-designed scheduler-allocation spec. sim_decision_eq() must + * still compare DISPATCH decisions correctly (Req 5 AC 15 does not + * exempt them), so that comparator is proven directly here rather than + * left unreachable. + * + * Integration-level: drives the full sched_sim_driver_run() (this task's + * other Observable) against a fixture built from a REAL + * QUEUE_UPDATE+JOBS_PUSH event pair (one queue, one JOB_STAT_PEND job) + * feeding the REAL builtin_fcfs picker -- since core.c's own + * sched_job_is_ready()/sched_queue_is_ready() are unconditional `return + * 1` stubs (task 2.4's own documented, still-live scope note), any queue/ + * job pair is immediately "ready", so this always produces exactly one + * live PEND decision via make_pendreason() (preason.c), whose + * xce_job_preason_t is always freshly calloc()'d (main_reason=0, no host/ + * res reasons) -- a fully deterministic, reproducible target to compare + * against. `--sim-verify`'s ERROR()-logged mismatch report is observed by + * redirecting STDOUT_FILENO around the sched_sim_driver_run() call, the + * same dup()/dup2() capture technique + * test_apply_correctness_verification.c's own + * test_apply_divergent_loud_fence_log_omits_false_peer_hwm_clause + * establishes (that file's own doc comment explains WHY this works: with + * tlog never tlog_init()'d in this test process -- true here too, nothing + * in this TU calls it -- every ERROR()/WARN() falls through tlog's + * early-print path, a raw write() to STDOUT_FILENO). + * + * Fixture strategy: same production-code-reuse technique test_sim_startup.c/ + * test_sim_event_loop.c already establish for the header+snapshot prefix + * (sched_record_start() against a small real xce_scheduler_t fixture) PLUS + * hand-constructed ReplayEvent/ReplaySession messages appended afterward via + * the real encode_ReplayEvent_ptr()/encode_ReplaySession_ptr()/ + * write_capnp_message() codec, same technique test_sim_event_loop.c's own + * append_jobstatechange_event()/append_replay_session() and + * test_e2e_capnp.c's test_dispatch_decision_round_trip()/ + * test_pend_decision_round_trip() already prove works. + * + * White-box #include of the same TU set test_sim_event_loop.c uses (see + * that file's own header comment) PLUS ../src/sim_verify.c (the code under + * test). + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "controller.capnp.h" +#include "kvmgr.h" +#include "preason.h" +#include "scheduler.h" +#include "xstring.h" + +/* Pull in the real functions under test into this TU (white-box #include, + * same convention as test_sim_event_loop.c/test_sim_startup.c). */ +#include "../src/core.c" +#include "../src/core_clock.c" +#include "../src/core_events.c" +#include "../src/core_session.c" +#include "../src/decision.c" +#include "../src/plugin.c" +#include "../src/plugins/builtin_fcfs.c" +#include "../src/recording.c" +#include "../src/sim.c" +#include "../src/sim_loop.c" +#include "../src/sim_verify.c" +#include "../src/snapshot.c" +#include "../src/snapshot_load.c" +#include "../src/snapshot_wire.c" + +void kill_sched_job(void* arg) { (void)arg; } +void kill_sched_queue(void* arg) { (void)arg; } +void kill_sched_worker(void* arg) { (void)arg; } +void kill_sched_user(void* arg) { (void)arg; } +void kill_sched_project(void* arg) { (void)arg; } +void kill_sched_resval(xce_val_t* val) { (void)val; } + +/* core_reset_workers()'s (core_session.c) linker dependency -- same stub + * test_sim_event_loop.c/test_core_full_cycle.c already establish; never + * invoked (no worker is ever added here). */ +void sched_worker_release_host_ctx(xce_sched_worker_t* w) { (void)w; } + +/* plugin_prepare_allocation()'s (plugin.c, the "limit" plugin category) + * linker dependency -- never invoked (this file never drives real + * allocation). */ +int sched_fill_limit_requests(xce_scheduler_t* sched, xvec_idval_t* vec, + xlist_t* limits) +{ + (void)sched; + (void)vec; + (void)limits; + return 0; +} + +/* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same + * copy test_sim_event_loop.c/test_sim_startup.c/test_core_full_cycle.c each + * carry (schedobj.c is not linked here: it would need plugin_register(), + * which requires a genuine xce_scheduler_t* neither this file's fixtures + * nor sim_loop.c's own core-scoped registration path have, Req 5 AC 17). */ +int sched_resolve_job_picker(xce_sched_core_t* core, xce_sched_plugin_t** out) +{ + xce_sched_plugin_t* plugin = NULL; + int rc = 0; + + if ((core == NULL) || (out == NULL)) { + return -EINVAL; + } + + *out = NULL; + + if (core->job_pick_plugin == NULL) { + return -EINVAL; + } + + lock(core->plugins, RTTI_LOCK_READ); + rc = key_find(core->plugins, core->job_pick_plugin, (void**)&plugin); + unlock(core->plugins); + + if (rc <= 0) { + return (rc < 0) ? rc : -ENOENT; + } + + if ((plugin == NULL) || (plugin->picker_iface == NULL) + || (plugin->picker_iface->next_job == NULL) + || (plugin->picker_iface->begin_session == NULL) + || (plugin->picker_iface->end_session == NULL) + || (plugin->picker_iface->on_dispatch == NULL) + || (plugin->picker_iface->on_pend == NULL)) { + return -EINVAL; + } + + *out = plugin; + + return 0; +} + +void setUp(void) { return; } +void tearDown(void) { return; } + +/* ---------------------------------------------------------------------- */ +/* Unit-level fixture helpers: hand-built xce_job_alloc_t trees. */ +/* ---------------------------------------------------------------------- */ + +static xce_alloc_t* make_test_alloc(const char* resname, const char* reserve, + uint64_t instid, double value, + uint64_t from, int64_t duration) +{ + xce_alloc_t* a = (xce_alloc_t*)calloc(1, sizeof(*a)); + + TEST_ASSERT_NOT_NULL(a); + a->resname = strdup(resname); + a->reserve = strdup(reserve); + a->instid = instid; + a->value = value; + a->from = from; + a->duration = duration; + + return a; +} + +static xce_host_alloc_t* make_test_host_alloc(const char* host, + xce_alloc_t** allocs, int nallocs) +{ + xce_host_alloc_t* h = (xce_host_alloc_t*)calloc(1, sizeof(*h)); + + TEST_ASSERT_NOT_NULL(h); + h->host = strdup(host); + h->nallocs = nallocs; + h->allocs = allocs; + + return h; +} + +static xce_job_alloc_t* +make_test_job_alloc(uint64_t jobid, xce_host_alloc_t** hosts, int nhosts) +{ + xce_job_alloc_t* j = (xce_job_alloc_t*)calloc(1, sizeof(*j)); + + TEST_ASSERT_NOT_NULL(j); + j->jobid = jobid; + j->nhosts = nhosts; + j->allocs = hosts; + j->nres_shared = 0; + j->allocs_shared = NULL; + + return j; +} + +static replay_decision_t make_replay_dispatch_decision(xce_job_alloc_t* alloc) +{ + replay_decision_t d; + sched_dispatch_decision_t* dd + = (sched_dispatch_decision_t*)calloc(1, sizeof(*dd)); + + TEST_ASSERT_NOT_NULL(dd); + dd->alloc = alloc; + + memset(&d, 0, sizeof(d)); + d.body_which = ReplayDecision_body_dispatch; + d.body.dispatch = dd; + + return d; +} + +static replay_decision_t make_replay_pend_decision(uint64_t jobid, + int main_reason) +{ + replay_decision_t d; + sched_pend_decision_t* pd = (sched_pend_decision_t*)calloc(1, sizeof(*pd)); + xce_job_preason_t* reason = (xce_job_preason_t*)calloc(1, sizeof(*reason)); + + TEST_ASSERT_NOT_NULL(pd); + TEST_ASSERT_NOT_NULL(reason); + reason->main_reason = main_reason; + pd->jobid = jobid; + pd->reason = reason; + + memset(&d, 0, sizeof(d)); + d.body_which = ReplayDecision_body_pend; + d.body.pend = pd; + + return d; +} + +/* ---------------------------------------------------------------------- */ +/* Unit-level tests: sim_decision_eq() directly. */ +/* ---------------------------------------------------------------------- */ + +static void test_dispatch_decisions_with_identical_content_match(void) +{ + xce_alloc_t* res_a[1] + = { make_test_alloc("gpu", "resv1", 1, 2.0, 100, 1800) }; + xce_host_alloc_t* host_a[1] = { make_test_host_alloc("hostA", res_a, 1) }; + xce_job_alloc_t* expected_alloc = make_test_job_alloc(42, host_a, 1); + + xce_alloc_t* res_b[1] + = { make_test_alloc("gpu", "resv1", 1, 2.0, 100, 1800) }; + xce_host_alloc_t* host_b[1] = { make_test_host_alloc("hostA", res_b, 1) }; + xce_job_alloc_t* actual_alloc = make_test_job_alloc(42, host_b, 1); + + replay_decision_t expected = make_replay_dispatch_decision(expected_alloc); + xce_decision_t* actual = make_dispatch_decision(actual_alloc); + + /* Two structurally-identical, but never pointer-equal, allocation + * trees: proves the comparison is real structural equality, not a + * pointer/memcmp shortcut (this file's own header comment). */ + TEST_ASSERT_NOT_EQUAL(expected_alloc, actual_alloc); + TEST_ASSERT_TRUE(sim_decision_eq(&expected, actual)); + + printf("test_dispatch_decisions_with_identical_content_match: PASS\n"); +} + +static void test_dispatch_decisions_differing_jobid_mismatch(void) +{ + xce_alloc_t* res[1] = { make_test_alloc("gpu", "r", 1, 1.0, 0, 0) }; + xce_host_alloc_t* host[1] = { make_test_host_alloc("hostA", res, 1) }; + xce_job_alloc_t* expected_alloc = make_test_job_alloc(42, host, 1); + + xce_alloc_t* res2[1] = { make_test_alloc("gpu", "r", 1, 1.0, 0, 0) }; + xce_host_alloc_t* host2[1] = { make_test_host_alloc("hostA", res2, 1) }; + xce_job_alloc_t* actual_alloc = make_test_job_alloc(43, host2, 1); + + replay_decision_t expected = make_replay_dispatch_decision(expected_alloc); + xce_decision_t* actual = make_dispatch_decision(actual_alloc); + + TEST_ASSERT_FALSE(sim_decision_eq(&expected, actual)); + + printf("test_dispatch_decisions_differing_jobid_mismatch: PASS\n"); +} + +static void test_dispatch_decisions_differing_host_mismatch(void) +{ + xce_alloc_t* res[1] = { make_test_alloc("gpu", "r", 1, 1.0, 0, 0) }; + xce_host_alloc_t* host[1] = { make_test_host_alloc("hostA", res, 1) }; + xce_job_alloc_t* expected_alloc = make_test_job_alloc(42, host, 1); + + xce_alloc_t* res2[1] = { make_test_alloc("gpu", "r", 1, 1.0, 0, 0) }; + xce_host_alloc_t* host2[1] = { make_test_host_alloc("hostB", res2, 1) }; + xce_job_alloc_t* actual_alloc = make_test_job_alloc(42, host2, 1); + + replay_decision_t expected = make_replay_dispatch_decision(expected_alloc); + xce_decision_t* actual = make_dispatch_decision(actual_alloc); + + TEST_ASSERT_FALSE(sim_decision_eq(&expected, actual)); + + printf("test_dispatch_decisions_differing_host_mismatch: PASS\n"); +} + +static void test_dispatch_decisions_differing_resource_value_mismatch(void) +{ + xce_alloc_t* res[1] = { make_test_alloc("gpu", "r", 1, 1.0, 0, 0) }; + xce_host_alloc_t* host[1] = { make_test_host_alloc("hostA", res, 1) }; + xce_job_alloc_t* expected_alloc = make_test_job_alloc(42, host, 1); + + xce_alloc_t* res2[1] = { make_test_alloc("gpu", "r", 1, 2.0, 0, 0) }; + xce_host_alloc_t* host2[1] = { make_test_host_alloc("hostA", res2, 1) }; + xce_job_alloc_t* actual_alloc = make_test_job_alloc(42, host2, 1); + + replay_decision_t expected = make_replay_dispatch_decision(expected_alloc); + xce_decision_t* actual = make_dispatch_decision(actual_alloc); + + TEST_ASSERT_FALSE(sim_decision_eq(&expected, actual)); + + printf("test_dispatch_decisions_differing_resource_value_mismatch: " + "PASS\n"); +} + +static void test_pend_decisions_with_identical_content_match(void) +{ + replay_decision_t expected = make_replay_pend_decision(9001ULL, 3); + xce_pendreason_t* reason = make_pendreason(9001ULL); + xce_decision_t* actual; + + reason->reasons->main_reason = 3; + actual = make_pend_decision(reason); + + TEST_ASSERT_TRUE(sim_decision_eq(&expected, actual)); + + printf("test_pend_decisions_with_identical_content_match: PASS\n"); +} + +static void test_pend_decisions_differing_jobid_mismatch(void) +{ + replay_decision_t expected = make_replay_pend_decision(9001ULL, 0); + xce_decision_t* actual = make_pend_decision(make_pendreason(9002ULL)); + + TEST_ASSERT_FALSE(sim_decision_eq(&expected, actual)); + + printf("test_pend_decisions_differing_jobid_mismatch: PASS\n"); +} + +static void test_pend_decisions_differing_main_reason_mismatch(void) +{ + replay_decision_t expected = make_replay_pend_decision(9001ULL, 3); + xce_decision_t* actual = make_pend_decision(make_pendreason(9001ULL)); + + /* actual's main_reason stays 0 (make_pendreason()'s own fresh-calloc + * default) -- differs from expected's 3. */ + TEST_ASSERT_FALSE(sim_decision_eq(&expected, actual)); + + printf("test_pend_decisions_differing_main_reason_mismatch: PASS\n"); +} + +static void test_decision_kind_mismatch_reported(void) +{ + replay_decision_t expected = make_replay_pend_decision(1ULL, 0); + xce_job_alloc_t* alloc = make_test_job_alloc(1, NULL, 0); + xce_decision_t* actual = make_dispatch_decision(alloc); + + /* Recorded PEND vs. live DISPATCH for the SAME jobid: still a + * mismatch -- op/body_which disagreement is reported, not ignored. */ + TEST_ASSERT_FALSE(sim_decision_eq(&expected, actual)); + + printf("test_decision_kind_mismatch_reported: PASS\n"); +} + +static void test_preempt_decision_always_reports_mismatch(void) +{ + xce_preempt_t preempt; + xce_decision_t actual; + replay_decision_t expected; + + memset(&preempt, 0, sizeof(preempt)); + preempt.jobid = 55ULL; + memset(&actual, 0, sizeof(actual)); + actual.op = XCE_DECISION_PREEMPT; + actual.u.preempt = &preempt; + + /* Even a recorded entry that ALSO claims PREEMPT for the same jobid + * still reports a mismatch -- recording_session.c's own documented + * "no live path produces this decision type today" makes any PREEMPT + * comparison unreachable-in-practice and therefore untrustworthy. */ + memset(&expected, 0, sizeof(expected)); + expected.body_which = ReplayDecision_body_preempt; + + TEST_ASSERT_FALSE(sim_decision_eq(&expected, &actual)); + + printf("test_preempt_decision_always_reports_mismatch: PASS\n"); +} + +static void test_decision_eq_defends_against_null_inputs(void) +{ + xce_decision_t* actual = make_pend_decision(make_pendreason(1ULL)); + replay_decision_t expected = make_replay_pend_decision(1ULL, 0); + + TEST_ASSERT_FALSE(sim_decision_eq(NULL, actual)); + TEST_ASSERT_FALSE(sim_decision_eq(&expected, NULL)); + TEST_ASSERT_FALSE(sim_decision_eq(NULL, NULL)); + + printf("test_decision_eq_defends_against_null_inputs: PASS\n"); +} + +/* ---------------------------------------------------------------------- */ +/* Unit-level test: sim_log_mismatch()'s captured output content. */ +/* ---------------------------------------------------------------------- */ + +static char* capture_stdout_around(void (*fn)(void)) +{ + char tmp[] = "/tmp/xce_simverify_logcap_XXXXXX"; + int fd = mkstemp(tmp); + int saved_stdout; + off_t len; + char* buf; + + TEST_ASSERT_TRUE_MESSAGE(fd >= 0, "mkstemp for stdout capture must " + "succeed"); + + fflush(stdout); + saved_stdout = dup(STDOUT_FILENO); + TEST_ASSERT_TRUE_MESSAGE(saved_stdout >= 0, "dup(STDOUT_FILENO) must " + "succeed"); + TEST_ASSERT_TRUE_MESSAGE(dup2(fd, STDOUT_FILENO) >= 0, + "dup2 redirect of STDOUT_FILENO must succeed"); + + fn(); + + fflush(stdout); + dup2(saved_stdout, STDOUT_FILENO); + close(saved_stdout); + + len = lseek(fd, 0, SEEK_CUR); + TEST_ASSERT_TRUE(len >= 0); + + buf = (char*)calloc(1, (size_t)len + 1); + TEST_ASSERT_NOT_NULL(buf); + TEST_ASSERT_TRUE(lseek(fd, 0, SEEK_SET) == 0); + if (len > 0) { + ssize_t nread = read(fd, buf, (size_t)len); + TEST_ASSERT_TRUE(nread == len); + } + close(fd); + unlink(tmp); + + return buf; +} + +static void log_one_mismatch_for_capture(void) +{ + replay_decision_t expected = make_replay_pend_decision(9001ULL, 3); + xce_decision_t* actual = make_pend_decision(make_pendreason(9001ULL)); + + sim_log_mismatch(12345ULL, 2, &expected, actual); +} + +static void test_log_mismatch_reports_logical_time_and_decision_detail(void) +{ + char* buf = capture_stdout_around(log_one_mismatch_for_capture); + + TEST_ASSERT_NOT_NULL_MESSAGE(strstr(buf, "MISMATCH"), + "log_mismatch must report a MISMATCH"); + TEST_ASSERT_NOT_NULL_MESSAGE( + strstr(buf, "begin_round=12345"), + "log_mismatch must report the session's exact logical time"); + TEST_ASSERT_NOT_NULL_MESSAGE( + strstr(buf, "decision_index=2"), + "log_mismatch must report the decision's position within the " + "session"); + TEST_ASSERT_NOT_NULL_MESSAGE( + strstr(buf, "jobid=9001"), + "log_mismatch must report actionable decision detail (jobid)"); + + free(buf); + printf("test_log_mismatch_reports_logical_time_and_decision_detail: " + "PASS\n"); +} + +/* ---------------------------------------------------------------------- */ +/* Integration-level fixture helpers: a real .replay stream driving the */ +/* REAL builtin_fcfs picker to a genuine live PEND decision. */ +/* ---------------------------------------------------------------------- */ + +static void make_tmp_dir(char* out, size_t out_len, const char* suffix) +{ + char tmpl[PATH_MAX]; + + snprintf(tmpl, sizeof(tmpl), "/tmp/sched_simverify%s_XXXXXX", suffix); + TEST_ASSERT_NOT_NULL(mkdtemp(tmpl)); + TEST_ASSERT_TRUE(strlen(tmpl) < out_len); + strcpy(out, tmpl); +} + +/* Builds a small, real xce_scheduler_t fixture purely to drive + * sched_record_start() and produce a REAL, valid `.replay` header+snapshot + * prefix (same technique test_sim_startup.c/test_sim_event_loop.c's own + * build_fixture_scheduler() establish) -- but, unlike + * test_sim_event_loop.c's copy, deliberately records schedPolicy= + * "builtin_fcfs" (a string literal, never freed -- this throwaway fixture + * scheduler is never torn down, matching this whole suite's documented + * accepted-leak convention) so sim_open_and_validate_header() carries + * "builtin_fcfs" into sim->core.job_pick_plugin and + * sim_register_default_picker() (sim_loop.c) registers the REAL + * builtin_fcfs plugin -- this file's own Observable needs the real picker, + * not a stub, to prove sim_decision_eq() against a genuinely-produced live + * decision. No picker needs to be key_add'ed into THIS throwaway + * scheduler's own core.plugins: xce_sched_core_save_snapshot() does not + * require a resolvable picker (Req 5 AC 21: "Picker state is NOT stored in + * snapshot"). + */ +static void build_fixture_scheduler(xce_scheduler_t* sched, char* record_dir) +{ + int rc; + + memset(sched, 0, sizeof(*sched)); + mylock_init(&(sched->record_lock)); + + rc = xce_sched_core_init(&(sched->core)); + TEST_ASSERT_EQUAL_INT(0, rc); + + sched->core.job_pick_plugin = (char*)"builtin_fcfs"; + sched->record_dir = record_dir; +} + +/* Same "leave the file open, skip sched_record_stop()'s own endSession + * marker" technique test_sim_event_loop.c's own write_header_and_snapshot() + * establishes -- see that function's doc comment for the full rationale. */ +static FILE* write_header_and_snapshot(char* out_path, size_t out_path_len, + const char* suffix) +{ + char record_dir[PATH_MAX]; + xce_scheduler_t sched; + int rc; + + make_tmp_dir(record_dir, sizeof(record_dir), suffix); + build_fixture_scheduler(&sched, record_dir); + + rc = sched_record_start(&sched, "fixture.replay"); + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_TRUE(strlen(sched.record_file_path) < out_path_len); + strcpy(out_path, sched.record_file_path); + + return sched.record_file; +} + +/* Appends one ReplayEvent{queueUpdate} message -- creates `queue_name` in + * the core (core_events.c's core_upsert_queue(), fires + * on_queue_update(picker, q) against the now-registered real builtin_fcfs). + * status/priority are set but never gate readiness: core.c's own + * sched_queue_is_ready() is an unconditional `return 1` stub (task 2.4's + * still-live documented scope note) -- any value here works. */ +static void append_queue_update_event(FILE* f, uint64_t ts, + const char* queue_name) +{ + xce_queue_t queue; + replay_event_t src; + struct capn c; + struct capn_segment* cs; + ReplayEvent_ptr p; + + memset(&queue, 0, sizeof(queue)); + queue.queue = (char*)queue_name; + queue.status = QUEUE_STAT_OPEN; + queue.priority = 1; + + memset(&src, 0, sizeof(src)); + src.ts = ts; + src.body_which = ReplayEvent_body_queueUpdate; + src.body.queue_update = &queue; + + capn_init_malloc(&c); + cs = capn_root(&c).seg; + encode_ReplayEvent_ptr(cs, &p, &src); + capn_setp(capn_root(&c), 0, p.p); + + TEST_ASSERT_EQUAL_INT(0, write_capnp_message(f, &c)); + capn_free(&c); +} + +/* Appends one ReplayEvent{jobsPush} message with exactly one + * JOB_STAT_PEND job assigned to `queue_name` -- MUST be appended AFTER + * append_queue_update_event() for the same queue name within the same + * session's accumulated batch: core_build_job() (core_events.c) looks the + * queue up via sched_get_queue() with no get-or-create fallback (unlike + * user/project), so the queue must already exist. Drives + * core_events.c's real JOBS_PUSH handling: sched_add_job() + + * on_job_push(picker, job) against the real builtin_fcfs -- which (since + * sched_job_is_ready()/sched_queue_is_ready() are unconditional `return 1` + * stubs) makes this job an immediate next_job() candidate, so the picker + * thread PENDs it for real via make_pend_decision(make_pendreason(jobid)) + * (sim_loop.c's own sim_picker_thread_fn). + */ +static void append_jobs_push_event(FILE* f, uint64_t ts, uint64_t jobid, + const char* queue_name) +{ + job_submit_data_t submit; + xce_job_t job; + xce_job_t* jobs[1]; + xce_jobs_data_t jobs_data; + replay_event_t src; + struct capn c; + struct capn_segment* cs; + ReplayEvent_ptr p; + + memset(&submit, 0, sizeof(submit)); + submit.queue = (char*)queue_name; + submit.user = (char*)"alice"; + submit.mintasks = 1; + submit.maxtasks = 1; + + memset(&job, 0, sizeof(job)); + job.jobid = jobid; + job.state = JOB_STAT_PEND; + job.submit = &submit; + + jobs[0] = &job; + + memset(&jobs_data, 0, sizeof(jobs_data)); + jobs_data.n_total = 1; + jobs_data.count = 1; + jobs_data.jobs = jobs; + + memset(&src, 0, sizeof(src)); + src.ts = ts; + src.body_which = ReplayEvent_body_jobsPush; + src.body.jobs_push = &jobs_data; + + capn_init_malloc(&c); + cs = capn_root(&c).seg; + encode_ReplayEvent_ptr(cs, &p, &src); + capn_setp(capn_root(&c), 0, p.p); + + TEST_ASSERT_EQUAL_INT(0, write_capnp_message(f, &c)); + capn_free(&c); +} + +/* Appends the .replay stream's whole-stream termination marker + * (ReplayEvent{endSession}, Req 5 AC 20) -- same technique + * test_sim_event_loop.c's own append_end_of_stream_marker() establishes. */ +static void append_end_of_stream_marker(FILE* f) +{ + replay_event_t src; + struct capn c; + struct capn_segment* cs; + ReplayEvent_ptr p; + + memset(&src, 0, sizeof(src)); + src.body_which = ReplayEvent_body_endSession; + + capn_init_malloc(&c); + cs = capn_root(&c).seg; + encode_ReplayEvent_ptr(cs, &p, &src); + capn_setp(capn_root(&c), 0, p.p); + + TEST_ASSERT_EQUAL_INT(0, write_capnp_message(f, &c)); + capn_free(&c); +} + +/* Appends one ReplaySession carrying exactly one recorded PendDecision + * (jobid, main_reason) -- the "recorded output" side of this file's own + * Observable, built via the real encode_ReplaySession_ptr() codec + * (recording_session.c's own real encode call site, not a hand-rolled + * byte layout). + */ +static void append_replay_session_with_pend(FILE* f, uint64_t begin_round, + uint64_t jobid, int main_reason) +{ + xce_job_preason_t reason; + sched_pend_decision_t pend; + replay_decision_t decision; + replay_decision_t* decisions[1]; + replay_session_t session; + struct capn c; + struct capn_segment* cs; + ReplaySession_ptr sp; + + memset(&reason, 0, sizeof(reason)); + reason.main_reason = main_reason; + + memset(&pend, 0, sizeof(pend)); + pend.jobid = jobid; + pend.reason = &reason; + + memset(&decision, 0, sizeof(decision)); + decision.ts = begin_round; + decision.body_which = ReplayDecision_body_pend; + decision.body.pend = &pend; + + decisions[0] = &decision; + + memset(&session, 0, sizeof(session)); + session.begin_round = begin_round; + session.decisions = decisions; + session.n_decisions = 1; + session.summary = NULL; + + capn_init_malloc(&c); + cs = capn_root(&c).seg; + encode_ReplaySession_ptr(cs, &sp, &session); + capn_setp(capn_root(&c), 0, sp.p); + + TEST_ASSERT_EQUAL_INT(0, write_capnp_message(f, &c)); + capn_free(&c); +} + +/* Runs sched_sim_driver_run() against `path` with --sim-verify set to + * `verify`, capturing everything written to stdout during the call (the + * same dup()/dup2() technique capture_stdout_around() above uses, inlined + * here since sched_sim_driver_run() needs live arguments, not a bare + * void(void) callback). */ +static char* run_sim_and_capture_stdout(const char* path, bool verify, + int* out_rc) +{ + sched_sim_args_t args; + xce_sched_sim_driver_t sim; + char tmp[] = "/tmp/xce_simverify_run_XXXXXX"; + int fd = mkstemp(tmp); + int saved_stdout; + off_t len; + char* buf; + + TEST_ASSERT_TRUE_MESSAGE(fd >= 0, "mkstemp for stdout capture must " + "succeed"); + + fflush(stdout); + saved_stdout = dup(STDOUT_FILENO); + TEST_ASSERT_TRUE_MESSAGE(saved_stdout >= 0, "dup(STDOUT_FILENO) must " + "succeed"); + TEST_ASSERT_TRUE_MESSAGE(dup2(fd, STDOUT_FILENO) >= 0, + "dup2 redirect of STDOUT_FILENO must succeed"); + + memset(&args, 0, sizeof(args)); + args.replay_path = path; + args.sim_start_ts = SCHED_SIM_START_TS_LATEST; + args.sim_speed = 0; /* frozen -- deterministic, no wall-clock wait */ + args.verify = verify; + + *out_rc = sched_sim_driver_run(&sim, &args); + + fflush(stdout); + dup2(saved_stdout, STDOUT_FILENO); + close(saved_stdout); + + len = lseek(fd, 0, SEEK_CUR); + TEST_ASSERT_TRUE(len >= 0); + + buf = (char*)calloc(1, (size_t)len + 1); + TEST_ASSERT_NOT_NULL(buf); + TEST_ASSERT_TRUE(lseek(fd, 0, SEEK_SET) == 0); + if (len > 0) { + ssize_t nread = read(fd, buf, (size_t)len); + TEST_ASSERT_TRUE(nread == len); + } + close(fd); + unlink(tmp); + + return buf; +} + +/* ---------------------------------------------------------------------- */ +/* Integration-level tests: full sched_sim_driver_run(), --sim-verify. */ +/* ---------------------------------------------------------------------- */ + +/* + * Observable (a): replaying a fixture against its own previously-recorded + * output (here: the exact PEND decision the real picker deterministically + * produces -- main_reason=0, make_pendreason()'s own fresh-calloc default) + * with verification enabled reports ZERO mismatches. + */ +static void test_verify_against_self_reports_zero_mismatches(void) +{ + char path[PATH_MAX]; + FILE* f; + int rc; + char* captured; + + f = write_header_and_snapshot(path, sizeof(path), "_selfmatch"); + TEST_ASSERT_NOT_NULL(f); + + append_queue_update_event(f, 1000ULL, "normal"); + append_jobs_push_event(f, 1500ULL, 777ULL, "normal"); + append_replay_session_with_pend(f, 2000ULL, 777ULL, 0); + append_end_of_stream_marker(f); + fclose(f); + + captured = run_sim_and_capture_stdout(path, true, &rc); + + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_NULL_MESSAGE( + strstr(captured, "MISMATCH"), + "verifying a fixture against its own previously-recorded decision " + "must report zero mismatches"); + + free(captured); + printf("test_verify_against_self_reports_zero_mismatches: PASS\n"); +} + +/* + * Observable (b): replaying against a deliberately-altered copy (recorded + * main_reason flipped to 99, which the real picker thread can never + * produce -- make_pendreason() always calloc()s a fresh, all-zero + * xce_job_preason_t) reports the EXACT mismatch: MISMATCH is logged, + * naming the specific job. + */ +static void test_verify_against_altered_copy_reports_exact_mismatch(void) +{ + char path[PATH_MAX]; + FILE* f; + int rc; + char* captured; + + f = write_header_and_snapshot(path, sizeof(path), "_altered"); + TEST_ASSERT_NOT_NULL(f); + + append_queue_update_event(f, 1000ULL, "normal"); + append_jobs_push_event(f, 1500ULL, 777ULL, "normal"); + append_replay_session_with_pend(f, 2000ULL, 777ULL, 99); + append_end_of_stream_marker(f); + fclose(f); + + captured = run_sim_and_capture_stdout(path, true, &rc); + + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_NOT_NULL_MESSAGE( + strstr(captured, "MISMATCH"), + "verifying against a deliberately-altered recorded decision must " + "report a mismatch"); + TEST_ASSERT_NOT_NULL_MESSAGE( + strstr(captured, "jobid=777"), + "the mismatch report must name the specific job"); + + free(captured); + printf("test_verify_against_altered_copy_reports_exact_mismatch: " + "PASS\n"); +} + +/* + * sim->verify == false must skip verification entirely, even against the + * SAME deliberately-altered fixture the previous test proves DOES mismatch + * when verification is on -- proves the --sim-verify gate itself, not just + * the comparator. + */ +static void test_verify_disabled_reports_no_mismatches_even_when_altered(void) +{ + char path[PATH_MAX]; + FILE* f; + int rc; + char* captured; + + f = write_header_and_snapshot(path, sizeof(path), "_verifyoff"); + TEST_ASSERT_NOT_NULL(f); + + append_queue_update_event(f, 1000ULL, "normal"); + append_jobs_push_event(f, 1500ULL, 777ULL, "normal"); + append_replay_session_with_pend(f, 2000ULL, 777ULL, 99); + append_end_of_stream_marker(f); + fclose(f); + + captured = run_sim_and_capture_stdout(path, false, &rc); + + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_NULL_MESSAGE( + strstr(captured, "MISMATCH"), + "sim->verify == false must never report a mismatch, even against a " + "fixture the verify-enabled test proves mismatches"); + + free(captured); + printf("test_verify_disabled_reports_no_mismatches_even_when_altered: " + "PASS\n"); +} + +int main(void) +{ + UNITY_BEGIN(); + + RUN_TEST(test_dispatch_decisions_with_identical_content_match); + RUN_TEST(test_dispatch_decisions_differing_jobid_mismatch); + RUN_TEST(test_dispatch_decisions_differing_host_mismatch); + RUN_TEST(test_dispatch_decisions_differing_resource_value_mismatch); + RUN_TEST(test_pend_decisions_with_identical_content_match); + RUN_TEST(test_pend_decisions_differing_jobid_mismatch); + RUN_TEST(test_pend_decisions_differing_main_reason_mismatch); + RUN_TEST(test_decision_kind_mismatch_reported); + RUN_TEST(test_preempt_decision_always_reports_mismatch); + RUN_TEST(test_decision_eq_defends_against_null_inputs); + RUN_TEST(test_log_mismatch_reports_logical_time_and_decision_detail); + RUN_TEST(test_verify_against_self_reports_zero_mismatches); + RUN_TEST(test_verify_against_altered_copy_reports_exact_mismatch); + RUN_TEST(test_verify_disabled_reports_no_mismatches_even_when_altered); + + return UNITY_END(); +}