feat(scheduler-core): implement simulation decision verification (task 11.4)
Add sim_verify.c (sim_decision_eq/sim_log_mismatch: real structural comparison of dispatch/pend decision payloads, not a jobid shortcut) and wire it into sim_loop.c's ReplaySession drain loop via decode_ReplaySession_ptr() (first real consumer of the generated capnp decode path). --sim-verify now detects and reports genuine decision divergence with logical-time + detail, per Req 5 AC 15/18.
This commit is contained in:
parent
b8d2f5f48d
commit
a6083d803e
SSH key fingerprint:
SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
7 changed files with 1469 additions and 12 deletions
|
|
@ -330,11 +330,12 @@
|
|||
- _Depends: 11.1_
|
||||
- _Verified: 2026-08-06 — new `test_sim_isolation.c` (548 lines) enforces AC 17 across `sim.h`/`sim.c`/`sim_loop.c` in three layers, extending `test_core_isolation.c`'s two-layer pattern rather than duplicating it: (1) source-text scan for forbidden `#include`s and forbidden-symbol call sites, reusing `test_core_isolation.c`'s exact `FORBIDDEN_SYMBOLS` list; (2) a structural check `test_core_isolation.c` has no equivalent of — extracts only the `typedef struct {...} xce_sched_sim_driver_t;` body text (via a comment-stripping pass first, since `sim.h`'s own boundary doc comment legitimately names `xcecli_ctx_t`/`xce_cluster_config_t` in prose) and scans that narrower region for forbidden field types; (3) the literal build/link-time proof — `sim.c` and `sim_loop.c` compiled together into a new isolated `sched_sim_isolation_lib` static_library (`meson.build`), inspected via `nm(1)`'s undefined-symbol table to prove what the compiled object actually calls, independent of transitive header declarations. Deliberately excludes the wall-clock ban (`FORBIDDEN_WALLCLOCK`) — out of this task's Req 5 AC 17 scope, and `sim.c`/`sim_loop.c` legitimately call `clock_gettime()` to seed/advance the core's virtual clock from the driver side, already documented at those call sites. 5/5 new sub-tests pass; 29/29 full scheduler suite passes; whole-project build clean. Independently reviewed and approved.
|
||||
|
||||
- [ ] 11.4 Implement decision verification
|
||||
- [x] 11.4 Implement decision verification
|
||||
- Compare each core-emitted decision against the corresponding recorded decision at the same logical position when verification is enabled; report any mismatch with the logical time and decision detail
|
||||
- Observable: replaying a fixture file with verification enabled against its own previously-recorded output reports zero mismatches; replaying against a deliberately-altered copy reports the exact mismatch
|
||||
- _Requirements: 5.15_
|
||||
- _Depends: 11.2_
|
||||
- _Verified: 2026-08-06 — new `sim_verify.c` (`sim_decision_eq()` + `sim_log_mismatch()`) and changes to `sim_loop.c`'s `ReplaySession` branch implement AC 15/18: `decode_ReplaySession_ptr()` decodes the session's recorded `decisions` list (a deep-copied `replay_session_t*`, generated capnp typemap code already present from task 7.x, its first real consumer) before `capn_free()`, kept alive across the whole per-session drain loop and freed via `free_ReplaySession_ptr()` on every exit path. `sim_decision_eq()` is a genuine field-by-field structural comparison (`xce_alloc_t`/`xce_host_alloc_t`/`xce_job_alloc_t` for DISPATCH, `xce_job_preason_t` including nested host/resource reason lists for PEND; PREEMPT is defensively false-with-log, matching `recording_session.c`'s own documented "not producible on any live path today" scope), not a jobid-only shortcut. Comparison runs positionally (index `di`) only when `sim->verify` is true (confirmed already fully wired end-to-end from `--sim-verify` through `main.c`→`sched_sim_args_t`→`sim.c`, no gap); both live-longer-than-recorded and recorded-longer-than-live length mismatches are surfaced, not silently dropped. New `test_sim_verify.c` (14 tests: 11 unit-level direct calls on `sim_decision_eq`/`sim_log_mismatch` + 3 integration-level through a real `sched_sim_driver_run()` fixture) covers both Observables: verify-against-self reports zero mismatches, verify-against-a-deliberately-altered copy reports the exact mismatch, verify=false reports nothing even when altered. RED phase independently reproduced by review (neutered `sim_decision_eq` to always-match, 8/14 tests failed including the altered-copy integration test, reverted, confirmed GREEN). 30/30 scheduler suite (was 29), whole-project build clean. Review APPROVED with two non-blocking FYI notes (a stale test-count label in a doc comment; `sim_log_mismatch()`'s signature intentionally drops AC 18 pseudocode's illustrative `sim` parameter and skips a wasteful allocate-then-discard `from_replay_decision()` conversion step, documented in-file, no behavioral gap against AC 15's actual contract).
|
||||
|
||||
- [ ] 11.5 Implement what-if branching and simulation output recording
|
||||
- Support loading the snapshot from a base replay file while replaying events from a separate alternate-events file
|
||||
|
|
|
|||
|
|
@ -16,6 +16,7 @@ scheduler_srcs = [
|
|||
'src/recording_session.c',
|
||||
'src/sim.c',
|
||||
'src/sim_loop.c',
|
||||
'src/sim_verify.c',
|
||||
'src/schedobj.c',
|
||||
'src/plugin.c',
|
||||
'src/resource.c',
|
||||
|
|
|
|||
|
|
@ -14,18 +14,21 @@
|
|||
|
||||
/*
|
||||
* sim_loop.c - simulation-mode driver event loop (scheduler-core spec, task
|
||||
* 11.2, Req 5 AC 14/18). Completes Req 5 AC 13's five-step startup
|
||||
* sequence step 5 (sim.c/task 11.1 implements steps 1-4) and hosts
|
||||
* sched_sim_driver_run() itself -- the design.md-literal SimDriver entry
|
||||
* point sim.h declares.
|
||||
* 11.2, Req 5 AC 14/18; decision verification added by task 11.4, Req 5 AC
|
||||
* 15/18). Completes Req 5 AC 13's five-step startup sequence step 5
|
||||
* (sim.c/task 11.1 implements steps 1-4) and hosts sched_sim_driver_run()
|
||||
* itself -- the design.md-literal SimDriver entry point sim.h declares.
|
||||
*
|
||||
* Scope boundary vs. sibling tasks 11.4/11.5 (both depend on this task):
|
||||
* task 11.4 owns comparing each drained decision against the recorded
|
||||
* ReplaySession.decisions when --sim-verify is set; task 11.5 owns writing
|
||||
* decisions to a --sim-output file and loading --sim-alt-events. Neither
|
||||
* branch is built here -- sim_print_decision() below is this task's own,
|
||||
* narrower Req 5 AC 19 default ("if --sim-output is not specified,
|
||||
* decisions are printed to stdout only").
|
||||
* Scope boundary vs. sibling task 11.5 (depends on this task): task 11.5
|
||||
* owns writing decisions to a --sim-output file and loading
|
||||
* --sim-alt-events. That branch is not built here -- sim_print_decision()
|
||||
* below is this task's own, narrower Req 5 AC 19 default ("if --sim-output
|
||||
* is not specified, decisions are printed to stdout only"). Task 11.4 (this
|
||||
* revision) owns comparing each drained decision against the recorded
|
||||
* ReplaySession.decisions when --sim-verify is set -- see the ReplaySession
|
||||
* branch in sched_sim_driver_run() below, and sim_verify.c (sibling file,
|
||||
* same task) for the sim_decision_eq()/sim_log_mismatch() comparison
|
||||
* logic itself.
|
||||
*
|
||||
* Picker-driving thread (sim_picker_thread_fn, below): Req 5 AC 18's
|
||||
* pseudocode calls `core_go()` then blocks draining `core->decisions` until
|
||||
|
|
@ -58,6 +61,20 @@
|
|||
extern int msg_root_has_shape(struct capn* c, capn_ptr* out, size_t want_words,
|
||||
size_t want_ptrs);
|
||||
|
||||
/*
|
||||
* Shared with sim_verify.c (task 11.4, Req 5 AC 15/18) -- same "extern
|
||||
* declared where consumed" idiom as msg_root_has_shape() above. Declared
|
||||
* locally here rather than in sim.h, which stays capnp-type-free (main.c
|
||||
* and other non-capnp-aware callers include it) -- see sim.h's own file
|
||||
* header comment for that boundary, and sim_verify.c's own file header
|
||||
* comment for the full rationale.
|
||||
*/
|
||||
extern int sim_decision_eq(const replay_decision_t* expected,
|
||||
const xce_decision_t* actual);
|
||||
extern void sim_log_mismatch(uint64_t begin_round, int decision_index,
|
||||
const replay_decision_t* expected,
|
||||
const xce_decision_t* actual);
|
||||
|
||||
/*
|
||||
* sim_advance_virtual_clock - Req 5 AC 14 (task 11.2): re-anchors
|
||||
* sim->core's virtual clock so xce_core_now(&sim->core) reports exactly
|
||||
|
|
@ -505,13 +522,37 @@ int sched_sim_driver_run(xce_sched_sim_driver_t* sim,
|
|||
ReplaySession_pointer_count)) {
|
||||
ReplaySession_ptr sp;
|
||||
uint64_t begin_round;
|
||||
replay_session_t* decoded_session = NULL;
|
||||
xce_decision_t* d;
|
||||
int push_rc;
|
||||
int di;
|
||||
|
||||
sp.p = root;
|
||||
begin_round = ReplaySession_get_beginRound(sp);
|
||||
|
||||
/* Req 5 AC 15/18 (task 11.4): decode the FULL session --
|
||||
* including its `decisions` list -- before the capnp arena is
|
||||
* freed. decode_ReplaySession_ptr() deep-copies every payload
|
||||
* out of the arena into plain heap structures first (same
|
||||
* "safe to capn_free() immediately after decode" contract
|
||||
* sim_process_replay_event_message()'s own doc comment already
|
||||
* establishes for decode_ReplayEvent(), verified against the
|
||||
* generated decode_ReplaySession()/decode_ReplayDecision()/
|
||||
* decode_DispatchDecision()/decode_PendDecision() chain in
|
||||
* controller.capnp.c -- every one calloc()s fresh storage and
|
||||
* copies field values out, never aliasing capn arena memory),
|
||||
* so `decoded_session` stays valid across the whole drain loop
|
||||
* below, independent of `c`'s lifetime. */
|
||||
decode_ReplaySession_ptr(&decoded_session, sp);
|
||||
capn_free(&c);
|
||||
|
||||
if (decoded_session == NULL) {
|
||||
ERROR("%s out of memory decoding ReplaySession at "
|
||||
"begin_round=%llu -- decision verification skipped "
|
||||
"for this round",
|
||||
__func__, (unsigned long long)begin_round);
|
||||
}
|
||||
|
||||
/* Req 5 AC 18 pseudocode: push the accumulated batch first,
|
||||
* THEN (mirroring requirements.md's own "beginRound =
|
||||
* xce_core_now(core) on core_go()" prose) jump the clock to
|
||||
|
|
@ -529,13 +570,47 @@ int sched_sim_driver_run(xce_sched_sim_driver_t* sim,
|
|||
ERROR("%s xce_core_go failed -- skipping this round's "
|
||||
"decision drain",
|
||||
__func__);
|
||||
free_ReplaySession_ptr(&decoded_session);
|
||||
continue;
|
||||
}
|
||||
|
||||
/* Req 5 AC 15 (task 11.4): when --sim-verify is set, walk the
|
||||
* decoded `decisions` array positionally (di) alongside the
|
||||
* live drained decisions, exactly matching AC 18's pseudocode
|
||||
* `di < msg.session.decisions.len` bound check -- never compare
|
||||
* past the end of the recorded array. A live decision with no
|
||||
* corresponding recorded entry is itself reported as a
|
||||
* mismatch (expected=NULL) rather than silently skipped; the
|
||||
* inverse case (recorded decisions left over after the live
|
||||
* stream ends) is reported the same way just below the loop. */
|
||||
di = 0;
|
||||
while ((cbuf_get(&(sim->core.decisions), &d) == 0) && (d != NULL)) {
|
||||
if (sim->verify && (decoded_session != NULL)) {
|
||||
const replay_decision_t* expected
|
||||
= (di < decoded_session->n_decisions)
|
||||
? decoded_session->decisions[di]
|
||||
: NULL;
|
||||
|
||||
if (!sim_decision_eq(expected, d)) {
|
||||
sim_log_mismatch(begin_round, di, expected, d);
|
||||
}
|
||||
di++;
|
||||
}
|
||||
sim_print_decision(d);
|
||||
}
|
||||
|
||||
if (sim->verify && (decoded_session != NULL)
|
||||
&& (di < decoded_session->n_decisions)) {
|
||||
int j;
|
||||
|
||||
for (j = di; j < decoded_session->n_decisions; j++) {
|
||||
sim_log_mismatch(begin_round, j,
|
||||
decoded_session->decisions[j], NULL);
|
||||
}
|
||||
}
|
||||
|
||||
free_ReplaySession_ptr(&decoded_session);
|
||||
|
||||
xce_core_end_session(&(sim->core));
|
||||
continue;
|
||||
}
|
||||
|
|
|
|||
431
src/scheduler/src/sim_verify.c
Normal file
431
src/scheduler/src/sim_verify.c
Normal file
|
|
@ -0,0 +1,431 @@
|
|||
#include <stdio.h>
|
||||
#include <string.h>
|
||||
|
||||
#include "controller.capnp.h"
|
||||
#include "decision.h"
|
||||
#include "utils.h"
|
||||
#include "xce/types.h"
|
||||
|
||||
/*
|
||||
* sim_verify.c - deterministic replay verification (scheduler-core spec,
|
||||
* task 11.4, Req 5 AC 15/18: "In simulation mode with --sim-verify, the
|
||||
* driver shall compare each core-emitted decision against the recorded
|
||||
* decisions in the replay file. Any mismatch shall be reported as a
|
||||
* regression with exact logical-time and decision-detail context.").
|
||||
*
|
||||
* Split out of sim_loop.c (task 11.2's own file) as a sibling rather than
|
||||
* appended there directly -- the exact same 800-line-cap precedent task 4.3
|
||||
* set splitting core_events.c out of core.c (tasks.md's "Implementation
|
||||
* Notes": "Task 4.3 split its implementation into a NEW file core_events.c
|
||||
* ... because implementing xce_core_push_events() directly in core.c would
|
||||
* have pushed it to 977 lines, over this project's 800-line hard cap"),
|
||||
* later repeated for core_session.c (task 4.4) and core_clock.c (task 4.5),
|
||||
* and again for sim_loop.c itself (task 11.2, sibling of sim.c). This
|
||||
* codebase's own per-function doc-comment convention (every function in
|
||||
* sim.c/sim_loop.c carries a substantial rationale comment) made the same
|
||||
* call here: the full structural-equality + mismatch-reporting logic below,
|
||||
* documented to the same standard, would not comfortably fit inside
|
||||
* sim_loop.c's own 800-line budget alongside its existing content.
|
||||
*
|
||||
* Owns exactly two entry points, both consumed by sim_loop.c's
|
||||
* sched_sim_driver_run() via a local `extern` declaration -- the same
|
||||
* "extern declared where consumed" idiom sim.c/sim_loop.c already use to
|
||||
* share msg_root_has_shape(), and recording.c/recording_session.c use to
|
||||
* share write_capnp_message() -- rather than adding a capnp-typed
|
||||
* declaration to sim.h, which stays deliberately capnp-type-free (main.c
|
||||
* and other non-capnp-aware callers include it; see sim.h's own file
|
||||
* header comment for that boundary):
|
||||
*
|
||||
* sim_decision_eq() -- Req 5 AC 18 pseudocode's `decision_eq()`: true
|
||||
* iff a decoded recorded ReplayDecision and a live
|
||||
* core-emitted xce_decision_t represent the same
|
||||
* scheduling outcome.
|
||||
* sim_log_mismatch() -- Req 5 AC 18 pseudocode's `log_mismatch()`:
|
||||
* reports one mismatch (logical time + decision
|
||||
* detail) via this codebase's ERROR() logging
|
||||
* convention (utils.h) -- the same macro every
|
||||
* anomaly path in sim.c/sim_loop.c already uses,
|
||||
* not a new mechanism. Deliberately takes
|
||||
* (begin_round, decision_index, expected, actual)
|
||||
* directly rather than AC 18's literal
|
||||
* `log_mismatch(sim, &expected, d)` signature --
|
||||
* nothing here reads xce_sched_sim_driver_t itself,
|
||||
* so threading it through would add a needless
|
||||
* coupling for zero behavioral difference.
|
||||
*
|
||||
* Comparison strategy (Req 5 AC 15: "compare each core-emitted decision
|
||||
* against the corresponding recorded decision"): structural, not pointer,
|
||||
* equality. A live xce_decision_t's xce_job_alloc_t* or xce_pendreason_t*
|
||||
* payload is ALWAYS a distinct heap allocation from the recorded
|
||||
* ReplayDecision's decoded replay_decision_t payload -- one came from the
|
||||
* picker/allocator thread's own make_dispatch_decision()/
|
||||
* make_pend_decision() calls (decision.c), the other from
|
||||
* decode_ReplaySession_ptr()'s deep-copy-from-capnp-arena walk
|
||||
* (controller.capnp.c). Pointer or memcmp equality would therefore always
|
||||
* report a false mismatch; every comparator below instead walks both trees
|
||||
* field-by-field, matching the exact xce_job_alloc_t/xce_job_preason_t
|
||||
* shapes recording_session.c's own encode side (the inverse direction)
|
||||
* already documents.
|
||||
*/
|
||||
|
||||
/*
|
||||
* sim_verify_str_eq - NULL-safe string equality. Both trees under
|
||||
* comparison route every string through STRING_DUP()/makeString()-style
|
||||
* heap copies on their own independent path (decode_* on one side,
|
||||
* make_pend_decision()/the picker's own construction on the other), so a
|
||||
* NULL/non-NULL string field is itself real signal, not noise -- (a==b==
|
||||
* NULL) counts as equal, exactly one of the two being NULL never does.
|
||||
*/
|
||||
static int sim_verify_str_eq(const char* a, const char* b)
|
||||
{
|
||||
if (a == b) {
|
||||
return 1;
|
||||
}
|
||||
if ((a == NULL) || (b == NULL)) {
|
||||
return 0;
|
||||
}
|
||||
return strcmp(a, b) == 0;
|
||||
}
|
||||
|
||||
/*
|
||||
* sim_verify_alloc_eq - xce_alloc_t (one resource allocation record, e.g.
|
||||
* one GPU/memory grant) field-by-field equality. `value` is compared with
|
||||
* plain `==`: both sides are exact copies (decode/construction, no
|
||||
* arithmetic in between) of the same double, so bit-exact comparison is
|
||||
* correct here, not the usual floating-point-tolerance concern.
|
||||
*/
|
||||
static int sim_verify_alloc_eq(const xce_alloc_t* a, const xce_alloc_t* b)
|
||||
{
|
||||
if (a == b) {
|
||||
return 1;
|
||||
}
|
||||
if ((a == NULL) || (b == NULL)) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
return (a->instid == b->instid) && (a->value == b->value)
|
||||
&& (a->from == b->from) && (a->duration == b->duration)
|
||||
&& sim_verify_str_eq(a->resname, b->resname)
|
||||
&& sim_verify_str_eq(a->reserve, b->reserve);
|
||||
}
|
||||
|
||||
/*
|
||||
* sim_verify_host_alloc_eq - xce_host_alloc_t equality: same host name,
|
||||
* same count and (in-order) content of per-resource xce_alloc_t entries.
|
||||
* Order-sensitive comparison is correct here: both the recorded and the
|
||||
* live tree are built by the SAME deterministic scheduling round replaying
|
||||
* against the SAME core state (Req 5's whole simulation-mode premise), so
|
||||
* the host/resource allocation arrays are expected to come out in the same
|
||||
* order, not just the same multiset.
|
||||
*/
|
||||
static int sim_verify_host_alloc_eq(const xce_host_alloc_t* a,
|
||||
const xce_host_alloc_t* b)
|
||||
{
|
||||
int i;
|
||||
|
||||
if (a == b) {
|
||||
return 1;
|
||||
}
|
||||
if ((a == NULL) || (b == NULL)) {
|
||||
return 0;
|
||||
}
|
||||
if (a->nallocs != b->nallocs) {
|
||||
return 0;
|
||||
}
|
||||
if (!sim_verify_str_eq(a->host, b->host)) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
for (i = 0; i < a->nallocs; i++) {
|
||||
if (!sim_verify_alloc_eq(a->allocs[i], b->allocs[i])) {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
return 1;
|
||||
}
|
||||
|
||||
/*
|
||||
* sim_verify_job_alloc_eq - xce_job_alloc_t (a DISPATCH decision's full
|
||||
* payload) equality: jobid, per-host allocation list (xce_host_alloc_t,
|
||||
* via sim_verify_host_alloc_eq above), and the shared-resource allocation
|
||||
* list (xce_alloc_t, via sim_verify_alloc_eq).
|
||||
*/
|
||||
static int sim_verify_job_alloc_eq(const xce_job_alloc_t* a,
|
||||
const xce_job_alloc_t* b)
|
||||
{
|
||||
int i;
|
||||
|
||||
if (a == b) {
|
||||
return 1;
|
||||
}
|
||||
if ((a == NULL) || (b == NULL)) {
|
||||
return 0;
|
||||
}
|
||||
if ((a->jobid != b->jobid) || (a->nhosts != b->nhosts)
|
||||
|| (a->nres_shared != b->nres_shared)) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
for (i = 0; i < a->nhosts; i++) {
|
||||
if (!sim_verify_host_alloc_eq(a->allocs[i], b->allocs[i])) {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
for (i = 0; i < a->nres_shared; i++) {
|
||||
if (!sim_verify_alloc_eq(a->allocs_shared[i], b->allocs_shared[i])) {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
return 1;
|
||||
}
|
||||
|
||||
/*
|
||||
* sim_verify_job_preason_eq - xce_job_preason_t (a PEND decision's reason
|
||||
* payload) equality: main_reason plus (as a deeper, not-strictly-mandatory
|
||||
* but cheap-to-add check per this task's own "don't leave an obviously-
|
||||
* wrong shallow check if the deep one is a few extra lines" guidance) the
|
||||
* full host-based and resource-based reason lists.
|
||||
*/
|
||||
static int sim_verify_job_preason_eq(const xce_job_preason_t* a,
|
||||
const xce_job_preason_t* b)
|
||||
{
|
||||
int i;
|
||||
|
||||
if (a == b) {
|
||||
return 1;
|
||||
}
|
||||
if ((a == NULL) || (b == NULL)) {
|
||||
return 0;
|
||||
}
|
||||
if ((a->main_reason != b->main_reason)
|
||||
|| (a->nhost_reasons != b->nhost_reasons)
|
||||
|| (a->nres_reasons != b->nres_reasons)) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
for (i = 0; i < a->nhost_reasons; i++) {
|
||||
const xce_host_reason_t* ha = a->host_based_reasons[i];
|
||||
const xce_host_reason_t* hb = b->host_based_reasons[i];
|
||||
|
||||
if (ha == hb) {
|
||||
continue;
|
||||
}
|
||||
if ((ha == NULL) || (hb == NULL)) {
|
||||
return 0;
|
||||
}
|
||||
if ((ha->reason != hb->reason)
|
||||
|| !sim_verify_str_eq(ha->host, hb->host)) {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
for (i = 0; i < a->nres_reasons; i++) {
|
||||
const xce_res_reason_t* ra = a->res_based_reasons[i];
|
||||
const xce_res_reason_t* rb = b->res_based_reasons[i];
|
||||
|
||||
if (ra == rb) {
|
||||
continue;
|
||||
}
|
||||
if ((ra == NULL) || (rb == NULL)) {
|
||||
return 0;
|
||||
}
|
||||
if ((ra->reason != rb->reason)
|
||||
|| !sim_verify_str_eq(ra->resource, rb->resource)
|
||||
|| !sim_verify_str_eq(ra->host, rb->host)) {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
return 1;
|
||||
}
|
||||
|
||||
/*
|
||||
* sim_decision_eq - Req 5 AC 15/18's `decision_eq()`. `expected` is one
|
||||
* decoded ReplayDecision from the recorded replay file (NULL means "no
|
||||
* recorded entry at this position" -- always a mismatch); `actual` is one
|
||||
* live decision the core just emitted (NULL is defensively treated the
|
||||
* same way, though sim_loop.c's drain loop never calls this with a NULL
|
||||
* actual today). Dispatches on `actual->op`, requiring `expected`'s own
|
||||
* body_which to name the SAME decision kind before comparing payloads --
|
||||
* an op/body_which mismatch (e.g. recorded PEND, live DISPATCH) is itself
|
||||
* reported as a mismatch, not silently ignored.
|
||||
*/
|
||||
int sim_decision_eq(const replay_decision_t* expected,
|
||||
const xce_decision_t* actual)
|
||||
{
|
||||
if ((expected == NULL) || (actual == NULL)) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
switch (actual->op) {
|
||||
case XCE_DECISION_DISPATCH:
|
||||
if (expected->body_which != ReplayDecision_body_dispatch) {
|
||||
return 0;
|
||||
}
|
||||
return (expected->body.dispatch != NULL)
|
||||
&& sim_verify_job_alloc_eq(expected->body.dispatch->alloc,
|
||||
actual->u.alloc);
|
||||
|
||||
case XCE_DECISION_PEND:
|
||||
if (expected->body_which != ReplayDecision_body_pend) {
|
||||
return 0;
|
||||
}
|
||||
if ((expected->body.pend == NULL) || (actual->u.reason == NULL)) {
|
||||
return 0;
|
||||
}
|
||||
return (expected->body.pend->jobid == actual->u.reason->jobid)
|
||||
&& sim_verify_job_preason_eq(expected->body.pend->reason,
|
||||
actual->u.reason->reasons);
|
||||
|
||||
case XCE_DECISION_PREEMPT:
|
||||
/* recording_session.c's own documented reason (see its
|
||||
* sched_record_session_add_decision(), XCE_DECISION_PREEMPT case):
|
||||
* no live allocator path produces this decision type today
|
||||
* (scheduler-preemption spec territory). A decision this driver
|
||||
* could never emit for real cannot be meaningfully structurally
|
||||
* verified either -- log loud and report a mismatch, rather than
|
||||
* silently treating an unreachable-today comparison as a "pass". */
|
||||
ERROR("%s PREEMPT decision comparison is not supported "
|
||||
"(scheduler-preemption spec territory -- no live path "
|
||||
"produces this decision type today) -- jobid=%llu reported "
|
||||
"as a mismatch",
|
||||
__func__,
|
||||
(unsigned long long)((actual->u.preempt != NULL)
|
||||
? actual->u.preempt->jobid
|
||||
: 0ULL));
|
||||
return 0;
|
||||
|
||||
default:
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
#define SIM_VERIFY_DESC_BUF_SZ (256)
|
||||
|
||||
/*
|
||||
* sim_verify_describe_replay_decision - formats one recorded ReplayDecision
|
||||
* into `buf` for sim_log_mismatch()'s actionable detail. Never fails --
|
||||
* worst case (a malformed/NULL payload) produces a clearly-labeled
|
||||
* placeholder string instead of crashing on a mismatch report itself.
|
||||
*/
|
||||
static void sim_verify_describe_replay_decision(const replay_decision_t* d,
|
||||
char* buf, size_t buf_sz)
|
||||
{
|
||||
if (d == NULL) {
|
||||
snprintf(buf, buf_sz, "<none>");
|
||||
return;
|
||||
}
|
||||
|
||||
switch (d->body_which) {
|
||||
case ReplayDecision_body_dispatch:
|
||||
if ((d->body.dispatch != NULL) && (d->body.dispatch->alloc != NULL)) {
|
||||
snprintf(buf, buf_sz, "DISPATCH jobid=%llu nhosts=%d",
|
||||
(unsigned long long)d->body.dispatch->alloc->jobid,
|
||||
d->body.dispatch->alloc->nhosts);
|
||||
} else {
|
||||
snprintf(buf, buf_sz, "DISPATCH <no alloc>");
|
||||
}
|
||||
break;
|
||||
|
||||
case ReplayDecision_body_pend:
|
||||
if (d->body.pend != NULL) {
|
||||
snprintf(buf, buf_sz, "PEND jobid=%llu main_reason=%d",
|
||||
(unsigned long long)d->body.pend->jobid,
|
||||
(d->body.pend->reason != NULL)
|
||||
? d->body.pend->reason->main_reason
|
||||
: -1);
|
||||
} else {
|
||||
snprintf(buf, buf_sz, "PEND <no reason>");
|
||||
}
|
||||
break;
|
||||
|
||||
case ReplayDecision_body_preempt:
|
||||
snprintf(buf, buf_sz, "PREEMPT jobid=%llu",
|
||||
(d->body.preempt != NULL)
|
||||
? (unsigned long long)d->body.preempt->jobid
|
||||
: 0ULL);
|
||||
break;
|
||||
|
||||
default:
|
||||
snprintf(buf, buf_sz, "<unknown body_which=%d>", d->body_which);
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
* sim_verify_describe_decision - same job as
|
||||
* sim_verify_describe_replay_decision() above, for the live xce_decision_t
|
||||
* side.
|
||||
*/
|
||||
static void sim_verify_describe_decision(const xce_decision_t* d, char* buf,
|
||||
size_t buf_sz)
|
||||
{
|
||||
if (d == NULL) {
|
||||
snprintf(buf, buf_sz, "<none>");
|
||||
return;
|
||||
}
|
||||
|
||||
switch (d->op) {
|
||||
case XCE_DECISION_DISPATCH:
|
||||
snprintf(buf, buf_sz, "DISPATCH jobid=%llu nhosts=%d",
|
||||
(d->u.alloc != NULL) ? (unsigned long long)d->u.alloc->jobid
|
||||
: 0ULL,
|
||||
(d->u.alloc != NULL) ? d->u.alloc->nhosts : 0);
|
||||
break;
|
||||
|
||||
case XCE_DECISION_PEND:
|
||||
snprintf(buf, buf_sz, "PEND jobid=%llu main_reason=%d",
|
||||
(d->u.reason != NULL) ? (unsigned long long)d->u.reason->jobid
|
||||
: 0ULL,
|
||||
((d->u.reason != NULL) && (d->u.reason->reasons != NULL))
|
||||
? d->u.reason->reasons->main_reason
|
||||
: -1);
|
||||
break;
|
||||
|
||||
case XCE_DECISION_PREEMPT:
|
||||
snprintf(buf, buf_sz, "PREEMPT jobid=%llu",
|
||||
(d->u.preempt != NULL)
|
||||
? (unsigned long long)d->u.preempt->jobid
|
||||
: 0ULL);
|
||||
break;
|
||||
|
||||
default:
|
||||
snprintf(buf, buf_sz, "<op=%d>", d->op);
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
* sim_log_mismatch - Req 5 AC 15/18's `log_mismatch()`: reports one
|
||||
* decision-verification mismatch with the exact logical time
|
||||
* (`begin_round` -- the enclosing ReplaySession's own timestamp, same
|
||||
* value AC 18's pseudocode reads off `msg.session`) plus each side's
|
||||
* decision detail, via ERROR() (utils.h) -- the same logging convention
|
||||
* every other anomaly in sim.c/sim_loop.c already uses (e.g.
|
||||
* `xce_core_push_events failed`, `xce_core_go failed`), not a new
|
||||
* mechanism. `decision_index` is this decision's position within the
|
||||
* session's decisions list (`di` in AC 18's pseudocode). Either `expected`
|
||||
* or `actual` may be NULL (a length mismatch between the recorded and live
|
||||
* decision counts for this session -- sim_loop.c's own drain loop reports
|
||||
* that case through this same function rather than a separate one).
|
||||
*/
|
||||
void sim_log_mismatch(uint64_t begin_round, int decision_index,
|
||||
const replay_decision_t* expected,
|
||||
const xce_decision_t* actual)
|
||||
{
|
||||
char expected_desc[SIM_VERIFY_DESC_BUF_SZ];
|
||||
char actual_desc[SIM_VERIFY_DESC_BUF_SZ];
|
||||
uint64_t expected_ts = (expected != NULL) ? expected->ts : 0ULL;
|
||||
|
||||
sim_verify_describe_replay_decision(expected, expected_desc,
|
||||
sizeof(expected_desc));
|
||||
sim_verify_describe_decision(actual, actual_desc, sizeof(actual_desc));
|
||||
|
||||
ERROR("%s decision verification MISMATCH at begin_round=%llu "
|
||||
"decision_index=%d expected_ts=%llu -- expected: %s -- actual: %s",
|
||||
__func__, (unsigned long long)begin_round, decision_index,
|
||||
(unsigned long long)expected_ts, expected_desc, actual_desc);
|
||||
}
|
||||
|
|
@ -535,6 +535,37 @@ sched_sim_isolation_test = executable(
|
|||
+ '"'],
|
||||
install: false)
|
||||
|
||||
# Contract test for decision verification (scheduler-core spec, task 11.4,
|
||||
# Req 5 AC 15/18): each core-emitted decision is compared against the
|
||||
# corresponding recorded decision at the same logical position when
|
||||
# --sim-verify is set, and any mismatch is reported with the exact logical
|
||||
# time and decision detail. Unit-level: sim_decision_eq()/sim_log_mismatch()
|
||||
# (new ../src/sim_verify.c, task 11.4's sibling file to sim_loop.c) driven
|
||||
# directly against hand-built DISPATCH/PEND/PREEMPT decision pairs --
|
||||
# including DISPATCH, which sim_loop.c's own real picker thread can never
|
||||
# produce live (Req 5 AC 17-adjacent scope boundary: real resource-matching
|
||||
# allocation belongs to the not-yet-designed scheduler-allocation spec), so
|
||||
# this is the only way to prove that comparator for real. Integration-level:
|
||||
# a full sched_sim_driver_run() run against a real QUEUE_UPDATE+JOBS_PUSH
|
||||
# fixture feeding the real builtin_fcfs picker to a genuine live PEND
|
||||
# decision, --sim-verify both on (self-match reports zero mismatches,
|
||||
# altered-copy reports the exact mismatch) and off (no mismatch reported
|
||||
# even against an altered fixture) -- mismatch output observed via the same
|
||||
# STDOUT_FILENO dup()/dup2() capture technique
|
||||
# test_apply_correctness_verification.c's own
|
||||
# test_apply_divergent_loud_fence_log_omits_false_peer_hwm_clause
|
||||
# establishes for ERROR()'s tlog early-print path. Same white-box #include
|
||||
# pattern as sched_sim_event_loop_test (core.c, core_clock.c, core_events.c,
|
||||
# core_session.c, decision.c, plugin.c, plugins/builtin_fcfs.c, recording.c,
|
||||
# sim.c, sim_loop.c, snapshot.c/snapshot_wire.c/snapshot_load.c) PLUS the
|
||||
# new ../src/sim_verify.c under test.
|
||||
sched_sim_verify_test = executable(
|
||||
'sched_sim_verify_test',
|
||||
['test_sim_verify.c'],
|
||||
include_directories:[incdir],
|
||||
dependencies: [scheduler_test_deps],
|
||||
install: false)
|
||||
|
||||
# Register as meson tests so `meson test --suite scheduler` runs them all.
|
||||
# detect_leaks=0 matches the suite convention (fixtures are not torn down).
|
||||
test('scheduler test', scheduler_test, suite: 'scheduler',
|
||||
|
|
@ -613,3 +644,6 @@ test('scheduler sim event loop test',
|
|||
test('scheduler sim isolation test',
|
||||
sched_sim_isolation_test, suite: 'scheduler',
|
||||
env: ['ASAN_OPTIONS=detect_leaks=0'])
|
||||
test('scheduler sim verify test',
|
||||
sched_sim_verify_test, suite: 'scheduler',
|
||||
env: ['ASAN_OPTIONS=detect_leaks=0'])
|
||||
|
|
|
|||
|
|
@ -55,6 +55,12 @@
|
|||
* same Req 5 AC 17 boundary sim.c/sim_loop.c themselves respect) PLUS
|
||||
* ../src/recording.c (fixture-building only, same as test_sim_startup.c)
|
||||
* PLUS ../src/sim.c/../src/sim_loop.c (the code under test) PLUS
|
||||
* ../src/sim_verify.c (task 11.4's sibling file -- sim_loop.c's
|
||||
* sched_sim_driver_run() now calls sim_decision_eq()/sim_log_mismatch() as
|
||||
* extern functions defined there; this white-box TU must include it too or
|
||||
* those calls are undefined references at link time, even though neither
|
||||
* test in this file exercises --sim-verify itself -- that is
|
||||
* test_sim_verify.c's own job) PLUS
|
||||
* ../src/snapshot.c/snapshot_wire.c/snapshot_load.c (task 8.1, needed by
|
||||
* both the fixture-building and sim.c's own snapshot load).
|
||||
*/
|
||||
|
|
@ -85,6 +91,7 @@
|
|||
#include "../src/recording.c"
|
||||
#include "../src/sim.c"
|
||||
#include "../src/sim_loop.c"
|
||||
#include "../src/sim_verify.c"
|
||||
#include "../src/snapshot.c"
|
||||
#include "../src/snapshot_load.c"
|
||||
#include "../src/snapshot_wire.c"
|
||||
|
|
|
|||
908
src/scheduler/testsuite/test_sim_verify.c
Normal file
908
src/scheduler/testsuite/test_sim_verify.c
Normal file
|
|
@ -0,0 +1,908 @@
|
|||
/*
|
||||
* test_sim_verify.c - contract test for decision verification
|
||||
* (scheduler-core spec, task 11.4, Req 5 AC 15/18).
|
||||
*
|
||||
* Observable under test (task 11.4's own text):
|
||||
* - replaying a fixture file with verification enabled against its own
|
||||
* previously-recorded output reports zero mismatches.
|
||||
* - replaying against a deliberately-altered copy reports the exact
|
||||
* mismatch.
|
||||
*
|
||||
* Two granularities, same split test_sim_event_loop.c's own header comment
|
||||
* already establishes for task 11.2:
|
||||
*
|
||||
* Unit-level (white-box): calls sim_decision_eq()/sim_log_mismatch()
|
||||
* (sim_verify.c, task 11.4's own sibling file to sim_loop.c) directly
|
||||
* against hand-built replay_decision_t/xce_decision_t pairs -- DISPATCH
|
||||
* match/mismatch (jobid, host, resource value), PEND match/mismatch
|
||||
* (jobid, main_reason), a body_which/op type mismatch, and PREEMPT's
|
||||
* documented always-false path. This is the only way to exercise DISPATCH
|
||||
* comparison at all: sim_loop.c's real picker thread
|
||||
* (sim_picker_thread_fn) only ever emits PEND decisions (its own doc
|
||||
* comment: "conservatively PENDs every candidate ... rather than
|
||||
* fabricate a DISPATCH decision with no real worker/resource assignment
|
||||
* behind it") -- real resource-matching allocation belongs to the
|
||||
* not-yet-designed scheduler-allocation spec. sim_decision_eq() must
|
||||
* still compare DISPATCH decisions correctly (Req 5 AC 15 does not
|
||||
* exempt them), so that comparator is proven directly here rather than
|
||||
* left unreachable.
|
||||
*
|
||||
* Integration-level: drives the full sched_sim_driver_run() (this task's
|
||||
* other Observable) against a fixture built from a REAL
|
||||
* QUEUE_UPDATE+JOBS_PUSH event pair (one queue, one JOB_STAT_PEND job)
|
||||
* feeding the REAL builtin_fcfs picker -- since core.c's own
|
||||
* sched_job_is_ready()/sched_queue_is_ready() are unconditional `return
|
||||
* 1` stubs (task 2.4's own documented, still-live scope note), any queue/
|
||||
* job pair is immediately "ready", so this always produces exactly one
|
||||
* live PEND decision via make_pendreason() (preason.c), whose
|
||||
* xce_job_preason_t is always freshly calloc()'d (main_reason=0, no host/
|
||||
* res reasons) -- a fully deterministic, reproducible target to compare
|
||||
* against. `--sim-verify`'s ERROR()-logged mismatch report is observed by
|
||||
* redirecting STDOUT_FILENO around the sched_sim_driver_run() call, the
|
||||
* same dup()/dup2() capture technique
|
||||
* test_apply_correctness_verification.c's own
|
||||
* test_apply_divergent_loud_fence_log_omits_false_peer_hwm_clause
|
||||
* establishes (that file's own doc comment explains WHY this works: with
|
||||
* tlog never tlog_init()'d in this test process -- true here too, nothing
|
||||
* in this TU calls it -- every ERROR()/WARN() falls through tlog's
|
||||
* early-print path, a raw write() to STDOUT_FILENO).
|
||||
*
|
||||
* Fixture strategy: same production-code-reuse technique test_sim_startup.c/
|
||||
* test_sim_event_loop.c already establish for the header+snapshot prefix
|
||||
* (sched_record_start() against a small real xce_scheduler_t fixture) PLUS
|
||||
* hand-constructed ReplayEvent/ReplaySession messages appended afterward via
|
||||
* the real encode_ReplayEvent_ptr()/encode_ReplaySession_ptr()/
|
||||
* write_capnp_message() codec, same technique test_sim_event_loop.c's own
|
||||
* append_jobstatechange_event()/append_replay_session() and
|
||||
* test_e2e_capnp.c's test_dispatch_decision_round_trip()/
|
||||
* test_pend_decision_round_trip() already prove works.
|
||||
*
|
||||
* White-box #include of the same TU set test_sim_event_loop.c uses (see
|
||||
* that file's own header comment) PLUS ../src/sim_verify.c (the code under
|
||||
* test).
|
||||
*/
|
||||
#include <errno.h>
|
||||
#include <limits.h>
|
||||
#include <stdatomic.h>
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <unity/unity.h>
|
||||
|
||||
#include "controller.capnp.h"
|
||||
#include "kvmgr.h"
|
||||
#include "preason.h"
|
||||
#include "scheduler.h"
|
||||
#include "xstring.h"
|
||||
|
||||
/* Pull in the real functions under test into this TU (white-box #include,
|
||||
* same convention as test_sim_event_loop.c/test_sim_startup.c). */
|
||||
#include "../src/core.c"
|
||||
#include "../src/core_clock.c"
|
||||
#include "../src/core_events.c"
|
||||
#include "../src/core_session.c"
|
||||
#include "../src/decision.c"
|
||||
#include "../src/plugin.c"
|
||||
#include "../src/plugins/builtin_fcfs.c"
|
||||
#include "../src/recording.c"
|
||||
#include "../src/sim.c"
|
||||
#include "../src/sim_loop.c"
|
||||
#include "../src/sim_verify.c"
|
||||
#include "../src/snapshot.c"
|
||||
#include "../src/snapshot_load.c"
|
||||
#include "../src/snapshot_wire.c"
|
||||
|
||||
void kill_sched_job(void* arg) { (void)arg; }
|
||||
void kill_sched_queue(void* arg) { (void)arg; }
|
||||
void kill_sched_worker(void* arg) { (void)arg; }
|
||||
void kill_sched_user(void* arg) { (void)arg; }
|
||||
void kill_sched_project(void* arg) { (void)arg; }
|
||||
void kill_sched_resval(xce_val_t* val) { (void)val; }
|
||||
|
||||
/* core_reset_workers()'s (core_session.c) linker dependency -- same stub
|
||||
* test_sim_event_loop.c/test_core_full_cycle.c already establish; never
|
||||
* invoked (no worker is ever added here). */
|
||||
void sched_worker_release_host_ctx(xce_sched_worker_t* w) { (void)w; }
|
||||
|
||||
/* plugin_prepare_allocation()'s (plugin.c, the "limit" plugin category)
|
||||
* linker dependency -- never invoked (this file never drives real
|
||||
* allocation). */
|
||||
int sched_fill_limit_requests(xce_scheduler_t* sched, xvec_idval_t* vec,
|
||||
xlist_t* limits)
|
||||
{
|
||||
(void)sched;
|
||||
(void)vec;
|
||||
(void)limits;
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same
|
||||
* copy test_sim_event_loop.c/test_sim_startup.c/test_core_full_cycle.c each
|
||||
* carry (schedobj.c is not linked here: it would need plugin_register(),
|
||||
* which requires a genuine xce_scheduler_t* neither this file's fixtures
|
||||
* nor sim_loop.c's own core-scoped registration path have, Req 5 AC 17). */
|
||||
int sched_resolve_job_picker(xce_sched_core_t* core, xce_sched_plugin_t** out)
|
||||
{
|
||||
xce_sched_plugin_t* plugin = NULL;
|
||||
int rc = 0;
|
||||
|
||||
if ((core == NULL) || (out == NULL)) {
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
*out = NULL;
|
||||
|
||||
if (core->job_pick_plugin == NULL) {
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
lock(core->plugins, RTTI_LOCK_READ);
|
||||
rc = key_find(core->plugins, core->job_pick_plugin, (void**)&plugin);
|
||||
unlock(core->plugins);
|
||||
|
||||
if (rc <= 0) {
|
||||
return (rc < 0) ? rc : -ENOENT;
|
||||
}
|
||||
|
||||
if ((plugin == NULL) || (plugin->picker_iface == NULL)
|
||||
|| (plugin->picker_iface->next_job == NULL)
|
||||
|| (plugin->picker_iface->begin_session == NULL)
|
||||
|| (plugin->picker_iface->end_session == NULL)
|
||||
|| (plugin->picker_iface->on_dispatch == NULL)
|
||||
|| (plugin->picker_iface->on_pend == NULL)) {
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
*out = plugin;
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
void setUp(void) { return; }
|
||||
void tearDown(void) { return; }
|
||||
|
||||
/* ---------------------------------------------------------------------- */
|
||||
/* Unit-level fixture helpers: hand-built xce_job_alloc_t trees. */
|
||||
/* ---------------------------------------------------------------------- */
|
||||
|
||||
static xce_alloc_t* make_test_alloc(const char* resname, const char* reserve,
|
||||
uint64_t instid, double value,
|
||||
uint64_t from, int64_t duration)
|
||||
{
|
||||
xce_alloc_t* a = (xce_alloc_t*)calloc(1, sizeof(*a));
|
||||
|
||||
TEST_ASSERT_NOT_NULL(a);
|
||||
a->resname = strdup(resname);
|
||||
a->reserve = strdup(reserve);
|
||||
a->instid = instid;
|
||||
a->value = value;
|
||||
a->from = from;
|
||||
a->duration = duration;
|
||||
|
||||
return a;
|
||||
}
|
||||
|
||||
static xce_host_alloc_t* make_test_host_alloc(const char* host,
|
||||
xce_alloc_t** allocs, int nallocs)
|
||||
{
|
||||
xce_host_alloc_t* h = (xce_host_alloc_t*)calloc(1, sizeof(*h));
|
||||
|
||||
TEST_ASSERT_NOT_NULL(h);
|
||||
h->host = strdup(host);
|
||||
h->nallocs = nallocs;
|
||||
h->allocs = allocs;
|
||||
|
||||
return h;
|
||||
}
|
||||
|
||||
static xce_job_alloc_t*
|
||||
make_test_job_alloc(uint64_t jobid, xce_host_alloc_t** hosts, int nhosts)
|
||||
{
|
||||
xce_job_alloc_t* j = (xce_job_alloc_t*)calloc(1, sizeof(*j));
|
||||
|
||||
TEST_ASSERT_NOT_NULL(j);
|
||||
j->jobid = jobid;
|
||||
j->nhosts = nhosts;
|
||||
j->allocs = hosts;
|
||||
j->nres_shared = 0;
|
||||
j->allocs_shared = NULL;
|
||||
|
||||
return j;
|
||||
}
|
||||
|
||||
static replay_decision_t make_replay_dispatch_decision(xce_job_alloc_t* alloc)
|
||||
{
|
||||
replay_decision_t d;
|
||||
sched_dispatch_decision_t* dd
|
||||
= (sched_dispatch_decision_t*)calloc(1, sizeof(*dd));
|
||||
|
||||
TEST_ASSERT_NOT_NULL(dd);
|
||||
dd->alloc = alloc;
|
||||
|
||||
memset(&d, 0, sizeof(d));
|
||||
d.body_which = ReplayDecision_body_dispatch;
|
||||
d.body.dispatch = dd;
|
||||
|
||||
return d;
|
||||
}
|
||||
|
||||
static replay_decision_t make_replay_pend_decision(uint64_t jobid,
|
||||
int main_reason)
|
||||
{
|
||||
replay_decision_t d;
|
||||
sched_pend_decision_t* pd = (sched_pend_decision_t*)calloc(1, sizeof(*pd));
|
||||
xce_job_preason_t* reason = (xce_job_preason_t*)calloc(1, sizeof(*reason));
|
||||
|
||||
TEST_ASSERT_NOT_NULL(pd);
|
||||
TEST_ASSERT_NOT_NULL(reason);
|
||||
reason->main_reason = main_reason;
|
||||
pd->jobid = jobid;
|
||||
pd->reason = reason;
|
||||
|
||||
memset(&d, 0, sizeof(d));
|
||||
d.body_which = ReplayDecision_body_pend;
|
||||
d.body.pend = pd;
|
||||
|
||||
return d;
|
||||
}
|
||||
|
||||
/* ---------------------------------------------------------------------- */
|
||||
/* Unit-level tests: sim_decision_eq() directly. */
|
||||
/* ---------------------------------------------------------------------- */
|
||||
|
||||
static void test_dispatch_decisions_with_identical_content_match(void)
|
||||
{
|
||||
xce_alloc_t* res_a[1]
|
||||
= { make_test_alloc("gpu", "resv1", 1, 2.0, 100, 1800) };
|
||||
xce_host_alloc_t* host_a[1] = { make_test_host_alloc("hostA", res_a, 1) };
|
||||
xce_job_alloc_t* expected_alloc = make_test_job_alloc(42, host_a, 1);
|
||||
|
||||
xce_alloc_t* res_b[1]
|
||||
= { make_test_alloc("gpu", "resv1", 1, 2.0, 100, 1800) };
|
||||
xce_host_alloc_t* host_b[1] = { make_test_host_alloc("hostA", res_b, 1) };
|
||||
xce_job_alloc_t* actual_alloc = make_test_job_alloc(42, host_b, 1);
|
||||
|
||||
replay_decision_t expected = make_replay_dispatch_decision(expected_alloc);
|
||||
xce_decision_t* actual = make_dispatch_decision(actual_alloc);
|
||||
|
||||
/* Two structurally-identical, but never pointer-equal, allocation
|
||||
* trees: proves the comparison is real structural equality, not a
|
||||
* pointer/memcmp shortcut (this file's own header comment). */
|
||||
TEST_ASSERT_NOT_EQUAL(expected_alloc, actual_alloc);
|
||||
TEST_ASSERT_TRUE(sim_decision_eq(&expected, actual));
|
||||
|
||||
printf("test_dispatch_decisions_with_identical_content_match: PASS\n");
|
||||
}
|
||||
|
||||
static void test_dispatch_decisions_differing_jobid_mismatch(void)
|
||||
{
|
||||
xce_alloc_t* res[1] = { make_test_alloc("gpu", "r", 1, 1.0, 0, 0) };
|
||||
xce_host_alloc_t* host[1] = { make_test_host_alloc("hostA", res, 1) };
|
||||
xce_job_alloc_t* expected_alloc = make_test_job_alloc(42, host, 1);
|
||||
|
||||
xce_alloc_t* res2[1] = { make_test_alloc("gpu", "r", 1, 1.0, 0, 0) };
|
||||
xce_host_alloc_t* host2[1] = { make_test_host_alloc("hostA", res2, 1) };
|
||||
xce_job_alloc_t* actual_alloc = make_test_job_alloc(43, host2, 1);
|
||||
|
||||
replay_decision_t expected = make_replay_dispatch_decision(expected_alloc);
|
||||
xce_decision_t* actual = make_dispatch_decision(actual_alloc);
|
||||
|
||||
TEST_ASSERT_FALSE(sim_decision_eq(&expected, actual));
|
||||
|
||||
printf("test_dispatch_decisions_differing_jobid_mismatch: PASS\n");
|
||||
}
|
||||
|
||||
static void test_dispatch_decisions_differing_host_mismatch(void)
|
||||
{
|
||||
xce_alloc_t* res[1] = { make_test_alloc("gpu", "r", 1, 1.0, 0, 0) };
|
||||
xce_host_alloc_t* host[1] = { make_test_host_alloc("hostA", res, 1) };
|
||||
xce_job_alloc_t* expected_alloc = make_test_job_alloc(42, host, 1);
|
||||
|
||||
xce_alloc_t* res2[1] = { make_test_alloc("gpu", "r", 1, 1.0, 0, 0) };
|
||||
xce_host_alloc_t* host2[1] = { make_test_host_alloc("hostB", res2, 1) };
|
||||
xce_job_alloc_t* actual_alloc = make_test_job_alloc(42, host2, 1);
|
||||
|
||||
replay_decision_t expected = make_replay_dispatch_decision(expected_alloc);
|
||||
xce_decision_t* actual = make_dispatch_decision(actual_alloc);
|
||||
|
||||
TEST_ASSERT_FALSE(sim_decision_eq(&expected, actual));
|
||||
|
||||
printf("test_dispatch_decisions_differing_host_mismatch: PASS\n");
|
||||
}
|
||||
|
||||
static void test_dispatch_decisions_differing_resource_value_mismatch(void)
|
||||
{
|
||||
xce_alloc_t* res[1] = { make_test_alloc("gpu", "r", 1, 1.0, 0, 0) };
|
||||
xce_host_alloc_t* host[1] = { make_test_host_alloc("hostA", res, 1) };
|
||||
xce_job_alloc_t* expected_alloc = make_test_job_alloc(42, host, 1);
|
||||
|
||||
xce_alloc_t* res2[1] = { make_test_alloc("gpu", "r", 1, 2.0, 0, 0) };
|
||||
xce_host_alloc_t* host2[1] = { make_test_host_alloc("hostA", res2, 1) };
|
||||
xce_job_alloc_t* actual_alloc = make_test_job_alloc(42, host2, 1);
|
||||
|
||||
replay_decision_t expected = make_replay_dispatch_decision(expected_alloc);
|
||||
xce_decision_t* actual = make_dispatch_decision(actual_alloc);
|
||||
|
||||
TEST_ASSERT_FALSE(sim_decision_eq(&expected, actual));
|
||||
|
||||
printf("test_dispatch_decisions_differing_resource_value_mismatch: "
|
||||
"PASS\n");
|
||||
}
|
||||
|
||||
static void test_pend_decisions_with_identical_content_match(void)
|
||||
{
|
||||
replay_decision_t expected = make_replay_pend_decision(9001ULL, 3);
|
||||
xce_pendreason_t* reason = make_pendreason(9001ULL);
|
||||
xce_decision_t* actual;
|
||||
|
||||
reason->reasons->main_reason = 3;
|
||||
actual = make_pend_decision(reason);
|
||||
|
||||
TEST_ASSERT_TRUE(sim_decision_eq(&expected, actual));
|
||||
|
||||
printf("test_pend_decisions_with_identical_content_match: PASS\n");
|
||||
}
|
||||
|
||||
static void test_pend_decisions_differing_jobid_mismatch(void)
|
||||
{
|
||||
replay_decision_t expected = make_replay_pend_decision(9001ULL, 0);
|
||||
xce_decision_t* actual = make_pend_decision(make_pendreason(9002ULL));
|
||||
|
||||
TEST_ASSERT_FALSE(sim_decision_eq(&expected, actual));
|
||||
|
||||
printf("test_pend_decisions_differing_jobid_mismatch: PASS\n");
|
||||
}
|
||||
|
||||
static void test_pend_decisions_differing_main_reason_mismatch(void)
|
||||
{
|
||||
replay_decision_t expected = make_replay_pend_decision(9001ULL, 3);
|
||||
xce_decision_t* actual = make_pend_decision(make_pendreason(9001ULL));
|
||||
|
||||
/* actual's main_reason stays 0 (make_pendreason()'s own fresh-calloc
|
||||
* default) -- differs from expected's 3. */
|
||||
TEST_ASSERT_FALSE(sim_decision_eq(&expected, actual));
|
||||
|
||||
printf("test_pend_decisions_differing_main_reason_mismatch: PASS\n");
|
||||
}
|
||||
|
||||
static void test_decision_kind_mismatch_reported(void)
|
||||
{
|
||||
replay_decision_t expected = make_replay_pend_decision(1ULL, 0);
|
||||
xce_job_alloc_t* alloc = make_test_job_alloc(1, NULL, 0);
|
||||
xce_decision_t* actual = make_dispatch_decision(alloc);
|
||||
|
||||
/* Recorded PEND vs. live DISPATCH for the SAME jobid: still a
|
||||
* mismatch -- op/body_which disagreement is reported, not ignored. */
|
||||
TEST_ASSERT_FALSE(sim_decision_eq(&expected, actual));
|
||||
|
||||
printf("test_decision_kind_mismatch_reported: PASS\n");
|
||||
}
|
||||
|
||||
static void test_preempt_decision_always_reports_mismatch(void)
|
||||
{
|
||||
xce_preempt_t preempt;
|
||||
xce_decision_t actual;
|
||||
replay_decision_t expected;
|
||||
|
||||
memset(&preempt, 0, sizeof(preempt));
|
||||
preempt.jobid = 55ULL;
|
||||
memset(&actual, 0, sizeof(actual));
|
||||
actual.op = XCE_DECISION_PREEMPT;
|
||||
actual.u.preempt = &preempt;
|
||||
|
||||
/* Even a recorded entry that ALSO claims PREEMPT for the same jobid
|
||||
* still reports a mismatch -- recording_session.c's own documented
|
||||
* "no live path produces this decision type today" makes any PREEMPT
|
||||
* comparison unreachable-in-practice and therefore untrustworthy. */
|
||||
memset(&expected, 0, sizeof(expected));
|
||||
expected.body_which = ReplayDecision_body_preempt;
|
||||
|
||||
TEST_ASSERT_FALSE(sim_decision_eq(&expected, &actual));
|
||||
|
||||
printf("test_preempt_decision_always_reports_mismatch: PASS\n");
|
||||
}
|
||||
|
||||
static void test_decision_eq_defends_against_null_inputs(void)
|
||||
{
|
||||
xce_decision_t* actual = make_pend_decision(make_pendreason(1ULL));
|
||||
replay_decision_t expected = make_replay_pend_decision(1ULL, 0);
|
||||
|
||||
TEST_ASSERT_FALSE(sim_decision_eq(NULL, actual));
|
||||
TEST_ASSERT_FALSE(sim_decision_eq(&expected, NULL));
|
||||
TEST_ASSERT_FALSE(sim_decision_eq(NULL, NULL));
|
||||
|
||||
printf("test_decision_eq_defends_against_null_inputs: PASS\n");
|
||||
}
|
||||
|
||||
/* ---------------------------------------------------------------------- */
|
||||
/* Unit-level test: sim_log_mismatch()'s captured output content. */
|
||||
/* ---------------------------------------------------------------------- */
|
||||
|
||||
static char* capture_stdout_around(void (*fn)(void))
|
||||
{
|
||||
char tmp[] = "/tmp/xce_simverify_logcap_XXXXXX";
|
||||
int fd = mkstemp(tmp);
|
||||
int saved_stdout;
|
||||
off_t len;
|
||||
char* buf;
|
||||
|
||||
TEST_ASSERT_TRUE_MESSAGE(fd >= 0, "mkstemp for stdout capture must "
|
||||
"succeed");
|
||||
|
||||
fflush(stdout);
|
||||
saved_stdout = dup(STDOUT_FILENO);
|
||||
TEST_ASSERT_TRUE_MESSAGE(saved_stdout >= 0, "dup(STDOUT_FILENO) must "
|
||||
"succeed");
|
||||
TEST_ASSERT_TRUE_MESSAGE(dup2(fd, STDOUT_FILENO) >= 0,
|
||||
"dup2 redirect of STDOUT_FILENO must succeed");
|
||||
|
||||
fn();
|
||||
|
||||
fflush(stdout);
|
||||
dup2(saved_stdout, STDOUT_FILENO);
|
||||
close(saved_stdout);
|
||||
|
||||
len = lseek(fd, 0, SEEK_CUR);
|
||||
TEST_ASSERT_TRUE(len >= 0);
|
||||
|
||||
buf = (char*)calloc(1, (size_t)len + 1);
|
||||
TEST_ASSERT_NOT_NULL(buf);
|
||||
TEST_ASSERT_TRUE(lseek(fd, 0, SEEK_SET) == 0);
|
||||
if (len > 0) {
|
||||
ssize_t nread = read(fd, buf, (size_t)len);
|
||||
TEST_ASSERT_TRUE(nread == len);
|
||||
}
|
||||
close(fd);
|
||||
unlink(tmp);
|
||||
|
||||
return buf;
|
||||
}
|
||||
|
||||
static void log_one_mismatch_for_capture(void)
|
||||
{
|
||||
replay_decision_t expected = make_replay_pend_decision(9001ULL, 3);
|
||||
xce_decision_t* actual = make_pend_decision(make_pendreason(9001ULL));
|
||||
|
||||
sim_log_mismatch(12345ULL, 2, &expected, actual);
|
||||
}
|
||||
|
||||
static void test_log_mismatch_reports_logical_time_and_decision_detail(void)
|
||||
{
|
||||
char* buf = capture_stdout_around(log_one_mismatch_for_capture);
|
||||
|
||||
TEST_ASSERT_NOT_NULL_MESSAGE(strstr(buf, "MISMATCH"),
|
||||
"log_mismatch must report a MISMATCH");
|
||||
TEST_ASSERT_NOT_NULL_MESSAGE(
|
||||
strstr(buf, "begin_round=12345"),
|
||||
"log_mismatch must report the session's exact logical time");
|
||||
TEST_ASSERT_NOT_NULL_MESSAGE(
|
||||
strstr(buf, "decision_index=2"),
|
||||
"log_mismatch must report the decision's position within the "
|
||||
"session");
|
||||
TEST_ASSERT_NOT_NULL_MESSAGE(
|
||||
strstr(buf, "jobid=9001"),
|
||||
"log_mismatch must report actionable decision detail (jobid)");
|
||||
|
||||
free(buf);
|
||||
printf("test_log_mismatch_reports_logical_time_and_decision_detail: "
|
||||
"PASS\n");
|
||||
}
|
||||
|
||||
/* ---------------------------------------------------------------------- */
|
||||
/* Integration-level fixture helpers: a real .replay stream driving the */
|
||||
/* REAL builtin_fcfs picker to a genuine live PEND decision. */
|
||||
/* ---------------------------------------------------------------------- */
|
||||
|
||||
static void make_tmp_dir(char* out, size_t out_len, const char* suffix)
|
||||
{
|
||||
char tmpl[PATH_MAX];
|
||||
|
||||
snprintf(tmpl, sizeof(tmpl), "/tmp/sched_simverify%s_XXXXXX", suffix);
|
||||
TEST_ASSERT_NOT_NULL(mkdtemp(tmpl));
|
||||
TEST_ASSERT_TRUE(strlen(tmpl) < out_len);
|
||||
strcpy(out, tmpl);
|
||||
}
|
||||
|
||||
/* Builds a small, real xce_scheduler_t fixture purely to drive
|
||||
* sched_record_start() and produce a REAL, valid `.replay` header+snapshot
|
||||
* prefix (same technique test_sim_startup.c/test_sim_event_loop.c's own
|
||||
* build_fixture_scheduler() establish) -- but, unlike
|
||||
* test_sim_event_loop.c's copy, deliberately records schedPolicy=
|
||||
* "builtin_fcfs" (a string literal, never freed -- this throwaway fixture
|
||||
* scheduler is never torn down, matching this whole suite's documented
|
||||
* accepted-leak convention) so sim_open_and_validate_header() carries
|
||||
* "builtin_fcfs" into sim->core.job_pick_plugin and
|
||||
* sim_register_default_picker() (sim_loop.c) registers the REAL
|
||||
* builtin_fcfs plugin -- this file's own Observable needs the real picker,
|
||||
* not a stub, to prove sim_decision_eq() against a genuinely-produced live
|
||||
* decision. No picker needs to be key_add'ed into THIS throwaway
|
||||
* scheduler's own core.plugins: xce_sched_core_save_snapshot() does not
|
||||
* require a resolvable picker (Req 5 AC 21: "Picker state is NOT stored in
|
||||
* snapshot").
|
||||
*/
|
||||
static void build_fixture_scheduler(xce_scheduler_t* sched, char* record_dir)
|
||||
{
|
||||
int rc;
|
||||
|
||||
memset(sched, 0, sizeof(*sched));
|
||||
mylock_init(&(sched->record_lock));
|
||||
|
||||
rc = xce_sched_core_init(&(sched->core));
|
||||
TEST_ASSERT_EQUAL_INT(0, rc);
|
||||
|
||||
sched->core.job_pick_plugin = (char*)"builtin_fcfs";
|
||||
sched->record_dir = record_dir;
|
||||
}
|
||||
|
||||
/* Same "leave the file open, skip sched_record_stop()'s own endSession
|
||||
* marker" technique test_sim_event_loop.c's own write_header_and_snapshot()
|
||||
* establishes -- see that function's doc comment for the full rationale. */
|
||||
static FILE* write_header_and_snapshot(char* out_path, size_t out_path_len,
|
||||
const char* suffix)
|
||||
{
|
||||
char record_dir[PATH_MAX];
|
||||
xce_scheduler_t sched;
|
||||
int rc;
|
||||
|
||||
make_tmp_dir(record_dir, sizeof(record_dir), suffix);
|
||||
build_fixture_scheduler(&sched, record_dir);
|
||||
|
||||
rc = sched_record_start(&sched, "fixture.replay");
|
||||
TEST_ASSERT_EQUAL_INT(0, rc);
|
||||
TEST_ASSERT_TRUE(strlen(sched.record_file_path) < out_path_len);
|
||||
strcpy(out_path, sched.record_file_path);
|
||||
|
||||
return sched.record_file;
|
||||
}
|
||||
|
||||
/* Appends one ReplayEvent{queueUpdate} message -- creates `queue_name` in
|
||||
* the core (core_events.c's core_upsert_queue(), fires
|
||||
* on_queue_update(picker, q) against the now-registered real builtin_fcfs).
|
||||
* status/priority are set but never gate readiness: core.c's own
|
||||
* sched_queue_is_ready() is an unconditional `return 1` stub (task 2.4's
|
||||
* still-live documented scope note) -- any value here works. */
|
||||
static void append_queue_update_event(FILE* f, uint64_t ts,
|
||||
const char* queue_name)
|
||||
{
|
||||
xce_queue_t queue;
|
||||
replay_event_t src;
|
||||
struct capn c;
|
||||
struct capn_segment* cs;
|
||||
ReplayEvent_ptr p;
|
||||
|
||||
memset(&queue, 0, sizeof(queue));
|
||||
queue.queue = (char*)queue_name;
|
||||
queue.status = QUEUE_STAT_OPEN;
|
||||
queue.priority = 1;
|
||||
|
||||
memset(&src, 0, sizeof(src));
|
||||
src.ts = ts;
|
||||
src.body_which = ReplayEvent_body_queueUpdate;
|
||||
src.body.queue_update = &queue;
|
||||
|
||||
capn_init_malloc(&c);
|
||||
cs = capn_root(&c).seg;
|
||||
encode_ReplayEvent_ptr(cs, &p, &src);
|
||||
capn_setp(capn_root(&c), 0, p.p);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, write_capnp_message(f, &c));
|
||||
capn_free(&c);
|
||||
}
|
||||
|
||||
/* Appends one ReplayEvent{jobsPush} message with exactly one
|
||||
* JOB_STAT_PEND job assigned to `queue_name` -- MUST be appended AFTER
|
||||
* append_queue_update_event() for the same queue name within the same
|
||||
* session's accumulated batch: core_build_job() (core_events.c) looks the
|
||||
* queue up via sched_get_queue() with no get-or-create fallback (unlike
|
||||
* user/project), so the queue must already exist. Drives
|
||||
* core_events.c's real JOBS_PUSH handling: sched_add_job() +
|
||||
* on_job_push(picker, job) against the real builtin_fcfs -- which (since
|
||||
* sched_job_is_ready()/sched_queue_is_ready() are unconditional `return 1`
|
||||
* stubs) makes this job an immediate next_job() candidate, so the picker
|
||||
* thread PENDs it for real via make_pend_decision(make_pendreason(jobid))
|
||||
* (sim_loop.c's own sim_picker_thread_fn).
|
||||
*/
|
||||
static void append_jobs_push_event(FILE* f, uint64_t ts, uint64_t jobid,
|
||||
const char* queue_name)
|
||||
{
|
||||
job_submit_data_t submit;
|
||||
xce_job_t job;
|
||||
xce_job_t* jobs[1];
|
||||
xce_jobs_data_t jobs_data;
|
||||
replay_event_t src;
|
||||
struct capn c;
|
||||
struct capn_segment* cs;
|
||||
ReplayEvent_ptr p;
|
||||
|
||||
memset(&submit, 0, sizeof(submit));
|
||||
submit.queue = (char*)queue_name;
|
||||
submit.user = (char*)"alice";
|
||||
submit.mintasks = 1;
|
||||
submit.maxtasks = 1;
|
||||
|
||||
memset(&job, 0, sizeof(job));
|
||||
job.jobid = jobid;
|
||||
job.state = JOB_STAT_PEND;
|
||||
job.submit = &submit;
|
||||
|
||||
jobs[0] = &job;
|
||||
|
||||
memset(&jobs_data, 0, sizeof(jobs_data));
|
||||
jobs_data.n_total = 1;
|
||||
jobs_data.count = 1;
|
||||
jobs_data.jobs = jobs;
|
||||
|
||||
memset(&src, 0, sizeof(src));
|
||||
src.ts = ts;
|
||||
src.body_which = ReplayEvent_body_jobsPush;
|
||||
src.body.jobs_push = &jobs_data;
|
||||
|
||||
capn_init_malloc(&c);
|
||||
cs = capn_root(&c).seg;
|
||||
encode_ReplayEvent_ptr(cs, &p, &src);
|
||||
capn_setp(capn_root(&c), 0, p.p);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, write_capnp_message(f, &c));
|
||||
capn_free(&c);
|
||||
}
|
||||
|
||||
/* Appends the .replay stream's whole-stream termination marker
|
||||
* (ReplayEvent{endSession}, Req 5 AC 20) -- same technique
|
||||
* test_sim_event_loop.c's own append_end_of_stream_marker() establishes. */
|
||||
static void append_end_of_stream_marker(FILE* f)
|
||||
{
|
||||
replay_event_t src;
|
||||
struct capn c;
|
||||
struct capn_segment* cs;
|
||||
ReplayEvent_ptr p;
|
||||
|
||||
memset(&src, 0, sizeof(src));
|
||||
src.body_which = ReplayEvent_body_endSession;
|
||||
|
||||
capn_init_malloc(&c);
|
||||
cs = capn_root(&c).seg;
|
||||
encode_ReplayEvent_ptr(cs, &p, &src);
|
||||
capn_setp(capn_root(&c), 0, p.p);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, write_capnp_message(f, &c));
|
||||
capn_free(&c);
|
||||
}
|
||||
|
||||
/* Appends one ReplaySession carrying exactly one recorded PendDecision
|
||||
* (jobid, main_reason) -- the "recorded output" side of this file's own
|
||||
* Observable, built via the real encode_ReplaySession_ptr() codec
|
||||
* (recording_session.c's own real encode call site, not a hand-rolled
|
||||
* byte layout).
|
||||
*/
|
||||
static void append_replay_session_with_pend(FILE* f, uint64_t begin_round,
|
||||
uint64_t jobid, int main_reason)
|
||||
{
|
||||
xce_job_preason_t reason;
|
||||
sched_pend_decision_t pend;
|
||||
replay_decision_t decision;
|
||||
replay_decision_t* decisions[1];
|
||||
replay_session_t session;
|
||||
struct capn c;
|
||||
struct capn_segment* cs;
|
||||
ReplaySession_ptr sp;
|
||||
|
||||
memset(&reason, 0, sizeof(reason));
|
||||
reason.main_reason = main_reason;
|
||||
|
||||
memset(&pend, 0, sizeof(pend));
|
||||
pend.jobid = jobid;
|
||||
pend.reason = &reason;
|
||||
|
||||
memset(&decision, 0, sizeof(decision));
|
||||
decision.ts = begin_round;
|
||||
decision.body_which = ReplayDecision_body_pend;
|
||||
decision.body.pend = &pend;
|
||||
|
||||
decisions[0] = &decision;
|
||||
|
||||
memset(&session, 0, sizeof(session));
|
||||
session.begin_round = begin_round;
|
||||
session.decisions = decisions;
|
||||
session.n_decisions = 1;
|
||||
session.summary = NULL;
|
||||
|
||||
capn_init_malloc(&c);
|
||||
cs = capn_root(&c).seg;
|
||||
encode_ReplaySession_ptr(cs, &sp, &session);
|
||||
capn_setp(capn_root(&c), 0, sp.p);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, write_capnp_message(f, &c));
|
||||
capn_free(&c);
|
||||
}
|
||||
|
||||
/* Runs sched_sim_driver_run() against `path` with --sim-verify set to
|
||||
* `verify`, capturing everything written to stdout during the call (the
|
||||
* same dup()/dup2() technique capture_stdout_around() above uses, inlined
|
||||
* here since sched_sim_driver_run() needs live arguments, not a bare
|
||||
* void(void) callback). */
|
||||
static char* run_sim_and_capture_stdout(const char* path, bool verify,
|
||||
int* out_rc)
|
||||
{
|
||||
sched_sim_args_t args;
|
||||
xce_sched_sim_driver_t sim;
|
||||
char tmp[] = "/tmp/xce_simverify_run_XXXXXX";
|
||||
int fd = mkstemp(tmp);
|
||||
int saved_stdout;
|
||||
off_t len;
|
||||
char* buf;
|
||||
|
||||
TEST_ASSERT_TRUE_MESSAGE(fd >= 0, "mkstemp for stdout capture must "
|
||||
"succeed");
|
||||
|
||||
fflush(stdout);
|
||||
saved_stdout = dup(STDOUT_FILENO);
|
||||
TEST_ASSERT_TRUE_MESSAGE(saved_stdout >= 0, "dup(STDOUT_FILENO) must "
|
||||
"succeed");
|
||||
TEST_ASSERT_TRUE_MESSAGE(dup2(fd, STDOUT_FILENO) >= 0,
|
||||
"dup2 redirect of STDOUT_FILENO must succeed");
|
||||
|
||||
memset(&args, 0, sizeof(args));
|
||||
args.replay_path = path;
|
||||
args.sim_start_ts = SCHED_SIM_START_TS_LATEST;
|
||||
args.sim_speed = 0; /* frozen -- deterministic, no wall-clock wait */
|
||||
args.verify = verify;
|
||||
|
||||
*out_rc = sched_sim_driver_run(&sim, &args);
|
||||
|
||||
fflush(stdout);
|
||||
dup2(saved_stdout, STDOUT_FILENO);
|
||||
close(saved_stdout);
|
||||
|
||||
len = lseek(fd, 0, SEEK_CUR);
|
||||
TEST_ASSERT_TRUE(len >= 0);
|
||||
|
||||
buf = (char*)calloc(1, (size_t)len + 1);
|
||||
TEST_ASSERT_NOT_NULL(buf);
|
||||
TEST_ASSERT_TRUE(lseek(fd, 0, SEEK_SET) == 0);
|
||||
if (len > 0) {
|
||||
ssize_t nread = read(fd, buf, (size_t)len);
|
||||
TEST_ASSERT_TRUE(nread == len);
|
||||
}
|
||||
close(fd);
|
||||
unlink(tmp);
|
||||
|
||||
return buf;
|
||||
}
|
||||
|
||||
/* ---------------------------------------------------------------------- */
|
||||
/* Integration-level tests: full sched_sim_driver_run(), --sim-verify. */
|
||||
/* ---------------------------------------------------------------------- */
|
||||
|
||||
/*
|
||||
* Observable (a): replaying a fixture against its own previously-recorded
|
||||
* output (here: the exact PEND decision the real picker deterministically
|
||||
* produces -- main_reason=0, make_pendreason()'s own fresh-calloc default)
|
||||
* with verification enabled reports ZERO mismatches.
|
||||
*/
|
||||
static void test_verify_against_self_reports_zero_mismatches(void)
|
||||
{
|
||||
char path[PATH_MAX];
|
||||
FILE* f;
|
||||
int rc;
|
||||
char* captured;
|
||||
|
||||
f = write_header_and_snapshot(path, sizeof(path), "_selfmatch");
|
||||
TEST_ASSERT_NOT_NULL(f);
|
||||
|
||||
append_queue_update_event(f, 1000ULL, "normal");
|
||||
append_jobs_push_event(f, 1500ULL, 777ULL, "normal");
|
||||
append_replay_session_with_pend(f, 2000ULL, 777ULL, 0);
|
||||
append_end_of_stream_marker(f);
|
||||
fclose(f);
|
||||
|
||||
captured = run_sim_and_capture_stdout(path, true, &rc);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, rc);
|
||||
TEST_ASSERT_NULL_MESSAGE(
|
||||
strstr(captured, "MISMATCH"),
|
||||
"verifying a fixture against its own previously-recorded decision "
|
||||
"must report zero mismatches");
|
||||
|
||||
free(captured);
|
||||
printf("test_verify_against_self_reports_zero_mismatches: PASS\n");
|
||||
}
|
||||
|
||||
/*
|
||||
* Observable (b): replaying against a deliberately-altered copy (recorded
|
||||
* main_reason flipped to 99, which the real picker thread can never
|
||||
* produce -- make_pendreason() always calloc()s a fresh, all-zero
|
||||
* xce_job_preason_t) reports the EXACT mismatch: MISMATCH is logged,
|
||||
* naming the specific job.
|
||||
*/
|
||||
static void test_verify_against_altered_copy_reports_exact_mismatch(void)
|
||||
{
|
||||
char path[PATH_MAX];
|
||||
FILE* f;
|
||||
int rc;
|
||||
char* captured;
|
||||
|
||||
f = write_header_and_snapshot(path, sizeof(path), "_altered");
|
||||
TEST_ASSERT_NOT_NULL(f);
|
||||
|
||||
append_queue_update_event(f, 1000ULL, "normal");
|
||||
append_jobs_push_event(f, 1500ULL, 777ULL, "normal");
|
||||
append_replay_session_with_pend(f, 2000ULL, 777ULL, 99);
|
||||
append_end_of_stream_marker(f);
|
||||
fclose(f);
|
||||
|
||||
captured = run_sim_and_capture_stdout(path, true, &rc);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, rc);
|
||||
TEST_ASSERT_NOT_NULL_MESSAGE(
|
||||
strstr(captured, "MISMATCH"),
|
||||
"verifying against a deliberately-altered recorded decision must "
|
||||
"report a mismatch");
|
||||
TEST_ASSERT_NOT_NULL_MESSAGE(
|
||||
strstr(captured, "jobid=777"),
|
||||
"the mismatch report must name the specific job");
|
||||
|
||||
free(captured);
|
||||
printf("test_verify_against_altered_copy_reports_exact_mismatch: "
|
||||
"PASS\n");
|
||||
}
|
||||
|
||||
/*
|
||||
* sim->verify == false must skip verification entirely, even against the
|
||||
* SAME deliberately-altered fixture the previous test proves DOES mismatch
|
||||
* when verification is on -- proves the --sim-verify gate itself, not just
|
||||
* the comparator.
|
||||
*/
|
||||
static void test_verify_disabled_reports_no_mismatches_even_when_altered(void)
|
||||
{
|
||||
char path[PATH_MAX];
|
||||
FILE* f;
|
||||
int rc;
|
||||
char* captured;
|
||||
|
||||
f = write_header_and_snapshot(path, sizeof(path), "_verifyoff");
|
||||
TEST_ASSERT_NOT_NULL(f);
|
||||
|
||||
append_queue_update_event(f, 1000ULL, "normal");
|
||||
append_jobs_push_event(f, 1500ULL, 777ULL, "normal");
|
||||
append_replay_session_with_pend(f, 2000ULL, 777ULL, 99);
|
||||
append_end_of_stream_marker(f);
|
||||
fclose(f);
|
||||
|
||||
captured = run_sim_and_capture_stdout(path, false, &rc);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, rc);
|
||||
TEST_ASSERT_NULL_MESSAGE(
|
||||
strstr(captured, "MISMATCH"),
|
||||
"sim->verify == false must never report a mismatch, even against a "
|
||||
"fixture the verify-enabled test proves mismatches");
|
||||
|
||||
free(captured);
|
||||
printf("test_verify_disabled_reports_no_mismatches_even_when_altered: "
|
||||
"PASS\n");
|
||||
}
|
||||
|
||||
int main(void)
|
||||
{
|
||||
UNITY_BEGIN();
|
||||
|
||||
RUN_TEST(test_dispatch_decisions_with_identical_content_match);
|
||||
RUN_TEST(test_dispatch_decisions_differing_jobid_mismatch);
|
||||
RUN_TEST(test_dispatch_decisions_differing_host_mismatch);
|
||||
RUN_TEST(test_dispatch_decisions_differing_resource_value_mismatch);
|
||||
RUN_TEST(test_pend_decisions_with_identical_content_match);
|
||||
RUN_TEST(test_pend_decisions_differing_jobid_mismatch);
|
||||
RUN_TEST(test_pend_decisions_differing_main_reason_mismatch);
|
||||
RUN_TEST(test_decision_kind_mismatch_reported);
|
||||
RUN_TEST(test_preempt_decision_always_reports_mismatch);
|
||||
RUN_TEST(test_decision_eq_defends_against_null_inputs);
|
||||
RUN_TEST(test_log_mismatch_reports_logical_time_and_decision_detail);
|
||||
RUN_TEST(test_verify_against_self_reports_zero_mismatches);
|
||||
RUN_TEST(test_verify_against_altered_copy_reports_exact_mismatch);
|
||||
RUN_TEST(test_verify_disabled_reports_no_mismatches_even_when_altered);
|
||||
|
||||
return UNITY_END();
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue