lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/src/scheduler/testsuite/test_record_lifecycle.c
Rongsong Shen 320f88a432
feat(scheduler-sim-decision-trace): compile trace sources into white-box tests of hooked files 
Task 1.6: the 27 tests that #include a hooked scheduler source now also
include sched_trace.c and sched_trace_capture.c.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 08:29:56 +00:00

1564 lines
60 KiB
C
Raw Blame History

/*
* test_record_lifecycle.c - contract test for the record start/stop
* lifecycle, the driver-loop recording-write hooks, periodic snapshot
* checkpointing, and recording error handling (scheduler-core spec, tasks
* 9.2/9.3, Req 5 AC 10/AC 11).
*
* Observable under test (task 9.2's own text):
* - start -> the file begins with a decodable ReplayHeader then a
* decodable SnapshotEntry.
* - start -> submit an event batch -> complete one session -> stop
* produces a file with events and exactly one ReplaySession in
* between, ending in a ReplayEvent{endSession} marker.
* - when `sched->recording` is false throughout, no file I/O occurs.
* - double-start fails cleanly; stop-when-not-recording fails cleanly.
* - a TOCTOU-style symlink/hardlink swap between path validation and the
* eventual open() is rejected, not silently followed.
*
* Observable under test (task 9.3's own text, added this revision):
* - recording through a configured `record_snapshot_interval` produces
* the expected NUMBER of extra periodic SnapshotEntry messages in the
* file (interval=3, 7 events across 7 calls -> 2 extra checkpoints;
* interval=0 -> zero extra checkpoints regardless of event count; one
* large single-call batch crossing two interval multiples -> exactly
* ONE checkpoint, proving/documenting this implementation's "once per
* call" boundary choice -- see recording_events.c's own comment on
* sched_record_write_events() for the full rationale).
* - simulating a write failure leaves the scheduler still processing
* sessions afterward with recording cleanly disabled: a genuine fd-level
* write failure (not a simulated return code -- see the fflush()+
* setvbuf(_IONBF)+close(fd) sequence in
* test_write_failure_disables_recording_and_driver_keeps_running(), and
* .wolf/cerebrum.md's matching Key Learning for why plain fd-closing
* alone is not sufficient against a buffered stdio stream) triggers
* sched_record_abort_on_error()'s cleanup (recording=false,
* record_file/record_file_path both NULL), then a full real
* normal-mode round still completes with no crash/hang and no further
* recording I/O attempted.
*
* White-box #include of ../src/core.c, ../src/core_clock.c,
* ../src/core_events.c, ../src/core_session.c, ../src/driver.c,
* ../src/plugin.c (same file set/rationale as
* test_driver_session_ordering.c -- a real picker pthread + stand-in
* allocator thread is needed to drive xce_sched_driver_run_normal_mode_round()
* through one genuine session, jbuf's hardcoded capacity-1 rendezvous
* requires it) PLUS ../src/recording.c, ../src/recording_events.c,
* ../src/recording_session.c (the code under test) PLUS ../src/snapshot.c,
* ../src/snapshot_wire.c (task 8.1's REAL xce_sched_core_save_snapshot(),
* needed by sched_record_start() -- NOT snapshot_load.c, which nothing
* here ever calls). No jobs are ever pushed (same "picker produces zero
* decisions" fixture shape as test_driver_session_ordering.c), so
* decision.c is not needed either.
*
* Decoding: this file reads its own produced `.replay` files back with the
* SAME generated read_X()/decode_X() functions (controller.capnp.h) group
* 7's own round-trip tests (test_e2e_capnp.c) use, via capn_init_fp()
* (sequential per-message stream reads, since a `.replay` file is
* consecutive self-framed capnp messages with no outer framing, Req 5
* AC 20) -- never by hand-parsing bytes.
*
* Stubs below are the exact same set test_driver_session_ordering.c
* already establishes for this core.c/core_clock.c/core_events.c/
* core_session.c/driver.c/plugin.c combination (kill_sched_*,
* sched_worker_release_host_ctx, sched_publish_dispatch/
* sched_publish_pend_reason, sched_fill_limit_requests,
* sched_resolve_job_picker's real reimplementation) -- see that file's own
* header comment for the full rationale. plugin_unregister is NOT stubbed
* (real, from the genuinely-linked plugin.c). No stub is needed for
* xce_core_now()/xce_sched_core_save_snapshot() here (unlike
* test_recording_path_validation.c's narrower TU): core_clock.c and
* snapshot.c/snapshot_wire.c are genuinely linked into this file.
*/
#include <errno.h>
#include <fcntl.h>
#include <limits.h>
#include <pthread.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <unistd.h>
#include <unity/unity.h>
#include "controller.capnp.h"
#include "kvmgr.h"
#include "preason.h"
#include "scheduler.h"
#include "xstring.h"
/* Pull in the real functions under test into this TU (white-box
* #include, same convention as test_driver_session_ordering.c). */
#include "../src/core.c"
#include "../src/core_clock.c"
#include "../src/core_events.c"
#include "../src/core_session.c"
#include "../src/driver.c"
/* Hooked sources compile SCHED_TRACE bodies (scheduler-sim-decision-trace). */
#include "../src/sched_trace.c"
#include "../src/sched_trace_capture.c"
#include "../src/plugin.c"
#include "../src/recording.c"
#include "../src/recording_events.c"
#include "../src/recording_session.c"
#include "../src/snapshot.c"
#include "../src/snapshot_wire.c"
void kill_sched_job(void* arg) { (void)arg; }
void kill_sched_queue(void* arg) { (void)arg; }
void kill_sched_worker(void* arg) { (void)arg; }
void kill_sched_user(void* arg) { (void)arg; }
void kill_sched_project(void* arg) { (void)arg; }
void kill_sched_resval(xce_val_t* val) { (void)val; }
/* core_reset_workers()'s (core_session.c) linker dependency -- never
* invoked (no worker is ever added here). */
void sched_worker_release_host_ctx(xce_sched_worker_t* w) { (void)w; }
/* resource-accounting.md's reconciliation (resource.c, not linked here --
* this file only #includes core_events.c). core_events.c's
* XCE_CORE_EVENT_JOB_RUSAGE handler calls this whenever job->accounts !=
* NULL; this fixture never populates job->accounts, so the real body is
* never reached -- link-satisfying stub only, same rationale as every
* other stub in this block (see test_job_update.c's identical stub). */
int sched_reconcile_job_resource_usage(xce_sched_core_t* c,
xce_job_account_t* a,
xce_job_rusages_t* r)
{
(void)c; (void)a; (void)r; return 0;
}
/* Link-satisfying stubs for scheduler.c's sched_publish_dispatch()/
* sched_publish_pend_reason() -- never actually invoked (this fixture
* pushes no jobs, so the driver's decision-drain loop never observes a
* non-NULL decision). */
int sched_publish_dispatch(xce_scheduler_t* sched, xce_job_dispatch_t* d)
{
(void)sched;
(void)d;
return 0;
}
int sched_publish_pend_reason(xce_scheduler_t* sched, xce_preason_set_t* r)
{
(void)sched;
(void)r;
return 0;
}
/* Link-satisfying stub for resource.c's sched_fill_limit_requests() --
* plugin.c's plugin_prepare_allocation() references it; never actually
* invoked here. */
int sched_fill_limit_requests(xce_sched_alloc_req_t* alloc_req,
xlist_t* limits, xlist_t* rusages, int nhosts,
int nslots)
{
(void)alloc_req;
(void)limits;
(void)rusages;
(void)nhosts;
(void)nslots;
return 0;
}
int sched_account_jobs(xce_sched_core_t* core)
{
(void)core;
return 0;
}
/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision
* Reconciliation capacity gate): link-satisfying stubs, mirroring this
* file's existing sched_account_jobs() no-op precedent immediately above --
* sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls
* sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to
* re-validate a recorded decision against live capacity, and
* sched_charge_account() (resource.c) to keep that check's own view of
* capacity up to date across multiple candidates in the same round. This
* file never exercises real resource contention (same rationale as the
* existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so
* the satisfy checks trivially always succeed and the charge call is a
* no-op. */
int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w,
xvec_t* uncomitted, xvec_idval_t* wanted,
xvec_idval_t* unmeet)
{
(void)core;
(void)w;
(void)uncomitted;
(void)wanted;
(void)unmeet;
return 1;
}
int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted,
xvec_idval_t* wanted, xvec_idval_t* unmeet)
{
(void)core;
(void)uncomitted;
(void)wanted;
(void)unmeet;
return 1;
}
int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct)
{
(void)core;
(void)acct;
return 0;
}
int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct)
{
(void)core;
(void)acct;
return 0;
}
/* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same
* rationale/algorithm as test_driver_session_ordering.c's own copy
* (schedobj.c is not linked). */
int sched_resolve_job_picker(xce_sched_core_t* core, xce_sched_plugin_t** out)
{
xce_sched_plugin_t* plugin = NULL;
int rc = 0;
if ((core == NULL) || (out == NULL)) {
return -EINVAL;
}
*out = NULL;
if (core->job_pick_plugin == NULL) {
return -EINVAL;
}
lock(core->plugins, RTTI_LOCK_READ);
rc = key_find(core->plugins, core->job_pick_plugin, (void**)&plugin);
unlock(core->plugins);
if (rc <= 0) {
return (rc < 0) ? rc : -ENOENT;
}
if ((plugin == NULL) || (plugin->picker_iface == NULL)
|| (plugin->picker_iface->next_job == NULL)
|| (plugin->picker_iface->begin_session == NULL)
|| (plugin->picker_iface->end_session == NULL)
|| (plugin->picker_iface->on_dispatch == NULL)
|| (plugin->picker_iface->on_pend == NULL)) {
return -EINVAL;
}
*out = plugin;
return 0;
}
void setUp(void) { return; }
void tearDown(void) { return; }
/* ---------------------------------------------------------------------- */
/* Stub picker plugin -- identical shape to */
/* test_driver_session_ordering.c's own: no jobs are ever pushed, so */
/* next_job() always returns NULL immediately. Its SOLE purpose is proving */
/* the wbuf/jbuf rendezvous timing lets a real session complete. */
/* ---------------------------------------------------------------------- */
static int stub_picker_begin_session(void* picker)
{
(void)picker;
return 0;
}
static xce_sched_job_t* stub_picker_next_job(void* picker)
{
(void)picker;
return NULL;
}
static int stub_picker_end_session(void* picker)
{
(void)picker;
return 0;
}
static int stub_picker_jobid_noop(void* picker, uint64_t jobid)
{
(void)picker;
(void)jobid;
return 0;
}
static int stub_picker_queue_update_noop(void* picker, xce_sched_queue_t* q)
{
(void)picker;
(void)q;
return 0;
}
static int stub_picker_job_noop(void* picker, xce_sched_job_t* job)
{
(void)picker;
(void)job;
return 0;
}
static int stub_picker_job_state_change_noop(void* picker, xce_sched_job_t* job,
uint32_t state)
{
(void)picker;
(void)job;
(void)state;
return 0;
}
static xce_job_picker_iface_t g_stub_picker_iface
= { .init = NULL,
.fini = NULL,
.begin_session = stub_picker_begin_session,
.next_job = stub_picker_next_job,
.end_session = stub_picker_end_session,
.on_dispatch = stub_picker_jobid_noop,
.on_pend = stub_picker_jobid_noop,
.on_queue_update = stub_picker_queue_update_noop,
.on_job_push = stub_picker_job_noop,
.on_job_state_change = stub_picker_job_state_change_noop,
.on_job_update = stub_picker_job_noop,
.on_config = NULL,
.on_job_rusage = NULL };
static xce_sched_plugin_t g_stub_picker_plugin
= { .name = "test_record_lifecycle_picker",
.picker_iface = &g_stub_picker_iface };
static void* picker_thread_fn(void* arg)
{
xce_sched_core_t* core = (xce_sched_core_t*)arg;
xce_sched_plugin_t* job_picker = NULL;
int rc;
int go = 0;
uint64_t jobid;
rc = sched_resolve_job_picker(core, &job_picker);
if (rc < 0) {
return NULL;
}
cbuf_get(&(core->wbuf), &go);
job_picker->picker_iface->begin_session(job_picker);
for (;;) {
xce_sched_job_t* job = job_picker->picker_iface->next_job(job_picker);
if (job == NULL) {
break;
}
jobid = job->jobid;
cbuf_put(&(core->jbuf), &jobid);
}
jobid = 0;
cbuf_put(&(core->jbuf), &jobid);
job_picker->picker_iface->end_session(job_picker);
return NULL;
}
static void* stand_in_allocator_thread_fn(void* arg)
{
xce_sched_core_t* core = (xce_sched_core_t*)arg;
uint64_t jobid;
xce_decision_t* decision = NULL;
cbuf_get(&(core->jbuf), &jobid);
(void)jobid;
cbuf_put(&(core->decisions), &decision);
return NULL;
}
/* ---------------------------------------------------------------------- */
/* bug-1843 fault injection: -Wl,--wrap=realloc (meson.build, this target */
/* only) redirects every `realloc` reference in this executable -- */
/* including inside the white-box-#included recording_session.c above -- */
/* to __wrap_realloc() below. Disabled (passes straight through to the */
/* real realloc() via __real_realloc(), the symbol `ld --wrap` reserves */
/* for it) unless a test arms it by setting g_realloc_fail_after; the one */
/* test that does always disarms it again immediately after its one target */
/* call, so the other tests in this binary see completely normal */
/* allocator behavior. */
/* ---------------------------------------------------------------------- */
extern void* __real_realloc(void* ptr, size_t size);
static int g_realloc_fail_after = -1; /* -1 == disabled */
static int g_realloc_call_count = 0;
void* __wrap_realloc(void* ptr, size_t size)
{
if (g_realloc_fail_after >= 0) {
int this_call = g_realloc_call_count++;
if (this_call == g_realloc_fail_after) {
return NULL;
}
}
return __real_realloc(ptr, size);
}
/* ---------------------------------------------------------------------- */
/* Fixture helpers */
/* ---------------------------------------------------------------------- */
static void make_tmp_dir(char* out, size_t out_len, const char* suffix)
{
char tmpl[PATH_MAX];
snprintf(tmpl, sizeof(tmpl), "/tmp/sched_reclife%s_XXXXXX", suffix);
TEST_ASSERT_NOT_NULL(mkdtemp(tmpl));
TEST_ASSERT_TRUE(strlen(tmpl) < out_len);
strcpy(out, tmpl);
}
/* Real xce_sched_core_init() + manually dataq_init'd pend_updates + the
* stub picker plugin registered -- same pattern as
* test_driver_session_ordering.c's fixture. `record_dir` must outlive
* `sched` (caller-owned storage). Non-const to match
* xce_scheduler_t.record_dir's own `char*` type (scheduler.h) -- every
* caller passes a mutable local buffer anyway. */
static void fixture_init(xce_scheduler_t* sched, char* record_dir)
{
int rc;
memset(sched, 0, sizeof(*sched));
/* record_lock (scheduler.h field comments, task 9.4 fix round) is
* normally initialized by sched_init(), which this white-box fixture
* deliberately bypasses (same rationale as this function's own header
* comment) -- initialize it here too, same pairing as the
* xce_sched_core_init() call just below initializing core.lock. Every
* sched_record_start()/sched_record_stop() call in this file now takes
* this lock. */
mylock_init(&(sched->record_lock));
rc = xce_sched_core_init(&(sched->core));
TEST_ASSERT_EQUAL_INT(0, rc);
sched->core.pend_updates[0] = (dataq_t*)calloc(1, sizeof(dataq_t));
dataq_init(sched->core.pend_updates[0], NULL);
sched->core.pend_updates[1] = (dataq_t*)calloc(1, sizeof(dataq_t));
dataq_init(sched->core.pend_updates[1], NULL);
lock(sched->core.plugins, RTTI_LOCK_WRITE);
rc = key_add(sched->core.plugins, (void*)g_stub_picker_plugin.name,
(void*)&g_stub_picker_plugin);
TEST_ASSERT_GREATER_OR_EQUAL(0, rc);
unlock(sched->core.plugins);
sched->core.job_pick_plugin = g_stub_picker_plugin.name;
sched->record_dir = record_dir;
}
/* Runs one complete normal-mode round (real picker thread + stand-in
* allocator thread, same design as
* test_driver_session_ordering.c's own single test) and blocks until it
* completes. */
static void run_one_real_session(xce_scheduler_t* sched)
{
pthread_t picker_tid;
pthread_t allocator_tid;
int rc;
rc = pthread_create(&picker_tid, NULL, picker_thread_fn, &(sched->core));
TEST_ASSERT_EQUAL_INT(0, rc);
rc = pthread_create(&allocator_tid, NULL, stand_in_allocator_thread_fn,
&(sched->core));
TEST_ASSERT_EQUAL_INT(0, rc);
xce_sched_driver_run_normal_mode_round(sched);
rc = pthread_join(picker_tid, NULL);
TEST_ASSERT_EQUAL_INT(0, rc);
rc = pthread_join(allocator_tid, NULL);
TEST_ASSERT_EQUAL_INT(0, rc);
}
/* ---------------------------------------------------------------------- */
/* Test 1: start() writes a decodable ReplayHeader then a decodable */
/* SnapshotEntry -- this task's own Observable, "byte-decodable by the */
/* round-trip decoder from group 7". */
/* ---------------------------------------------------------------------- */
static void test_start_writes_decodable_header_then_snapshot_entry(void)
{
char record_dir[PATH_MAX];
xce_scheduler_t sched;
char path_copy[PATH_MAX];
int rc;
FILE* f;
struct capn c1;
struct capn c2;
ReplayHeader_ptr hdr;
SnapshotEntry_ptr snap;
capn_text rv;
make_tmp_dir(record_dir, sizeof(record_dir), "_hdr");
fixture_init(&sched, record_dir);
rc = sched_record_start(&sched, "session.replay");
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_TRUE(sched.recording);
TEST_ASSERT_NOT_NULL(sched.record_file_path);
TEST_ASSERT_NOT_NULL(sched.record_file);
strcpy(path_copy, sched.record_file_path);
rc = sched_record_stop(&sched);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_FALSE(sched.recording);
f = fopen(path_copy, "r");
TEST_ASSERT_NOT_NULL(f);
/* msg 1: ReplayHeader */
TEST_ASSERT_EQUAL_INT(0, capn_init_fp(&c1, f, 0));
hdr.p = capn_getp(capn_root(&c1), 0, 1);
rv = ReplayHeader_get_recorderVersion(hdr);
TEST_ASSERT_EQUAL_STRING("1.0", rv.str);
capn_free(&c1);
/* msg 2: SnapshotEntry -- MUST be the first data message after
* ReplayHeader (Req 5 AC 20). */
TEST_ASSERT_EQUAL_INT(0, capn_init_fp(&c2, f, 0));
snap.p = capn_getp(capn_root(&c2), 0, 1);
{
SnapshotData_ptr sdata = SnapshotEntry_get_data(snap);
TEST_ASSERT_FALSE(sdata.p.type == CAPN_NULL);
}
capn_free(&c2);
fclose(f);
printf("test_start_writes_decodable_header_then_snapshot_entry: PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 2: start -> submit an event batch -> complete one session -> stop */
/* produces events + exactly one ReplaySession + a trailing endSession */
/* marker -- this task's own literal Observable. */
/* ---------------------------------------------------------------------- */
static void
test_full_cycle_start_event_session_stop_produces_expected_stream(void)
{
char record_dir[PATH_MAX];
xce_scheduler_t sched;
xce_sched_update_request_t* req;
char path_copy[PATH_MAX];
int rc;
FILE* f;
struct capn c;
make_tmp_dir(record_dir, sizeof(record_dir), "_cycle");
fixture_init(&sched, record_dir);
rc = sched_record_start(&sched, "cycle.replay");
TEST_ASSERT_EQUAL_INT(0, rc);
strcpy(path_copy, sched.record_file_path);
/* "submit an event batch": one JSTATE_CHANGE update, drained by
* xce_sched_driver_translate_pending_events() (called from
* xce_sched_driver_run_normal_mode_round() below) -- the SAME
* pend_updates dataq mechanism handler.c populates in production. */
req = (xce_sched_update_request_t*)calloc(1, sizeof(*req));
req->op = XCE_SCHED_OP_JSTATE_CHANGE;
req->u.job_state_change.jobid = 42ULL;
req->u.job_state_change.state = 7u;
dataq_enq(sched.core.pend_updates[0], &(req->link));
/* "complete a session": one full normal-mode round through a real
* picker thread + stand-in allocator thread (no jobs pushed, so the
* session produces zero decisions -- still exactly one ReplaySession
* message, matching this task's Observable which requires "one
* session in between", not any particular decision count). */
run_one_real_session(&sched);
TEST_ASSERT_EQUAL_UINT64(1ULL, sched.record_event_count);
rc = sched_record_stop(&sched);
TEST_ASSERT_EQUAL_INT(0, rc);
f = fopen(path_copy, "r");
TEST_ASSERT_NOT_NULL(f);
/* msg 1: ReplayHeader */
TEST_ASSERT_EQUAL_INT(0, capn_init_fp(&c, f, 0));
capn_free(&c);
/* msg 2: SnapshotEntry (initial) */
TEST_ASSERT_EQUAL_INT(0, capn_init_fp(&c, f, 0));
capn_free(&c);
/* msg 3: ReplayEvent{jobStateChange, jobid=42} -- the submitted batch. */
{
ReplayEvent_ptr p;
struct ReplayEvent wire;
replay_event_t dst;
TEST_ASSERT_EQUAL_INT(0, capn_init_fp(&c, f, 0));
p.p = capn_getp(capn_root(&c), 0, 1);
read_ReplayEvent(&wire, p);
memset(&dst, 0, sizeof(dst));
decode_ReplayEvent(&dst, &wire);
TEST_ASSERT_EQUAL_INT((int)ReplayEvent_body_jobStateChange,
(int)dst.body_which);
TEST_ASSERT_NOT_NULL(dst.body.job_state_change);
TEST_ASSERT_EQUAL_UINT64(42ULL, dst.body.job_state_change->jobid);
free_ReplayEvent(&dst);
capn_free(&c);
}
/* msg 4: exactly one ReplaySession. */
{
ReplaySession_ptr p;
struct ReplaySession wire;
TEST_ASSERT_EQUAL_INT(0, capn_init_fp(&c, f, 0));
p.p = capn_getp(capn_root(&c), 0, 1);
read_ReplaySession(&wire, p);
(void)wire; /* begin_round/decisions checked structurally only --
* this fixture pushes no jobs, so 0 decisions is
* correct and expected. */
capn_free(&c);
}
/* msg 5: the end-of-recording marker, ReplayEvent{endSession}. */
{
ReplayEvent_ptr p;
struct ReplayEvent wire;
replay_event_t dst;
TEST_ASSERT_EQUAL_INT(0, capn_init_fp(&c, f, 0));
p.p = capn_getp(capn_root(&c), 0, 1);
read_ReplayEvent(&wire, p);
memset(&dst, 0, sizeof(dst));
decode_ReplayEvent(&dst, &wire);
TEST_ASSERT_EQUAL_INT((int)ReplayEvent_body_endSession,
(int)dst.body_which);
free_ReplayEvent(&dst);
capn_free(&c);
}
/* Exactly 5 messages -- nothing more to read. */
TEST_ASSERT_NOT_EQUAL(0, capn_init_fp(&c, f, 0));
fclose(f);
printf("test_full_cycle_start_event_session_stop_produces_expected_stream:"
" PASS\n");
}
/* ---------------------------------------------------------------------- */
/* bug-1852: a recorded event must carry its payload as RECEIVED, not what */
/* is left of it after the core applied it. core_apply_worker_resvals() */
/* MOVE_PTRs each resval out of the WORKERS_PUSH payload; recording after */
/* xce_core_push_events() wrote the worker's resval names with NULL values, */
/* so a replay never learned the worker's capacity (live soak: a recording */
/* opened before the worker's first value push replayed as "no slots" -- */
/* 3 live DISPATCHes became sim PENDs). */
/* ---------------------------------------------------------------------- */
static void skip_header_and_initial_snapshot(FILE* f); /* defined below */
static void test_recorded_workers_push_keeps_resval_values(void)
{
char record_dir[PATH_MAX];
xce_scheduler_t sched;
char path_copy[PATH_MAX];
xce_sched_update_request_t* rr;
xce_sched_update_request_t* rw;
xce_resource_t* wire_res;
xce_resource_t** res_ptrs;
xce_worker_t* wire_w;
xce_worker_t** w_ptrs;
xce_name_val_t* nv;
int rc;
int found = 0;
FILE* f;
struct capn c;
make_tmp_dir(record_dir, sizeof(record_dir), "_1852");
fixture_init(&sched, record_dir);
rc = sched_record_start(&sched, "workers.replay");
TEST_ASSERT_EQUAL_INT(0, rc);
strcpy(path_copy, sched.record_file_path);
/* Heap payloads throughout: release_translated_payload() frees them
* like genuinely wire-decoded ones (test_resval_ownership.c's own
* warning about stack/literal payloads applies verbatim). */
wire_res = (xce_resource_t*)calloc(1, sizeof(*wire_res));
wire_res->name = strdup("slots");
res_ptrs = (xce_resource_t**)calloc(1, sizeof(*res_ptrs));
res_ptrs[0] = wire_res;
nv = (xce_name_val_t*)calloc(1, sizeof(*nv));
nv->name = strdup("slots");
nv->val = (xce_val_t*)calloc(1, sizeof(*nv->val));
nv->val->which = Value_u_fvalue;
nv->val->u.fvalue = 22.0;
wire_w = (xce_worker_t*)calloc(1, sizeof(*wire_w));
wire_w->name = strdup("hostA");
wire_w->nresvals_to_sched = 1;
wire_w->resvals_to_sched = (xce_name_val_t**)calloc(1, sizeof(nv));
wire_w->resvals_to_sched[0] = nv;
w_ptrs = (xce_worker_t**)calloc(1, sizeof(*w_ptrs));
w_ptrs[0] = wire_w;
/* Resource definition first, so the core resolves "slots" and actually
* adopts (moves) the worker's value -- the path that emptied the
* recorded payload. */
rr = (xce_sched_update_request_t*)calloc(1, sizeof(*rr));
rr->op = XCE_SCHED_OP_UPDATE_RESOURCES;
rr->u.resourceupdate.nresources = 1;
rr->u.resourceupdate.resources = res_ptrs;
dataq_enq(sched.core.pend_updates[0], &(rr->link));
rw = (xce_sched_update_request_t*)calloc(1, sizeof(*rw));
rw->op = XCE_SCHED_OP_UPDATE_WORKER;
rw->u.workerupdate.nworkers = 1;
rw->u.workerupdate.workers = w_ptrs;
dataq_enq(sched.core.pend_updates[0], &(rw->link));
rc = xce_sched_driver_translate_pending_events(&(sched.core), &sched);
TEST_ASSERT_EQUAL_INT(2, rc);
/* Precondition: the live core did adopt the value. */
{
xce_sched_worker_t* w = sched_get_worker(&(sched.core), "hostA");
xce_resource_t* r = NULL;
TEST_ASSERT_NOT_NULL(w);
TEST_ASSERT_TRUE(key_find(sched.core.resmgr, "slots", (void**)&r) > 0);
TEST_ASSERT_NOT_NULL(zarray_get(&(w->resvals), r->idx));
}
rc = sched_record_stop(&sched);
TEST_ASSERT_EQUAL_INT(0, rc);
f = fopen(path_copy, "r");
TEST_ASSERT_NOT_NULL(f);
skip_header_and_initial_snapshot(f);
while (capn_init_fp(&c, f, 0) == 0) {
ReplayEvent_ptr p;
struct ReplayEvent wire;
replay_event_t dst;
p.p = capn_getp(capn_root(&c), 0, 1);
read_ReplayEvent(&wire, p);
memset(&dst, 0, sizeof(dst));
decode_ReplayEvent(&dst, &wire);
if (dst.body_which == ReplayEvent_body_workersPush) {
xce_worker_t* rw0;
xce_name_val_t* rnv;
found = 1;
TEST_ASSERT_NOT_NULL(dst.body.workers_push);
TEST_ASSERT_EQUAL_INT(1, dst.body.workers_push->count);
rw0 = dst.body.workers_push->workers[0];
TEST_ASSERT_EQUAL_STRING("hostA", rw0->name);
TEST_ASSERT_EQUAL_INT(1, rw0->nresvals_to_sched);
rnv = rw0->resvals_to_sched[0];
TEST_ASSERT_EQUAL_STRING("slots", rnv->name);
TEST_ASSERT_NOT_NULL_MESSAGE(rnv->val,
"the recorded WORKERS_PUSH must keep the resval's value, "
"not the NULL the core's MOVE_PTR left behind (bug-1852)");
TEST_ASSERT_EQUAL_FLOAT(22.0f, (float)rnv->val->u.fvalue);
}
free_ReplayEvent(&dst);
capn_free(&c);
}
fclose(f);
TEST_ASSERT_TRUE_MESSAGE(found, "a WORKERS_PUSH event must be recorded");
printf("test_recorded_workers_push_keeps_resval_values: PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 3: when recording is inactive, the driver-loop hooks touch no */
/* file/record state -- zero file I/O (Req 5 AC 10, "Performance"). */
/* ---------------------------------------------------------------------- */
static void test_recording_inactive_hooks_are_zero_cost_noops(void)
{
char record_dir[PATH_MAX];
xce_scheduler_t sched;
xce_core_event_t events[1];
sched_record_session_ctx_t* ctx;
int rc;
make_tmp_dir(record_dir, sizeof(record_dir), "_inactive");
fixture_init(&sched, record_dir);
/* sched.recording deliberately never set true. */
memset(events, 0, sizeof(events));
events[0].type = XCE_CORE_EVENT_JOB_STATE_CHANGE;
events[0].u.job_state_change.jobid = 99ULL;
rc = sched_record_write_events(&sched, events, 1);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_NULL(sched.record_file);
TEST_ASSERT_NULL(sched.record_file_path);
ctx = sched_record_session_begin(&sched, 123ULL);
TEST_ASSERT_NULL(ctx);
TEST_ASSERT_NULL(sched.record_file);
/* Driving a full real round with recording inactive must not create
* any file either. */
run_one_real_session(&sched);
TEST_ASSERT_FALSE(sched.recording);
TEST_ASSERT_NULL(sched.record_file);
TEST_ASSERT_NULL(sched.record_file_path);
TEST_ASSERT_EQUAL_UINT64(0ULL, sched.record_event_count);
}
/* ---------------------------------------------------------------------- */
/* Test 4: double-start / stop-when-not-recording fail cleanly. */
/* ---------------------------------------------------------------------- */
static void test_double_start_fails_cleanly(void)
{
char record_dir[PATH_MAX];
xce_scheduler_t sched;
int rc;
make_tmp_dir(record_dir, sizeof(record_dir), "_dblstart");
fixture_init(&sched, record_dir);
rc = sched_record_start(&sched, "a.replay");
TEST_ASSERT_EQUAL_INT(0, rc);
rc = sched_record_start(&sched, "b.replay");
TEST_ASSERT_EQUAL_INT(-EALREADY, rc);
/* Original recording untouched by the rejected second start. */
TEST_ASSERT_TRUE(sched.recording);
TEST_ASSERT_TRUE(strstr(sched.record_file_path, "a.replay") != NULL);
rc = sched_record_stop(&sched);
TEST_ASSERT_EQUAL_INT(0, rc);
}
static void test_stop_when_not_recording_fails_cleanly(void)
{
char record_dir[PATH_MAX];
xce_scheduler_t sched;
int rc;
make_tmp_dir(record_dir, sizeof(record_dir), "_nostart");
fixture_init(&sched, record_dir);
rc = sched_record_stop(&sched);
TEST_ASSERT_EQUAL_INT(-EALREADY, rc);
rc = sched_record_start(&sched, "c.replay");
TEST_ASSERT_EQUAL_INT(0, rc);
rc = sched_record_stop(&sched);
TEST_ASSERT_EQUAL_INT(0, rc);
/* Stopping again after a clean stop also fails cleanly. */
rc = sched_record_stop(&sched);
TEST_ASSERT_EQUAL_INT(-EALREADY, rc);
}
static void test_null_sched_returns_einval(void)
{
TEST_ASSERT_EQUAL_INT(-EINVAL, sched_record_start(NULL, "x.replay"));
TEST_ASSERT_EQUAL_INT(-EINVAL, sched_record_stop(NULL));
}
/* ---------------------------------------------------------------------- */
/* Test 5: TOCTOU -- a symlink/hard link swapped in at the resolved path */
/* between path validation and the eventual open() is rejected, not */
/* silently followed (recording.h's own documented mandate for this task). */
/* */
/* Genuinely racing a background thread against the exact few-instruction */
/* window inside sched_record_start() between its internal */
/* sched_validate_record_filename() call and its open() call would be */
/* inherently flaky (no test hook exists to pause mid-function, nor should */
/* one be added to production code just for this). Instead, both tests */
/* below reproduce the STATE a genuine race would produce -- a malicious */
/* symlink/hard link occupying the target path at the moment */
/* sched_record_start() is invoked, planted via real symlink(2)/link(2) */
/* syscalls (no mocking) -- and prove the full end-to-end call rejects it, */
/* AND (the part that specifically proves the open()-time hardening this */
/* task added, not just sched_validate_record_filename()'s pre-existing */
/* containment check) that the exact open()+fstat()+st_nlink==1 sequence */
/* sched_record_start() runs internally -- reproduced here verbatim, same */
/* flags -- independently rejects the swap. Either layer alone would be */
/* enough in production; proving the second layer here is what closes the */
/* TOCTOU gap this task's own docstring (recording.h) flags: a real race */
/* would land AFTER sched_validate_record_filename() already accepted the */
/* path, i.e. exactly the scenario where ONLY the open()-time check saves */
/* it. */
/* ---------------------------------------------------------------------- */
static int toctou_hardened_open(const char* path)
{
int fd;
struct stat st;
/* Verbatim copy of sched_record_start()'s own hardening sequence
* (recording.c) -- see this test's own header comment for why this is
* reproduced rather than racing a real thread. Deliberately NO
* O_TRUNC: it truncates atomically as part of open() itself, before
* the fstat()/st_nlink check below ever runs -- against a hard-linked
* victim (same inode, no symlink involved) that would corrupt the
* victim regardless of what the check then decides. ftruncate() only
* runs after the check passes. */
fd = open(path, O_WRONLY | O_CREAT | O_NOFOLLOW, 0640);
if (fd < 0) {
return (errno > 0) ? -errno : -EIO;
}
if (fstat(fd, &st) != 0) {
int rc = (errno > 0) ? -errno : -EIO;
close(fd);
return rc;
}
if (!S_ISREG(st.st_mode) || (st.st_nlink != 1)) {
close(fd);
return -EACCES;
}
if (ftruncate(fd, 0) != 0) {
int rc = (errno > 0) ? -errno : -EIO;
close(fd);
return rc;
}
close(fd);
return 0;
}
static void test_toctou_symlink_planted_after_validation_rejected(void)
{
char record_dir[PATH_MAX];
char outside_dir[PATH_MAX];
char outside_secret[PATH_MAX];
char resolved[PATH_MAX];
char target[PATH_MAX];
const char* secret_content = "do-not-touch";
int rc;
FILE* f;
char readback[64];
make_tmp_dir(record_dir, sizeof(record_dir), "_toctou_sym_src");
make_tmp_dir(outside_dir, sizeof(outside_dir), "_toctou_sym_dst");
snprintf(outside_secret, sizeof(outside_secret), "%s/secret", outside_dir);
f = fopen(outside_secret, "w");
TEST_ASSERT_NOT_NULL(f);
fputs(secret_content, f);
fclose(f);
/* Simulates "validation just resolved this path and found nothing
* there yet" (a fresh filename -- sched_validate_record_filename()
* itself accepts this, since lstat() on a nonexistent entry short-
* circuits its containment checks entirely). */
rc = sched_validate_record_filename(record_dir, "race.replay", resolved,
sizeof(resolved));
TEST_ASSERT_EQUAL_INT(0, rc);
/* The race: a symlink appears at the just-validated path, pointing
* OUTSIDE record_dir, in the window before open() runs. */
snprintf(target, sizeof(target), "%s/race.replay", record_dir);
TEST_ASSERT_EQUAL_INT(0, symlink(outside_secret, target));
/* Layer 1: the open()-time hardening sequence itself rejects it. */
rc = toctou_hardened_open(resolved);
TEST_ASSERT_TRUE(rc < 0);
/* Layer 2 (defense in depth): a real end-to-end sched_record_start()
* call against the same record_dir/filename is ALSO rejected --
* sched_validate_record_filename() re-runs and now sees the symlink,
* closing the loop even though this particular ordering happens to be
* caught one layer earlier than the pure race would be. */
{
xce_scheduler_t sched;
fixture_init(&sched, record_dir);
rc = sched_record_start(&sched, "race.replay");
TEST_ASSERT_TRUE(rc < 0);
TEST_ASSERT_FALSE(sched.recording);
TEST_ASSERT_NULL(sched.record_file);
}
/* The outside secret file was never opened for writing/truncated. */
f = fopen(outside_secret, "r");
TEST_ASSERT_NOT_NULL(f);
memset(readback, 0, sizeof(readback));
TEST_ASSERT_NOT_NULL(fgets(readback, sizeof(readback), f));
fclose(f);
TEST_ASSERT_EQUAL_STRING(secret_content, readback);
}
static void test_toctou_hardlink_planted_after_validation_rejected(void)
{
char record_dir[PATH_MAX];
char outside_dir[PATH_MAX];
char victim[PATH_MAX];
char resolved[PATH_MAX];
char aliased[PATH_MAX];
const char* victim_content = "do-not-alias";
int rc;
FILE* f;
char readback[64];
make_tmp_dir(record_dir, sizeof(record_dir), "_toctou_hl_src");
make_tmp_dir(outside_dir, sizeof(outside_dir), "_toctou_hl_dst");
snprintf(victim, sizeof(victim), "%s/victim", outside_dir);
f = fopen(victim, "w");
TEST_ASSERT_NOT_NULL(f);
fputs(victim_content, f);
fclose(f);
rc = sched_validate_record_filename(record_dir, "alias.replay", resolved,
sizeof(resolved));
TEST_ASSERT_EQUAL_INT(0, rc);
/* The race: a hard link to the OUTSIDE victim file appears at the
* just-validated path -- same inode, no symlink involved, so
* O_NOFOLLOW alone would NOT catch this (recording.h's own doc
* comment: "no symlink is involved in that race, so O_NOFOLLOW never
* triggers") -- only the fstat()+st_nlink==1 check does. */
snprintf(aliased, sizeof(aliased), "%s/alias.replay", record_dir);
TEST_ASSERT_EQUAL_INT(0, link(victim, aliased));
rc = toctou_hardened_open(resolved);
TEST_ASSERT_EQUAL_INT(-EACCES, rc);
{
xce_scheduler_t sched;
fixture_init(&sched, record_dir);
rc = sched_record_start(&sched, "alias.replay");
TEST_ASSERT_TRUE(rc < 0);
TEST_ASSERT_FALSE(sched.recording);
TEST_ASSERT_NULL(sched.record_file);
}
f = fopen(victim, "r");
TEST_ASSERT_NOT_NULL(f);
memset(readback, 0, sizeof(readback));
TEST_ASSERT_NOT_NULL(fgets(readback, sizeof(readback), f));
fclose(f);
TEST_ASSERT_EQUAL_STRING(victim_content, readback);
}
/* ---------------------------------------------------------------------- */
/* Test 6: periodic snapshot checkpointing (task 9.3, Req 5 AC 11) -- */
/* interval N produces the expected NUMBER of extra SnapshotEntry */
/* messages, decoded and counted (not just file-size-checked) from the */
/* real generated capnp decoders, same convention as the tests above. */
/* ---------------------------------------------------------------------- */
/* Builds and submits one synthetic JOB_STATE_CHANGE event via
* sched_record_write_events() -- the direct entry point recording_events.c
* exposes, bypassing the need to drive a real session (xce_sched_core_t's
* `session_active` stays 0 throughout, which is exactly what
* write_snapshot_entry()'s xce_sched_core_save_snapshot() call requires --
* core.h's own documented precondition -- so calling this hook directly,
* without ever running a session, is a faithful, simpler stand-in for the
* real driver.c call site). `jobid` varies per call purely so decoded
* events are individually distinguishable in assertions below. */
static void submit_one_jobstate_event(xce_scheduler_t* sched, uint64_t jobid)
{
xce_core_event_t ev;
memset(&ev, 0, sizeof(ev));
ev.type = XCE_CORE_EVENT_JOB_STATE_CHANGE;
ev.u.job_state_change.jobid = jobid;
/* Same two steps driver.c takes per batch (bug-1852): record the
* events, (apply them,) then the periodic checkpoint. */
TEST_ASSERT_EQUAL_INT(0, sched_record_write_events(sched, &ev, 1));
TEST_ASSERT_EQUAL_INT(0, sched_record_checkpoint_after_apply(sched));
}
/* Reads exactly one capnp message off `f` and decodes it as a ReplayEvent,
* asserting it is a jobStateChange event for `expected_jobid`. */
static void expect_next_msg_is_jobstate_event(FILE* f, uint64_t expected_jobid)
{
struct capn c;
ReplayEvent_ptr p;
struct ReplayEvent wire;
replay_event_t dst;
TEST_ASSERT_EQUAL_INT(0, capn_init_fp(&c, f, 0));
p.p = capn_getp(capn_root(&c), 0, 1);
read_ReplayEvent(&wire, p);
memset(&dst, 0, sizeof(dst));
decode_ReplayEvent(&dst, &wire);
TEST_ASSERT_EQUAL_INT((int)ReplayEvent_body_jobStateChange,
(int)dst.body_which);
TEST_ASSERT_NOT_NULL(dst.body.job_state_change);
TEST_ASSERT_EQUAL_UINT64(expected_jobid, dst.body.job_state_change->jobid);
free_ReplayEvent(&dst);
capn_free(&c);
}
/* Reads exactly one capnp message off `f` and decodes it as a
* SnapshotEntry, asserting its `data` field is present (same structural
* check test 1 above already establishes for the initial SnapshotEntry). */
static void expect_next_msg_is_snapshot_entry(FILE* f)
{
struct capn c;
SnapshotEntry_ptr snap;
SnapshotData_ptr sdata;
TEST_ASSERT_EQUAL_INT(0, capn_init_fp(&c, f, 0));
snap.p = capn_getp(capn_root(&c), 0, 1);
sdata = SnapshotEntry_get_data(snap);
TEST_ASSERT_FALSE(sdata.p.type == CAPN_NULL);
capn_free(&c);
}
/* Reads exactly one capnp message off `f` and decodes it as the
* end-of-recording marker, ReplayEvent{endSession}. */
static void expect_next_msg_is_end_session_marker(FILE* f)
{
struct capn c;
ReplayEvent_ptr p;
struct ReplayEvent wire;
replay_event_t dst;
TEST_ASSERT_EQUAL_INT(0, capn_init_fp(&c, f, 0));
p.p = capn_getp(capn_root(&c), 0, 1);
read_ReplayEvent(&wire, p);
memset(&dst, 0, sizeof(dst));
decode_ReplayEvent(&dst, &wire);
TEST_ASSERT_EQUAL_INT((int)ReplayEvent_body_endSession,
(int)dst.body_which);
free_ReplayEvent(&dst);
capn_free(&c);
}
/* Skips the ReplayHeader + initial SnapshotEntry that sched_record_start()
* always writes first (Req 5 AC 20) -- every test below only cares about
* what comes AFTER those two, so this consumes them structurally (same
* "structural check only" pattern test 2 above already uses for its own
* ReplaySession message). */
static void skip_header_and_initial_snapshot(FILE* f)
{
struct capn c;
TEST_ASSERT_EQUAL_INT(0, capn_init_fp(&c, f, 0)); /* ReplayHeader */
capn_free(&c);
TEST_ASSERT_EQUAL_INT(0,
capn_init_fp(&c, f, 0)); /* initial SnapshotEntry */
capn_free(&c);
}
/* interval=3, 7 events submitted ONE PER CALL (7 separate
* sched_record_write_events() calls) -- record_event_count runs
* 1,2,3,4,5,6,7, landing exactly on the interval's multiples at 3 and 6.
* Expect exactly 2 extra periodic SnapshotEntry messages, positioned
* immediately after the 3rd and 6th events respectively (this task's own
* Observable, using this task's own example numbers). */
static void
test_periodic_checkpoint_interval_produces_expected_snapshot_count(void)
{
char record_dir[PATH_MAX];
xce_scheduler_t sched;
char path_copy[PATH_MAX];
FILE* f;
struct capn c;
int i;
make_tmp_dir(record_dir, sizeof(record_dir), "_periodic");
fixture_init(&sched, record_dir);
sched.record_snapshot_interval = 3ULL;
TEST_ASSERT_EQUAL_INT(0, sched_record_start(&sched, "periodic.replay"));
strcpy(path_copy, sched.record_file_path);
for (i = 0; i < 7; i++) {
submit_one_jobstate_event(&sched, (uint64_t)(100 + i));
}
TEST_ASSERT_EQUAL_UINT64(7ULL, sched.record_event_count);
TEST_ASSERT_TRUE(sched.recording); /* no abort fired along the way */
TEST_ASSERT_EQUAL_INT(0, sched_record_stop(&sched));
f = fopen(path_copy, "r");
TEST_ASSERT_NOT_NULL(f);
skip_header_and_initial_snapshot(f);
expect_next_msg_is_jobstate_event(f, 100ULL);
expect_next_msg_is_jobstate_event(f, 101ULL);
expect_next_msg_is_jobstate_event(f, 102ULL);
expect_next_msg_is_snapshot_entry(f); /* count==3, extra checkpoint #1 */
expect_next_msg_is_jobstate_event(f, 103ULL);
expect_next_msg_is_jobstate_event(f, 104ULL);
expect_next_msg_is_jobstate_event(f, 105ULL);
expect_next_msg_is_snapshot_entry(f); /* count==6, extra checkpoint #2 */
expect_next_msg_is_jobstate_event(f, 106ULL);
expect_next_msg_is_end_session_marker(f);
/* Nothing more to read -- exactly 2 extra SnapshotEntry messages, no
* more, no fewer. */
TEST_ASSERT_NOT_EQUAL(0, capn_init_fp(&c, f, 0));
fclose(f);
printf("test_periodic_checkpoint_interval_produces_expected_snapshot_count:"
" PASS\n");
}
/* record_snapshot_interval == 0 (the default/disabled case) -- many events
* recorded, zero extra SnapshotEntry messages produced regardless. */
static void test_periodic_checkpoint_interval_zero_disables_checkpointing(void)
{
char record_dir[PATH_MAX];
xce_scheduler_t sched;
char path_copy[PATH_MAX];
FILE* f;
struct capn c;
int i;
make_tmp_dir(record_dir, sizeof(record_dir), "_nointerval");
fixture_init(&sched, record_dir);
TEST_ASSERT_EQUAL_UINT64(0ULL,
sched.record_snapshot_interval); /* default */
TEST_ASSERT_EQUAL_INT(0, sched_record_start(&sched, "nointerval.replay"));
strcpy(path_copy, sched.record_file_path);
for (i = 0; i < 10; i++) {
submit_one_jobstate_event(&sched, (uint64_t)(200 + i));
}
TEST_ASSERT_EQUAL_UINT64(10ULL, sched.record_event_count);
TEST_ASSERT_EQUAL_INT(0, sched_record_stop(&sched));
f = fopen(path_copy, "r");
TEST_ASSERT_NOT_NULL(f);
skip_header_and_initial_snapshot(f);
for (i = 0; i < 10; i++) {
expect_next_msg_is_jobstate_event(f, (uint64_t)(200 + i));
}
expect_next_msg_is_end_session_marker(f);
/* No SnapshotEntry anywhere in that stream -- exactly the 10 events
* plus the end marker, nothing more. */
TEST_ASSERT_NOT_EQUAL(0, capn_init_fp(&c, f, 0));
fclose(f);
printf("test_periodic_checkpoint_interval_zero_disables_checkpointing:"
" PASS\n");
}
/* interval=3, but all 6 events are submitted in ONE single
* sched_record_write_events() call (one batch) -- record_event_count jumps
* straight from 0 to 6 in that single call, crossing BOTH the count==3 and
* count==6 multiples within it. Documents/proves this task's chosen
* "once per call" semantics: only ONE checkpoint is written for the call,
* not one per crossed boundary (see recording_events.c's own comment on
* sched_record_write_events() for the full rationale) -- this is the exact
* scenario where "once per crossed boundary" and "once per call" would
* observably differ, and this test proves which one this implementation
* chose. */
static void
test_periodic_checkpoint_single_batch_crossing_two_boundaries_checkpoints_once(
void)
{
char record_dir[PATH_MAX];
xce_scheduler_t sched;
char path_copy[PATH_MAX];
xce_core_event_t events[6];
FILE* f;
struct capn c;
int i;
make_tmp_dir(record_dir, sizeof(record_dir), "_batchboundary");
fixture_init(&sched, record_dir);
sched.record_snapshot_interval = 3ULL;
TEST_ASSERT_EQUAL_INT(0,
sched_record_start(&sched, "batchboundary.replay"));
strcpy(path_copy, sched.record_file_path);
memset(events, 0, sizeof(events));
for (i = 0; i < 6; i++) {
events[i].type = XCE_CORE_EVENT_JOB_STATE_CHANGE;
events[i].u.job_state_change.jobid = (uint64_t)(300 + i);
}
TEST_ASSERT_EQUAL_INT(0, sched_record_write_events(&sched, events, 6));
TEST_ASSERT_EQUAL_INT(0, sched_record_checkpoint_after_apply(&sched));
TEST_ASSERT_EQUAL_UINT64(6ULL, sched.record_event_count);
TEST_ASSERT_TRUE(sched.recording);
TEST_ASSERT_EQUAL_INT(0, sched_record_stop(&sched));
f = fopen(path_copy, "r");
TEST_ASSERT_NOT_NULL(f);
skip_header_and_initial_snapshot(f);
for (i = 0; i < 6; i++) {
expect_next_msg_is_jobstate_event(f, (uint64_t)(300 + i));
}
expect_next_msg_is_snapshot_entry(f); /* exactly ONE, not two */
expect_next_msg_is_end_session_marker(f);
TEST_ASSERT_NOT_EQUAL(0, capn_init_fp(&c, f, 0));
fclose(f);
printf("test_periodic_checkpoint_single_batch_crossing_two_boundaries_"
"checkpoints_once: PASS\n");
}
/* A periodic checkpoint's OWN write can fail independently of the
* per-event writes that precede it in the same call -- proven here by
* forcing xce_sched_core_save_snapshot() (which write_snapshot_entry()
* calls first) to fail via core.h's own documented `session_active != 0`
* precondition, entirely independent of file-descriptor health. Proves
* task 9.3's "route through the SAME sched_record_abort_on_error() path,
* do not special-case it" requirement for the periodic-checkpoint branch
* specifically (the general write-failure test below already covers the
* per-event-write branch). */
static void
test_periodic_checkpoint_write_failure_disables_recording_cleanly(void)
{
char record_dir[PATH_MAX];
xce_scheduler_t sched;
xce_core_event_t ev;
make_tmp_dir(record_dir, sizeof(record_dir), "_ckptfail");
fixture_init(&sched, record_dir);
sched.record_snapshot_interval = 1ULL; /* every single event checkpoints */
TEST_ASSERT_EQUAL_INT(0, sched_record_start(&sched, "ckptfail.replay"));
TEST_ASSERT_TRUE(sched.recording);
/* Force xce_sched_core_save_snapshot()'s precondition check to fail
* (core.h: "-EBUSY ... checked BEFORE any mutation") -- no real session
* is ever driven; this only fakes the flag write_snapshot_entry()'s
* save_snapshot() call reads. */
sched.core.session_active = 1;
memset(&ev, 0, sizeof(ev));
ev.type = XCE_CORE_EVENT_JOB_STATE_CHANGE;
ev.u.job_state_change.jobid = 400ULL;
/* The event itself writes fine; the periodic checkpoint that follows
* it (interval=1, bug-1852's separate after-apply step) is what fails
* and aborts. */
TEST_ASSERT_EQUAL_INT(0, sched_record_write_events(&sched, &ev, 1));
TEST_ASSERT_TRUE(sched_record_checkpoint_after_apply(&sched) < 0);
TEST_ASSERT_FALSE(sched.recording);
TEST_ASSERT_NULL(sched.record_file);
TEST_ASSERT_NULL(sched.record_file_path);
sched.core.session_active = 0; /* restore -- avoid tripping other state */
}
/* ---------------------------------------------------------------------- */
/* Test 7: write-failure end-to-end (task 9.3's own Observable) -- */
/* a genuine write failure (the underlying fd closed out from under the */
/* buffered FILE*, then forced unbuffered so the very next fwrite() hits */
/* the syscall directly instead of silently buffering past the break) */
/* cleanly disables recording via sched_record_abort_on_error(), AND the */
/* driver loop keeps working afterward -- a full real session still */
/* completes with no crash/hang, recording simply staying inactive. */
/* ---------------------------------------------------------------------- */
static void test_write_failure_disables_recording_and_driver_keeps_running(void)
{
char record_dir[PATH_MAX];
xce_scheduler_t sched;
xce_core_event_t ev;
int fd;
uint64_t event_count_before_failure;
make_tmp_dir(record_dir, sizeof(record_dir), "_writefail");
fixture_init(&sched, record_dir);
TEST_ASSERT_EQUAL_INT(0, sched_record_start(&sched, "writefail.replay"));
TEST_ASSERT_TRUE(sched.recording);
TEST_ASSERT_NOT_NULL(sched.record_file);
/* Commit the header+initial-snapshot bytes sched_record_start() already
* buffered to the still-healthy fd, then force the stream unbuffered so
* every subsequent fwrite() hits the underlying fd directly (no stdio
* buffering to mask a failure), THEN close that fd out from under the
* FILE* -- "close the underlying fd out from under the FILE* before a
* write", per this task's own suggested technique. */
TEST_ASSERT_EQUAL_INT(0, fflush(sched.record_file));
TEST_ASSERT_EQUAL_INT(0, setvbuf(sched.record_file, NULL, _IONBF, 0));
fd = fileno(sched.record_file);
TEST_ASSERT_TRUE(fd >= 0);
TEST_ASSERT_EQUAL_INT(0, close(fd));
memset(&ev, 0, sizeof(ev));
ev.type = XCE_CORE_EVENT_JOB_STATE_CHANGE;
ev.u.job_state_change.jobid = 500ULL;
/* The write genuinely fails now (EBADF on the closed fd, surfaced
* immediately since the stream is unbuffered) -- proving a REAL I/O
* failure, not a simulated return code. */
TEST_ASSERT_TRUE(sched_record_write_events(&sched, &ev, 1) < 0);
/* sched_record_abort_on_error()'s cleanup fired: recording cleanly
* disabled, file handle closed+NULLed, path freed+NULLed -- this
* task's own Observable, "recording cleanly disabled". */
TEST_ASSERT_FALSE(sched.recording);
TEST_ASSERT_NULL(sched.record_file);
TEST_ASSERT_NULL(sched.record_file_path);
event_count_before_failure = sched.record_event_count;
/* "the scheduler still processing sessions afterward" -- drive one full
* real normal-mode round (real picker thread + stand-in allocator
* thread, same helper the earlier full-cycle test uses) through the
* now-recording-inactive scheduler. No crash, no hang, and no further
* recording I/O is attempted (recording_inactive_hooks test above
* already proves the zero-I/O contract for `recording == false`; this
* proves it holds true specifically AFTER a real abort, not just from a
* fixture that never started recording). */
run_one_real_session(&sched);
TEST_ASSERT_FALSE(sched.recording);
TEST_ASSERT_NULL(sched.record_file);
TEST_ASSERT_NULL(sched.record_file_path);
TEST_ASSERT_EQUAL_UINT64(event_count_before_failure,
sched.record_event_count);
printf("test_write_failure_disables_recording_and_driver_keeps_running:"
" PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 8: bug-1843 regression -- a decision-batch growth failure inside */
/* sched_record_session_add_decision() must not silently produce a */
/* truncated-but-looks-complete ReplaySession. */
/* */
/* Root cause (confirmed via a real captured .replay file plus targeted */
/* diagnostic instrumentation, not just code reading): the decisions-array */
/* growth inside sched_record_session_add_decision() (realloc(), doubling */
/* from 8) silently dropped the current decision and returned on a failed */
/* growth, WITHOUT marking the round's accumulator incomplete -- */
/* sched_record_session_finalize() then wrote whatever prefix HAD */
/* accumulated as an apparently complete, genuine ReplaySession. The LIVE */
/* scheduling round was completely unaffected (on_decision() still fires */
/* for every decision regardless of recording) -- only the .replay FILE's */
/* own record of that one round silently lost its tail. --sim-verify's */
/* replay comparison (sim_loop.c, a completely different file -- NOT the */
/* bug, despite that being the leading theory before this fix) then had no */
/* way to distinguish that from a genuinely smaller live round, and */
/* reported the missing decisions as spurious "expected: <none>" MISMATCH */
/* lines for jobs that were, in reality, correctly decided live and simply */
/* never recorded. */
/* */
/* Forces a REAL realloc() failure (via this executable's own */
/* -Wl,--wrap=realloc, meson.build) on the growth triggered by the 9th */
/* decision (8 -> 16) -- the first growth (0 -> 8) is left unarmed so the */
/* ctx's own normal growth path runs first, same shape as the original */
/* capture (many decisions successfully recorded before the one growth */
/* that failed). */
/* ---------------------------------------------------------------------- */
static void
test_decision_batch_growth_failure_drops_incomplete_session(void)
{
char record_dir[PATH_MAX];
xce_scheduler_t sched;
sched_record_session_ctx_t* ctx;
xce_pendreason_t prs[9];
xce_decision_t dec;
int i;
int rc;
make_tmp_dir(record_dir, sizeof(record_dir), "_decgrowfail");
fixture_init(&sched, record_dir);
TEST_ASSERT_EQUAL_INT(0, sched_record_start(&sched, "decgrowfail.replay"));
TEST_ASSERT_TRUE(sched.recording);
TEST_ASSERT_NOT_NULL(sched.record_file);
ctx = sched_record_session_begin(&sched, xce_core_now(&(sched.core)));
TEST_ASSERT_NOT_NULL(ctx);
dec.op = XCE_DECISION_PEND;
/* First 8 decisions: normal, unarmed growth (0 -> 8). */
for (i = 0; i < 8; i++) {
prs[i].jobid = 3000ULL + (uint64_t)i;
prs[i].reasons = NULL;
dec.u.reason = &(prs[i]);
sched_record_session_add_decision(ctx, xce_core_now(&(sched.core)),
&dec);
}
/* 9th decision triggers the 8 -> 16 growth -- arm the wrap to fail
* exactly this ONE realloc() call, then disarm immediately after. */
g_realloc_call_count = 0;
g_realloc_fail_after = 0;
prs[8].jobid = 3008ULL;
prs[8].reasons = NULL;
dec.u.reason = &(prs[8]);
sched_record_session_add_decision(ctx, xce_core_now(&(sched.core)), &dec);
g_realloc_fail_after = -1;
rc = sched_record_session_end(&sched, ctx, xce_core_now(&(sched.core)));
TEST_ASSERT_TRUE(rc < 0);
/* bug-1843 fix's Observable: routed through the SAME
* sched_record_abort_on_error() cleanup the write-failure test above
* asserts -- recording cleanly disabled, nothing left half-written. */
TEST_ASSERT_FALSE(sched.recording);
TEST_ASSERT_NULL(sched.record_file);
TEST_ASSERT_NULL(sched.record_file_path);
printf("test_decision_batch_growth_failure_drops_incomplete_session:"
" PASS\n");
}
int main(void)
{
UNITY_BEGIN();
RUN_TEST(test_start_writes_decodable_header_then_snapshot_entry);
RUN_TEST(test_full_cycle_start_event_session_stop_produces_expected_stream);
RUN_TEST(test_recorded_workers_push_keeps_resval_values);
RUN_TEST(test_recording_inactive_hooks_are_zero_cost_noops);
RUN_TEST(test_double_start_fails_cleanly);
RUN_TEST(test_stop_when_not_recording_fails_cleanly);
RUN_TEST(test_null_sched_returns_einval);
RUN_TEST(test_toctou_symlink_planted_after_validation_rejected);
RUN_TEST(test_toctou_hardlink_planted_after_validation_rejected);
RUN_TEST(
test_periodic_checkpoint_interval_produces_expected_snapshot_count);
RUN_TEST(test_periodic_checkpoint_interval_zero_disables_checkpointing);
RUN_TEST(
test_periodic_checkpoint_single_batch_crossing_two_boundaries_checkpoints_once);
RUN_TEST(test_periodic_checkpoint_write_failure_disables_recording_cleanly);
RUN_TEST(test_write_failure_disables_recording_and_driver_keeps_running);
RUN_TEST(test_decision_batch_growth_failure_drops_incomplete_session);
return UNITY_END();
}
Reference in a new issue View git blame Copy permalink