lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(scheduler): recorder silently truncated a round's decisions on realloc failure (bug-1843)

Browse source 
sched_record_session_add_decision() (recording_session.c) grows its
per-round decisions array via realloc(), doubling from 8. On a failed
growth it silently dropped the current decision and every one after it
that round, without marking the round incomplete -- finalize() then
wrote the short array as an ordinary, apparently-complete ReplaySession.
--sim-verify's replay comparison had no way to distinguish that from a
genuinely smaller live round, producing spurious "expected: <none>"
mismatches for jobs that were, in reality, correctly decided live and
simply never recorded. Live scheduling itself was unaffected either
way -- only the .replay file's own record of that one round lost its
tail.

This refutes the leading theory from the prior investigation (replay
seeing a job's first appearance "one round early" via coarser event
batching in sim_loop.c) -- confirmed via targeted diagnostic
instrumentation against the original captured .replay file: replay
deterministically reproduces the full candidate set every round;
the recording itself has the one-round gap.

Fix: track truncation on the session ctx; finalize() now refuses to
write an incomplete session and fails loudly through the same
sched_record_abort_on_error() path every other recording failure in
this subsystem already uses.

TDD: added a regression test forcing a real, deterministic realloc()
failure via -Wl,--wrap=realloc (scoped to this one test binary --
no prior malloc/realloc fault-injection harness existed in this
codebase). Verified genuine red -> green: the test fails without the
fix, passes with it. Full 85/85 scheduler suite green.

Note: this prevents future recordings from losing decisions this way;
it cannot retroactively repair an already-captured .replay file.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-23 08:37:42 +00:00
parent 88b370ee84
commit 5ee329b1b8
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
5 changed files with 224 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

12
src/scheduler/testsuite/meson.build
View file

@ -520,6 +520,18 @@ sched_record_lifecycle_test = executable(
['test_record_lifecycle.c'],
include_directories:[incdir],
dependencies: [scheduler_test_deps],
# bug-1843 regression: `-Wl,--wrap=realloc` linker symbol interposition
# (same technique/idiom as scheduler_decision_publish_retry_test's own
# xcecli_* wraps above), so
# test_decision_batch_growth_failure_drops_incomplete_session() can force
# a REAL, deterministic realloc() failure inside
# sched_record_session_add_decision()'s own decisions-array growth --
# this codebase has no other malloc/realloc fault-injection harness, and
# bug-1843's root cause was exactly this growth-failure path silently
# dropping decisions instead of aborting the recording. __wrap_realloc()
# passes through to __real_realloc() for every call except the one this
# test arms, so the other 12 tests in this same binary are unaffected.
link_args: ['-Wl,--wrap=realloc'],
install: false)
# Contract test for the scheduler-side OPCode_schedAdmin request handler

119
src/scheduler/testsuite/test_record_lifecycle.c
View file

@ -381,6 +381,36 @@ static void* stand_in_allocator_thread_fn(void* arg)
return NULL;
}
/* ---------------------------------------------------------------------- */
/* bug-1843 fault injection: -Wl,--wrap=realloc (meson.build, this target */
/* only) redirects every `realloc` reference in this executable -- */
/* including inside the white-box-#included recording_session.c above -- */
/* to __wrap_realloc() below. Disabled (passes straight through to the */
/* real realloc() via __real_realloc(), the symbol `ld --wrap` reserves */
/* for it) unless a test arms it by setting g_realloc_fail_after; the one */
/* test that does always disarms it again immediately after its one target */
/* call, so the other tests in this binary see completely normal */
/* allocator behavior. */
/* ---------------------------------------------------------------------- */
extern void* __real_realloc(void* ptr, size_t size);
static int g_realloc_fail_after = -1; /* -1 == disabled */
static int g_realloc_call_count = 0;
void* __wrap_realloc(void* ptr, size_t size)
{
if (g_realloc_fail_after >= 0) {
int this_call = g_realloc_call_count++;
if (this_call == g_realloc_fail_after) {
return NULL;
}
}
return __real_realloc(ptr, size);
}
/* ---------------------------------------------------------------------- */
/* Fixture helpers */
/* ---------------------------------------------------------------------- */
@ -1282,6 +1312,94 @@ static void test_write_failure_disables_recording_and_driver_keeps_running(void)
" PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 8: bug-1843 regression -- a decision-batch growth failure inside */
/* sched_record_session_add_decision() must not silently produce a */
/* truncated-but-looks-complete ReplaySession. */
/* */
/* Root cause (confirmed via a real captured .replay file plus targeted */
/* diagnostic instrumentation, not just code reading): the decisions-array */
/* growth inside sched_record_session_add_decision() (realloc(), doubling */
/* from 8) silently dropped the current decision and returned on a failed */
/* growth, WITHOUT marking the round's accumulator incomplete -- */
/* sched_record_session_finalize() then wrote whatever prefix HAD */
/* accumulated as an apparently complete, genuine ReplaySession. The LIVE */
/* scheduling round was completely unaffected (on_decision() still fires */
/* for every decision regardless of recording) -- only the .replay FILE's */
/* own record of that one round silently lost its tail. --sim-verify's */
/* replay comparison (sim_loop.c, a completely different file -- NOT the */
/* bug, despite that being the leading theory before this fix) then had no */
/* way to distinguish that from a genuinely smaller live round, and */
/* reported the missing decisions as spurious "expected: <none>" MISMATCH */
/* lines for jobs that were, in reality, correctly decided live and simply */
/* never recorded. */
/* */
/* Forces a REAL realloc() failure (via this executable's own */
/* -Wl,--wrap=realloc, meson.build) on the growth triggered by the 9th */
/* decision (8 -> 16) -- the first growth (0 -> 8) is left unarmed so the */
/* ctx's own normal growth path runs first, same shape as the original */
/* capture (many decisions successfully recorded before the one growth */
/* that failed). */
/* ---------------------------------------------------------------------- */
static void
test_decision_batch_growth_failure_drops_incomplete_session(void)
{
char record_dir[PATH_MAX];
xce_scheduler_t sched;
sched_record_session_ctx_t* ctx;
xce_pendreason_t prs[9];
xce_decision_t dec;
int i;
int rc;
make_tmp_dir(record_dir, sizeof(record_dir), "_decgrowfail");
fixture_init(&sched, record_dir);
TEST_ASSERT_EQUAL_INT(0, sched_record_start(&sched, "decgrowfail.replay"));
TEST_ASSERT_TRUE(sched.recording);
TEST_ASSERT_NOT_NULL(sched.record_file);
ctx = sched_record_session_begin(&sched, xce_core_now(&(sched.core)));
TEST_ASSERT_NOT_NULL(ctx);
dec.op = XCE_DECISION_PEND;
/* First 8 decisions: normal, unarmed growth (0 -> 8). */
for (i = 0; i < 8; i++) {
prs[i].jobid = 3000ULL + (uint64_t)i;
prs[i].reasons = NULL;
dec.u.reason = &(prs[i]);
sched_record_session_add_decision(ctx, xce_core_now(&(sched.core)),
&dec);
}
/* 9th decision triggers the 8 -> 16 growth -- arm the wrap to fail
* exactly this ONE realloc() call, then disarm immediately after. */
g_realloc_call_count = 0;
g_realloc_fail_after = 0;
prs[8].jobid = 3008ULL;
prs[8].reasons = NULL;
dec.u.reason = &(prs[8]);
sched_record_session_add_decision(ctx, xce_core_now(&(sched.core)), &dec);
g_realloc_fail_after = -1;
rc = sched_record_session_end(&sched, ctx, xce_core_now(&(sched.core)));
TEST_ASSERT_TRUE(rc < 0);
/* bug-1843 fix's Observable: routed through the SAME
* sched_record_abort_on_error() cleanup the write-failure test above
* asserts -- recording cleanly disabled, nothing left half-written. */
TEST_ASSERT_FALSE(sched.recording);
TEST_ASSERT_NULL(sched.record_file);
TEST_ASSERT_NULL(sched.record_file_path);
printf("test_decision_batch_growth_failure_drops_incomplete_session:"
" PASS\n");
}
int main(void)
{
UNITY_BEGIN();
@ -1301,6 +1419,7 @@ int main(void)
test_periodic_checkpoint_single_batch_crossing_two_boundaries_checkpoints_once);
RUN_TEST(test_periodic_checkpoint_write_failure_disables_recording_cleanly);
RUN_TEST(test_write_failure_disables_recording_and_driver_keeps_running);
RUN_TEST(test_decision_batch_growth_failure_drops_incomplete_session);
return UNITY_END();
}