fix(scheduler): recorder silently truncated a round's decisions on realloc failure (bug-1843)

sched_record_session_add_decision() (recording_session.c) grows its
per-round decisions array via realloc(), doubling from 8. On a failed
growth it silently dropped the current decision and every one after it
that round, without marking the round incomplete -- finalize() then
wrote the short array as an ordinary, apparently-complete ReplaySession.
--sim-verify's replay comparison had no way to distinguish that from a
genuinely smaller live round, producing spurious "expected: <none>"
mismatches for jobs that were, in reality, correctly decided live and
simply never recorded. Live scheduling itself was unaffected either
way -- only the .replay file's own record of that one round lost its
tail.

This refutes the leading theory from the prior investigation (replay
seeing a job's first appearance "one round early" via coarser event
batching in sim_loop.c) -- confirmed via targeted diagnostic
instrumentation against the original captured .replay file: replay
deterministically reproduces the full candidate set every round;
the recording itself has the one-round gap.

Fix: track truncation on the session ctx; finalize() now refuses to
write an incomplete session and fails loudly through the same
sched_record_abort_on_error() path every other recording failure in
this subsystem already uses.

TDD: added a regression test forcing a real, deterministic realloc()
failure via -Wl,--wrap=realloc (scoped to this one test binary --
no prior malloc/realloc fault-injection harness existed in this
codebase). Verified genuine red -> green: the test fails without the
fix, passes with it. Full 85/85 scheduler suite green.

Note: this prevents future recordings from losing decisions this way;
it cannot retroactively repair an already-captured .replay file.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-23 08:37:42 +00:00
parent 88b370ee84
commit 5ee329b1b8
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
5 changed files with 224 additions and 18 deletions

View file

@ -299,6 +299,15 @@ sched_record_session_ctx_new(uint64_t begin_round_ts);
* handled loudly (ERROR log) rather than silently dropped or crashing) --
* see driver.c for the shared rationale. XCE_DECISION_NONE/
* XCE_DECISION_NEW_SESSION are likewise skipped (nothing to record).
*
* bug-1843 fix: if growing `ctx`'s internal decisions array fails
* (realloc() returns NULL), this decision -- and every one still to come
* this round -- goes unrecorded; `ctx` is marked internally so
* sched_record_session_finalize() below refuses to write the resulting
* incomplete ReplaySession as though it were a genuine, smaller round (see
* that function's own doc comment). The live scheduling round itself is
* entirely unaffected either way -- only this round's .replay FILE
* representation is dropped.
*/
extern void sched_record_session_add_decision(sched_record_session_ctx_t* ctx,
uint64_t ts,
@ -310,8 +319,10 @@ extern void sched_record_session_add_decision(sched_record_session_ctx_t* ctx,
* computed from `end_ts` and the tallies sched_record_session_add_decision()
* collected) to `sched->record_file`, then frees `ctx` and everything it
* owns. A NULL `ctx` is a no-op (0 is returned; nothing to free). On a
* write failure, recording is disabled the same way
* sched_record_write_events() does. Always frees `ctx` before returning,
* write failure -- including bug-1843's incomplete-`ctx` case, see
* sched_record_session_finalize()'s own doc comment below -- recording is
* disabled the same way sched_record_write_events() does. Always frees
* `ctx` before returning,
* even on failure or when `sched`'s recording state changed out from under
* the caller between sched_record_session_begin() and this call.
*/
@ -330,8 +341,16 @@ extern int sched_record_session_end(xce_scheduler_t* sched,
* so a caller with no xce_scheduler_t* (the simulation driver,
* sim_record.c, Req 5 AC 17) can finalize+write its own accumulated
* session against its own FILE* (sim->record_file) without that gate.
* Returns 0 on success, a negative errno from write_capnp_message() on
* failure -- same contract sched_record_session_end() itself documents.
* Returns 0 on success, a negative errno on failure -- either from
* write_capnp_message(), or -ENOMEM (bug-1843 fix), without writing
* anything, when `ctx` was marked truncated by a decision-batch growth
* failure inside sched_record_session_add_decision() -- writing `ctx`'s
* necessarily-incomplete `decisions` prefix would otherwise produce a
* ReplaySession wire message indistinguishable from a genuine, smaller
* live round, corrupting the recording in a way --sim-verify has no way to
* detect (bug-1843's own "expected: <none>" spurious mismatches). Same
* "always frees ctx" contract either way -- same contract
* sched_record_session_end() itself documents.
* `ctx` must be non-NULL (unlike sched_record_session_end(), this function
* has no NULL-ctx no-op case -- the caller already knows it has a live ctx
* by construction, since it only ever calls this after a successful