fix(scheduler): recorder silently truncated a round's decisions on realloc failure (bug-1843)
sched_record_session_add_decision() (recording_session.c) grows its per-round decisions array via realloc(), doubling from 8. On a failed growth it silently dropped the current decision and every one after it that round, without marking the round incomplete -- finalize() then wrote the short array as an ordinary, apparently-complete ReplaySession. --sim-verify's replay comparison had no way to distinguish that from a genuinely smaller live round, producing spurious "expected: <none>" mismatches for jobs that were, in reality, correctly decided live and simply never recorded. Live scheduling itself was unaffected either way -- only the .replay file's own record of that one round lost its tail. This refutes the leading theory from the prior investigation (replay seeing a job's first appearance "one round early" via coarser event batching in sim_loop.c) -- confirmed via targeted diagnostic instrumentation against the original captured .replay file: replay deterministically reproduces the full candidate set every round; the recording itself has the one-round gap. Fix: track truncation on the session ctx; finalize() now refuses to write an incomplete session and fails loudly through the same sched_record_abort_on_error() path every other recording failure in this subsystem already uses. TDD: added a regression test forcing a real, deterministic realloc() failure via -Wl,--wrap=realloc (scoped to this one test binary -- no prior malloc/realloc fault-injection harness existed in this codebase). Verified genuine red -> green: the test fails without the fix, passes with it. Full 85/85 scheduler suite green. Note: this prevents future recordings from losing decisions this way; it cannot retroactively repair an already-captured .replay file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
88b370ee84
commit
5ee329b1b8
5 changed files with 224 additions and 18 deletions
|
|
@ -299,6 +299,15 @@ sched_record_session_ctx_new(uint64_t begin_round_ts);
|
|||
* handled loudly (ERROR log) rather than silently dropped or crashing) --
|
||||
* see driver.c for the shared rationale. XCE_DECISION_NONE/
|
||||
* XCE_DECISION_NEW_SESSION are likewise skipped (nothing to record).
|
||||
*
|
||||
* bug-1843 fix: if growing `ctx`'s internal decisions array fails
|
||||
* (realloc() returns NULL), this decision -- and every one still to come
|
||||
* this round -- goes unrecorded; `ctx` is marked internally so
|
||||
* sched_record_session_finalize() below refuses to write the resulting
|
||||
* incomplete ReplaySession as though it were a genuine, smaller round (see
|
||||
* that function's own doc comment). The live scheduling round itself is
|
||||
* entirely unaffected either way -- only this round's .replay FILE
|
||||
* representation is dropped.
|
||||
*/
|
||||
extern void sched_record_session_add_decision(sched_record_session_ctx_t* ctx,
|
||||
uint64_t ts,
|
||||
|
|
@ -310,8 +319,10 @@ extern void sched_record_session_add_decision(sched_record_session_ctx_t* ctx,
|
|||
* computed from `end_ts` and the tallies sched_record_session_add_decision()
|
||||
* collected) to `sched->record_file`, then frees `ctx` and everything it
|
||||
* owns. A NULL `ctx` is a no-op (0 is returned; nothing to free). On a
|
||||
* write failure, recording is disabled the same way
|
||||
* sched_record_write_events() does. Always frees `ctx` before returning,
|
||||
* write failure -- including bug-1843's incomplete-`ctx` case, see
|
||||
* sched_record_session_finalize()'s own doc comment below -- recording is
|
||||
* disabled the same way sched_record_write_events() does. Always frees
|
||||
* `ctx` before returning,
|
||||
* even on failure or when `sched`'s recording state changed out from under
|
||||
* the caller between sched_record_session_begin() and this call.
|
||||
*/
|
||||
|
|
@ -330,8 +341,16 @@ extern int sched_record_session_end(xce_scheduler_t* sched,
|
|||
* so a caller with no xce_scheduler_t* (the simulation driver,
|
||||
* sim_record.c, Req 5 AC 17) can finalize+write its own accumulated
|
||||
* session against its own FILE* (sim->record_file) without that gate.
|
||||
* Returns 0 on success, a negative errno from write_capnp_message() on
|
||||
* failure -- same contract sched_record_session_end() itself documents.
|
||||
* Returns 0 on success, a negative errno on failure -- either from
|
||||
* write_capnp_message(), or -ENOMEM (bug-1843 fix), without writing
|
||||
* anything, when `ctx` was marked truncated by a decision-batch growth
|
||||
* failure inside sched_record_session_add_decision() -- writing `ctx`'s
|
||||
* necessarily-incomplete `decisions` prefix would otherwise produce a
|
||||
* ReplaySession wire message indistinguishable from a genuine, smaller
|
||||
* live round, corrupting the recording in a way --sim-verify has no way to
|
||||
* detect (bug-1843's own "expected: <none>" spurious mismatches). Same
|
||||
* "always frees ctx" contract either way -- same contract
|
||||
* sched_record_session_end() itself documents.
|
||||
* `ctx` must be non-NULL (unlike sched_record_session_end(), this function
|
||||
* has no NULL-ctx no-op case -- the caller already knows it has a live ctx
|
||||
* by construction, since it only ever calls this after a successful
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue