fix(scheduler): recorder silently truncated a round's decisions on realloc failure (bug-1843)

sched_record_session_add_decision() (recording_session.c) grows its
per-round decisions array via realloc(), doubling from 8. On a failed
growth it silently dropped the current decision and every one after it
that round, without marking the round incomplete -- finalize() then
wrote the short array as an ordinary, apparently-complete ReplaySession.
--sim-verify's replay comparison had no way to distinguish that from a
genuinely smaller live round, producing spurious "expected: <none>"
mismatches for jobs that were, in reality, correctly decided live and
simply never recorded. Live scheduling itself was unaffected either
way -- only the .replay file's own record of that one round lost its
tail.

This refutes the leading theory from the prior investigation (replay
seeing a job's first appearance "one round early" via coarser event
batching in sim_loop.c) -- confirmed via targeted diagnostic
instrumentation against the original captured .replay file: replay
deterministically reproduces the full candidate set every round;
the recording itself has the one-round gap.

Fix: track truncation on the session ctx; finalize() now refuses to
write an incomplete session and fails loudly through the same
sched_record_abort_on_error() path every other recording failure in
this subsystem already uses.

TDD: added a regression test forcing a real, deterministic realloc()
failure via -Wl,--wrap=realloc (scoped to this one test binary --
no prior malloc/realloc fault-injection harness existed in this
codebase). Verified genuine red -> green: the test fails without the
fix, passes with it. Full 85/85 scheduler suite green.

Note: this prevents future recordings from losing decisions this way;
it cannot retroactively repair an already-captured .replay file.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

.wolf/buglog.json

@@ -24995,22 +24995,21 @@
       "timestamp": "2026-09-22T19:35:00Z",
       "error_message": "Same user report as bug-1842: replaying scheduler-replay-20260922-095207.replay via --sim --sim-verify also logs a small number of 'expected: <none> -- actual: PEND jobid=... main_reason=0' mismatches -- live decisions with no recorded counterpart at all (beyond decoded_session->n_decisions), distinct from bug-1842's DISPATCH-vs-DISPATCH false positives (already fixed). 61 occurrences total across the full 643-session replay file (0.02% of the 3431 live PEND decisions produced).",
       "file": "src/scheduler/src/sim_loop.c (sched_sim_driver_run()'s decision-drain loop, sim_reconcile_unaccounted_jobs()), src/scheduler/src/sched_election.c (sched_job_picker()/sched_allocator())",
-      "root_cause": "NOT YET ROOT-CAUSED -- investigated but not fixed. Confirmed NOT the same mechanism as the historical bug-1611/bug-1613/bug-1779/bug-1759 family (all independently re-verified FIXED as of 2026-09-19 per .kiro/steering/scheduler-driver-equivalence.md -- sim_reconcile_unaccounted_jobs() does call on_dispatch() via sim_reconcile_invoke_on_dispatch(), and normal-mode/sim-mode now share the identical sched_job_picker()/sched_allocator() thread pair). Empirical characterization from a clean reproduction (xce_scheduler --sim --sim-verify --sim-speed=0 against the user's replay file, --logfile capture): every occurrence is a job's FIRST-ever appearance as a live PEND candidate landing exactly one round earlier during replay than the session it was recorded in -- every SUBSEQUENT round for the same job (it stays genuinely PEND, apparently resource-contention-bound, for many further rounds -- confirmed via repeated identical '[sim] PEND jobid=...' lines every round thereafter) matches the recording perfectly; only the very first appearance mismatches. The picker thread's own per-round loop (sched_job_picker(), sched_election.c: cbuf_get(wbuf) -> begin_session() -> next_job() until NULL -> end_session(), one full deterministic sweep per xce_core_signal_go() token) is not inherently racy in isolation, which argues against 'live thread free-running past the round boundary' as the mechanism. Leading hypothesis, NOT confirmed: sched_sim_driver_run()'s main loop buffers ALL ReplayEvent messages between two ReplaySession markers and applies them atomically via one xce_core_push_events() call immediately before the NEXT session begins (sim_loop.c ~line 1521) -- this is a strictly coarser, round-granular discretization than how the ORIGINAL live recording's normal-mode driver actually received the corresponding JOBS_PUSH/JOB_STATE_CHANGE events (over a real network connection, at whatever real-time instant they actually arrived relative to the live picker's own session-begin instant) and recorded them into the SAME or a DIFFERENT ReplaySession's event window depending on live timing down to the millisecond. If the live picker's round-N session had already begun (begin_session() already called) at the exact real moment the recording process received a newly-arriving job's JOBS_PUSH, that job would correctly miss round N and only appear round N+1 in the recording, even though the .replay file's own coarser round-boundary bookkeeping could group its JOBS_PUSH event into round N's pre-session batch instead -- making it visible to replay's round-N sweep, one round earlier than the live original saw it. Correlates in the user's log with clusters of 'xce_core_push_events JOBS_PUSH: refusing resync state 0x1 ... existing job is RUN' WARN lines for already-dispatched sibling jobs arriving in overlapping/duplicate batches shortly after -- consistent with, but not proof of, this event-batching-granularity theory. NOT YET verified by direct .replay file inspection (would need to correlate each mismatching jobid's exact recorded ReplayEvent position against its enclosing ReplaySession boundaries in the raw capnp stream) or by comparing against the ORIGINAL live (normal-mode, non-replay) recording run's own real-time event-arrival log, neither of which this session had available.",
-      "fix": "NOT FIXED -- open, reported for follow-up. Low apparent severity: does not affect actual scheduling correctness in either mode (the affected jobs correctly stay PEND under real resource contention and are eventually correctly dispatched in both the recording and the replay -- only --sim-verify's own reporting is affected, with one spurious MISMATCH line per affected job's first-round appearance). Before attempting a fix: (1) confirm the event-batching-granularity hypothesis by direct inspection of the .replay file's raw ReplayEvent/ReplaySession boundaries around one of the 61 mismatching jobids, or by instrumenting sim_loop.c's event-buffering loop to log which round each event lands in; (2) if confirmed, decide whether the fix belongs in the RECORDER (recording_events.c -- capture live event-arrival timing with round-boundary precision the replay can reproduce) or is instead an accepted, documented limitation of round-granular replay (no code change, just a --sim-verify caveat in scheduler-driver-equivalence.md). Do not attempt a speculative fix to sim_loop.c's event-buffering or session-boundary sequencing without that confirmation first -- this logic is shared by the project's own documented driver-equivalence invariant (scheduler-driver-equivalence.md) and any change here needs the resource-contention-fixture testing discipline that doc already mandates.",
+      "root_cause": "CONFIRMED (2026-09-23), and the leading hypothesis from the prior session's investigation (event-batching-granularity in sim_loop.c's replay driver -- 'a job's first appearance lands one round earlier during replay than the session it was recorded in') is REFUTED by direct measurement, not just re-reasoned about: added temporary per-round diagnostic logging to sched_sim_driver_run() (sim_loop.c, reverted after use -- not part of the fix) and replayed the exact scheduler-replay-20260922-095207.replay capture this bug was originally filed against with --sim --sim-verify --sim-speed=0. For every one of the 61 mismatches, the LIVE (replay) side deterministically re-offers the full, correct set of already-known PEND candidates every round (e.g. round begin_round=...789016869984 produced 38 PEND decisions for jobs [22]..[59], with ZERO new events pushed into that round -- core state was provably identical to the immediately preceding round, which also produced 38). The RECORDED side, for that exact same round, has only 32 decisions (jobs [22]..[53]) -- missing exactly the newest-added cohort, jobs [54]..[59] -- while the immediately preceding and following recorded rounds both have all 38. This is the true shape of the bug: an isolated, self-healing gap in the ORIGINAL LIVE RECORDING, not a replay-side batching/discretization artifact (sim_loop.c's event-buffer-then-push-before-begin_session mechanism was verified, by direct code reading of both sim_loop.c's replay-side push loop and driver.c's live-side xce_sched_driver_translate_pending_events()/sched_record_write_events() call order, to be a byte-for-byte-faithful reproduction of the recorder's own file ordering -- there is no live-vs-replay discretization gap to close there). True root cause: sched_record_session_add_decision() (recording_session.c) grows its per-round ctx->decisions array via realloc(), doubling from an initial capacity of 8 (8 -> 16 -> 32 -> 64...). If that realloc() call fails, the existing code silently drops the CURRENT decision (free_record_decision(rd); return;) and leaves ctx->n/ctx->cap exactly where they were -- every SUBSEQUENT decision that round hits the same n==cap growth check and is ALSO silently dropped, so the round effectively truncates at whichever doubling boundary (8/16/32/64) the failing realloc() landed on (n=32 here -- exactly one of those boundaries, and exactly matching the empirically observed truncation point). Nothing marks the round as incomplete: sched_record_session_finalize() (recording_session.c) then serializes and writes ctx's now-short decisions array as an ordinary, apparently-complete ReplaySession -- indistinguishable on replay from a genuinely smaller live round. This is the ONE inconsistent failure-handling path in this file: every OTHER failure in the same recording subsystem (write_capnp_message() failures, XCE_DECISION_PREEMPT/NONE cases) is either propagated as a hard error (sched_record_abort_on_error()) or logged loudly by design -- only this one growth failure was silent-and-partial. The live SCHEDULING round itself was never affected either way (on_decision() still fires for every decision drained from core.decisions regardless of recording state) -- only the .replay FILE's own record of that one round silently lost its tail, which --sim-verify's replay comparison then had no way to distinguish from a genuinely smaller live round.",
+      "fix": "FIXED (2026-09-23, TDD). recording_session.c: added a `bool truncated` field to sched_record_session_ctx_s; sched_record_session_add_decision()'s realloc-failure branch now sets ctx->truncated = true (in addition to its existing ERROR log) before dropping the decision. sched_record_session_finalize() now checks ctx->truncated FIRST and, if set, frees everything and returns -ENOMEM WITHOUT encoding/writing anything -- routing through the exact same sched_record_abort_on_error() cleanup path (via sched_record_session_end(), driver.c) that every other recording failure in this subsystem already uses, instead of silently writing a truncated-looking-complete ReplaySession. sim_loop.c's own --sim-output path (sched_record_session_finalize() called directly) already treated any negative return as fatal, so it is covered for free with zero changes there. recording.h's doc comments for sched_record_session_add_decision()/sched_record_session_end()/sched_record_session_finalize() updated to describe the new contract. TDD: added test_decision_batch_growth_failure_drops_incomplete_session() (test_record_lifecycle.c) -- this codebase had no existing malloc/realloc fault-injection harness, so added `-Wl,--wrap=realloc` linker interposition scoped to ONLY this test executable's meson target (same established `-Wl,--wrap=` idiom already used elsewhere in this test suite for xcecli_*/sched_meet_host_selection, meson.build), with a `__wrap_realloc()` that passes through to `__real_realloc()` unless the one new test arms it. The test forces a REAL, deterministic realloc() failure on the 9th decision's array growth (8 -> 16) and asserts sched_record_session_end() returns an error and recording is cleanly disabled (matching test_write_failure_disables_recording_and_driver_keeps_running()'s existing assertion style). Verified genuine red->green: temporarily reverted just the production fix and reran -- scheduler record lifecycle test FAILED (exit 1) without the fix, confirming the new test actually exercises the bug; restored the fix and reran -- full 85/85 scheduler suite green. NOTE: this fix prevents FUTURE recordings from losing decisions this way -- it cannot retroactively repair the already-captured scheduler-replay-20260922-095207.replay file, which will continue to show the same 61 --sim-verify mismatches when replayed (now fully explained, not a live bug) unless/until it is re-captured.",
       "tags": [
         "scheduler",
         "sim",
         "sim-verify",
         "replay",
-        "reconciliation",
-        "picker",
-        "round-boundary",
-        "event-batching",
-        "user-reported",
-        "unfixed",
-        "needs-diagnosis",
-        "low-severity",
-        "reporting-only"
+        "recording",
+        "recording_session.c",
+        "realloc-failure",
+        "silent-data-loss",
+        "fixed",
+        "tdd",
+        "wrap-realloc",
+        "fault-injection"
       ],
       "related_bugs": [
         "bug-1842",
@@ -25020,7 +25019,7 @@
         "bug-1759"
       ],
       "occurrences": 1,
-      "last_seen": "2026-09-22T19:35:00Z"
+      "last_seen": "2026-09-23T08:00:00Z"
     }
   ]
 }