From 984a6869746b397aacb2f9c79de920647a98f60d Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sat, 19 Sep 2026 14:01:09 +0000 Subject: [PATCH] docs(scheduler): correct stale bug-1611 status and driver-equivalence steering The dimension-1 audit (previous commit) reported bug-1611 as only "partially fixed" -- but that assessment worked from bug-1611's own stale narrative text without checking scheduler-sim-eager-reconciliation (a separate spec, completed 2026-09-11, predating this session's audit). That spec already decoupled sim_reconcile_unaccounted_jobs()'s trigger from the JOB_IS_RUN gate, so a job's DISPATCH decision is captured in the same round it is recorded, never deferred to a later confirmation event -- closing bug-1611's original motivating scenario for good. The "residual gap" my own first correction pass invented (an already-RUN-at-snapshot-load job) is also not real: such a job's allocation comes from the snapshot directly (confirmed field-complete by this session's dimension-4 audit), so it's never a reconciliation candidate in the first place. Corrected bug-1611's buglog entry to FIXED and updated scheduler-driver-equivalence.md's stale "Known violations" section (still described both bug-1611 and bug-1613 as open) to reflect current reality, with a pointer to re-verify rather than re-trust either framing. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC --- .kiro/steering/scheduler-driver-equivalence.md | 7 ++++--- .wolf/buglog.json | 13 +++++++------ 2 files changed, 11 insertions(+), 9 deletions(-) diff --git a/.kiro/steering/scheduler-driver-equivalence.md b/.kiro/steering/scheduler-driver-equivalence.md index 6d54d000..9a1ba0f3 100644 --- a/.kiro/steering/scheduler-driver-equivalence.md +++ b/.kiro/steering/scheduler-driver-equivalence.md @@ -20,9 +20,10 @@ Similarly, a `ReplaySession` carries a `decisions` list — the recorded decisio ## Known violations (evidence trail) -- **bug-1613** (confirmed): `sched_account_jobs()` is called by `driver.c` but never by `sim_loop.c`, which still uses the older monolithic `xce_core_go()` with no interleaving point for it. Compounding this, `sched_account_jobs()`/`sched_charge_account()`/`sched_update_account()`/`sched_update_res_accounts()` are signatured on `xce_scheduler_t*` (the normal-mode-only driver struct) even though their bodies only ever touch `sched->core` — `xce_sched_sim_driver_t` deliberately has no `xce_scheduler_t*` field (Req 5 AC 17, simulation-isolation boundary), so these functions cannot even be called from the sim driver today without a signature change. `sched_resolve_job_picker()` went through this exact `xce_scheduler_t*` → `xce_sched_core_t*` relocation already (scheduler.h documents it) — the precedent to follow, not a new pattern to invent. -- **bug-1611** (suspected same root cause): `sim_loop.c` only reads `decoded_session->decisions` when `sim->verify` is set, purely to compare against freshly-drained live decisions. It is never applied to reconstruct state for a job the live picker in this replay isn't independently re-deciding. This plausibly explains both bug-1611 (no `DISPATCH` decision ever visible for such jobs) and the accounting gap in bug-1613 (their usage is never counted, not even transiently) as one underlying gap, not two. -- Both are tracked in `.wolf/buglog.json`; a dedicated fix spec is scoped separately (see `.kiro/specs/` — do not fix inline as a drive-by patch, per the testing discipline below). +**UPDATE 2026-09-19 (`/xce-audit-invariants scheduler`, re-run after `scheduler-sim-live-allocation`): both violations below are now FIXED.** The `scheduler-sim-live-allocation` feature made `driver.c`'s and `sim_loop.c`'s per-round sequence call the identical, single-implementation election/allocation/accounting chain (see `.kiro/specs/scheduler-sim-live-allocation/`); `sim_reconcile_unaccounted_jobs()` was independently narrowed to a timing-gap safety net whose own trigger no longer requires a job to have reached `JOB_IS_RUN` (`scheduler-sim-eager-reconciliation`, completed 2026-09-11, predating this update) — closing bug-1611's original real-world scenario (a dispatch and its confirmation landing many rounds apart) for good. See `.wolf/buglog.json`'s bug-1611/bug-1613 entries for full evidence. This section is kept as a historical record of the violation and its resolution — do not re-flag either as open without first re-verifying `sim_loop.c`'s current per-round sequence against `driver.c`'s, per the Testing discipline below. + +- **bug-1613** (FIXED, 2026-08-24 / confirmed still fixed 2026-09-19): `sched_account_jobs()` is called by `driver.c` but never by `sim_loop.c`, which still uses the older monolithic `xce_core_go()` with no interleaving point for it. Compounding this, `sched_account_jobs()`/`sched_charge_account()`/`sched_update_account()`/`sched_update_res_accounts()` are signatured on `xce_scheduler_t*` (the normal-mode-only driver struct) even though their bodies only ever touch `sched->core` — `xce_sched_sim_driver_t` deliberately has no `xce_scheduler_t*` field (Req 5 AC 17, simulation-isolation boundary), so these functions cannot even be called from the sim driver today without a signature change. `sched_resolve_job_picker()` went through this exact `xce_scheduler_t*` → `xce_sched_core_t*` relocation already (scheduler.h documents it) — the precedent to follow, not a new pattern to invent. +- **bug-1611** (FIXED, 2026-09-11 via `scheduler-sim-eager-reconciliation`, general case additionally superseded by `scheduler-sim-live-allocation`'s live election/allocation path): `sim_loop.c` only read `decoded_session->decisions` when `sim->verify` was set, purely to compare against freshly-drained live decisions — never applied to reconstruct state for a job the live picker in that replay wasn't independently re-deciding. Same underlying root cause as bug-1613 (both traced to `sim_loop.c` not mirroring a per-round obligation `driver.c` upholds). ## Testing discipline diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 5c04c8a3..d68c9b0b 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -19831,7 +19831,7 @@ "error_message": "Discovered analyzing a real 108-session replay (xce_scheduler -i -s scheduler-replay-20260820-070103.replay, continue to EOF): sim_print_decision() (sim_loop.c) never printed a single DISPATCH or PREEMPT line across the entire replay -- only PEND -- even though 22 of the 30 jobs pushed at round 32 clearly transitioned to running (confirmed by ongoing JOB_UPDATE/JOB_RUSAGE ticks for jobids [0]-[21] every few rounds through round 107). An operator watching only decision output has zero confirmation those 22 jobs ever dispatched; they just silently stop appearing in PEND.", "file": "src/scheduler/src/sim_loop.c", "root_cause": "Not yet root-caused with certainty. Plausible, benign explanation: in --sim/-s REPLAY mode, jobs already resolved (dispatched) in the ORIGINAL recorded run may be replayed purely as state deltas (JOB_UPDATE/JOB_RUSAGE) without the live picker/allocator re-deriving a fresh XCE_DECISION_DISPATCH -- only jobs still outstanding (PEND) get freshly re-decided every round, which is why PEND is the only decision op ever observed. Unconfirmed against sim_loop.c's actual replay-vs-live-decide branching; needs a source read of how sim_loop.c distinguishes replayed-decision playback from freshly-computed decisions before concluding this is 'working as designed' vs a real visibility gap.", - "fix": "Not yet fixed. If confirmed benign (dispatch legitimately has no fresh decision object in replay mode), the round-summary rollup being designed for bug-1609/1610 should not label its dispatch count misleadingly (a permanent 'X dispatch' column reading 0 for an entire replay looks like a bug even when it isn't) -- consider deriving a running-job delta from info-jobs-equivalent state instead of the decision tally for replay-mode display. If NOT benign (a real decision-visibility gap), this is a separate, more serious finding than the formatting issues in bug-1609/1610.\n\n--- STATUS UPDATE 2026-08-24 (xce-audit-invariants scheduler audit) ---\n\nRoot cause confirmed and the literal mechanism is now built: sim_reconcile_unaccounted_jobs() (sim_loop.c:566-709, landed via scheduler-sim-decision-parity tasks 2.1-3, 2026-08-22) reconstructs and reports a decision for a RUN-state job whose DISPATCH decision was only in the recorded replay stream. BUT its trigger gate (sim_loop.c:592) requires JOB_IS_RUN(job->state) before searching, and the search only scans the CURRENT round's decoded_session (decoded fresh at sim_loop.c:871, freed at line 999 -- no cross-round persistence). So reconciliation only fires when a job's DISPATCH decision and its RUN confirmation land in the exact same replay round. This bug's own original scenario (dispatch at round 32, RUN confirmation through round 107) never matches -- the round-32 DISPATCH decision is gone by the time the job reaches RUN state. The scheduler-sim-decision-parity spec's own task 4.3 implementation notes explicitly acknowledge this residual does NOT fix bug-1611's dominant real-world manifestation. A follow-up spec already exists for this exact residual: .kiro/specs/scheduler-sim-eager-reconciliation/ (created 2026-08-22, phase=requirements-generated, ready_for_implementation=false -- design/tasks not yet generated). Its diagnosis: decouple allocation CAPTURE from the JOB_IS_RUN gate (capture as soon as a DISPATCH decision is seen for a job with job->accounts == NULL, regardless of state), leaving sched_account_jobs()'s existing JOB_IS_RUN + job->accounts != NULL gate to keep deferring actual charging until state transitions to RUN. Fix location: sim_reconcile_unaccounted_jobs() (sim_loop.c:566-709), trigger gate at line 592, plus its ownership-transfer logic (609-693) needs a double-capture guard since capture would no longer imply state==RUN. Do NOT fix inline as a drive-by patch per scheduler-driver-equivalence.md's testing discipline -- promote scheduler-sim-eager-reconciliation to design/tasks instead.\n\n--- STATUS UPDATE 2026-08-27 (scheduler-sim-eager-reconciliation task 1.1) ---\n\nCore production fix landed: sim_reconcile_unaccounted_jobs() trigger gate (sim_loop.c:592) had !JOB_IS_RUN(job->state) removed, leaving job->accounts == NULL as the sole capture-eligibility gate -- exactly the diagnosis from the 2026-08-24 status update. Match/capture/report mechanics (609-693) were confirmed already safe for early (pre-RUN) capture without changes (see scheduler-sim-eager-reconciliation/research.md's uncharge-safety research log -- sched_uncharge_account() has zero production callers, accounting is reset-and-resum not incremental, so no double-capture guard beyond the existing job->accounts != NULL check was needed). Old test test_reconcile_skips_non_run_state_job (which asserted the bug as correct behavior) replaced with test_reconcile_captures_non_run_state_job_when_dispatch_recorded (inverted expectation). RED/GREEN verified: new test failed under the old gate (job->allocs stayed NULL, confirmed via a real build+run before the fix), then passed after the one-line fix; full sched_sim_reconcile_unaccounted_test suite green (10/10, all prior regression tests unchanged). This is task 1.1 of the spec only -- task 1.2 (capture-safety unit coverage), task 2.1 (resInUse deferral integration test), and task 3.x (contention/--sim-verify/full-regression validation) are still pending before the spec itself is implementation-complete. STATUS UPDATE 2026-09-19 (/xce-audit-invariants scheduler dimension-1 audit, following scheduler-sim-live-allocation): the general-case symptom (an entire replay showing zero DISPATCH decisions) is now closed -- sim_loop.c genuinely spawns the real sched_job_picker()/sched_allocator() thread pair (scheduler-sim-live-allocation feature), which dispatches/pends jobs live instead of unconditionally pending. sim_reconcile_unaccounted_jobs() is now explicitly and correctly narrowed (by design, documented in requirements.md/design.md) to a timing-gap safety net rather than the general-case source of dispatch decisions. A residual gap remains BY DESIGN, not oversight: a job whose only DISPATCH evidence falls entirely outside what a given round's live-election-or-reconciliation pass ever revisits (e.g. already-RUN at snapshot load, before the replay file's own start) is still not captured -- tracked separately as the accepted scope boundary of scheduler-sim-eager-reconciliation (not yet designed). This entry stays open/tagged accordingly (PARTIALLY FIXED, not FIXED) until that follow-up lands.", + "fix": "Not yet fixed. If confirmed benign (dispatch legitimately has no fresh decision object in replay mode), the round-summary rollup being designed for bug-1609/1610 should not label its dispatch count misleadingly (a permanent 'X dispatch' column reading 0 for an entire replay looks like a bug even when it isn't) -- consider deriving a running-job delta from info-jobs-equivalent state instead of the decision tally for replay-mode display. If NOT benign (a real decision-visibility gap), this is a separate, more serious finding than the formatting issues in bug-1609/1610.\n\n--- STATUS UPDATE 2026-08-24 (xce-audit-invariants scheduler audit) ---\n\nRoot cause confirmed and the literal mechanism is now built: sim_reconcile_unaccounted_jobs() (sim_loop.c:566-709, landed via scheduler-sim-decision-parity tasks 2.1-3, 2026-08-22) reconstructs and reports a decision for a RUN-state job whose DISPATCH decision was only in the recorded replay stream. BUT its trigger gate (sim_loop.c:592) requires JOB_IS_RUN(job->state) before searching, and the search only scans the CURRENT round's decoded_session (decoded fresh at sim_loop.c:871, freed at line 999 -- no cross-round persistence). So reconciliation only fires when a job's DISPATCH decision and its RUN confirmation land in the exact same replay round. This bug's own original scenario (dispatch at round 32, RUN confirmation through round 107) never matches -- the round-32 DISPATCH decision is gone by the time the job reaches RUN state. The scheduler-sim-decision-parity spec's own task 4.3 implementation notes explicitly acknowledge this residual does NOT fix bug-1611's dominant real-world manifestation. A follow-up spec already exists for this exact residual: .kiro/specs/scheduler-sim-eager-reconciliation/ (created 2026-08-22, phase=requirements-generated, ready_for_implementation=false -- design/tasks not yet generated). Its diagnosis: decouple allocation CAPTURE from the JOB_IS_RUN gate (capture as soon as a DISPATCH decision is seen for a job with job->accounts == NULL, regardless of state), leaving sched_account_jobs()'s existing JOB_IS_RUN + job->accounts != NULL gate to keep deferring actual charging until state transitions to RUN. Fix location: sim_reconcile_unaccounted_jobs() (sim_loop.c:566-709), trigger gate at line 592, plus its ownership-transfer logic (609-693) needs a double-capture guard since capture would no longer imply state==RUN. Do NOT fix inline as a drive-by patch per scheduler-driver-equivalence.md's testing discipline -- promote scheduler-sim-eager-reconciliation to design/tasks instead.\n\n--- STATUS UPDATE 2026-08-27 (scheduler-sim-eager-reconciliation task 1.1) ---\n\nCore production fix landed: sim_reconcile_unaccounted_jobs() trigger gate (sim_loop.c:592) had !JOB_IS_RUN(job->state) removed, leaving job->accounts == NULL as the sole capture-eligibility gate -- exactly the diagnosis from the 2026-08-24 status update. Match/capture/report mechanics (609-693) were confirmed already safe for early (pre-RUN) capture without changes (see scheduler-sim-eager-reconciliation/research.md's uncharge-safety research log -- sched_uncharge_account() has zero production callers, accounting is reset-and-resum not incremental, so no double-capture guard beyond the existing job->accounts != NULL check was needed). Old test test_reconcile_skips_non_run_state_job (which asserted the bug as correct behavior) replaced with test_reconcile_captures_non_run_state_job_when_dispatch_recorded (inverted expectation). RED/GREEN verified: new test failed under the old gate (job->allocs stayed NULL, confirmed via a real build+run before the fix), then passed after the one-line fix; full sched_sim_reconcile_unaccounted_test suite green (10/10, all prior regression tests unchanged). This is task 1.1 of the spec only -- task 1.2 (capture-safety unit coverage), task 2.1 (resInUse deferral integration test), and task 3.x (contention/--sim-verify/full-regression validation) are still pending before the spec itself is implementation-complete. CORRECTED STATUS UPDATE 2026-09-19 (superseding the immediately-preceding note in this same entry, which itself worked from stale premises): scheduler-sim-eager-reconciliation (spec completed 2026-09-11, BEFORE this session's audit ran) already decoupled sim_reconcile_unaccounted_jobs()'s trigger from the JOB_IS_RUN gate -- the function now fires on job->accounts == NULL alone, regardless of PEND/RUN state (sim_loop.c:560-572), so a job's DISPATCH decision is captured in the SAME round it is recorded, never deferred until a later, separately-recorded RUN-state confirmation event. This closes the exact real-world scenario that motivated this bug (dispatch round 32, confirmation round 107) for good -- the previous note's claimed \"residual gap\" (a job already-RUN before the replay window starts) is not a real gap either: such a job's allocation is populated directly from the snapshot at load time (xce_sched_core_load_snapshot(), confirmed field-complete for job->allocs/accounts by this same audit's dimension-4 pass), so job->accounts is already non-NULL immediately post-snapshot-load and such a job is never a reconciliation candidate in the first place. Combined with scheduler-sim-reconcile-terminal-guard (bug-1764, excludes terminal jobs from false rematch) and scheduler-sim-live-allocation (the live picker/allocator now handles most jobs directly, reconciliation is a narrow safety net), this entry is FULLY FIXED. STATUS: FIXED, not PARTIALLY FIXED.", "tags": [ "repl", "sim", @@ -19841,15 +19841,16 @@ "scheduler-sim-diagnostics", "scheduler-sim-eager-reconciliation", "fixed", - "closed-by-scheduler-sim-live-allocation", - "partially-fixed" + "closed-by-scheduler-sim-live-allocation" ], "related_bugs": [ - "bug-1609", - "bug-1610" + "bug-1764", + "bug-1613", + "bug-1610", + "bug-1609" ], "occurrences": 3, - "last_seen": "2026-09-19T09:30:00Z" + "last_seen": "2026-09-19T14:00:00Z" }, { "id": "bug-1612",