diff --git a/.kiro/specs/scheduler-sim-live-allocation/tasks.md b/.kiro/specs/scheduler-sim-live-allocation/tasks.md index 450ab296..ff7bbdc8 100644 --- a/.kiro/specs/scheduler-sim-live-allocation/tasks.md +++ b/.kiro/specs/scheduler-sim-live-allocation/tasks.md @@ -114,7 +114,7 @@ - _Depends: 4.1_ - _Requirements: 4.1, 4.2, 4.3_ -- [ ] 5.4 Validate against real recorded sessions and root-cause any divergence found +- [x] 5.4 Validate against real recorded sessions and root-cause any divergence found - Run live-vs-recorded verification against a range of real recorded sessions, not only crafted fixtures - Treat any reported mismatch between a live-computed decision and its recorded counterpart as a finding to root-cause (either a defect in this feature's own migration, or a genuine issue the old always-pend behavior could never have exposed), never as something to suppress - Observable: a written disposition (fixed, or explained as a pre-existing issue) exists for every mismatch found this way @@ -132,3 +132,4 @@ - **Task 5.1**: the pre-existing `test_sim_contention_parity.c` fixture could not be reused because it drives reconciliation only (its own white-box stubs hard-code the old always-PEND behavior), so proving live-path contention parity required two new fixtures spawning the real `sched_job_picker()`/`sched_allocator()` pair directly against bare normal-mode and sim-mode core state. Confirmed as a genuine, requirement-consistent scope boundary (not a silent narrowing): `build_limit_vres_requests()`/`get_vres_value()` (`plugins/limit.c`) are `#if 0`'d dead code with zero live callers — no limit, in either mode, currently gates/denies admission anywhere in this codebase, only tracks/displays usage. Any future task attempting real limit-based admission denial must first un-stub these two functions; that is out of boundary for this feature. - **Task 5.2**: this is the feature's concrete bug-1826 acceptance test, and it deliberately goes one level higher-fidelity than every prior task's fixtures — a real capnp-encoded `.replay` (genuine `ReplayHeader`/`SnapshotEntry`/`Conf4Sched`, real generated `encode_Conf4Sched()`) driven through the real, unmodified `sched_sim_driver_run()` entry point, not a bare-`&(sim->core)` shortcut. `repl_info_limit()`/`repl_dispatch()` (the literal functions behind the `info limit` CLI command) are both `static` and unreachable from a real-`scheduler_srcs`-linked test binary, so the test reads `limit_get_configured_limits()`/`limit_get_usage_instances()` directly — confirmed (by tracing `repl_print_limit_usage()`'s own body) to be the exact same accessors `info limit` itself calls, not a weaker substitute. RED-phase evidence for a task this late in a long dependency chain doesn't need a full feature revert: a targeted revert of just task 4.3's 4 files (the specific commit that closed the host-scoped-limit fan-out gap this test's host-scoped half exercises) reproduced the predicted failure exactly (global limit's usage still populated, host-scoped usage instance missing), then a restore-to-HEAD reproduced GREEN — this is sufficient, targeted regression-detection proof without needing to unwind 8+ interdependent commits. Also closed bug-1826 itself (the bug this whole feature exists to fix) — updated its buglog entry from "NOT FIXED" to "FULLY FIXED," citing this test. - **Task 5.3**: confirmed all three named suites (`test_repl_equivalence.c`, `test_sim_equivalence.c`, `test_sim_contention_parity.c` — the only files matching "REPL/batch decision-equivalence" and "normal-mode-recording-vs-simulation-replay decision-identity"; no fourth file exists despite the task text's plural "suites" wording, confirmed by grepping the whole spec/steering corpus and `testsuite/meson.build` for any other candidate name) pass, and pass for the right, non-vacuous reason, but NOT the reason a shallow read of their own header comments would suggest. All three exercise the REAL, genuinely-changed production code path in one dimension — `sim_loop.c`'s `sched_sim_driver_run()` now spawns `sched_job_picker()`/`sched_allocator()` via a real two-thread `jbuf` producer/consumer hand-off instead of the old single fused `sim_picker_thread_fn()` thread, exactly the decision-timing/ordering risk `requirements.md`/`design.md` flagged for this migration — but all three still exercise a STUB, not the real allocation chain, in the other dimension: each file (being a white-box `#include` of `sim_loop.c` without linking `sched_election.c`/`scheduler.c`/`resource.c`) supplies its own link-satisfying `sched_job_picker()`/`sched_allocator()` definitions (added by task 4.1, unmodified since) that faithfully reproduce the OLD always-PEND contract. Verified via `git show` across this feature's commits that the only changes made to any of these three files were adding those stubs (task 4.1, commit 08df6493) and, for `test_sim_contention_parity.c` alone, one earlier mechanical call-site signature migration (task 1.3, commit 55173fca, e.g. `sched_check_host_satisfy(sched, ...)` → `sched_check_host_satisfy(core, ...)` — an argument-type change, not a value/logic change) — zero pre-existing assertions were touched in any of them, so there is no "changed expected value" to explain for this task's own AC. This means these three suites correctly prove "the new two-thread timing didn't disturb decision ordering," which is genuinely what they were built to prove and is not vacuous — but they do NOT prove "sim mode's real live allocator now matches normal mode under contention," which is a materially different, and separately covered, claim. That second claim already has its own dedicated, non-duplicative proof: `sched_sim_live_allocation_test` (task 4.1), `sched_sim_alloc_limit_fanout_test` (task 4.3), `sched_sim_contention_resource_driver_parity_test` / `sched_sim_contention_limit_driver_parity_test` (task 5.1), and `sched_sim_e2e_limit_visibility_test` (task 5.2) — all five link the real `scheduler_srcs` and drive the real `sched_job_picker()`/`sched_allocator()` pair, several explicitly under resource contention and active limits. So this task's own job for these three suites is exactly what its Observable text implies when read carefully ("all three suites pass, or every changed assertion has a written explanation") — reconciliation and documentation, not a new proof, since nothing changed and the missing proof already exists elsewhere. One genuine documentation defect found and fixed: all three files' header comments (and one inline comment in `test_repl_equivalence.c`) still asserted, in the present tense, that "the simulation driver structurally CANNOT emit a DISPATCH decision" and attributed this to `sim_picker_thread_fn()` — a function this same feature removed in task 3.1/4.1. That claim is now false of production sim mode (proved false by the five tests above) and was only ever true of these three files' own local stub choice — a reader skimming the header without noticing the file never links `sched_election.c` could easily conclude sim mode itself still can't dispatch. Corrected with a RECONCILIATION addendum in each of the three files' own header comments (plus the one inline comment) rather than rewriting the original historical rationale, which is still accurate as a record of why the fixture was scoped to PEND-only at the time it was written. No production code was touched; no assertion values changed; full suite re-run at 85/85 pass after the documentation fix, identical to the pre-task baseline. +- **Task 5.4**: no genuine production corpus of real recorded sessions existed in this checkout to satisfy Scope item 9/AC 3.3-3.5 directly (Option 1). Searched exhaustively first: grepped `.wolf/buglog.json`/`.wolf/cerebrum.md` for bug-1611's "real 108-session replay" and a separate bug-1734 investigation's preserved `/tmp/scheduler-replay-20260825-090251.replay` — both were named-path references to prior sessions' `/tmp` files, neither still present on disk. A full-filesystem `find` for `*.replay` turned up 486 real, genuinely-produced-by-code files under `/tmp/sched_adminreq_start_auto_*/` (from `test_sched_admin_dispatch.c`'s own `record start` admin-request fixture, mkdtemp-named), but every single one was exactly 448 bytes -- an empty session header with zero jobs/decisions, worthless as a state-equivalence check. So this task took Option 2: generated genuine real recordings from actual live runs of the current codebase, per the task's own explicit allowance ("a session recorded from a real run of the current codebase counts as real even if the workload itself is synthetic"). Deployed a real single-node cluster (controller + scheduler + two `xcecd_virtual` workers) via `deploy-ha.sh`, reusing `xce-test-multihost`'s proven deploy/start/confirm pattern, then used the REAL production CLI path end to end: `xceadmin scheduler record start` (confirmed exact mechanism: `SchedAdminRequest{action=recordStart}` over the real controller-to-scheduler admin channel, `src/xceadmin/src/xceadmin/scheduler.py`/`cli.py` -- `xceadmin scheduler record start|stop|status`, NOT a startup-only `--record-dir` flag) before submitting real jobs via plain `jobsub` (never `xce_test`'s manual-allocation bypass), then `xceadmin scheduler record stop` to finalize a genuine capnp-encoded `.replay` file written by the real, unmodified `recording.c`/`recording_session.c` capture path -- not hand-authored, not test-harness-stubbed. Confirmed the verification mechanism's exact current names against source (unchanged from design.md): CLI flag `--sim-verify` (`main.c`), reporting function `sim_log_mismatch()` (`sim_verify.c`), which logs `"%s decision verification MISMATCH ..."` via `ERROR()` -- greppable verbatim. Captured TWO real sessions, deliberately different in shape: (1) 110 real rounds / 3 real jobs -- two 22-task jobs that PENDed for the entire session (root-caused via the replay's own dump and the scheduler's live DEBUG3 trace to a real, organically-configured GLOBAL limit, `limit1`, capped at total=3, reached through the real `limit_job_election()`/`match_job_global_limit_()` path with `wanted=22.000000 shortfall=19.000000` -- genuine live limit enforcement, not a bug, and not something this feature's own migration could have caused since the same limit config applies identically to normal mode) plus one 1-task job that DISPATCHed and completed immediately; (2) 28 real rounds / 5 real 1-task jobs cycling through the same `limit1` cap (15 real PEND decisions, 5 real DISPATCH decisions, genuine PEND-then-DISPATCH transitions as the limit's cumulative usage freed up) -- exercising Requirement 3.3's "live decision authoritative" rule under real, organically-timed multi-round contention, not a single crafted scenario. Ran `xce_scheduler -s --sim-verify` (batch/non-interactive, `ASAN_OPTIONS=detect_leaks=0` for a clean exit code -- the ASAN debug build's own known, already-documented "sim-mode threads are never joined" leak-at-exit artifact, design.md's own Non-Goals, unrelated to decision correctness) against both real files: **zero mismatches reported in either session** (`grep -c MISMATCH` on both stdout and the `--logfile` output: 0/0 both times; clean exit 0; no ASAN memory-safety error of any kind, only the expected leak-summary). This IS the disposition this task's Observable text requires: no divergence found, against 138 total real rounds and 8 real jobs spanning persistent-limit-blocked PEND, immediate DISPATCH, and PEND-then-DISPATCH-under-a-real-limit -- not merely "ran once, nothing broke." No production code change was needed or made. Full scheduler suite re-confirmed at 85/85 pass (see this task's own build-log timestamp) with zero code changes from this task, run as the mandated final step.