fix(scheduler): sim resource over-commit, byte-unit display, sync job-state-at-capture
- bug-1799/1801: fix resInUse over-commit in sim replay -- sched_account_jobs's per-round re-derive pass now skips RECONCILING/UNAVAIL workers the same way core_reset_workers already does (bug-1751), closing the gap that let a worker's used slots exceed its own capacity across a resync replay - bug-1804: revise scheduler-sim-eager-reconciliation spec (user-directed) so a captured job's job->state is set to RUN synchronously at dispatch capture, mirroring sched_commit_decision's normal-mode semantics (bug-1624), instead of deferring until its separately-recorded JOB_STATE_CHANGE event replays - bug-1800: render bytes-valued CONSUMABLE resources (mem/swp/tmp) with the same human-scaled unit formatting already used for maxmem/maxswap/maxtmp - bug-1802/1803: add then revert a break-on-ANY-event breakpoint kind after live feedback that it over-fires inside busy sessions; replace it with a prescan-based "next non-dummy session" hint in info summary instead Full scheduler suite: 77/77 tests pass (asan profile). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
parent
ffca959ed9
commit
25fb5c115a
22 changed files with 1827 additions and 303 deletions
|
|
@ -21666,6 +21666,199 @@
|
|||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-10T06:57:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1799",
|
||||
"timestamp": "2026-09-11T07:32:00Z",
|
||||
"error_message": "User-reported (xce_scheduler -i -s <replay>, live cluster-captured replay /tmp/scheduler-replay-20260911-064117.replay, worker pc13v, 22 real slots): `info worker pc13v` at session=98 shows `resource slots: value=22.00 used=22.00` while the printed `assigned job:` list for that same worker sums to 26 slots (jobs [5]-[9] each *4 = 20, plus six single-slot jobs from a second batch = 6); the very next session (99, no new DISPATCH/PEND events, same assigned-jobs list) shows `used=26.00` -- exceeding the worker's own `value=22.00` capacity, an admission-invariant violation (max slots exceeded). Session 98's own event stream shows two back-to-back `WORKERS_PUSH`+`JOB_PUSH` (njobs=20) events for the same worker/jobids, consistent with a live scheduler-connection resync/reconnect being faithfully replayed (matching bug-1734/bug-1739's documented resync scenario), plus `sched_add_job failed ... rc=-17, merging state into existing tracked job instead of discarding` warnings for nearly every job in the resync batch, and one `resync claims RUN for jobid ...[7]/[8] with no allocation data -- refusing to merge a phantom RUN` (bug-1739's guard firing as designed).",
|
||||
"file": "src/scheduler/src/scheduler.c, src/scheduler/src/core_session.c",
|
||||
"root_cause": "Interaction between two independently-reasonable pieces of code that were never reconciled with each other. (1) bug-1751 (2026-09-01, worker-rejoin-resync task 5) deliberately added an `IS_WORKER_RECONCILING(w->status)` skip to `core_reset_workers()` (core_session.c:136-142, the sim/normal-mode-shared per-round resInUse-reset helper called from `xce_core_begin_session()`): a worker mid-resync has its `w->resInUse` zeroing SKIPPED that round, on purpose, so its in-flight usage isn't wiped while reconciliation is still in progress. (2) `sched_account_jobs()` (scheduler.c:1434-1461) is the ONLY thing that re-derives `resInUse`/`sharedInUse` each round -- it walks every job in `core->jobs` and, for each one with `JOB_IS_RUN(job->state) && job->accounts != NULL`, calls `sched_charge_account()` (an unconditional ADD via `sched_update_res_accounts(..., charge=1)`, resource.c:608-633) -- with NO equivalent RECONCILING check and NO reset of its own. Its own documented contract (core.h:311-322) is explicit that callers 'must call this strictly after the session's resval reset' -- a precondition bug-1751's reset-skip silently violates for exactly the RECONCILING window it targets. Consequence: for every round a worker stays RECONCILING (entered via a WORKERS_PUSH re-announce during a scheduler reconnect/resync, exactly what session 98's two WORKERS_PUSH events reflect), every currently-RUN job's slot charge on that worker gets RE-ADDED on top of the NOT-reset resInUse from the prior round, instead of the intended zero-then-resum. This produces exactly the observed shape: `used=` (accumulating additively) trails one round behind the true `assigned jobs` sum at first (22 vs 26 at session 98, since only some of the newly-RUN jobs' charges had been double-counted-in by that round's pass), then overtakes real capacity outright (26 at session 99). This is NOT the same mechanism as bug-1739 (JOBS_PUSH resync-merge phantom-RUN guard, which governs whether `job->allocs`/`state` get attached at all) nor bug-1624 (already-fixed dispatch-to-RUN synchronization gap) nor bug-1727 (already-fixed `info worker` display double-count of `value=`) -- all four live in the same resource-accounting neighborhood but are structurally distinct bugs. Root cause identified via static trace only (codegraph_explore + direct read of core_session.c/scheduler.c/core.h); not yet reproduced with an isolated minimal repro or instrumented trace confirming the exact round-by-round resInUse progression.",
|
||||
"fix": "NOT YET FIXED -- investigation only, per user request (this session's live replay-driven Q&A, not yet escalated to an implementation task). Likely fix direction: give `sched_account_jobs()` the SAME `IS_WORKER_RECONCILING` awareness `core_reset_workers()`/`sched_reset_worker_()` already have (bug-1751) -- either (a) skip charging jobs whose host is currently RECONCILING (mirroring the reset-skip so the pair stays consistent: neither reset nor resum touches a RECONCILING worker's resInUse until it clears), or (b) have `core_reset_workers()` NOT skip the reset for a RECONCILING worker but instead have `sched_account_jobs()` re-derive from job state alone regardless of RECONCILING (i.e. resInUse is always a clean per-round sum, and RECONCILING-safety is enforced elsewhere, e.g. admission control refusing new dispatches to a RECONCILING host, which per bug-1751/bug-1745 already exists as a separate candidate-filter check). Needs the worker-rejoin-resync spec's own design intent re-read (.kiro/specs/worker-rejoin-resync/) before picking (a) vs (b), plus a live repro (re-run this exact replay file with an instrumented sched_account_jobs() logging resInUse before/after each round) to confirm the round-by-round mechanism before implementing, and full TDD (scheduler-driver-equivalence steering requires normal-mode/sim-mode parity for any fix here).",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"sim",
|
||||
"replay",
|
||||
"resource-accounting",
|
||||
"resInUse",
|
||||
"sched_account_jobs",
|
||||
"worker-rejoin-resync",
|
||||
"IS_WORKER_RECONCILING",
|
||||
"over-commit",
|
||||
"slots",
|
||||
"max-capacity-exceeded",
|
||||
"core_reset_workers",
|
||||
"bug-1751-interaction",
|
||||
"not-fixed",
|
||||
"user-reported",
|
||||
"found-live"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1751",
|
||||
"bug-1739",
|
||||
"bug-1734",
|
||||
"bug-1624",
|
||||
"bug-1727",
|
||||
"bug-1613"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-11T06:43:54Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1800",
|
||||
"timestamp": "2026-09-11T07:55:00Z",
|
||||
"error_message": "User request (live replay session, worker pc13v): `info worker`'s CONSUMABLE value=/used= line for a bytes-valued resource (mem/swp/tmp) printed raw byte counts with no unit scaling -- e.g. `resource mem: value=20684476416.00 used=87960930222080.00` -- unlike every OTHER bytes-valued field in the same listing (maxmem/maxswap/maxtmp already render `value=33182072832.000000 (30.90 GB)`, and mem/tmp/swp themselves render human-scaled via the Load Metrics path when NOT flagged consumable). bug-1727's own header comment had explicitly scoped this out: 'no bytes-valued CONSUMABLE resource is exercised anywhere in this codebase's tests today.'",
|
||||
"file": "src/scheduler/src/repl_info.c",
|
||||
"root_cause": "repl_info_worker_one()'s and repl_info_resource_one()'s CONSUMABLE Value_u_fvalue branches both hardcoded `%.2f`/`%.2f` for value=/used=, deliberately bypassing repl_format_numeric_resource_value()'s existing units_mode-aware/repl_is_bytes_valued_resource()-gated formatting that every other numeric field in this listing already goes through -- correct for a non-bytes-valued consumable (slots, licenses) but wrong for mem/tmp/swp, which are bytes-valued CONSUMABLE resources in live practice even though no test exercised that combination before this session.",
|
||||
"fix": "FIXED (2026-09-11). Added a new shared static helper repl_format_consumable_value_and_used() (repl_info.c, right after repl_format_numeric_resource_value()) that routes value=/used= through the existing repl_format_numeric_resource_value() formatter (units_mode-aware, plus the swp MB->bytes special case) when repl_is_bytes_valued_resource(rname) or rname==\"swp\" is true, and keeps the exact pre-existing \"%.2f\"/\"%.2f\" formatting otherwise (slots/licenses unaffected). Wired into both repl_info_worker_one()'s per-host CONSUMABLE case and repl_info_resource_one()'s GLOBAL/SHARED CONSUMABLE case, which share the fix via this one helper so they can't disagree. Updated the stale doc comments at repl_info.c's Resource Value Formatter section header and both call sites (previously asserted this pair was 'NOT units_mode-aware' and that no bytes-valued CONSUMABLE resource existed in tests -- no longer true). TDD: added 4 new tests to test_repl_resource_table.c (test_info_worker_units_mode_both_renders_raw_and_human_for_consumable_bytes_resource, test_info_worker_units_mode_raw_renders_bare_values_for_consumable_bytes_resource, test_info_worker_units_mode_converts_swp_megabytes_for_consumable_resource, test_info_resource_units_mode_both_renders_raw_and_human_for_consumable_bytes_resource) covering both/raw modes, the swp MB special case, and the info-resource twin. RED confirmed first (4 new failures, 76 pre-existing tests still green, `xce-build.sh scheduler asan xce.build --incremental`); GREEN confirmed after the fix (77/77 Ok, zero regressions). All existing pinned CONSUMABLE value=/used= tests (bug-1601/1613/1727, using \"slots\"/\"licenses\"/\"gpu_licenses\") were verified to use non-bytes-valued resource names before the fix, so they were unaffected by construction, not just by accident.",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"repl",
|
||||
"info-worker",
|
||||
"info-resource",
|
||||
"resource-accounting",
|
||||
"units-mode",
|
||||
"bytes-human-format",
|
||||
"mem",
|
||||
"swp",
|
||||
"tmp",
|
||||
"consumable",
|
||||
"display-feature",
|
||||
"user-requested",
|
||||
"fixed",
|
||||
"tdd"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1727",
|
||||
"bug-1601",
|
||||
"bug-1613"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-11T07:55:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1801",
|
||||
"timestamp": "2026-09-11T08:35:00Z",
|
||||
"error_message": "Follow-up fix for bug-1799 (user asked to 'fix bug-1799', this session): the resInUse over-commit caused by core_reset_workers()'s RECONCILING/UNAVAIL reset-skip (bug-1751) having no matching skip in sched_account_jobs()'s per-round re-derive pass.",
|
||||
"file": "src/scheduler/src/resource.c",
|
||||
"root_cause": "See bug-1799's own entry for the full mechanism. Confirmed exactly as diagnosed: sched_update_account() (resource.c, the function both sched_charge_account() and sched_account_jobs()'s per-round resum ultimately call) had no IS_WORKER_UNAVAIL/IS_WORKER_RECONCILING check in its per-host charge loop, so a job on a worker mid-resync got its charge re-added every round on top of the not-reset total.",
|
||||
"fix": "FIXED (2026-09-11). Added `if (IS_WORKER_UNAVAIL(w->status) || IS_WORKER_RECONCILING(w->status)) { continue; }` to sched_update_account()'s per-host loop (resource.c), right after resolving the worker `w`, before the sched_update_res_accounts() charge call -- the exact same combined status check core_reset_workers()/sched_reset_worker_() already use for the reset half (bug-1751), restoring the 'resum only what reset also touched' symmetry. Scoped to the per-host branch only: core->sharedInUse's own reset (core_reset_shared_resval()) is unconditional with no per-worker gating, so the shared-resource charge is untouched. This is a single-point fix (sched_update_account() backs both sched_charge_account()/sched_uncharge_account()), so it also applies (harmlessly/defensively -- a RECONCILING/UNAVAIL worker should never be a fresh DISPATCH target per the existing bug-1745/1751 candidate-filter exclusion) to the dispatch-time charge path via sched_commit_job_alloc(). TDD: added test_charge_account_skips_reconciling_or_unavail_worker_host_charge (test_resource_accounting.c) -- charges a job account against a worker flagged RECONCILING then UNAVAIL (per-host resInUse must stay 0 both times, shared resInUse still charges normally), then clears the status bit and re-charges (must resume normally on the very next call, proving this is a per-call skip, not a permanent block). RED confirmed by temporarily reverting the skip and rebuilding (`xce-build.sh scheduler asan xce.build --incremental`): exactly this one new test failed (`Expected 0 Was 2.5`), 76 other tests green. GREEN confirmed after restoring the fix: 77/77 Ok, zero regressions.",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"resource-accounting",
|
||||
"resInUse",
|
||||
"sched_account_jobs",
|
||||
"sched_update_account",
|
||||
"worker-rejoin-resync",
|
||||
"IS_WORKER_RECONCILING",
|
||||
"IS_WORKER_UNAVAIL",
|
||||
"over-commit",
|
||||
"slots",
|
||||
"bug-1751-interaction",
|
||||
"fixed",
|
||||
"tdd",
|
||||
"user-requested"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1799",
|
||||
"bug-1751",
|
||||
"bug-1739",
|
||||
"bug-1734",
|
||||
"bug-1624",
|
||||
"bug-1727"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-11T08:35:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1802",
|
||||
"timestamp": "2026-09-11T08:35:00Z",
|
||||
"error_message": "User request (same live replay session as bug-1799/1800): 'how can we step to the next session which is not dummy(something occurs)?' -- the REPL's `break <EVENT_TYPE>` vocabulary required setting all ten per-event breakpoint kinds individually to catch 'the next session with ANY event at all,' with no single convenience kind for that common workflow.",
|
||||
"file": "src/scheduler/src/repl_break.c, src/scheduler/src/repl_internal.h",
|
||||
"root_cause": "Not a bug -- a missing convenience feature. BP_KIND_TABLE (repl_break.c) only offered the ten specific per-event kinds (CONFIG_PUSH/JOB_PUSH/.../JOB_RUSAGE) plus BEGIN_ROUND/DECISION; repl_breakpoint_matches_event() compared bp->kind for exact equality against repl_bp_kind_for_core_event(ev->type), with no wildcard option.",
|
||||
"fix": "FIXED (2026-09-11). Added a new XCE_REPL_BP_ANY enum member (repl_internal.h) and a `{\"ANY\", XCE_REPL_BP_ANY}` BP_KIND_TABLE entry (listed first for discoverability). repl_breakpoint_matches_event() (repl_break.c) now special-cases bp->kind == XCE_REPL_BP_ANY to match any event whose type maps to a real per-event kind (repl_bp_kind_for_core_event() returning nonzero) instead of requiring exact kind equality -- still subject to the breakpoint's own filters (though a bare `break ANY`, the intended usage, has none). Deliberately does NOT match BEGIN_ROUND (fires on every session including dummy ones -- matching it would defeat the whole point) or DECISION (a separate drained-decision hook). `break ANY` + `continue` now stops the REPL at the very next session that decodes at least one real event, transparently skipping any number of dummy (virtual-clock-only) sessions in between, in one command instead of ten. TDD: added test_any_breakpoint_skips_dummy_sessions_to_first_real_event (test_repl_breakpoints_inspection.c, full replay-file + REPL-script Group D fixture) -- two dummy sessions (rounds 1000/2000, zero events each) followed by one JOB_STATE_CHANGE event, `break ANY` + `continue` must land exactly at event #1 having processed both dummy rounds as 'idle' (round-rollup evidence), and must never print BEGIN_ROUND. Also extended the two existing BP_KIND_TABLE token-enumeration tests (test_repl_help.c's help-text test, test_repl_breakpoints_inspection.c's unrecognized-event-type-error test) to include \"ANY\" so both stay genuinely exhaustive against the real table. RED confirmed (the new test failed on the FIRST attempt for an unrelated reason -- the hit message names the breakpoint's own user-typed token, \"ANY\", not the underlying event's type name \"JOB_STATE_CHANGE\", which the test's first draft wrongly assumed; corrected the expected string, not the implementation, which was already correct). GREEN confirmed: 77/77 Ok, zero regressions, alongside bug-1801's fix in the same build.",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"repl",
|
||||
"sim",
|
||||
"replay",
|
||||
"breakpoints",
|
||||
"break",
|
||||
"ANY",
|
||||
"dummy-session",
|
||||
"cli-ux",
|
||||
"feature",
|
||||
"user-requested",
|
||||
"fixed",
|
||||
"tdd"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-11T08:35:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1803",
|
||||
"timestamp": "2026-09-11T09:35:00Z",
|
||||
"error_message": "User feedback (live replay session, immediately after bug-1802's `break ANY` shipped): 'It seems that break any command will introduce too many break in single session' -- live transcript showed `break ANY` + repeated `step` firing 'hit breakpoint 1 @ event #N: any' on EVERY event (JOB_UPDATE, JOB_STATE_CHANGE, ...) inside an already-busy session (session 40), forcing many single-event `step`s just to get back OUT of that session -- the opposite of the intended 'skip past dummy sessions in one hop' workflow. User explicitly requested removing `break ANY` and instead having `info summary` display the next non-dummy session id so `step N` can jump there directly.",
|
||||
"file": "src/scheduler/src/repl_break.c, src/scheduler/src/repl_internal.h, src/scheduler/src/sim.c, src/scheduler/src/repl_info.c",
|
||||
"root_cause": "Design mismatch, not a code defect: a breakpoint is inherently a persistent, repeatedly-firing mechanism (correct behavior for its stated purpose -- it fires on every matching event until deleted), but the user's actual need ('skip past a run of dummy sessions to the next one worth looking at') is a ONE-TIME navigation query, not a recurring pause condition. Modeling it as a breakpoint kind (bug-1802) meant it kept firing on every subsequent matching event for the rest of the session, well past the point it had already served its purpose, with no natural way to say 'just this once.'",
|
||||
"fix": "REVERTED `break ANY` entirely (bug-1802's XCE_REPL_BP_ANY enum member, BP_KIND_TABLE entry, repl_breakpoint_matches_event() special case, and its 3 tests) back to the exact pre-bug-1802 code, confirmed via `grep -rn XCE_REPL_BP_ANY` returning nothing but the historical doc-comment note left in repl_internal.h. REPLACED with a one-time informational display: `info summary` now also prints 'next non-dummy session: N -- run `step M` to reach it' (or 'none remaining'), computed by a new sim_prescan_next_nonempty_session() (sim.c, exported via sim.h) that peeks ahead in sim->replay_file from the current read position (same ftell()/fseek()-around-a-shape-only-scan technique sim_prescan_session_count() already establishes for the startup session-count banner) for the next ReplaySession boundary preceded by at least one real event, then rewinds -- a pure read-only query, never a recurring pause. Returns a RELATIVE 'sessions ahead' count (not an absolute session number) specifically to avoid needing xce_sched_repl_state_t's full definition inside sim.c (an incomplete type there by the project's own Simulation Isolation boundary -- caught by a real compile error on the first attempt, `invalid use of incomplete typedef`, when the function tried to read sim->repl_state->session_count directly); repl_info.c (which has the full repl_internal.h definition) adds the relative count to rs->session_count for the absolute display number. TDD: added test_info_summary_reports_next_nondummy_session_when_present and test_info_summary_reports_none_remaining_when_all_dummy (test_repl_breakpoints_inspection.c). RED confirmed by temporarily hardcoding the summary's prescan result to a sentinel non-0/non-1 value and rebuilding: exactly these 2 new tests failed, 75 others green. GREEN confirmed after restoring: 77/77 Ok, zero regressions. The existing test_info_summary_reports_counts (a bare in-memory fixture with no `.replay` file, sim->replay_file==NULL) is correctly unaffected -- the new line is silently omitted whenever the prescan can't run, mirroring sched_sim_driver_start()'s own 'purely informational, silently skipped on failure' precedent for its sibling prescan function.",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"repl",
|
||||
"sim",
|
||||
"replay",
|
||||
"break-ANY",
|
||||
"reverted",
|
||||
"info-summary",
|
||||
"next-non-dummy-session",
|
||||
"sim_prescan_next_nonempty_session",
|
||||
"cli-ux",
|
||||
"user-requested",
|
||||
"fixed",
|
||||
"tdd",
|
||||
"design-mismatch"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1802"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-11T09:35:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1804",
|
||||
"timestamp": "2026-09-11T09:55:00Z",
|
||||
"error_message": "User design assertion (same live replay session as bug-1799/1800/1803): 'according to the design, when a job has been scheduled by scheduler, we can say job is running in scheduler angle' -- directly challenging this session's own earlier explanation that a job showing DISPATCH-then-still-PEND for multiple sessions was 'by design, not a bug' (deferred-until-confirmation semantics from the scheduler-sim-eager-reconciliation spec). User asked for confirmation and, once confirmed as a genuine design fork (not a bug in the then-current implementation), explicitly directed adopting the OTHER interpretation: revise the spec to match normal-mode's own synchronous dispatch-commit semantics (bug-1624).",
|
||||
"file": ".kiro/specs/scheduler-sim-eager-reconciliation/{requirements,design,research,tasks,spec.json}, src/scheduler/src/sim_loop.c",
|
||||
"root_cause": "Not a bug in the sense of an implementation defect -- a genuine, previously-researched design fork between two defensible readings of 'job->state during replay': (a) reconstruct what the ORIGINAL live scheduler process's own job->state field actually held (RUN, synchronously, the instant sched_commit_decision() decided to dispatch -- bug-1624), vs (b) track the SEPARATELY-recorded JOB_STATE_CHANGE/worker-confirmation event's own timing as the operative signal (the scheduler-sim-eager-reconciliation spec's originally-shipped, explicitly-researched choice -- its research.md's own 'Does bug-1624's normal-mode fix already close this gap?' entry correctly noted bug-1624 is a live in-memory write, not a recordable event, so a recording's own JOB_STATE_CHANGE event genuinely CAN land rounds after its DISPATCH decision even post-bug-1624 -- a true fact, but the spec then chose interpretation (b) on top of that fact without it being forced). Live operator use surfaced interpretation (b)'s practical cost directly: a job showing PEND in `info jobs`/`info worker` for multiple sessions after its own `[sim] DISPATCH` line had already printed, reading as a scheduler bug even though it was working exactly as the shipped spec specified.",
|
||||
"fix": "FIXED (2026-09-11, user-directed). Adopted interpretation (a): `sim_reconcile_unaccounted_jobs()` (sim_loop.c)'s FOUND branch now sets `job->state = JOB_STAT_RUN;` synchronously, in the same operation as the allocation capture (`job->allocs`/`job->accounts`), mirroring `scheduler.c`'s `sched_commit_decision()` (bug-1624) exactly. `sched_account_jobs()`'s own unchanged `JOB_IS_RUN` gate therefore charges the job's resInUse starting the SAME round as capture, not a later confirmation round. A later, separately-recorded `JOB_STATE_CHANGE`/`JOB_UPDATE`-to-RUN event, when it replays, is now a safe idempotent re-affirmation (RUN->RUN, `core_events.c`'s handler reassigns `job->state` unconditionally with no charging side effect) rather than the sole state-transition trigger. Revised the full `scheduler-sim-eager-reconciliation` spec in place (requirements.md Requirement 1's ACs 1.2-1.5 rewritten, Boundary Context's out-of-scope bullet corrected; design.md's Overview/Non-Goals/Boundary Commitments/Architecture flowchart/System Flows sequence diagram/Requirements Traceability/Service Interface doc/Testing Strategy all updated with dated 'Correction (2026-09-11)' notes, not silently rewritten; research.md's own 'Does bug-1624 already close this gap' entry got a superseding revision note explaining the (a) vs (b) trade-off explicitly, since the original factual finding was correct and remains true -- only the DECISION built on top of it changed; tasks.md got a new task 4/4.1/4.2 documenting the revision, both marked done; spec.json's updated_at bumped). TDD: renamed/inverted test_captured_pend_job_excluded_from_resinuse_until_run to test_captured_job_included_in_resinuse_same_round_as_capture (test_sim_reconcile_unaccounted.c) -- asserts job->state==RUN and resInUse charged immediately after capture (no separate job->state write needed), plus a new Requirement-1.4 companion check that a redundant RUN->RUN confirmation event (driven through the REAL XCE_CORE_EVENT_JOB_STATE_CHANGE handler) doesn't touch the allocation and doesn't change what gets charged after a proper core_reset_workers()-then-sched_account_jobs() round boundary (the test's own first draft skipped that reset before its second sched_account_jobs() call, an artificial double-charge that can never happen in real per-round production flow -- caught by RED/GREEN, fixed in the test, not the production code). Also rewrote test_sim_contention_parity.c's entire second test (test_contention_capacity_check_correct_across_multi_round_dispatch_confirm_gap) -- a 300+ line fixture/assertion block built around the OLD 'deferred until confirmation, partial-then-full usage across rounds 1/4/5' narrative -- to the NEW 'both dispatched jobs saturate capacity immediately at round 1; the later confirmation events are safe no-ops' narrative, since the old assertions (capacity free at round 1, only-A-charged at round 4) are now factually backwards. Audited every other file calling/depending on sim_reconcile_unaccounted_jobs() (test_snapshot_save_load.c, test_sim_snapshot_reconcile_gap.c, test_repl_resource_table.c, test_sim_round_boundary_reconciliation.c, test_sim_e2e_original_motivation.c, test_sim_verify.c) and confirmed each is unaffected: their fixtures either put the DISPATCH and JOB_STATE_CHANGE in the SAME round (a no-op change under the new synchronous-capture semantics, since the job was already RUN via the explicit event before reconciliation ever ran) or never rely on job-state/resInUse timing at all (pure --sim-verify decision-content comparisons, or jobs that are never captured by this mechanism in the first place). RED confirmed (temporarily reverted the `job->state = JOB_STAT_RUN;` line, rebuilt: exactly the 2 updated tests failed for the expected reasons, 75 others green). GREEN confirmed after restoring: full scheduler suite 77/77 Ok, zero regressions.",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"sim",
|
||||
"replay",
|
||||
"sim_reconcile_unaccounted_jobs",
|
||||
"sched_commit_decision",
|
||||
"job-state",
|
||||
"job-lifecycle",
|
||||
"resource-accounting",
|
||||
"resInUse",
|
||||
"design-fork",
|
||||
"scheduler-sim-eager-reconciliation",
|
||||
"spec-revision",
|
||||
"user-directed",
|
||||
"bug-1624",
|
||||
"driver-equivalence",
|
||||
"fixed",
|
||||
"tdd"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1613",
|
||||
"bug-1611",
|
||||
"bug-1624",
|
||||
"bug-1799",
|
||||
"bug-1801"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-11T09:55:00Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue