fix(scheduler): sim mode force-dispatched recorded decisions past live capacity (bug-1836, bug-1838)

User-reported: replaying a captured session dispatched 44 single-slot jobs
onto a 22-slot host. Root cause traced through two layers:

- bug-1836: sim_reconcile_unaccounted_jobs() (the "Recorded-Decision
  Reconciliation" backstop) force-applied any recorded DISPATCH decision
  it found, with zero live capacity check. Added
  sim_reconciled_alloc_fits_capacity(), reusing the same
  sched_check_host_satisfy()/sched_check_shared_satisfy() admission checks
  the live picker/allocator path already uses.

- bug-1838: even with that gate, reconciliation still preempted every
  freshly-PEND job before the live picker/allocator got a turn each round
  -- unnecessary now that scheduler-sim-live-allocation gave sim mode the
  same real, capacity-checked picker/allocator threads normal mode uses.
  Narrowed reconciliation's trigger back to JOB_IS_RUN-only (undoing a
  2026-09-11 widening whose own justification no longer holds), so a
  freshly-pushed job now always flows through the same live, verified
  decision path as normal mode. Reconciliation now only ever fires for a
  job already RUN from before the replay's own observation window --
  the one case structurally invisible to the live picker.

Also logged (bug-1837, not fixed here, needs its own design decision):
sim_reconcile_unaccounted_jobs() calls a picker's on_dispatch() hook from
the driver thread instead of the allocator thread plugin.h documents it
belonging to -- currently inert since the only real picker plugin happens
to lock internally, but not contract-safe for a future lock-free one.

Fixing bug-1838 required re-examining every test exercising the old
PEND-inclusive trigger; several needed their fixtures corrected to
register a real worker (harmless before any capacity check existed), and
others needed their actual intent re-derived and their assertions
inverted or restated now that a PEND job is never eligible for
reconciliation. Dated correction notes added in place to the
scheduler-sim-eager-reconciliation spec, matching its own existing
2026-09-11 correction-note precedent.

Verified: full scheduler suite 85/85, independently re-confirmed by
running the four most affected binaries directly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-21 09:50:27 +00:00
parent cd9ecb4159
commit 94115b0024
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
44 changed files with 2769 additions and 333 deletions

View file

@ -1,13 +1,13 @@
{
"version": 1,
"meta": {
"lastScanned": "2026-09-21T03:39:53.909Z",
"fileCount": 666,
"lastScanned": "2026-09-21T09:31:36.646Z",
"fileCount": 668,
"hits": 0,
"misses": 0,
"renderedHash": "45c99420f9851c89e6b8a493a3c4c5f0a05400f4342a6cd486d0b86ccf3680a8",
"storeUpdatedAt": "2026-09-21T03:39:53.919Z",
"rootHash": "5dc33e10712cf857"
"renderedHash": "56a6cca5dea3c105bb5d64b41019c5e07814ffd9ab069c9aab6f152e32fce1f2",
"storeUpdatedAt": "2026-09-21T09:31:36.655Z",
"rootHash": "cca4c863d5077424"
},
"files": {
".agent-shell/transcripts/2026-08-11-11-39-59.md": {
@ -3225,20 +3225,20 @@
},
".kiro/specs/scheduler-sim-eager-reconciliation/design.md": {
"description": "Design Document",
"tokens": 6493,
"hash": "e1712bb6eb14a47b",
"size": 26074,
"mtimeMs": 1789122761693.9182,
"updatedAt": "2026-09-11T10:32:41.734Z",
"tokens": 7078,
"hash": "388510e52bf7c634",
"size": 28419,
"mtimeMs": 1789980075150.255,
"updatedAt": "2026-09-21T08:41:15.194Z",
"source": "hook"
},
".kiro/specs/scheduler-sim-eager-reconciliation/requirements.md": {
"description": "Requirements Document",
"tokens": 2957,
"hash": "d4e3125d696b25d8",
"size": 11826,
"mtimeMs": 1789122511639.8337,
"updatedAt": "2026-09-11T10:28:31.934Z",
"tokens": 3361,
"hash": "3892b8f845fdd9af",
"size": 13443,
"mtimeMs": 1789980097028.1904,
"updatedAt": "2026-09-21T08:41:37.065Z",
"source": "hook"
},
".kiro/specs/scheduler-sim-eager-reconciliation/research.md": {
@ -9836,11 +9836,11 @@
},
"src/scheduler/include/scheduler.h": {
"description": "if !defined(_SCHEDULER_H_)",
"tokens": 6125,
"hash": "b963930a3cd4f465",
"size": 22967,
"mtimeMs": 1789750425046.0798,
"updatedAt": "2026-09-18T16:53:45.094Z",
"tokens": 6140,
"hash": "9ebe74de3d64e5e7",
"size": 23023,
"mtimeMs": 1789967319105.591,
"updatedAt": "2026-09-21T05:08:39.157Z",
"source": "hook"
},
"src/scheduler/src/plugin.c": {
@ -9953,29 +9953,29 @@
},
"src/scheduler/src/sim_loop.c": {
"description": "include <errno.h>",
"tokens": 18113,
"hash": "74a4896cf97efc47",
"size": 67921,
"mtimeMs": 1789794049526.3513,
"updatedAt": "2026-09-19T05:00:49.564Z",
"tokens": 22511,
"hash": "6bacd553e27e51b7",
"size": 84414,
"mtimeMs": 1789977028368.3345,
"updatedAt": "2026-09-21T07:50:28.417Z",
"source": "hook"
},
"src/scheduler/testsuite/test_sim_reconcile_unaccounted.c": {
"description": "Declares documents",
"tokens": 30859,
"hash": "a7ee960113058168",
"size": 115721,
"mtimeMs": 1789794628587.9622,
"updatedAt": "2026-09-19T05:10:28.628Z",
"tokens": 34635,
"hash": "ae87004ae3cee9a6",
"size": 129879,
"mtimeMs": 1789979940956.2974,
"updatedAt": "2026-09-21T08:39:00.991Z",
"source": "hook"
},
"src/scheduler/testsuite/test_sim_contention_parity.c": {
"description": "Declares calls",
"tokens": 21701,
"hash": "907c7366ef4536fe",
"size": 81376,
"mtimeMs": 1789809732381.1213,
"updatedAt": "2026-09-19T09:22:12.429Z",
"tokens": 21740,
"hash": "d824c6b350e69b1d",
"size": 81524,
"mtimeMs": 1789978567598.1646,
"updatedAt": "2026-09-21T08:16:07.642Z",
"source": "hook"
},
"reports/sep06-sep12-2026.md": {
@ -11566,11 +11566,11 @@
},
"src/scheduler/include/core.h": {
"description": "if !defined(_CORE_H_)",
"tokens": 10892,
"hash": "949582dd4823e043",
"size": 40845,
"mtimeMs": 1789721930305.5815,
"updatedAt": "2026-09-18T08:58:50.343Z",
"tokens": 11223,
"hash": "cc63291ebec75cc8",
"size": 42083,
"mtimeMs": 1789969624960.9233,
"updatedAt": "2026-09-21T05:47:05.008Z",
"source": "hook"
},
"src/scheduler/include/sched/alloc_cache.h": {
@ -11677,8 +11677,8 @@
"tokens": 17581,
"hash": "81c8f86b16f80bf3",
"size": 65936,
"mtimeMs": 1789752400592.5513,
"updatedAt": "2026-09-18T17:26:40.636Z",
"mtimeMs": 1789969683658.6997,
"updatedAt": "2026-09-21T05:48:03.708Z",
"source": "hook"
},
"src/scheduler/src/schedobj.c": {
@ -11782,11 +11782,11 @@
},
"src/scheduler/testsuite/test_sim_live_allocation.c": {
"description": "Declares already",
"tokens": 5795,
"hash": "97569b95a4d71687",
"size": 21900,
"mtimeMs": 1789788543691.8958,
"updatedAt": "2026-09-19T03:29:03.742Z",
"tokens": 9156,
"hash": "0234199ee3e7f8de",
"size": 34503,
"mtimeMs": 1789983083813.7146,
"updatedAt": "2026-09-21T09:31:23.860Z",
"source": "hook"
},
"src/scheduler/src/sim_alloc.c": {
@ -12216,6 +12216,24 @@
"mtimeMs": 1789961993857.0706,
"updatedAt": "2026-09-21T03:39:53.909Z",
"source": "hook"
},
"src/scheduler/testsuite/test_sim_verify.c": {
"description": "Declares mismatch",
"tokens": 18569,
"hash": "ce46afa8186ebac0",
"size": 69631,
"mtimeMs": 1789978226655.7988,
"updatedAt": "2026-09-21T08:10:26.695Z",
"source": "hook"
},
"src/scheduler/src/sim_verify.c": {
"description": "include <stdio.h>",
"tokens": 4694,
"hash": "fe98da3155c5abee",
"size": 17601,
"mtimeMs": 1789983096595.9946,
"updatedAt": "2026-09-21T09:31:36.645Z",
"source": "hook"
}
}
}

View file

@ -1,7 +1,7 @@
# anatomy.md
> Auto-maintained by OpenWolf. Last scanned: 2026-09-21T03:39:53.909Z
> Files: 666 tracked | Anatomy hits: 0 | Misses: 0
> Auto-maintained by OpenWolf. Last scanned: 2026-09-21T09:31:36.646Z
> Files: 668 tracked | Anatomy hits: 0 | Misses: 0
## ./
@ -541,8 +541,8 @@
## .kiro/specs/scheduler-sim-eager-reconciliation/
- `design.md` — Design Document (~6493 tok)
- `requirements.md` — Requirements Document (~2957 tok)
- `design.md` — Design Document (~7078 tok)
- `requirements.md` — Requirements Document (~3361 tok)
- `research.md` — Research & Design Decisions (~5166 tok)
- `spec.json` (~135 tok)
- `tasks.md` — Implementation Plan (~2306 tok)
@ -957,9 +957,9 @@
## src/scheduler/include/
- `core.h` — if !defined(_CORE_H_) (~10892 tok)
- `core.h` — if !defined(_CORE_H_) (~11223 tok)
- `plugin.h` — if !defined(_PLUGIN_H_) (~5194 tok)
- `scheduler.h` — if !defined(_SCHEDULER_H_) (~6125 tok)
- `scheduler.h` — if !defined(_SCHEDULER_H_) (~6140 tok)
- `sim.h` — if !defined(_SIM_H_) (~4213 tok)
## src/scheduler/include/sched/
@ -988,7 +988,8 @@
- `scheduler.c` — include <errno.h> (~17581 tok)
- `sim_alloc.c` — include <stdlib.h> (~2585 tok)
- `sim_limit.c` — include <stdlib.h> (~2102 tok)
- `sim_loop.c` — include <errno.h> (~18113 tok)
- `sim_loop.c` — include <errno.h> (~22511 tok)
- `sim_verify.c` — include <stdio.h> (~4694 tok)
- `sim.c` — include <errno.h> (~7432 tok)
## src/scheduler/src/plugins/
@ -1027,15 +1028,16 @@
- `test_seqjob_rewrap_equivalence.c` — Declares char (~6448 tok)
- `test_sim_alloc_limit_fanout.c` — Declares char (~4716 tok)
- `test_sim_contention_limit_driver_parity.c` — Declares char (~6514 tok)
- `test_sim_contention_parity.c` — Declares calls (~21701 tok)
- `test_sim_contention_parity.c` — Declares calls (~21740 tok)
- `test_sim_contention_resource_driver_parity.c` — Declares char (~7336 tok)
- `test_sim_e2e_limit_visibility.c` — Declares char (~8076 tok)
- `test_sim_e2e_original_motivation.c` (~18078 tok)
- `test_sim_equivalence.c` — Declares BOTH (~11161 tok)
- `test_sim_live_allocation.c` — Declares already (~5795 tok)
- `test_sim_reconcile_unaccounted.c` — Declares documents (~30859 tok)
- `test_sim_live_allocation.c` — Declares already (~9156 tok)
- `test_sim_reconcile_unaccounted.c` — Declares documents (~34635 tok)
- `test_sim_resource_contention.c` — Declares is (~10149 tok)
- `test_sim_session_ordering.c` (~10531 tok)
- `test_sim_verify.c` — Declares mismatch (~18569 tok)
- `test_snapshot_save_load.c` (~29478 tok)
- `test.c` — define UNIT_TEST 1 (~7044 tok)

View file

@ -24776,6 +24776,101 @@
],
"occurrences": 1,
"last_seen": "2026-09-21T03:47:10Z"
},
{
"id": "bug-1836",
"timestamp": "2026-09-20T22:00:00Z",
"error_message": "User-reported live over-dispatch in xce_scheduler -i -s <replay>: replaying a captured session (JOB_PUSH of a 50-element array job) against a real single worker with 22 slots dispatched 44 concurrent single-slot jobs (`assigned jobs: 0-3-1789963203-0[0-43]`), exactly double the host's real capacity, while `info worker` simultaneously showed `resource slots: value=22.00 used=22.00` -- the accounting display looked self-consistent (used == value) even though double the job count the display's own 'used' figure implied was actually running. User confirmed normal mode (the live controller-connected driver) does not exhibit this under equivalent conditions, isolating the defect to simulation/replay mode specifically.",
"file": "src/scheduler/src/sim_loop.c",
"root_cause": "sim_reconcile_unaccounted_jobs() (the 'Recorded-Decision Reconciliation' mechanism, scheduler-sim-decision-parity/scheduler-sim-eager-reconciliation specs) unconditionally force-applied any recorded DISPATCH decision it found in decoded_session->decisions for a job with accounts==NULL -- setting job->state=JOB_STAT_RUN and job->allocs/job->accounts straight from the recording -- with ZERO call to sched_check_host_satisfy()/sched_check_shared_satisfy() or any other admission/capacity check. This directly contradicted the scheduler-sim-decision-parity spec's own design.md assumption underpinning its Requirement 2 ('capacity awareness'): 'sched_check_host_satisfy_()/sched_check_shared_satisfy_() already read resInUse/sharedInUse unconditionally; once those fields are correctly populated in simulation mode, the existing (unmodified) capacity-check code path produces the same outcome normal mode would' -- an assumption that only holds for jobs decided via the LIVE picker/allocator path (sched_election.c's sched_allocator(), which DOES call these checks via sched_try_alloc_on_host_()), not for jobs captured via this SEPARATE reconciliation backstop, which bypasses that code path entirely. Confirmed via the round sequence in sched_sim_driver_run(): xce_core_push_events() creates all 50 freshly-pushed jobs (PEND, accounts==NULL) BEFORE sim_reconcile_unaccounted_jobs() runs for that same round, so ALL 50 are eligible reconciliation candidates the moment the function starts -- for however many of them (44, in the live report) the session's OWN originally-recorded decisions said DISPATCH, reconciliation force-applied all of them with no regard for the CURRENT sim snapshot's actual worker capacity (22 slots), before the live picker/allocator (which runs afterward, per the fixed round order) ever got a chance to independently admission-check any of them -- explaining both the 44 dispatches and the 6 correctly-PENDed remainder (jobs whose OWN recorded decision was PEND, never matched by reconciliation's ReplayDecision_body_dispatch-only filter). This is a distinct mechanism from bug-1611/bug-1613 (both already fixed/superseded by scheduler-sim-live-allocation and scheduler-sim-eager-reconciliation) -- those closed a TIMING gap in when reconciliation's job->state transition happens; this bug is a CAPACITY-VALIDATION gap in whether it happens at all, independent of timing. Uncovered via /xce-audit-invariants scheduler, dimension 1 (driver state-equivalence).",
"fix": "FIXED (2026-09-21). Added sim_reconciled_alloc_fits_capacity() (sim_loop.c, static) -- re-validates a candidate recorded allocation's full resource footprint (every per-host allocation plus any shared/global allocation) against LIVE resInUse/sharedInUse using the SAME sched_check_host_satisfy()/sched_check_shared_satisfy() admission-check functions (resource.c) the live picker/allocator path already calls, resolving each allocation's resnames against core->resmgr fresh (the decoded allocation carries only resnames/values, not resolved indices). sim_reconcile_unaccounted_jobs()'s FOUND branch now calls this BEFORE taking ownership of a matched entry; a capacity miss is handled exactly like 'no matching recorded decision' (job left unaccounted this round, entry NOT consumed/freed, naturally retried later or correctly PENDed by the live picker once it runs). A newly-admitted candidate's usage is charged into the LIVE resInUse/sharedInUse immediately via sched_charge_account() (mirroring sched_commit_decision()'s own immediate charge for a live dispatch) so a LATER candidate in the SAME reconciliation pass correctly sees an EARLIER accepted candidate's usage -- essential for the exact multi-job-contention scenario this bug exercises. To make the capacity check see genuinely prior-round (already-running) jobs' usage too, added sim_reconcile_toggle_prior_charges() (sim_loop.c, static) -- seeds resInUse/sharedInUse from every already-RUN+accounted job at the start of sim_reconcile_unaccounted_jobs(), then fully undoes that seed plus every incremental charge made during the pass at the end, leaving the function's net effect on the real accounting arrays at exactly zero on return; the caller's own unchanged, immediately-following sched_account_jobs() call remains the sole, non-double-counting authoritative charge for the round (unchanged architecture/ordering). Deliberately NOT implemented via sched_account_jobs()/a new sched_unaccount_jobs() exported pair (tried first, then reverted): that exact by-name function is separately instrumented and call-count-asserted by test_sim_contention_parity.c's g_round_idx mechanism ('sched_account_jobs() called exactly once per round' is itself a protected invariant there) -- doubling its call count broke 2 of that file's tests (masked at first by a coincidental g_round_resinuse[8] array-capacity cap that hid the true +5/+10 call-count delta behind an apparent '+2' pattern; properly diagnosed by reading the array declaration, not guessed). sched_check_host_satisfy()/sched_check_shared_satisfy() were relocated from scheduler.h to core.h (pure declaration move, bodies unchanged in resource.c, following the exact sched_account_jobs()/sched_resolve_job_picker() precedent already established for this exact purpose) since sim_loop.c must never include scheduler.h (pulls in client.h/xce/cluster.h, forbidden by test_sim_isolation.c's Req 5 AC 17 enforcement). New regression test test_reconcile_respects_host_capacity_under_contention (test_sim_reconcile_unaccounted.c) -- two PEND jobs each recorded-DISPATCH to the same 1-slot-capacity host; asserts exactly one is admitted (not both, not neither), the rejected one's entry is not consumed, and post-round resInUse reads exactly 1.0 (never 2.0). Fixing this surfaced 12 pre-existing tests (9 in test_sim_reconcile_unaccounted.c, 3 in test_sim_verify.c) whose fixtures named a host in a recorded allocation without ever registering a real xce_sched_worker_t of that name -- harmless before this fix (nothing checked capacity), now correctly rejected by the new gate; fixed by registering a real (or, for test_sim_verify.c, which never white-box-includes resource.c, a bare link-satisfying) worker for each such fixture, generous capacity since none of those 12 tests are actually about capacity limits. Verified: full scheduler suite 85/85 Ok, 0 Fail (asan profile, xce-build.sh scheduler asan), independently re-confirmed by running sched_sim_reconcile_unaccounted_test (22/22) and sched_sim_verify_test (17/17) directly with ASAN_OPTIONS=detect_leaks=0 (this test-file family's own established convention, unrelated pre-existing leak reports otherwise printed by these binaries when run standalone outside meson's own suite-level ASAN_OPTIONS).",
"tags": [
"scheduler",
"sim",
"replay",
"resource-accounting",
"sim_reconcile_unaccounted_jobs",
"scheduler-driver-equivalence",
"over-commit",
"slots",
"capacity-check",
"xce-audit-invariants",
"user-reported",
"found-live",
"fixed"
],
"related_bugs": [
"bug-1611",
"bug-1613",
"bug-1799",
"bug-1764",
"bug-1759",
"bug-1779"
],
"occurrences": 1,
"last_seen": "2026-09-21T05:46:00Z"
},
{
"id": "bug-1837",
"timestamp": "2026-09-20T22:05:00Z",
"error_message": "Found during /xce-audit-invariants scheduler (dimension 3, jbuf capacity-1 rendezvous audit), not user-reported and not a live symptom: sim_reconcile_unaccounted_jobs() (sim_loop.c) calls picker->picker_iface->on_dispatch(picker, job->jobid) directly from the sim driver thread (sched_sim_driver_run()'s own thread, between xce_core_begin_session() and xce_core_signal_go()), but plugin.h documents on_dispatch()/on_pend() as 'Post-allocation hooks (MUST) -- allocator thread, after each allocation attempt' -- a MUST-level threading contract this call site violates by construction (driver-thread code calling an allocator-thread-only hook).",
"file": "src/scheduler/src/sim_loop.c",
"root_cause": "bug-1759/bug-1779's own fix added this on_dispatch() call to sim_reconcile_unaccounted_jobs() specifically to prevent sim_picker_thread_fn() from re-offering (and re-PENDing) an already-reconciled job on a later round -- a real, necessary fix for that narrower symptom (a duplicate PEND report) -- but the fix's own header comment justifies it only in those terms ('the picker never re-decides an already-resolved job'), never addressing thread-safety of calling an allocator-thread-only hook from the driver thread instead. It currently causes no observable corruption only because the sole real (non-no-op) picker plugin, builtin_fcfs.c, happens to guard its entire candidate queue with an internal lock (priv->lock, taken in every hook including end_session()) -- an implementation detail plugin.h's own documented contract does not require picker plugins to provide (\"the picker thread holds no lock the allocator thread needs\"). A future or alternate picker plugin implemented per that documented (lock-free) contract would race sim_reconcile_unaccounted_jobs()'s driver-thread on_dispatch() call against the picker thread's own end_session()/next_job() tail-end activity from the prior round, unsynchronized.",
"fix": "NOT FIXED -- audit finding only, logged per user request to run the full scheduler-driver-equivalence audit; not addressed in the same pass as bug-1836 to keep that fix's diff properly scoped and reviewable (a real fix here means either moving the on_dispatch() call onto the allocator thread's own loop -- a larger restructuring of the reconciliation mechanism's call boundary -- or documenting/enforcing that every picker plugin must be internally lock-safe against this specific driver-thread call, which would need to be added to plugin.h's own interface contract, not just builtin_fcfs.c's implementation). Currently dormant/inert: builtin_fcfs.c is the only registered non-no-op picker plugin in production, and its internal locking happens to make every call site safe today. Needs its own design decision (which of the two fix directions above) before implementation.",
"tags": [
"scheduler",
"sim",
"sim_reconcile_unaccounted_jobs",
"picker-plugin",
"threading-contract",
"jbuf",
"xce-audit-invariants",
"dormant",
"not-fixed",
"needs-design-decision",
"related-to-bug-1759",
"related-to-bug-1779"
],
"related_bugs": [
"bug-1759",
"bug-1779",
"bug-1836"
],
"occurrences": 1,
"last_seen": "2026-09-21T05:00:00Z"
},
{
"id": "bug-1838",
"timestamp": "2026-09-21T00:30:00Z",
"error_message": "Follow-up architectural correction after bug-1836 (found during user discussion, not a separately-reported live symptom): even with bug-1836's capacity gate in place, sim_reconcile_unaccounted_jobs() still unnecessarily PREEMPTED the live picker/allocator for any freshly-PEND job with a matching recorded DISPATCH decision, rather than letting the live path -- now genuinely capacity-checked, per scheduler-sim-live-allocation -- decide it on its own the way normal mode always has. User specifically asked why sim mode diverged from normal mode given they 'share the same core', then proposed the architecturally correct framing directly: the simulator should run the identical decision-making core (minus controller/worker I/O) and use the recording only to verify/reconstruct, not to short-circuit live decisions.",
"file": "src/scheduler/src/sim_loop.c",
"root_cause": "sim_reconcile_unaccounted_jobs()'s trigger condition was widened from JOB_IS_RUN-only to state-independent (PEND-or-RUN) by the scheduler-sim-eager-reconciliation spec (2026-09-11), to fix a real problem at the time: a job's recorded DISPATCH decision only exists in the ONE round's decoded_session it was captured in (freed at round end), so a RUN-only gate meant reconciliation could never find it once the job's SEPARATE, later-replayed RUN-confirmation event finally made it RUN. That widening was the only available fix at the time because sim mode's own live picker/allocator path had no working capacity accounting (bug-1613) and so could not be trusted to independently re-derive the SAME decision the original recording made -- reconciliation's job was to force in the recording's answer since the live path's own answer might diverge or be wrong. scheduler-sim-live-allocation (completed 2026-09-19) closed that gap for real: sim mode now spawns the SAME real, capacity-checked sched_job_picker()/sched_allocator() threads (sched_election.c) normal mode does, with sched_account_jobs() correctly wired into its round sequence -- its own design.md explicitly states a 'Live/Recorded Authority Rule' (live decision wins when the live path reaches a job; reconciliation is a narrow safety net, not a first-resort source of truth) but the actual per-round ORDERING (reconciliation runs unconditionally before the live picker/allocator gets a turn each round) never enforced that rule -- so the PEND-inclusive trigger left in place after scheduler-sim-live-allocation landed kept preempting jobs the live path was now fully capable of deciding correctly and verifiably on its own. This was the deeper root cause underlying bug-1836's exact reproduction (44 jobs force-dispatched via reconciliation before the live picker/allocator ever got a turn); bug-1836's capacity gate made that force-application SAFE (never exceeds real capacity) but did not stop it from happening. Also confirmed (via direct investigation): a snapshot-loaded job (snapshot_load.c's xce_sched_core_load_snapshot()) is registered with the live picker via on_job_push() for every restored job unconditionally, RUN or PEND -- so there is no 'invisible to the picker' case a RUN-only trigger could miss; every legitimate reconciliation candidate today is also, structurally, a job the live picker/allocator can independently resolve. The sole remaining genuine case: a job already RUN before this replay's own observation window (dispatched too far in the past for its DISPATCH decision to still be in any currently-decoded session, and never offered by next_job() since RUN jobs are never picker candidates) -- for that job alone, the recording remains the sole source of truth.",
"fix": "FIXED (2026-09-21). Narrowed sim_reconcile_unaccounted_jobs()'s trigger back to JOB_IS_RUN(job->state)-only (undoing the 2026-09-11 widening), so a freshly-PEND job now always flows through the same real, capacity-checked live picker/allocator normal mode uses -- verified via the pre-existing --sim-verify generate-then-compare mechanism -- rather than being force-applied from history. bug-1836's capacity gate (sim_reconciled_alloc_fits_capacity()/sim_reconcile_toggle_prior_charges()) remains in place unchanged as defense-in-depth for the narrower remaining RUN-only case. Fixing this correctly broke 6 test binaries whose fixtures depended on the old PEND-inclusive behavior; investigation of each was more nuanced than initially guessed: test_sim_verify.c's and test_sim_contention_parity.c's failures were NOT missing-worker-registration issues (the initial hypothesis) but rather that those two files each stub sched_job_picker()/sched_allocator() to a deliberately PEND-only pair (used to keep their OTHER, unrelated tests decoupled from real dispatch logic) -- under the new design a PEND job in those fixtures genuinely can never be dispatched by anything, so the correct fix was inverting each test's assertions to lock in the new, correct behavior (a PEND job stays PEND, and --sim-verify correctly reports the resulting MISMATCH against a history that shows it dispatched, since these minimal harnesses have no real capacity-tracked path to reproduce that decision) rather than trying to make the stub dispatch something it structurally cannot. The 5 directly-affected tests in test_sim_reconcile_unaccounted.c were each individually re-examined for their TRUE intent and fixed accordingly: one inverted/renamed to lock in 'PEND is now always ignored regardless of a matching decision' (test_reconcile_ignores_pend_state_job_even_when_dispatch_recorded), three converted from a PEND-starting to a RUN-starting fixture job (still meaningfully testing the same underlying resInUse-timing/retry/multi-round mechanics, now via the only state that's still eligible), and one (the bug-1759/1779 on_dispatch-retirement regression test) kept its job PEND-built-then-force-set-to-RUN specifically to reproduce the pathological 'RUN job the picker was never told about' state reconciliation's on_dispatch() safety net still needs to guard against, since no real production path can reach that shape anymore post-bug-1838. A sweep for other coincidentally-passing tests found and fixed one more: bug-1836's own new regression test (test_reconcile_respects_host_capacity_under_contention) also used two PEND jobs, converted to RUN with its 'loser' assertion corrected to match. Added a new live-dispatch-vs-recording round-trip test (test_sim_live_allocation.c) to restore the coverage inverting test_sim_verify.c's test removed (that a genuinely live-dispatched decision correctly reproduces a matching historical recording when both the job and the resources it wants are real). scheduler-sim-eager-reconciliation's own requirements.md/design.md got dated 'Correction (2026-09-21, bug-1838)' notes in place, following the exact precedent format its own 2026-09-11 correction already established, superseding (not deleting) the original PEND-inclusive Acceptance Criterion 1 text. Verified: full scheduler suite 85/85 Ok, 0 Fail (asan profile); independently re-confirmed by running sched_sim_reconcile_unaccounted_test (22/22), sched_sim_verify_test (17/17), sched_sim_contention_parity_test (2/2), and sched_sim_live_allocation_test (2/2) directly.",
"tags": [
"scheduler",
"sim",
"replay",
"sim_reconcile_unaccounted_jobs",
"scheduler-driver-equivalence",
"reconciliation-trigger",
"design-correction",
"architecture",
"user-directed",
"xce-audit-invariants",
"fixed",
"related-to-bug-1836"
],
"related_bugs": [
"bug-1836",
"bug-1837",
"bug-1611",
"bug-1613",
"bug-1759",
"bug-1779",
"bug-1804"
],
"occurrences": 1,
"last_seen": "2026-09-21T02:40:00Z"
}
]
}

View file

@ -242,3 +242,7 @@
- **(2026-09-15) `controller-status`'s spec.json tracks its delta/reopen history in a single top-level `delta` object, not an array** — when starting a new delta cycle on top of one that already reached `validate_impl.decision: "GO"`, preserve the just-closed cycle by nesting it under `delta.prior_delta` (reopened_at/reason/new_requirements/impl_status) rather than overwriting it outright, so the requirement-number provenance ("which delta added Req N") stays recoverable from spec.json alone without needing git blame.
- **(2026-09-20, bug-1833) `xceadmin`'s bundled JSON schemas exist in TWO copies on disk, and editing the wrong one is silently discarded on the next build.** `src/configs/schema/*.schema.json` is the REAL canonical source; `src/xceadmin/src/xcecommon/schema_files/json/*.schema.json` is a checked-in mirror that `meson-build.sh:104` unconditionally overwrites from the canonical copy on every `xceadmin` build (`cp "$SRC_DIR/configs/schema/"*.schema.json "$SCHEMA_DST/json/"`). Editing only the xceadmin-tree copy produces a schema change that appears correct in `git diff` but has ZERO effect on any real build or test run — confirmed by editing only that copy, rebuilding, and watching a new schema constraint simply not fire (no error, just silent no-op), then finding the sync line and editing both copies to fix it. **Before editing any file under `src/xceadmin/src/xcecommon/schema_files/`, grep `meson-build.sh` for a sync/cp step first** — the same "editing xceadmin source doesn't retroactively update an already-built snapshot" caution documented elsewhere in this file extends to specific FILES within the tree being silently resynced from elsewhere, not just whole-package staleness.
- **(2026-09-20, bug-1834) `var` (rtti's `typeinfo.h`) is `#define var void*` — a raw macro, not a typedef — so `var a, b, c;` does NOT declare three `void*`s.** It expands to `void* a, b, c;`, which in C only pointer-types the FIRST declarator; `b` and `c` silently fall back to an implicit `int` (a legacy K&R extension GCC still tolerates by default), producing `-Wincompatible-pointer-types`/`-Wint-conversion` errors at every later use, often far from the declaration and easy to misread as unrelated call-site bugs. Hit writing 4 new tests in `test_limit.c` that copied `var umgr, qmgr, hmgr, pmgr;` from a working single-variable pattern elsewhere in the file. **Always declare each `var` on its own line** (`var umgr;` / `var qmgr;` / ...) — this codebase's own existing tests (`test_build_limit_ok1` et al.) already do this consistently; the comma-list form is the anomaly, not the norm, despite looking harmless for an ordinary typedef'd type.
- **(2026-09-21, bug-1836, `/xce-audit-invariants scheduler`) A driver-equivalence audit needs to check every DECISION-PRODUCING path, not just the "obvious" live one — a second, backstop mechanism (Recorded-Decision Reconciliation, `sim_reconcile_unaccounted_jobs()`, sim_loop.c) can silently bypass the SAME admission-check plumbing the primary path (`sched_allocator()`/`sched_try_alloc_on_host_()`) correctly uses, and a design doc's own stated assumption ("once resInUse is correctly populated in sim mode, the existing capacity-check code path produces the same outcome normal mode would") can be TRUE for the primary path and silently FALSE for a secondary one added by a LATER spec that never re-verified it.** Symptom was a live user report (44 single-slot jobs dispatched onto a 22-slot host during `xce_scheduler -i -s <replay>`) that superficially looked like a resource-accounting display bug (`used=22.00` looked self-consistent) but was actually an admission-control bypass: the reconciliation backstop force-applied recorded DISPATCH decisions from a `.replay` file's OWN history with zero live capacity re-validation. Fix pattern: re-run the SAME `sched_check_host_satisfy()`/`sched_check_shared_satisfy()` calls the live path uses, resolving the candidate's resnames fresh against `core->resmgr` (a decoded/recorded allocation carries only names+values, not resolved indices).
- **When a fix needs to peek/seed then undo a resource-accounting baseline (e.g. "temporarily charge every already-running job so a capacity check sees real usage, then undo it before returning"), do NOT reach for the SAME by-name accounting entry point (`sched_account_jobs()`) an existing test elsewhere specifically instruments and call-count-asserts as a protected invariant ("must be called exactly once per round").** Doubling that exact symbol's call count broke `test_sim_contention_parity.c`'s `g_round_idx` mechanism — and the failure was initially confusing because a coincidental fixed-size counter array (`g_round_resinuse[8]`) CAPPED the observed count, making a 2-round fixture's true delta (+2, correctly proportional) look identical to a 5-round fixture's true delta (which should have been +5 but was capped at the array's own ceiling, also showing "+2" after the cap) — always check whether an assertion's input value could itself be silently clamped by an array bound before concluding two different-sized reproductions "coincidentally" show the same delta. Fix: build a small `static` local helper in the SAME file doing the identical `sched_charge_account()`/`sched_uncharge_account()` iteration, deliberately NOT calling the protected symbol by name, so the call-count invariant elsewhere stays intact while the underlying accounting effect is identical.
- **Adding a genuine admission/capacity check to test-owning code can turn dozens of PRE-EXISTING tests red for a completely mechanical, expected reason: their fixtures name a resource/host in a mock allocation without ever registering a real object of that name, because nothing previously validated it.** 12 tests across 2 files broke this way when bug-1836's fix added a `sched_get_worker(hostname) == NULL` check — each one just needed a real (or, in a file that stubs the checker entirely and never even reaches the worker lookup meaningfully, a bare/minimal) worker registered for whichever hostname its fixture already referenced, with generous capacity (these tests were about ownership/dedup/timing mechanics, not capacity limits). This class of breakage is a sign the new check is doing its job, not a sign the check is wrong — verify each failure's root cause individually before assuming a batch of new failures means the production fix itself is flawed.
- **(2026-09-21, bug-1838) When a design revision removes a secondary decision-producing shortcut so more work routes through "the real path," don't assume every test file's "real path" IS real — some test files deliberately stub the exact mechanism you're now relying on, for reasons unrelated to what you're testing.** Predicted two failing tests' root cause as "fixture never registered a real worker" (correct pattern from the bug-1836 experience immediately above) — actually wrong for both: `test_sim_verify.c` and `test_sim_contention_parity.c` each stub `sched_job_picker()`/`sched_allocator()` to a deliberately PEND-only pair, used to keep those files' OTHER tests decoupled from real dispatch logic. Narrowing `sim_reconcile_unaccounted_jobs()`'s trigger to stop force-applying a PEND job's recorded decision meant those fixtures' jobs now correctly stay PEND forever (nothing in that file can ever dispatch them) — the fix wasn't adding a missing worker, it was inverting the test's own assertion to lock in the new, correct behavior ("still PEND, `--sim-verify` correctly reports the resulting MISMATCH, since this minimal harness has no real capacity-tracked path to reproduce a historical dispatch"). **Lesson: before assuming a test failure needs a "make the fixture more realistic" fix, grep the failing test's OWN file for whether the specific mechanism your change now depends on (a picker, an allocator, a resource check) is the REAL production implementation or a same-named local override/stub — a same-file `grep -n "^void\* sched_job_picker\|^int sched_check_host_satisfy"` etc. before touching the test.** A prior, adjacent, and still-correct architectural point (a design doc's own "Live/Recorded Authority Rule" can be stated but never actually enforced by the code's real ordering) is a separate, valid finding from this same investigation — don't conflate "the design doc says X should happen" with "therefore the live mechanism in THIS test file is capable of making X happen."

View file

@ -5533,3 +5533,70 @@
| 20:38 | Edited src/include/typemap.h | 7→10 lines | ~151 |
| 20:39 | Edited src/controller/testsuite/test_e2e_capnp.c | modified test_ctrl_hello_request_key_round_trip() | ~1131 |
| 20:39 | Edited src/controller/testsuite/test_e2e_capnp.c | 2→3 lines | ~46 |
| 22:07 | Edited src/scheduler/include/core.h | 6→7 lines | ~39 |
| 22:08 | Edited src/scheduler/include/core.h | modified sched_account_jobs() | ~753 |
| 22:08 | Edited src/scheduler/include/scheduler.h | 13→14 lines | ~248 |
| 22:08 | Edited src/scheduler/src/scheduler.c | added 5 condition(s) | ~251 |
| 22:09 | Edited src/scheduler/src/sim_loop.c | 4→7 lines | ~83 |
| 22:10 | Edited src/scheduler/src/sim_loop.c | modified returns() | ~405 |
| 22:11 | Edited src/scheduler/src/sim_loop.c | modified for() | ~166 |
| 22:12 | Edited src/scheduler/src/sim_loop.c | added 1 condition(s) | ~659 |
| 22:12 | Edited src/scheduler/src/sim_loop.c | modified sched_account_jobs() | ~347 |
| 22:14 | Edited src/scheduler/src/sim_loop.c | added 16 condition(s) | ~1298 |
| 22:20 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | added 1 condition(s) | ~1328 |
| 22:32 | Edited src/scheduler/src/sim_loop.c | modified for() | ~90 |
| 22:33 | Edited src/scheduler/src/sim_loop.c | modified if() | ~293 |
| 22:33 | Edited src/scheduler/src/sim_loop.c | modified if() | ~114 |
| 22:43 | Edited src/scheduler/src/sim_loop.c | modified returns() | ~614 |
| 22:44 | Edited src/scheduler/src/sim_loop.c | sched_unaccount_jobs() → sim_reconcile_toggle_prior_charges() | ~331 |
| 22:45 | Edited src/scheduler/src/sim_loop.c | sched_unaccount_jobs() → sim_reconcile_toggle_prior_charges() | ~155 |
| 22:46 | Edited src/scheduler/src/sim_loop.c | added 4 condition(s) | ~526 |
| 22:47 | Edited src/scheduler/include/core.h | reduced (-15 lines) | ~119 |
| 22:48 | Edited src/scheduler/src/scheduler.c | removed 40 lines | ~13 |
| 23:13 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified test_reconcile_captures_non_run_state_job_when_dispatch_recorded() | ~283 |
| 23:13 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified test_reconcile_reconstructs_allocs_and_accounts_from_matching_dispatch() | ~168 |
| 23:13 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified test_reconcile_then_release_via_job_state_change_is_asan_clean() | ~149 |
| 23:14 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 9→14 lines | ~117 |
| 23:14 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 8→13 lines | ~127 |
| 23:14 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | expanded (+10 lines) | ~211 |
| 23:14 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 9→14 lines | ~119 |
| 23:15 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 9→14 lines | ~118 |
| 23:15 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | expanded (+10 lines) | ~207 |
| 23:19 | Edited src/scheduler/testsuite/test_sim_verify.c | modified add_bare_worker() | ~339 |
| 23:20 | Edited src/scheduler/testsuite/test_sim_verify.c | expanded (+7 lines) | ~129 |
| 23:28 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 8→13 lines | ~224 |
| 00:49 | Edited src/scheduler/src/sim_loop.c | modified condition() | ~1331 |
| 00:50 | Edited src/scheduler/src/sim_loop.c | 4→9 lines | ~117 |
| 01:05 | Edited src/scheduler/testsuite/test_sim_live_allocation.c | added 1 condition(s) | ~961 |
| 01:07 | Edited src/scheduler/testsuite/test_sim_live_allocation.c | modified comment() | ~2255 |
| 01:08 | Edited src/scheduler/testsuite/test_sim_verify.c | modified INVERTED() | ~671 |
| 01:09 | Edited src/scheduler/testsuite/test_sim_verify.c | reduced (-7 lines) | ~573 |
| 01:10 | Edited src/scheduler/testsuite/test_sim_verify.c | 2→2 lines | ~24 |
| 01:16 | Edited src/scheduler/testsuite/test_sim_contention_parity.c | modified CORRECTED() | ~1686 |
| 01:17 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified test_reconcile_ignores_pend_state_job_even_when_dispatch_recorded() | ~1005 |
| 01:18 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 2→2 lines | ~24 |
| 01:19 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified 12() | ~798 |
| 01:19 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 3→3 lines | ~63 |
| 01:20 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified REVISED() | ~803 |
| 01:27 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified 15() | ~582 |
| 01:27 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 3→3 lines | ~54 |
| 01:28 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | expanded (+9 lines) | ~530 |
| 01:33 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified test_reconcile_removes_reconciled_job_from_picker_candidates() | ~822 |
| 01:33 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 11→12 lines | ~161 |
| 01:35 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified 13() | ~458 |
| 01:35 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 6→7 lines | ~111 |
| 01:36 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified regression() | ~312 |
| 01:36 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 2→2 lines | ~33 |
| 01:38 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified if() | ~291 |
| 01:39 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 4→4 lines | ~35 |
| 01:41 | Edited .kiro/specs/scheduler-sim-eager-reconciliation/design.md | modified Correction() | ~1035 |
| 01:41 | Edited .kiro/specs/scheduler-sim-eager-reconciliation/requirements.md | modified Correction() | ~612 |
| 01:52 | Edited src/scheduler/testsuite/test_sim_live_allocation.c | 6→7 lines | ~42 |
| 01:52 | Edited src/scheduler/testsuite/test_sim_live_allocation.c | 36→38 lines | ~393 |
| 01:59 | Edited src/scheduler/testsuite/test_sim_live_allocation.c | 3→4 lines | ~43 |
| 02:13 | Edited src/scheduler/src/sim_verify.c | expanded (+12 lines) | ~220 |
| 02:20 | Edited src/scheduler/testsuite/test_sim_live_allocation.c | 2→2 lines | ~35 |
| 02:30 | Edited src/scheduler/testsuite/test_sim_live_allocation.c | 2→2 lines | ~35 |
| 02:31 | Edited src/scheduler/testsuite/test_sim_live_allocation.c | expanded (+22 lines) | ~556 |
| 02:31 | Edited src/scheduler/testsuite/test_sim_live_allocation.c | 4→3 lines | ~28 |
| 02:31 | Edited src/scheduler/src/sim_verify.c | removed 13 lines | ~17 |