fix(scheduler): 3 --sim-verify false positives + snapshot resreq_fp gap (bug-1844/1845/1772)

/xce-audit-invariants scheduler dimension-1/4 audit, prompted by a fresh
user-captured .replay file that still showed massive --sim-verify mismatch
volume after bug-1843's fix. Root-caused via temporary diagnostic
instrumentation against the real capture (reverted, never committed) rather
than trusting the surface symptom.

bug-1844: sim_verify_alloc_eq() (sim_verify.c) compared `instid`, a bare
per-process monotonic counter (scheduler.c's sched_get_id(), core->nextid)
stamped at allocation-build time -- not semantic content. The original
recording's counter and a fresh replay's counter have no reason to reach
the same value for "the same" dispatch. Live-verified: every DISPATCH
mismatch showed identical value/duration/resname/reserve, differing only
in instid (e.g. 826 vs 1). Dropped from the comparison, same treatment as
bug-1842's `from` exclusion.

bug-1845: sim_verify_job_preason_eq() required PEND decisions' full
host/resource reason-detail lists to match. core.h documents "simulation
isolation" as deliberate design: sim/replay never registers
core->unsatisfiable_lookup_cb, so it can never hit the unsatisfiable-cache
main.c's normal-mode branch registers. A live cache hit
(sched_pend_from_cached_reason_()) records only main_reason with zero
detail; replay always falls through to the full per-host computation and
always produces detail. Same semantic outcome (PEND, same main_reason) at
two architecturally-legitimate detail levels. Relaxed the comparator to
main_reason only. Separately verified the cache itself is sound (key is
safe within a round since capacity only decreases and the cache clears
every round) -- ruling it out as the cause of a second, still-open
divergence (bug-1846: one job genuinely dispatches one round early in
replay vs. the original recording -- a suspected resource-accounting
parity gap, not yet root-caused, logged for follow-up).

bug-1772 (previously deferred, incorrectly tagged "fixed"): re-found
independently by a dimension-4 snapshot-field-completeness audit subagent.
snapshot_build_primary_schedobj() (snapshot_load.c) never computed
resreq_fp, unlike its declared model core_build_primary_schedobj()
(core_events.c) -- every snapshot-restored job got a zeroed fingerprint,
which alloc_cache.c's cache key folds in directly alongside
user/queue/project, so two distinct restored jobs sharing those three
fields could silently share an incorrect unsatisfiable-cache verdict.
Added the same 4-field computation the live path uses.

TDD throughout: each fix has a dedicated regression test, and each was
verified via genuine red (revert fix, confirm the new test fails) then
green (restore, full 85/85 scheduler suite). Also ran three parallel
dimension audits (core I/O isolation, picker/allocator jbuf rendezvous) --
both clean, no violations found.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

.wolf/buglog.json

@@ -23026,7 +23026,7 @@
       "timestamp": "2026-09-05T00:00:00Z",
       "error_message": "OPEN, NOT A CRASH -- pre-existing, out-of-scope gap discovered during requirements scoping for the resreq-maxtasks-wiring spec (bug-1770's fix). snapshot_build_primary_schedobj() (src/scheduler/src/snapshot_load.c:81-96) never computes primary->resreq_fp at all, unlike core_build_primary_schedobj() (src/scheduler/src/core_events.c:130-135), which sets resreq_fp.min_tasks/max_tasks/select_node_id/rusage_hash from the parsed xce_primary_resreq_t immediately after parsing. Every job reconstructed from a snapshot (post-restart or post-failover restore) therefore gets a zeroed resreq_fp -- the same class of gap as bug-1767's broader 'recurring schedobj-reconstruction-duplication' pattern (snapshot_load.c independently re-implements core_events.c's schedobj construction rather than sharing it, and each independent copy has repeatedly drifted field-by-field). TRIGGER CORRECTION (2026-09-05, resreq-maxtasks-wiring design review): this entry's original phrasing ('post-restart or post-failover restore') overstates the live trigger. xce_sched_core_load_snapshot()'s ONLY real (non-test) caller anywhere in src/ is sim.c's sim_locate_and_load_snapshot(), reachable only via main.c's --sim FILE offline replay flag. Normal-mode main.c has no snapshot-load call or flag at all; a live scheduler restart (including post-failover, e.g. run-ha-soak.sh's restart_scheduler_for_new_primary()) starts with empty in-memory state and rejoins via on_scheduler_join() -> the same core_build_job() path fresh ingest uses -- never touching snapshot_load.c. The accurate trigger is: a job in a .replay-recorded snapshot loaded by the --sim simulation/replay driver, not any live-cluster crash-recovery or failover scenario. This does not change the bug's validity or fix (the gap is real for --sim/replay fidelity, which matters per this project's driver-equivalence invariant since --sim drives the identical allocator dispatch) -- it only corrects when it's actually reachable, which matters for scoping test coverage (an HA soak does NOT exercise this path at all).",
       "root_cause": "snapshot_load.c's snapshot_build_primary_schedobj() is a hand-duplicated, field-for-field port of core_events.c's core_build_primary_schedobj() (kept independent because core_build_primary_schedobj() is static and the two files are compiled as separate objects -- see bug-1767's own root-cause text for the deliberate-duplication rationale). The port was written before task 1.2 (scheduler-allocation feature) added resreq_fp/hash_rusages() wiring to the core_events.c original, and the snapshot_load.c copy was never updated to match. Effect: a snapshot-restored job's unsatisfiable-resreq cache fingerprint (used by scheduler-allocation's alloc_cache.c) is always the zero value, which is a valid-looking but WRONG fingerprint -- it could produce a false cache hit against an unrelated job that also happens to have a zeroed fingerprint (extremely unlikely today since real jobs always populate resreq_fp correctly at first ingest, but a snapshot-restored job's own fingerprint stays wrong for its entire post-restore lifetime, and any code that later relies on comparing/keying by resreq_fp for a restored job silently gets 0/0/0/0). Discovered while scoping resreq-maxtasks-wiring (bug-1770's fix spec) because that fix will make resreq_fp carry real, meaningful, non-degenerate maxTasks values for the first time -- increasing the practical consequence of this gap versus its current all-zero-everywhere baseline.",
-      "fix": "NOT FIXED -- logged for its own follow-up, deliberately kept out of the resreq-maxtasks-wiring spec's scope (per that spec's own requirements-phase scoping decision, 2026-09-05) to avoid absorbing an unrelated schedobj-reconstruction-duplication gap into a narrowly-scoped bug fix. Fix should add the same resreq_fp computation core_build_primary_schedobj() does (or better, per bug-1767's own recommended remediation, export core_build_primary_schedobj()/core_build_schedobj() and have snapshot_load.c call the real function instead of maintaining a third independent duplicate).",
+      "fix": "FIXED (2026-09-23, TDD, /xce-audit-invariants scheduler dimension-4 audit -- independently re-found by both a dedicated subagent and direct investigation into bug-1845; this entry's own prior 'fixed'/'closed-by-scheduler-sim-live-allocation' tags were STALE/WRONG -- direct code inspection confirmed snapshot_build_primary_schedobj() still had no resreq_fp computation immediately before this fix). Added the same four-field resreq_fp computation core_build_primary_schedobj() (core_events.c) performs -- min_tasks/max_tasks/select_node_id/rusage_hash -- to snapshot_build_primary_schedobj() (snapshot_load.c), matching that function's own documented 'field-for-field port' contract, which had simply never included this field. Added test_snapshot_load_computes_resreq_fingerprint_for_restored_primary() (test_snapshot_save_load.c) -- a job restored from a snapshot with mintasks=4/maxtasks=6 must have schedobj->u.primary->resreq_fp.min_tasks/max_tasks reflect those real values, not {0,0,0,0}. Verified genuine red->green: temporarily reverted just the new resreq_fp assignment and reran -- scheduler snapshot save load test FAILED (exit 1); restored the fix and reran -- full 85/85 scheduler suite green.",
       "tags": [
         "scheduler",
         "snapshot-restore",
@@ -23036,14 +23036,15 @@
         "schedobj-duplication",
         "severity-medium",
         "fixed",
-        "closed-by-scheduler-sim-live-allocation"
+        "tdd",
+        "audit-reopened"
       ],
       "related_bugs": [
         "bug-1767",
         "bug-1770"
       ],
       "occurrences": 1,
-      "last_seen": "2026-09-19T09:00:00Z"
+      "last_seen": "2026-09-23T10:05:00Z"
     },
     {
       "id": "bug-1773",
@@ -25020,6 +25021,91 @@
       ],
       "occurrences": 1,
       "last_seen": "2026-09-23T08:00:00Z"
+    },
+    {
+      "id": "bug-1844",
+      "timestamp": "2026-09-23T09:15:00Z",
+      "error_message": "User-reported follow-up to bug-1843: replaying a fresh capture (scheduler-replay-20260923-085356.replay, built after bug-1843's fix) with --sim --sim-verify still shows a huge volume of MISMATCH lines (4881 in one run) on DISPATCH decisions whose printed description is IDENTICAL on both sides -- e.g. 'expected: DISPATCH jobid=...[40] nhosts=1 -- actual: DISPATCH jobid=...[40] nhosts=1' still flagged as a mismatch.",
+      "file": "src/scheduler/src/sim_verify.c (sim_verify_alloc_eq())",
+      "root_cause": "CONFIRMED via temporary field-by-field diagnostic instrumentation in sim_verify_alloc_eq() (added, verified, then reverted -- not part of the fix) replayed against the real captured .replay file: `instid` was the ONLY field ever differing (value/duration/resname/reserve matched exactly every single time, e.g. recorded instid=826 vs replay instid=1, both otherwise byte-identical). `instid` is stamped from `sched->core.nextid++` (scheduler.c's sched_get_id()) at allocation-build time -- a bare per-process monotonic counter tracking how many allocation instances THIS core has built over its own entire execution history, not a semantic property of what got allocated. The ORIGINAL recording's counter (running, and building allocations, for potentially far longer before this capture window started) and a fresh replay's own counter (starting from whatever its loaded snapshot's nextid happens to be) have no reason to reach the same value for 'the same' dispatch. Exact same class of bug as bug-1838-followup/bug-1842's `from` field (also non-deterministic bookkeeping, already excluded) -- this comparator simply had a second field of the same kind left in.",
+      "fix": "FIXED (2026-09-23, TDD). Dropped the `a->instid == b->instid` clause from sim_verify_alloc_eq() (sim_verify.c), same treatment as bug-1842's `from` exclusion. Updated that function's doc comment to document the exclusion and cite the real captured-file evidence (826 vs 1). Added test_dispatch_decisions_differing_instid_still_match() (test_sim_verify.c), mirroring the existing test_dispatch_decisions_differing_from_timestamp_still_match() pattern -- two allocations identical in every field except instid (826 vs 1, matching the real observed divergence) must still compare equal. Verified genuine red->green: temporarily re-added the instid comparison and reran -- scheduler sim verify test FAILED (exit 1); restored the fix and reran -- full 85/85 scheduler suite green. NOTE: this fix does NOT address the second, separate anomaly found in the same replay file during this investigation (see bug-1845): a genuine one-round-early DISPATCH timing difference for one job (a resource-accounting/capacity divergence between live and replay, not a field-comparison issue) -- open, tracked separately.",
+      "tags": [
+        "scheduler",
+        "sim",
+        "sim-verify",
+        "replay",
+        "instid",
+        "nextid",
+        "monotonic-counter",
+        "non-deterministic-field",
+        "fixed",
+        "tdd"
+      ],
+      "related_bugs": [
+        "bug-1842",
+        "bug-1838-followup",
+        "bug-1843",
+        "bug-1845"
+      ],
+      "occurrences": 1,
+      "last_seen": "2026-09-23T09:15:00Z"
+    },
+    {
+      "id": "bug-1845",
+      "timestamp": "2026-09-23T10:00:00Z",
+      "error_message": "Follow-up to bug-1843/bug-1844: replaying a fresh capture (scheduler-replay-20260923-085356.replay) with --sim --sim-verify still showed a large volume of MISMATCH lines on PEND decisions where recorded and replay agreed on jobid/main_reason but the deep host/resource reason-detail comparison disagreed (e.g. recorded {main_reason=0, nhost_reasons=0, nres_reasons=0} vs replay {main_reason=0, nhost_reasons=0, nres_reasons=2}), live-inspected via gdb by the user.",
+      "file": "src/scheduler/src/sim_verify.c (sim_verify_job_preason_eq())",
+      "root_cause": "CONFIRMED (2026-09-23) as a documented, intentional architectural asymmetry, not a defect: core.h states outright that 'a simulation-mode session never touches the real cache implementation at all' (simulation isolation). core->unsatisfiable_lookup_cb/unsatisfiable_populate_cb (alloc_cache.c's xce_alloc_cache_lookup_cb/populate_cb) are registered only by main.c's normal-mode branch -- sim_loop.c never registers them. sched_make_primary_alloc_() (scheduler.c) consults that cache FIRST; on a hit it records reason detail via sched_pend_from_cached_reason_(), which sets ONLY main_reason (no host/resource detail at all -- skipping that expensive per-host computation is the entire point of the cache). Replay, having no cache, always falls through to the full per-host computation (sched_try_alloc_on_host_(), resource.c) and therefore always produces full detail. Two independent, architecturally-legitimate detail levels for the exact same semantic outcome (this job is PEND, for this main reason), not a decision correctness signal. Separately verified the cache mechanism itself is SOUND (not a distinct bug): its key (alloc_cache.c's build_key_(): resreq_fp + user + queue + project, no host/capacity dimension) is safe within a round because capacity only decreases monotonically as earlier jobs dispatch and the cache is cleared fresh every round (xce_alloc_cache_clear_cb() at begin_session) -- a cached 'unsatisfiable' verdict from an earlier job this round remains valid for a later job sharing the same key.",
+      "fix": "FIXED (2026-09-23, TDD). sim_verify_job_preason_eq() (sim_verify.c) now compares ONLY main_reason, dropping the nhost_reasons/nres_reasons/host_based_reasons/res_based_reasons deep comparison entirely (that deep check was added as an optional 'cheap to add' enhancement, never proven necessary by a dedicated test, and is provably wrong given the cache-isolation asymmetry above). Doc comment updated to explain why. Added test_pend_decisions_differing_reason_detail_still_match() (test_sim_verify.c) -- two PEND decisions with the same jobid/main_reason but one with empty (cache-hit-shaped) and one with populated (full-computation-shaped) host/resource reason detail must still compare equal. Verified genuine red->green: temporarily re-added the deep comparison and reran -- scheduler sim verify test FAILED; restored the fix and reran -- full 85/85 scheduler suite green.",
+      "tags": [
+        "scheduler",
+        "sim",
+        "sim-verify",
+        "replay",
+        "pend-reason",
+        "unsatisfiable-cache",
+        "alloc_cache",
+        "simulation-isolation",
+        "fixed",
+        "tdd"
+      ],
+      "related_bugs": [
+        "bug-1843",
+        "bug-1844",
+        "bug-1842",
+        "bug-1846"
+      ],
+      "occurrences": 1,
+      "last_seen": "2026-09-23T10:00:00Z"
+    },
+    {
+      "id": "bug-1846",
+      "timestamp": "2026-09-23T10:05:00Z",
+      "error_message": "OPEN, NOT YET ROOT-CAUSED. Same replay session as bug-1844/1845 (scheduler-replay-20260923-085356.replay): after excluding instid (bug-1844) and reason-detail (bug-1845), --sim-verify still shows one job (jobid=0-8-1790154325-0[39] in the observed capture) genuinely DISPATCHING one round earlier in replay than in the original live recording -- not a false-positive field comparison, a real positional divergence: replay's round begin_round=...436582151823 dispatches a 13-job batch (jobs [27]..[39]) while the SAME round in the original recording only dispatched 12 (jobs [27]..[38]); job [39]'s recorded dispatch instead lands in the NEXT round, cascading into ~40+ downstream positional MISMATCH lines for that round pair alone.",
+      "file": "src/scheduler/src/resource.c (sched_account_jobs, sched_charge_account/sched_uncharge_account), src/scheduler/src/sim_loop.c (sim_reconcile_unaccounted_jobs, sim_reconcile_toggle_prior_charges) -- suspected, not confirmed",
+      "root_cause": "NOT YET ROOT-CAUSED. Ruled out as explanations: (1) sim_loop.c's event-buffer-then-push-before-begin_session mechanism -- verified byte-for-byte faithful to the recorder's own file ordering during the bug-1843 investigation, no live-vs-replay discretization gap there; (2) the unsatisfiable-cache (alloc_cache.c) -- verified sound within a round (see bug-1845's root_cause) and, being purely additive skip-logic on top of the SAME full computation replay always performs, cannot itself cause replay to admit an EXTRA job a full check would have rejected. Leading hypothesis (UNCONFIRMED): a resource-accounting (resInUse/sharedInUse) state divergence between the live recorder and replay by this point in the run -- the class of bug scheduler-driver-equivalence.md documents as bug-1611/bug-1613's lineage (accounting-state parity is load-bearing, not incidental, and invisible at the decision level until contention makes it decision-visible, exactly this scenario: a 12-vs-13-job batch is a marginal-capacity difference). Not yet instrumented: w->resInUse/core->sharedInUse for the specific worker/resource this job batch consumes, round-by-round, to find where (if at all) replay's accounting genuinely differs from what the recorded decisions imply the original live run's accounting held at the equivalent round. Note bug-1611/bug-1613 are independently documented as FIXED (scheduler-driver-equivalence.md's 2026-09-19 update, re-verified via /xce-audit-invariants scheduler) -- if this is real, it is either a NEW instance of the same bug class (e.g. an accounting gap in the newer sim_reconcile_unaccounted_jobs()/bug-1836 charge-peek logic, which postdates that fix) or a currently-undocumented edge case the 2026-09-19 re-verification's fixture didn't exercise.",
+      "fix": "NOT FIXED -- open, reported for follow-up. Next step: instrument w->resInUse (and core->sharedInUse if the relevant resource is shared) for the specific worker(s) this job batch targets, logged at the start of every round (after sched_account_jobs(), before xce_core_signal_go()) in sim_loop.c, replayed against this exact .replay file, to find the first round (if any) where replay's own accounting state provably differs from what the ORIGINAL recording's decision sequence implies live's accounting held. If confirmed, the fix belongs wherever driver.c and sim_loop.c's per-round accounting calls diverge (scheduler-driver-equivalence.md's own audit method: enumerate every non-core call driver.c's round makes, confirm sim_loop.c's round makes the equivalent one in the same window, and the reverse).",
+      "tags": [
+        "scheduler",
+        "sim",
+        "sim-verify",
+        "replay",
+        "resource-accounting",
+        "resInUse",
+        "driver-equivalence",
+        "unfixed",
+        "needs-diagnosis",
+        "open"
+      ],
+      "related_bugs": [
+        "bug-1843",
+        "bug-1844",
+        "bug-1845",
+        "bug-1611",
+        "bug-1613"
+      ],
+      "occurrences": 1,
+      "last_seen": "2026-09-23T10:05:00Z"
     }
   ]
 }