lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(scheduler): 3 --sim-verify false positives + snapshot resreq_fp gap (bug-1844/1845/1772)

Browse source 
/xce-audit-invariants scheduler dimension-1/4 audit, prompted by a fresh
user-captured .replay file that still showed massive --sim-verify mismatch
volume after bug-1843's fix. Root-caused via temporary diagnostic
instrumentation against the real capture (reverted, never committed) rather
than trusting the surface symptom.

bug-1844: sim_verify_alloc_eq() (sim_verify.c) compared `instid`, a bare
per-process monotonic counter (scheduler.c's sched_get_id(), core->nextid)
stamped at allocation-build time -- not semantic content. The original
recording's counter and a fresh replay's counter have no reason to reach
the same value for "the same" dispatch. Live-verified: every DISPATCH
mismatch showed identical value/duration/resname/reserve, differing only
in instid (e.g. 826 vs 1). Dropped from the comparison, same treatment as
bug-1842's `from` exclusion.

bug-1845: sim_verify_job_preason_eq() required PEND decisions' full
host/resource reason-detail lists to match. core.h documents "simulation
isolation" as deliberate design: sim/replay never registers
core->unsatisfiable_lookup_cb, so it can never hit the unsatisfiable-cache
main.c's normal-mode branch registers. A live cache hit
(sched_pend_from_cached_reason_()) records only main_reason with zero
detail; replay always falls through to the full per-host computation and
always produces detail. Same semantic outcome (PEND, same main_reason) at
two architecturally-legitimate detail levels. Relaxed the comparator to
main_reason only. Separately verified the cache itself is sound (key is
safe within a round since capacity only decreases and the cache clears
every round) -- ruling it out as the cause of a second, still-open
divergence (bug-1846: one job genuinely dispatches one round early in
replay vs. the original recording -- a suspected resource-accounting
parity gap, not yet root-caused, logged for follow-up).

bug-1772 (previously deferred, incorrectly tagged "fixed"): re-found
independently by a dimension-4 snapshot-field-completeness audit subagent.
snapshot_build_primary_schedobj() (snapshot_load.c) never computed
resreq_fp, unlike its declared model core_build_primary_schedobj()
(core_events.c) -- every snapshot-restored job got a zeroed fingerprint,
which alloc_cache.c's cache key folds in directly alongside
user/queue/project, so two distinct restored jobs sharing those three
fields could silently share an incorrect unsatisfiable-cache verdict.
Added the same 4-field computation the live path uses.

TDD throughout: each fix has a dedicated regression test, and each was
verified via genuine red (revert fix, confirm the new test fails) then
green (restore, full 85/85 scheduler suite). Also ran three parallel
dimension audits (core I/O isolation, picker/allocator jbuf rendezvous) --
both clean, no violations found.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-23 13:40:16 +00:00
parent 5ee329b1b8
commit fcf119b4ac
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
6 changed files with 309 additions and 50 deletions
Download patch file Download diff file Expand all files Collapse all files

88
src/scheduler/testsuite/test_sim_verify.c
View file

@ -659,6 +659,42 @@ static void test_dispatch_decisions_differing_from_timestamp_still_match(void)
"PASS\n");
}
/*
* test_dispatch_decisions_differing_instid_still_match - bug-1844
* regression: `instid` (xce_alloc_t's per-process monotonic allocation-
* instance counter, stamped from `sched->core.nextid++` at allocation-build
* time -- scheduler.c's sched_get_id()) is an execution-order bookkeeping
* value, not part of a decision's semantic identity, exactly like `from`
* above (bug-1838-followup). The ORIGINAL recording's counter may have been
* running, and building allocations, for far longer before this capture
* window started; a fresh replay's own counter starts whatever its loaded
* snapshot's nextid happens to be -- the two have no reason to reach the
* same value for "the same" dispatch, even when every other field is
* byte-for-byte identical. Uses two `instid` values matching a real
* captured .replay file's own observed divergence (826 vs 1) to prove this
* is deliberate exclusion, not a fluke of small numbers happening to
* coincide.
*/
static void test_dispatch_decisions_differing_instid_still_match(void)
{
xce_alloc_t* res_a[1]
= { make_test_alloc("mem", "", 826ULL, 307200.0, 100, -1) };
xce_host_alloc_t* host_a[1] = { make_test_host_alloc("hostA", res_a, 1) };
xce_job_alloc_t* expected_alloc = make_test_job_alloc(42, host_a, 1);
xce_alloc_t* res_b[1]
= { make_test_alloc("mem", "", 1ULL, 307200.0, 100, -1) };
xce_host_alloc_t* host_b[1] = { make_test_host_alloc("hostA", res_b, 1) };
xce_job_alloc_t* actual_alloc = make_test_job_alloc(42, host_b, 1);
replay_decision_t expected = make_replay_dispatch_decision(expected_alloc);
xce_decision_t* actual = make_dispatch_decision(actual_alloc);
TEST_ASSERT_TRUE(sim_decision_eq(&expected, actual));
printf("test_dispatch_decisions_differing_instid_still_match: PASS\n");
}
static void test_pend_decisions_with_identical_content_match(void)
{
replay_decision_t expected = make_replay_pend_decision(9001ULL, 3);
@ -695,6 +731,56 @@ static void test_pend_decisions_differing_main_reason_mismatch(void)
printf("test_pend_decisions_differing_main_reason_mismatch: PASS\n");
}
/*
* test_pend_decisions_differing_reason_detail_still_match - bug-1845
* regression: the recorded (expected) side represents a live cache-hit
* PEND (core.h's documented "simulation isolation" -- sim/replay never
* registers core->unsatisfiable_lookup_cb, so it can never itself produce
* this shape, only ever receive it from a recorded normal-mode run) --
* main_reason set, zero host/resource reason detail
* (sched_pend_from_cached_reason_(), scheduler.c, never adds any). The
* live (actual) side represents replay's own full per-host computation for
* the SAME job/round -- same main_reason, but with real host- and
* resource-based reason detail attached (sched_try_alloc_on_host_(),
* resource.c). Same semantic outcome (PEND, same main reason) at two
* different, architecturally-legitimate detail levels -- must still
* compare equal.
*/
static void test_pend_decisions_differing_reason_detail_still_match(void)
{
replay_decision_t expected = make_replay_pend_decision(9001ULL, 0);
xce_pendreason_t* reason = make_pendreason(9001ULL);
xce_host_reason_t* hreasons[1];
xce_res_reason_t* rreasons[2];
xce_decision_t* actual;
hreasons[0] = (xce_host_reason_t*)calloc(1, sizeof(xce_host_reason_t));
hreasons[0]->host = strdup("workerA");
hreasons[0]->reason = 1;
rreasons[0] = (xce_res_reason_t*)calloc(1, sizeof(xce_res_reason_t));
rreasons[0]->resource = strdup("mem");
rreasons[0]->host = strdup("workerA");
rreasons[0]->reason = 2;
rreasons[1] = (xce_res_reason_t*)calloc(1, sizeof(xce_res_reason_t));
rreasons[1]->resource = strdup("slots");
rreasons[1]->host = strdup("workerA");
rreasons[1]->reason = 2;
reason->reasons->main_reason = 0;
reason->reasons->nhost_reasons = 1;
reason->reasons->host_based_reasons = hreasons;
reason->reasons->nres_reasons = 2;
reason->reasons->res_based_reasons = rreasons;
actual = make_pend_decision(reason);
TEST_ASSERT_TRUE(sim_decision_eq(&expected, actual));
printf("test_pend_decisions_differing_reason_detail_still_match: PASS\n");
}
static void test_decision_kind_mismatch_reported(void)
{
replay_decision_t expected = make_replay_pend_decision(1ULL, 0);
@ -1784,9 +1870,11 @@ int main(void)
RUN_TEST(test_dispatch_decisions_differing_host_mismatch);
RUN_TEST(test_dispatch_decisions_differing_resource_value_mismatch);
RUN_TEST(test_dispatch_decisions_differing_from_timestamp_still_match);
RUN_TEST(test_dispatch_decisions_differing_instid_still_match);
RUN_TEST(test_pend_decisions_with_identical_content_match);
RUN_TEST(test_pend_decisions_differing_jobid_mismatch);
RUN_TEST(test_pend_decisions_differing_main_reason_mismatch);
RUN_TEST(test_pend_decisions_differing_reason_detail_still_match);
RUN_TEST(test_decision_kind_mismatch_reported);
RUN_TEST(test_preempt_decision_always_reports_mismatch);
RUN_TEST(test_decision_eq_defends_against_null_inputs);