lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(scheduler): record events before the core applies them (bug-1852)

Browse source 
The recorder encoded each controller event batch after
xce_core_push_events() had applied it, and applying moves payload fields
out of the events: a WORKERS_PUSH's resvals (core_apply_worker_resvals
MOVE_PTR), a JOBS_PUSH's allocs, a JOB_RUSAGE's rusages, and a
CONFIG_PUSH's shared resvals (sched_apply_pending_config recorded after
sched_init_config). Recordings silently lost exactly those fields -- a
worker's resval names were written with NULL values, so --sim-verify
replayed "no slots" and turned 3 live DISPATCHes into PENDs.

driver.c now records the batch before the push; the periodic in-stream
SnapshotEntry moves out of sched_record_write_events() into new
sched_record_checkpoint_after_apply(), called after the push so a
checkpoint still reflects the events written ahead of it. The config
push is likewise recorded before sched_init_config().

Regression test: test_recorded_workers_push_keeps_resval_values (RED:
recorded value NULL; GREEN after). Periodic-checkpoint tests updated to
the write-then-checkpoint contract. meson test 109/109; 25 live soak
recordings replay with 0 mismatches.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-24 08:41:55 +00:00
parent ca4db13a79
commit 9c02184fb4
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
10 changed files with 258 additions and 33 deletions
Download patch file Download diff file Expand all files Collapse all files

17
.wolf/buglog.json
View file

@ -25187,6 +25187,23 @@
"resvals",
"xstring"
]
},
{
"id": "bug-1852",
"timestamp": "2026-09-24T08:10:00Z",
"error_message": "Scheduler replay-parity soak gate (run-ha-soak.sh --sequential 6, post bug-1851): 1 of 7 clean recordings failed --sim-verify with 3 MISMATCHes, all in its first decision round: live 'DISPATCH jobid=... nhosts=1' vs sim 'PEND ... main_reason=0' (sim: unsatisfy_ unmet resource 0 (slots) shortfall 1). Sim trace: snapshot at record start restored the worker with 0/0 resvals; the recording's only WORKERS_PUSH 'carries 10 resval(s)' but core_apply_worker_resvals 'applied 0' -- every recorded NameValue.val was NULL.",
"file": "src/scheduler/src/driver.c",
"root_cause": "The recorder encoded each event batch AFTER applying it: driver.c's xce_sched_driver_translate_pending_events() called xce_core_push_events() and only then sched_record_write_events() on the same xce_core_event_t payloads. Applying is destructive -- core_apply_worker_resvals() MOVE_PTRs each resval (nv->val -> worker->resvals), core_build_job MOVE_PTRs a JOBS_PUSH's allocs, JOB_RUSAGE transfers rusages, and core_apply_config_resvals() MOVE_PTRs a CONFIG_PUSH's shared resvals (bug-1847's new config-push recording also recorded after sched_init_config). So recordings silently lost exactly those fields: a worker's resval names were written with NULL values. Masked whenever the worker's values were already in the record-start snapshot; exposed when a recording opened after a scheduler restart, before the worker's first value push. Field-completeness violation (persistence.md). bug-1688's 'a repeat WORKERS_PUSH carries 20 resval names with nv->val==NULL' was traced from a replay file and was very likely this same stripping, not something the controller sends.",
"fix": "FIXED (2026-09-24, TDD). driver.c records the batch BEFORE xce_core_push_events(); the periodic SnapshotEntry checkpoint was split out of sched_record_write_events() into new sched_record_checkpoint_after_apply() (recording.h/recording_events.c), called AFTER the push so a checkpoint still reflects the events ahead of it in the stream. sched_apply_pending_config() (scheduler.c) likewise records the CONFIG_PUSH before sched_init_config() and checkpoints after. Stubs added to the 4 driver test units that stub the recorder. Regression test test_record_lifecycle.c::test_recorded_workers_push_keeps_resval_values (RESOURCES_PUSH 'slots' + WORKERS_PUSH hostA slots=22 through the real driver with recording on; decodes the recorded WORKERS_PUSH) -- RED (recorded val NULL), GREEN after; 3 periodic-checkpoint tests updated to the write-then-checkpoint contract. Full meson test 109/109 OK.",
"tags": [
"scheduler",
"recording",
"replay",
"sim-verify",
"field-completeness",
"MOVE_PTR",
"driver"
]
}
]
}