lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(scheduler): mid-round rejoin config push truncated live rounds (bug-1846 part 2)

Browse source 
sched_publish_retry_loop() calls sched_check_pending_rejoin() after every
successful publish -- inside the driver's decision-drain loop -- so after a
client-library reconnect the scheduler re-joins mid-round. The controller
answers every join (on_scheduler_join(), controller/sched.c) by re-pushing
config, workers and jobs. handler.c applied the config push synchronously
on the handler thread: sched_init_config() -> sched_init_queues() ->
builtin_fcfs on_queue_update(), which nulls the picker cursor when it
reinserts the queue the cursor rests on. next_job() then returned NULL and
the round ended with the rest of the queue never considered.

Evidence (scheduler-replay-20260923-085356.replay, temporary event-stream
diagnostics, reverted): 9 recorded rounds end after 11-12 decisions; every
one is followed by a workersPush + full 50-job jobsPush (the join's resync,
reporting just-dispatched jobs back as PEND), every full-length dispatching
round has none, and the config push never appears because it bypassed the
pending-update queue the recorder writes from -- which is also why replay
could not reproduce the truncation.

Fix (defer to the round edge, same direction as bug-1813): handler.c now
calls new sched_stash_config(), parking the Conf4Sched in new
xce_scheduler_t.pending_config under pending_config_lock (an older
unapplied push is freed). main.c's loop calls new
sched_apply_pending_config() before each round, running the unchanged
sched_init_config() strictly between sessions. sched_fini() frees a
never-applied push.

Tests:
- test_handlers.c: test_config_push_is_stashed_not_applied_on_handler_thread
  -- RED (sched_init_config called on the handler thread), GREEN after.
- new test_config_push_deferred.c (sched_config_push_deferred_test): real
  scheduler_req_handler() + real builtin_fcfs mid-session -- RED (next_job()
  returned NULL right after the push), GREEN (session walks the remaining
  jobs; configver unchanged until applied once between rounds).
- test_admin_request.c / test_job_array_expand.c / test_job_array_wire_e2e.c
  #include handler.c directly and get a link stub for sched_stash_config().
Full scheduler suite 87/87 (asan).

Not addressed: config pushes are still not written to a recording (a
replay spanning a config change won't see it); the ~100s reconnect churn
itself (see bug-1812/1813).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-24 06:30:34 +00:00
parent 77f047f2e2
commit a9feded88e
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
11 changed files with 413 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

10
src/scheduler/testsuite/test_job_array_expand.c
View file

@ -44,6 +44,16 @@ int sched_init_config(xce_scheduler_t* s, xce_conf4sched_t* c)
return 0;
}
/* bug-1846: handler.c's config-push path now stashes via sched_stash_config()
* (scheduler.c, not linked here) instead of calling sched_init_config() --
* link-satisfying stub, same rationale as the one above. */
int sched_stash_config(xce_scheduler_t* s, xce_conf4sched_t* c)
{
(void)s;
(void)c;
return 0;
}
/* sched_record_status (recording.c, scheduler-core spec, task 9.4) --
* handler.c's OPCode_schedAdmin/recordStatus path references it; this file
* never exercises OPCode_schedAdmin, so a link-satisfying stub is enough,