fix(scheduler): mid-round rejoin config push truncated live rounds (bug-1846 part 2)

sched_publish_retry_loop() calls sched_check_pending_rejoin() after every
successful publish -- inside the driver's decision-drain loop -- so after a
client-library reconnect the scheduler re-joins mid-round. The controller
answers every join (on_scheduler_join(), controller/sched.c) by re-pushing
config, workers and jobs. handler.c applied the config push synchronously
on the handler thread: sched_init_config() -> sched_init_queues() ->
builtin_fcfs on_queue_update(), which nulls the picker cursor when it
reinserts the queue the cursor rests on. next_job() then returned NULL and
the round ended with the rest of the queue never considered.

Evidence (scheduler-replay-20260923-085356.replay, temporary event-stream
diagnostics, reverted): 9 recorded rounds end after 11-12 decisions; every
one is followed by a workersPush + full 50-job jobsPush (the join's resync,
reporting just-dispatched jobs back as PEND), every full-length dispatching
round has none, and the config push never appears because it bypassed the
pending-update queue the recorder writes from -- which is also why replay
could not reproduce the truncation.

Fix (defer to the round edge, same direction as bug-1813): handler.c now
calls new sched_stash_config(), parking the Conf4Sched in new
xce_scheduler_t.pending_config under pending_config_lock (an older
unapplied push is freed). main.c's loop calls new
sched_apply_pending_config() before each round, running the unchanged
sched_init_config() strictly between sessions. sched_fini() frees a
never-applied push.

Tests:
- test_handlers.c: test_config_push_is_stashed_not_applied_on_handler_thread
  -- RED (sched_init_config called on the handler thread), GREEN after.
- new test_config_push_deferred.c (sched_config_push_deferred_test): real
  scheduler_req_handler() + real builtin_fcfs mid-session -- RED (next_job()
  returned NULL right after the push), GREEN (session walks the remaining
  jobs; configver unchanged until applied once between rounds).
- test_admin_request.c / test_job_array_expand.c / test_job_array_wire_e2e.c
  #include handler.c directly and get a link stub for sched_stash_config().
Full scheduler suite 87/87 (asan).

Not addressed: config pushes are still not written to a recording (a
replay spanning a config change won't see it); the ~100s reconnect churn
itself (see bug-1812/1813).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-24 06:30:34 +00:00
parent 77f047f2e2
commit a9feded88e
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
11 changed files with 413 additions and 7 deletions

View file

@ -291,12 +291,31 @@ typedef struct {
* sched_init(), destroyed by sched_fini() (scheduler.c) -- same
* lifecycle pairing as `core.lock` (core.c). */
mylock_t record_lock;
/* bug-1846: a config push received on the handler thread (handler.c's
* scheduler_req_handler()), parked here until the driver loop applies
* it between rounds via sched_apply_pending_config(). The controller
* re-pushes config on every scheduler join, and the scheduler re-joins
* from inside a round's decision publish (sched_check_pending_rejoin(),
* via sched_publish_retry_loop()) -- applying it on arrival reset the
* picker's cursor mid-session and ended the round early. Only the
* newest push is kept (each is a complete Conf4Sched). Owned by this
* struct until applied; freed by sched_fini() if never applied.
* Protected by `pending_config_lock`. */
xce_conf4sched_t* pending_config;
mylock_t pending_config_lock;
} xce_scheduler_t;
extern int sched_init(xce_scheduler_t*, xce_client_config_t*,
const xce_cluster_config_t*, xce_request_handler_t);
extern void sched_fini(xce_scheduler_t*);
extern int sched_init_config(xce_scheduler_t* sched, xce_conf4sched_t*);
/* bug-1846: see xce_scheduler_t.pending_config. sched_stash_config() is
* the handler-thread half (takes ownership of `config`, replacing -- and
* freeing -- any older unapplied push); sched_apply_pending_config() is the
* driver-thread half, run between rounds, and returns 1 if a config was
* applied, 0 if none was pending, or sched_init_config()'s negative rc. */
extern int sched_stash_config(xce_scheduler_t* sched, xce_conf4sched_t*);
extern int sched_apply_pending_config(xce_scheduler_t* sched);
/* Req 2 AC 1 (scheduler-core spec, task 1.3): applies the "builtin_fcfs"
* default to sched->job_pick_plugin when unset. See the field comment on
* xce_scheduler_t above -- this does NOT resolve/register the name. */