fix(scheduler): mid-round rejoin config push truncated live rounds (bug-1846 part 2)
sched_publish_retry_loop() calls sched_check_pending_rejoin() after every successful publish -- inside the driver's decision-drain loop -- so after a client-library reconnect the scheduler re-joins mid-round. The controller answers every join (on_scheduler_join(), controller/sched.c) by re-pushing config, workers and jobs. handler.c applied the config push synchronously on the handler thread: sched_init_config() -> sched_init_queues() -> builtin_fcfs on_queue_update(), which nulls the picker cursor when it reinserts the queue the cursor rests on. next_job() then returned NULL and the round ended with the rest of the queue never considered. Evidence (scheduler-replay-20260923-085356.replay, temporary event-stream diagnostics, reverted): 9 recorded rounds end after 11-12 decisions; every one is followed by a workersPush + full 50-job jobsPush (the join's resync, reporting just-dispatched jobs back as PEND), every full-length dispatching round has none, and the config push never appears because it bypassed the pending-update queue the recorder writes from -- which is also why replay could not reproduce the truncation. Fix (defer to the round edge, same direction as bug-1813): handler.c now calls new sched_stash_config(), parking the Conf4Sched in new xce_scheduler_t.pending_config under pending_config_lock (an older unapplied push is freed). main.c's loop calls new sched_apply_pending_config() before each round, running the unchanged sched_init_config() strictly between sessions. sched_fini() frees a never-applied push. Tests: - test_handlers.c: test_config_push_is_stashed_not_applied_on_handler_thread -- RED (sched_init_config called on the handler thread), GREEN after. - new test_config_push_deferred.c (sched_config_push_deferred_test): real scheduler_req_handler() + real builtin_fcfs mid-session -- RED (next_job() returned NULL right after the push), GREEN (session walks the remaining jobs; configver unchanged until applied once between rounds). - test_admin_request.c / test_job_array_expand.c / test_job_array_wire_e2e.c #include handler.c directly and get a link stub for sched_stash_config(). Full scheduler suite 87/87 (asan). Not addressed: config pushes are still not written to a recording (a replay spanning a config change won't see it); the ~100s reconnect churn itself (see bug-1812/1813). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
77f047f2e2
commit
a9feded88e
11 changed files with 413 additions and 7 deletions
|
|
@ -291,12 +291,31 @@ typedef struct {
|
|||
* sched_init(), destroyed by sched_fini() (scheduler.c) -- same
|
||||
* lifecycle pairing as `core.lock` (core.c). */
|
||||
mylock_t record_lock;
|
||||
/* bug-1846: a config push received on the handler thread (handler.c's
|
||||
* scheduler_req_handler()), parked here until the driver loop applies
|
||||
* it between rounds via sched_apply_pending_config(). The controller
|
||||
* re-pushes config on every scheduler join, and the scheduler re-joins
|
||||
* from inside a round's decision publish (sched_check_pending_rejoin(),
|
||||
* via sched_publish_retry_loop()) -- applying it on arrival reset the
|
||||
* picker's cursor mid-session and ended the round early. Only the
|
||||
* newest push is kept (each is a complete Conf4Sched). Owned by this
|
||||
* struct until applied; freed by sched_fini() if never applied.
|
||||
* Protected by `pending_config_lock`. */
|
||||
xce_conf4sched_t* pending_config;
|
||||
mylock_t pending_config_lock;
|
||||
} xce_scheduler_t;
|
||||
|
||||
extern int sched_init(xce_scheduler_t*, xce_client_config_t*,
|
||||
const xce_cluster_config_t*, xce_request_handler_t);
|
||||
extern void sched_fini(xce_scheduler_t*);
|
||||
extern int sched_init_config(xce_scheduler_t* sched, xce_conf4sched_t*);
|
||||
/* bug-1846: see xce_scheduler_t.pending_config. sched_stash_config() is
|
||||
* the handler-thread half (takes ownership of `config`, replacing -- and
|
||||
* freeing -- any older unapplied push); sched_apply_pending_config() is the
|
||||
* driver-thread half, run between rounds, and returns 1 if a config was
|
||||
* applied, 0 if none was pending, or sched_init_config()'s negative rc. */
|
||||
extern int sched_stash_config(xce_scheduler_t* sched, xce_conf4sched_t*);
|
||||
extern int sched_apply_pending_config(xce_scheduler_t* sched);
|
||||
/* Req 2 AC 1 (scheduler-core spec, task 1.3): applies the "builtin_fcfs"
|
||||
* default to sched->job_pick_plugin when unset. See the field comment on
|
||||
* xce_scheduler_t above -- this does NOT resolve/register the name. */
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue