lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(scheduler): mid-round rejoin config push truncated live rounds (bug-1846 part 2)

Browse source 
sched_publish_retry_loop() calls sched_check_pending_rejoin() after every
successful publish -- inside the driver's decision-drain loop -- so after a
client-library reconnect the scheduler re-joins mid-round. The controller
answers every join (on_scheduler_join(), controller/sched.c) by re-pushing
config, workers and jobs. handler.c applied the config push synchronously
on the handler thread: sched_init_config() -> sched_init_queues() ->
builtin_fcfs on_queue_update(), which nulls the picker cursor when it
reinserts the queue the cursor rests on. next_job() then returned NULL and
the round ended with the rest of the queue never considered.

Evidence (scheduler-replay-20260923-085356.replay, temporary event-stream
diagnostics, reverted): 9 recorded rounds end after 11-12 decisions; every
one is followed by a workersPush + full 50-job jobsPush (the join's resync,
reporting just-dispatched jobs back as PEND), every full-length dispatching
round has none, and the config push never appears because it bypassed the
pending-update queue the recorder writes from -- which is also why replay
could not reproduce the truncation.

Fix (defer to the round edge, same direction as bug-1813): handler.c now
calls new sched_stash_config(), parking the Conf4Sched in new
xce_scheduler_t.pending_config under pending_config_lock (an older
unapplied push is freed). main.c's loop calls new
sched_apply_pending_config() before each round, running the unchanged
sched_init_config() strictly between sessions. sched_fini() frees a
never-applied push.

Tests:
- test_handlers.c: test_config_push_is_stashed_not_applied_on_handler_thread
  -- RED (sched_init_config called on the handler thread), GREEN after.
- new test_config_push_deferred.c (sched_config_push_deferred_test): real
  scheduler_req_handler() + real builtin_fcfs mid-session -- RED (next_job()
  returned NULL right after the push), GREEN (session walks the remaining
  jobs; configver unchanged until applied once between rounds).
- test_admin_request.c / test_job_array_expand.c / test_job_array_wire_e2e.c
  #include handler.c directly and get a link stub for sched_stash_config().
Full scheduler suite 87/87 (asan).

Not addressed: config pushes are still not written to a recording (a
replay spanning a config change won't see it); the ~100s reconnect churn
itself (see bug-1812/1813).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-24 06:30:34 +00:00
parent 77f047f2e2
commit a9feded88e
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
11 changed files with 413 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

60
src/scheduler/testsuite/test_handlers.c
View file

@ -25,10 +25,27 @@
* and sched_get_job (job-array-single-wal-record task 5.3's per-element
* dedup check, XceData_data_jobArray — not tested here, see
* test_job_array_expand.c for the real kvmgr-backed sched_get_job). */
static int g_init_config_calls = 0;
static int g_stash_config_calls = 0;
static xce_conf4sched_t* g_stashed_config = NULL;
int sched_init_config(xce_scheduler_t* s, xce_conf4sched_t* c)
{
(void)s;
(void)c;
g_init_config_calls++;
return 0;
}
/* bug-1846: the config-push handler must hand the pushed config to the
* driver (applied between rounds) rather than calling sched_init_config()
* on the handler thread -- counted here so the test below can tell which
* path the real handler took. */
int sched_stash_config(xce_scheduler_t* s, xce_conf4sched_t* c)
{
(void)s;
g_stash_config_calls++;
g_stashed_config = c;
return 0;
}
@ -307,6 +324,48 @@ static void test_handle_notify_job_exit_maps_exit_code_to_done_or_exit(void)
TEST_ASSERT_EQUAL_INT(0, pop_op(&sched, NULL));
}
/* bug-1846: a config push arrives on the handler thread whenever the
* scheduler (re)joins -- including a rejoin triggered from inside a live
* round's decision publish. Applying it there (sched_init_config() ->
* sched_init_queues() -> picker on_queue_update()) reset the fcfs cursor
* mid-session and ended the round early. The handler must only stash it
* and take ownership (payload pointer nulled), never apply it. */
static void test_config_push_is_stashed_not_applied_on_handler_thread(void)
{
xce_scheduler_t sched;
xce_request_t request;
xce_configdata_t configdata;
xce_conf4sched_t* config = (xce_conf4sched_t*)calloc(1, sizeof(*config));
int rc;
fixture_init(&sched);
g_init_config_calls = 0;
g_stash_config_calls = 0;
g_stashed_config = NULL;
memset(&configdata, 0, sizeof(configdata));
configdata.data.conf4sched = config;
memset(&request, 0, sizeof(request));
request.opcode = OPCode_push;
request.body_which = Request_body_configpushmsg;
request.body.configpushmsg = &configdata;
rc = scheduler_req_handler(&request, &sched);
TEST_ASSERT_TRUE(rc >= 0);
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, g_init_config_calls,
"config push must not be applied synchronously on the handler "
"thread -- it can land mid-round and truncate the session");
TEST_ASSERT_EQUAL_INT(1, g_stash_config_calls);
TEST_ASSERT_EQUAL_PTR(config, g_stashed_config);
TEST_ASSERT_NULL_MESSAGE(configdata.data.conf4sched,
"handler must take ownership of the stashed "
"config so the request teardown won't free it");
free(config);
}
int main(int argc, char* argv[])
{
(void)argc;
@ -320,5 +379,6 @@ int main(int argc, char* argv[])
RUN_TEST(test_handle_resvals_full_skips_empty_enqueues_nonempty, 0);
RUN_TEST(test_handle_notify_job_dispatch_fail_enqueues_pend, 0);
RUN_TEST(test_handle_notify_job_exit_maps_exit_code_to_done_or_exit, 0);
RUN_TEST(test_config_push_is_stashed_not_applied_on_handler_thread, 0);
return UNITY_END();
}