fix(scheduler): mid-round rejoin config push truncated live rounds (bug-1846 part 2)

sched_publish_retry_loop() calls sched_check_pending_rejoin() after every
successful publish -- inside the driver's decision-drain loop -- so after a
client-library reconnect the scheduler re-joins mid-round. The controller
answers every join (on_scheduler_join(), controller/sched.c) by re-pushing
config, workers and jobs. handler.c applied the config push synchronously
on the handler thread: sched_init_config() -> sched_init_queues() ->
builtin_fcfs on_queue_update(), which nulls the picker cursor when it
reinserts the queue the cursor rests on. next_job() then returned NULL and
the round ended with the rest of the queue never considered.

Evidence (scheduler-replay-20260923-085356.replay, temporary event-stream
diagnostics, reverted): 9 recorded rounds end after 11-12 decisions; every
one is followed by a workersPush + full 50-job jobsPush (the join's resync,
reporting just-dispatched jobs back as PEND), every full-length dispatching
round has none, and the config push never appears because it bypassed the
pending-update queue the recorder writes from -- which is also why replay
could not reproduce the truncation.

Fix (defer to the round edge, same direction as bug-1813): handler.c now
calls new sched_stash_config(), parking the Conf4Sched in new
xce_scheduler_t.pending_config under pending_config_lock (an older
unapplied push is freed). main.c's loop calls new
sched_apply_pending_config() before each round, running the unchanged
sched_init_config() strictly between sessions. sched_fini() frees a
never-applied push.

Tests:
- test_handlers.c: test_config_push_is_stashed_not_applied_on_handler_thread
  -- RED (sched_init_config called on the handler thread), GREEN after.
- new test_config_push_deferred.c (sched_config_push_deferred_test): real
  scheduler_req_handler() + real builtin_fcfs mid-session -- RED (next_job()
  returned NULL right after the push), GREEN (session walks the remaining
  jobs; configver unchanged until applied once between rounds).
- test_admin_request.c / test_job_array_expand.c / test_job_array_wire_e2e.c
  #include handler.c directly and get a link stub for sched_stash_config().
Full scheduler suite 87/87 (asan).

Not addressed: config pushes are still not written to a recording (a
replay spanning a config change won't see it); the ~100s reconnect churn
itself (see bug-1812/1813).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.wolf/buglog.json

@@ -25082,9 +25082,9 @@
       "id": "bug-1846",
       "timestamp": "2026-09-23T10:05:00Z",
       "error_message": "OPEN, NOT YET ROOT-CAUSED. Same replay session as bug-1844/1845 (scheduler-replay-20260923-085356.replay): after excluding instid (bug-1844) and reason-detail (bug-1845), --sim-verify still shows one job (jobid=0-8-1790154325-0[39] in the observed capture) genuinely DISPATCHING one round earlier in replay than in the original live recording -- not a false-positive field comparison, a real positional divergence: replay's round begin_round=...436582151823 dispatches a 13-job batch (jobs [27]..[39]) while the SAME round in the original recording only dispatched 12 (jobs [27]..[38]); job [39]'s recorded dispatch instead lands in the NEXT round, cascading into ~40+ downstream positional MISMATCH lines for that round pair alone.",
-      "file": "src/scheduler/src/plugins/limit.c (match_job_host_limit_)",
-      "root_cause": "CONFIRMED (2026-09-23, via temporary per-decision diagnostics in the --sim-verify drain loop, since reverted) -- two distinct things. (1) ROOT-CAUSED + FIXED: match_job_host_limit_() (limit.c) runs on EVERY per-host allocation attempt (limit_prepare_allocation() via plugin_prepare_allocation()) and unconditionally xlist_store()d the matched vres into obj->hlimits[worker->idx]; obj->hlimits is only rebuilt by limit_job_election() on a configver bump. A job that kept pending on the same host accumulated one extra copy per attempt; sched_fill_limit_requests() staged every copy into ctx->alloc_req.host and seqjob's make_host_alloc() built a host alloc carrying all of them. In the captured replay the LIVE recording itself shows it (pc13v host allocs with mem+slots+70/79/94/112/136/175 identical limit=limits2/u=rshen/q=xce/p= entries, resetting only on a config push) -- i.e. a PRODUCTION bug, not sim-only. make_job_account() makes one res_account per entry and sched_update_res_accounts() charges each, so the host limit was charged (and later released) N-fold. Replay had the same accumulation with different counts, which is what --sim-verify flagged. (2) NOT ROOT-CAUSED, live-side: 9 recorded rounds (all containing DISPATCHes) end after only 11-12 decisions while replay processes the whole candidate list; the next recorded round then carries the leftovers. Ruled out: recorder growth/truncation (recording_session.c), a picker/round time budget (none exists; truncated rounds last 0.8-2.2s), fcfs readiness predicates (both return 1), the unsatisfiable-cache reason handoff (MOVE_PTR on both paths), queued controller updates (all dataq-enqueued). Remaining candidates: something live-only ending the fcfs session early (e.g. handler.c's synchronous config push -> sched_init_queues() -> on_queue_update() nulling priv->cursor mid-session, too rare to explain all 9), or sched_allocator's silent `alloc_rc < 0 -> continue` (no decision emitted) -- possibly related to (1)'s inflated limit charge. Needs the live scheduler.log for the capture window and/or a fresh capture from a fixed binary.",
-      "fix": "PARTIAL (2026-09-23, TDD). limit.c: new hlimits_contains_() pointer-identity check (vres is interned per name in privdata->instances) -- match_job_host_limit_() now stores a matched vres into obj->hlimits[worker] only once. New regression test test_limit_host_dedup.c (sched_limit_host_dedup_test, meson suite 'scheduler'): 5 plugin_prepare_allocation() attempts for one job on one worker must leave hlimits[worker]->count==1 and exactly one staged copy of the vres in alloc_req.host -- verified RED before the fix (Expected 1 Was 2 on attempt 2) and GREEN after. Replay re-run after the fix: replay-side host allocs are now always 3 entries; remaining DISPATCH alloc mismatches are the recording's own inflated entries (recorded with the buggy binary) and will disappear only with a fresh capture. Part (2) remains OPEN.",
+      "file": "src/scheduler/src/plugins/limit.c (match_job_host_limit_); src/scheduler/src/handler.c + main.c + scheduler.c (config push deferral)",
+      "root_cause": "CONFIRMED (2026-09-23, via temporary per-decision diagnostics in the --sim-verify drain loop, since reverted) -- two distinct things. (1) ROOT-CAUSED + FIXED: match_job_host_limit_() (limit.c) runs on EVERY per-host allocation attempt (limit_prepare_allocation() via plugin_prepare_allocation()) and unconditionally xlist_store()d the matched vres into obj->hlimits[worker->idx]; obj->hlimits is only rebuilt by limit_job_election() on a configver bump. A job that kept pending on the same host accumulated one extra copy per attempt; sched_fill_limit_requests() staged every copy into ctx->alloc_req.host and seqjob's make_host_alloc() built a host alloc carrying all of them. In the captured replay the LIVE recording itself shows it (pc13v host allocs with mem+slots+70/79/94/112/136/175 identical limit=limits2/u=rshen/q=xce/p= entries, resetting only on a config push) -- i.e. a PRODUCTION bug, not sim-only. make_job_account() makes one res_account per entry and sched_update_res_accounts() charges each, so the host limit was charged (and later released) N-fold. Replay had the same accumulation with different counts, which is what --sim-verify flagged. (2) ROOT-CAUSED (2026-09-23): the 9 live rounds cut off after 11-12 decisions were each ended by a mid-round scheduler REJOIN. sched_publish_retry_loop() (scheduler.c) calls sched_check_pending_rejoin() after every successful publish -- i.e. inside the driver's decision-drain loop -- and after a client-library reconnect that issues xcecli_join(). The controller answers every join (on_scheduler_join(), controller/sched.c) with xcesched_push_configs() + xcesched_push_workers() + xcesched_push_jobs(). handler.c applied the config push synchronously on the handler thread (sched_init_config() -> sched_init_queues() -> builtin_fcfs on_queue_update()), which nulls priv->cursor when it reinserts the queue the cursor rests on; next_job() then returns NULL and the round ends. Evidence from the capture (temporary event-stream diagnostics, reverted): every truncated round is followed by a workersPush + full 50-job jobsPush (ntotal=50, pos=0, the join's resync -- e.g. jobs just dispatched reported back as PEND), every full-length dispatching round has none, and the config push itself never appears because it bypassed the pending-update queue the recorder writes from. Rejoins came roughly every ~100s (the configver bump also explains the periodic reset of part (1)'s duplicate counts). The underlying connection churn is a separate issue (see bug-1812/1813's pc13v reconnect/resync storm).",
+      "fix": "PARTIAL (2026-09-23, TDD). limit.c: new hlimits_contains_() pointer-identity check (vres is interned per name in privdata->instances) -- match_job_host_limit_() now stores a matched vres into obj->hlimits[worker] only once. New regression test test_limit_host_dedup.c (sched_limit_host_dedup_test, meson suite 'scheduler'): 5 plugin_prepare_allocation() attempts for one job on one worker must leave hlimits[worker]->count==1 and exactly one staged copy of the vres in alloc_req.host -- verified RED before the fix (Expected 1 Was 2 on attempt 2) and GREEN after. Replay re-run after the fix: replay-side host allocs are now always 3 entries; remaining DISPATCH alloc mismatches are the recording's own inflated entries (recorded with the buggy binary) and will disappear only with a fresh capture. PART (2) FIXED (2026-09-23, TDD, user-chosen design 'defer to round edge'): handler.c's configpushmsg case now calls new sched_stash_config() (parks the Conf4Sched in new xce_scheduler_t.pending_config under pending_config_lock, freeing any older unapplied push) instead of sched_init_config(); main.c's loop calls new sched_apply_pending_config() before each xce_sched_driver_run_normal_mode_round(), which runs the unchanged sched_init_config() strictly between sessions; sched_fini() frees a never-applied push. Same deferral direction as bug-1813's array-job fix. Tests: test_handlers.c's test_config_push_is_stashed_not_applied_on_handler_thread (RED: sched_init_config called 1x on the handler thread; GREEN after) and new test_config_push_deferred.c (sched_config_push_deferred_test: real scheduler_req_handler() + real builtin_fcfs mid-session -- RED: next_job() returned NULL right after the config push, job B never considered; GREEN: session walks B and C, configver unchanged until sched_apply_pending_config() applies it once). Known remaining gap (pre-existing, not addressed): a config push is still not written to a recording, so a replay spanning a config CHANGE won't see it.",
       "tags": [
         "scheduler",
         "sim",