myxce/.wolf/bugs/bug-1760.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	legacy
bug-1760	fixed	2026-09-24	88b0ac99	scheduler	unknown	2026-09-02T04:10:00Z	2026-09-19T09:00:00Z	3	src/scheduler/src/scheduler.c (sched_needs_rejoin:487 sched_on_reconnect:507 sched_publish_check_rejoin:1474) src/scheduler/src/main.c	scheduler ha failover rejoin availability component-failover-adoption sched_publish_check_rejoin chicken-and-egg discovered-live severity-high-suspected verified-2026-09-04 partially-diagnosed live-repro-confirmed-2026-09-04 blocked-on-design-decision req-3-ac3-conflict needs-spec-amendment live-verified-2026-09-04		88b0ac99	true

bug-1760: Live-tested against a real 3-node HA cluster + xcecd_virtual worker + xce_scheduler…

Description

Live-tested against a real 3-node HA cluster + xcecd_virtual worker + xce_scheduler (worker-rejoin-resync soak-harness development): after the CURRENT primary is killed and a NEW primary is elected+reconverged, a job submitted through the new primary stays PEND for 60+ seconds with ZERO scheduler.log activity of any kind (no WARN/INFO/ERROR, no reconnect attempt) -- not a slow recovery, a genuinely stuck one. Reproduced twice in a row with the same shape.

Environment

Not recorded (migrated from buglog.json).

Reproduction

Not recorded (migrated from buglog.json).

Root cause

CONFIRMED via source read (not yet live-instrumented/gdb-confirmed): the scheduler has NO periodic connection-liveness/reconnect check analogous to the worker daemon's checking_connection() (src/worker/src/controller_connector.c:505, called every xcecd_ctx.update_interval seconds from xcecd.c's main loop via xcecli_hello()). The scheduler's ONLY rejoin mechanism is sched_publish_check_rejoin() (scheduler.c:1474), which re-issues xcecli_join() ONLY as a side effect of its own NEXT SUCCESSFUL publish call (sched_publish_dispatch()/sched_publish_pend_reason() via sched_publish_retry_loop(), which calls sched_publish_check_rejoin() only after attempt_fn() already returned >=0). sched_needs_rejoin itself is set correctly by sched_on_reconnect() (the client library's own transparent-reconnect callback), so the SCHEDULER's own underlying connection to the cluster likely IS aware it reconnected somewhere -- but nothing forces a FRESH xcecli_join() against the actual NEW primary until the scheduler has something to publish. After a real controller failover, the scheduler has nothing queued to publish (no job ever reaches its picker/allocator, since the CONTROLLER-side registration of "where the scheduler is" is presumed to be live-connection-scoped state on whichever controller instance the scheduler last explicitly joined -- i.e. the dead primary -- and does not appear to carry over to the newly-elected primary), so the scheduler can never publish anything to trigger its own rejoin: a chicken-and-egg deadlock. NOT YET independently confirmed via gdb/live instrumentation which exact half of this loop is stuck (client-library-level reconnect vs. controller-side scheduler-registration loss) -- this is a plausible, evidence-consistent hypothesis from static reading + the observed symptom, not a fully root-caused finding.

Fix

NOT FIXED -- discovered as a side effect of unrelated worker-rejoin-resync soak-harness development; deliberately not investigated further or fixed in that session (explicit instruction: log and continue). This is a potentially serious standing HA-availability gap (every job submitted after ANY controller failover could stay PEND forever until the scheduler process is manually restarted) that deserves its own dedicated debugging investigation (e.g. /kiro-debug) scoped to component-failover-adoption's scheduler-rejoin design, ideally with live gdb/log instrumentation to confirm which half of the chicken-and-egg loop is actually broken. Pragmatic workaround adopted ONLY in the worker-rejoin-resync soak harness (.claude/skills/xce-test-ha/run-ha-soak.sh): restart the scheduler process after each round's controller failover, the same way the harness already restarts the downed CONTROLLER node -- a fresh scheduler process calls sched_init(), which tries all 3 candidates in order and joins whichever is actually reachable, sidestepping the stuck reactive-rejoin path entirely. This workaround is scoped to test infrastructure only and does NOT fix the underlying production behavior.

UPDATE (2026-09-04): FIXED. sched_publish_check_rejoin() (scheduler.c) renamed to sched_check_pending_rejoin() and exposed (was static) via scheduler.h. main.c's round loop (while(1) { xce_sched_driver_run_normal_mode_round(); ...; sleep(1); }) now calls sched_check_pending_rejoin(&sched) unconditionally once per ~1s round, in addition to the original reactive call sched_publish_retry_loop() still makes after a successful decision publish. This closes the chicken-and-egg deadlock: even when a round has nothing to publish (the common case right after a failover, since the new primary has no record of this scheduler having joined it), the rejoin is now retried on a bounded ~1s cadence regardless.

Deliberately placed in main.c, not inside driver.c's xce_sched_driver_run_normal_mode_round() (the first attempt): driver.c is linked white-box by ~10 scheduler test binaries (sim/driver/admin-request tests) that never link scheduler.c (where sched_check_pending_rejoin/sched_needs_rejoin live) -- a driver.c call site broke all of those under -Wl,--no-undefined (undefined reference to sched_check_pending_rejoin). main.c is normal-mode-only and not linked into any test binary, so it has no such cross-boundary link cost.

TDD: two new tests added to test_scheduler_decision_publish_retry.c. test_check_pending_rejoin_invoked_directly_without_a_publish() proves the renamed/exposed function works standalone (reconnect simulated via the captured xcecli_set_reconnect_handler callback, then sched_check_pending_rejoin() called directly with zero decisions ever published -- verifies retry-on-failure/clear-on-success via the existing __wrap_xcecli_join double). test_main_loop_checks_pending_rejoin_unconditionally() is a source-text scan (same technique as test_scheduler_decision_redelivery.c's existing test_no_worker_resync_symbols_in_scheduler_publish_path) proving main.c's round loop calls sched_check_pending_rejoin(&sched) unconditionally, positioned after core.stop's break check but before sleep(1) -- i.e. every live round, not gated on publish success. Full scheduler ASAN suite (fresh build, isolated build dir to avoid a concurrent session's own build in xce.build): 62/64 Ok, 1 pre-existing Fail (bug-1758, sched_core_events_test, unrelated -- belongs to scheduler-dispatch-commit spec per that entry), 1 pre-existing Timeout (scheduler backpressure test, documented known issue). sched_scheduler_decision_publish_retry_test itself: 8/8 tests pass (ASAN_OPTIONS=detect_leaks=0, this suite's established convention for fixtures that call the real sched_init() without teardown).

Not independently live-verified against a real 3-node HA failover in this session (the fix is a straightforward, well-reasoned closing of a plainly-diagnosed gap -- a periodic check where only a reactive one existed -- verified by unit test rather than a live repro this time).

UPDATE (2026-09-04, live-repro + partial fix session): Root cause CONFIRMED live, exactly as hypothesized. Deployed a fresh 3-node HA cluster (ports 9981-9983, /tmp/xce-deploy-bug1760), submitted a job pre-failover (dispatched and finished normally), killed the primary (ctrl-1), waited for a real PreVote election (ctrl-3 became CTRL_PRIMARY term=3), then submitted a job through the client library (which correctly redirected to the new primary -- confirmed jobsubmit ok). That job stayed PEND indefinitely with ZERO scheduler.log activity of any kind for the whole observation window, reproducing the original symptom exactly.

PARTIAL FIX APPLIED: sched_publish_check_rejoin() (scheduler.c) renamed to sched_check_pending_rejoin(), made non-static (was static; now declared in scheduler.h), and called from a SECOND site: main()'s round loop (scheduler/src/main.c), unconditionally, once per ~1s round, in addition to its original reactive call from sched_publish_retry_loop() after a successful decision publish. This alone is a real, harmless improvement (any rejoin already flagged some other way is now retried within ~1s instead of only after a successful publish) but was PROVEN INSUFFICIENT by a second live test with this fix in place: the exact same stuck-PEND symptom reproduced identically, because sched_needs_rejoin was never set true in the first place -- see below.

FULL ROOT CAUSE, now fully confirmed (both halves of the original open question resolved): sched_needs_rejoin is set ONLY by sched_on_reconnect(), the client library's transparent-reconnect callback, which itself only fires as a side effect of an ACTUAL xcecli_* call (XCECLI_FAILOVER_CALL/XCECLI_FF_CALL, client_internal.h) detecting a dead connection and rotating to a live candidate. An idle scheduler round -- no jobs, no queue/worker events, nothing to publish -- makes ZERO client-library calls of any kind. So after a controller failover, if the scheduler has nothing already queued, the dead connection is NEVER NOTICED by anything, sched_needs_rejoin never flips true, and no amount of checking that flag more often (the periodic-call half of the fix above) helps. This is a pure detection gap, not a retry-cadence gap.

ARCHITECTURAL CONFLICT DISCOVERED (this is the actual remaining blocker, not further coding): the obvious fix -- add an active liveness probe (xcecli_hello()) call to the same periodic round-loop site, mirroring the worker's own checking_connection()/xcecli_hello() pattern (controller_connector.c) -- was implemented, and DOES work: with it in place, a third live test showed the scheduler's reconnect-and-rejoin cycle firing correctly and the post-failover job dispatching. However, this directly violates an existing, deliberately-authored structural regression test: test_scheduler_connect.c's test_no_periodic_heartbeat_call_in_source (component-failover-adoption spec, task 3.3, Requirement 3 AC3), which source-text-scans scheduler.c for the literal strings 'xcecli_hello' and 'heartbeat' and fails if either appears, specifically to prevent introducing exactly this kind of periodic call. That test's own header comment reasoning: 'a stalled reconnect retry [inside such a call] could delay/stall the scheduler's own round loop' -- i.e. a probe call that blocks while the client library is mid-retry against a dead cluster could stall the ENTIRE scheduling loop (not just rejoin), which is a real, distinct hazard from the one this bug describes. Requirement 3 AC3's actual literal text ('reconnect retries shall not delay or prevent the component's own heartbeat RESPONSES to a connected controller') is about a different concern (the scheduler has no heartbeat-response obligation at all, unlike the worker, so that specific AC is satisfied trivially/by-construction for the scheduler) -- the test's blanket string-ban is a broader, self-imposed guard than the letter of that AC actually requires, and did not anticipate this bug's distinct 'the scheduler never even detects a dead connection while idle' scenario.

DECISION MADE THIS SESSION: did NOT unilaterally override this existing, deliberately-tested design decision. Reverted the xcecli_hello probe (and its test-file/meson.build wrapping) back out, keeping only the harmless round-loop periodic flag-check. sched_check_pending_rejoin() now carries a doc comment explaining exactly this tension and pointing at this buglog entry.

NEXT STEP (needs a design decision, not just code): either (a) amend component-failover-adoption's Requirement 3 AC3 and its guarding test to explicitly allow a BOUNDED/non-blocking reconnect-detection probe (distinct from an unbounded heartbeat-response obligation) -- e.g. a probe with a short, hard timeout that cannot stall the round loop for more than that bound -- or (b) find an alternative reconnect-detection mechanism that does not require any application-level call at all, e.g. async RDHUP-driven detection at the transport/iomgr layer for client-role (non-controller-peer) connections, matching how this project's controller-to-controller connections detect failure without polling (see ha-async-io.md) -- if such passive detection genuinely doesn't exist for client-role connections today, that itself would need to be added at the client-library layer (src/lib/client.c/xcectx.c), not scheduler.c. Whoever picks this up should re-read Requirement 3's full text (component-failover-adoption/requirements.md) and the Req-3-AC3 test's header comment before choosing, and treat this as a spec amendment, not a quiet code change.

Cleanup: cluster (ports 9981-9983, /tmp/xce-deploy-bug1760) fully torn down, all started processes killed, confirmed via ps aux. Full scheduler ASAN suite re-run after the revert: 63/64 pass, the sole failure being the already-logged, unrelated bug-1758.

UPDATE (2026-09-04, live verification of the passive-detection fix): CONFIRMED FIXED end to end. The main session designed and implemented a second fix component this session (with the user, not unilaterally): a new public libxce accessor xcecli_is_connection_broken() (client.h/client.c) exposing the existing internal xcecli_ff_needs_reconnect() predicate (client_internal.h) -- a pure, socket-free, side-effect-free read of ctx->ctx.state == XCE_CTX_ERR, a flag ALREADY maintained asynchronously by handle_disconnect() (xcectx.c), the transport's RDHUP/EPOLLERR handler, with zero application calls involved. sched_check_pending_rejoin() (scheduler.c) now attempts xcecli_join() when EITHER sched_needs_rejoin is set OR xcecli_is_connection_broken(sched->apictx) reports true -- closing the idle-scheduler detection gap without adding any periodic active-probe call (no xcecli_hello, no heartbeat string anywhere in scheduler.c -- test_no_periodic_heartbeat_call_in_source, the Req-3-AC3 structural guard, passes unmodified).

Live re-verification (fresh 3-node cluster, ports 9971-9973, ASAN build in src/xce.build.verify1760, same repro recipe as this entry's earlier 2026-09-04 live-repro session that proved the FIRST partial fix alone was insufficient): deployed cluster, confirmed a baseline job dispatches and completes normally (RUN then DONE) pre-failover. Killed ctrl-1 (the primary). New primary (ctrl-2, term=3) elected 15s later. Critically, with ZERO jobs submitted in the interim (genuinely idle scheduler, the exact scenario the first partial fix could not handle): scheduler.log shows "sched_check_pending_rejoin: pending rejoin succeeded, cluster membership restored" firing on its own, ~300ms after the new primary was elected -- the scheduler re-registered itself with the new primary entirely on its own initiative, with no job ever having been submitted to trigger it reactively. Confirmed this is the NEW passive-detection code path (not the old reactive one, which structurally cannot fire with zero app calls in flight). A fresh job submitted afterward through the new primary dispatched immediately (RUN within ~180ms of submission, DONE ~2s later per its own sleep duration) -- no stuck-PEND, no manual intervention. Full scheduler ASAN suite re-run in the same build: 63/64 pass, sole failure the already-logged unrelated bug-1758. New unit tests (test_is_connection_broken_* in lib/testsuite/test_libxce_retry.c; test_check_pending_rejoin_triggers_on_passive_broken_signal / test_check_pending_rejoin_no_op_when_healthy_and_no_flag in scheduler/testsuite/test_scheduler_decision_publish_retry.c) all pass. Cluster fully torn down, all processes killed, confirmed via ps aux -- zero xce_controller/xce_scheduler/xcecd processes remain.

This closes bug-1760 -- both halves (reactive + passive detection) are now live-verified, not just unit-tested.

Verification

Not recorded (migrated from buglog.json).

References

Not recorded (migrated from buglog.json).