Live 4-process E2E run (3-node controller cluster + xcecd_virtual
worker + xce_scheduler + xce_apiserver, all on the full 3-candidate
list) with the primary killed mid-session. Worker reconnect is
solidly proven with real log evidence and unchanged PIDs across all
three components throughout. Scheduler's rejoin path was never
naturally exercised (no job reached its picker/allocator) -- same
partial-coverage shape already accepted for task 3.3.
Surfaced a likely-real, structural gap in the already-approved task
4.3/4.4 apiserver rejoin mechanism (bug-039): XCECLI_FF_CALL can
silently self-heal the transport within a single heartbeat tick,
making the cross-tick failure->success transition invisible to
heartbeat.go's detector on any failover fast enough to complete
within one heartbeat interval -- which is most real single-election
failovers in this codebase. Also found bug-040 (job pushes never
reach apiserver's store), very likely a downstream symptom of
bug-039's rejoin gap via on_apiserver_join()'s per-process,
unreplicated apiserver registration list. Both logged with detailed,
independently-verified root-cause analysis for a dedicated follow-up
-- fixing them is outside this task's cross-component-validation
boundary.
No src/ changes -- live validation only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Rongsong Shen
6cb8f22b08
