Starts a real worker (auto-detects xcecd_virtual vs. real xcecd, mirroring
xce-deploy-ha's own detection) + xce_scheduler alongside the controller
cluster, dispatches a genuine job, kills the primary while it's RUN, and
verifies the worker rejoins the new primary and the job recovers to DONE
via real reconciliation -- closing the "no real daemon-to-daemon smoke
test" gap flagged by worker-rejoin-resync's own feature validation.
Live-verified end-to-end multiple times (sequential and chaotic, 2 rounds
each) against a real 3-node cluster.
Also logs two bugs discovered along the way (not fixed, out of scope for
this test-infra change): bug-1759 (scheduler sim-mode dispatch/PEND
inconsistency) and bug-1760 (scheduler has no periodic post-failover
reconnect check, only a reactive one gated on its own next successful
publish -- a chicken-and-egg deadlock, live-confirmed via 60s+ of stuck
PEND with zero scheduler log activity). bug-1760 is worked around in this
harness only by restarting the scheduler process after each new-primary
election; the underlying gap is unfixed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019ypUeKgbAqLWUkaSJ79poN
Rongsong Shen
00fc0a43e0