test(xce-test-ha): add worker-rejoin-resync coverage to sequential/chaotic soak modes

Starts a real worker (auto-detects xcecd_virtual vs. real xcecd, mirroring
xce-deploy-ha's own detection) + xce_scheduler alongside the controller
cluster, dispatches a genuine job, kills the primary while it's RUN, and
verifies the worker rejoins the new primary and the job recovers to DONE
via real reconciliation -- closing the "no real daemon-to-daemon smoke
test" gap flagged by worker-rejoin-resync's own feature validation.
Live-verified end-to-end multiple times (sequential and chaotic, 2 rounds
each) against a real 3-node cluster.

Also logs two bugs discovered along the way (not fixed, out of scope for
this test-infra change): bug-1759 (scheduler sim-mode dispatch/PEND
inconsistency) and bug-1760 (scheduler has no periodic post-failover
reconnect check, only a reactive one gated on its own next successful
publish -- a chicken-and-egg deadlock, live-confirmed via 60s+ of stuck
PEND with zero scheduler log activity). bug-1760 is worked around in this
harness only by restarting the scheduler process after each new-primary
election; the underlying gap is unfixed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019ypUeKgbAqLWUkaSJ79poN

.wolf/buglog.json

@@ -20662,6 +20662,82 @@
       ],
       "occurrences": 1,
       "last_seen": "2026-09-02T02:45:00Z"
+    },
+    {
+      "id": "bug-1758",
+      "timestamp": "2026-09-02T03:23:00Z",
+      "error_message": "scheduler suite (meson test --suite scheduler): sched_core_events_test FAILs deterministically -- test_core_events.c:1364:test_job_rusage_reconciles_mem_reservation_against_usage:FAIL: Expected 4 Was 0. Discovered as one of 2 failures (the other is a pre-existing, unrelated libxce/meson.build ASAN-flag gap) during worker-rejoin-resync feature-level /kiro-validate-impl full-suite run; confirmed NOT caused by worker-rejoin-resync.",
+      "file": "src/scheduler/testsuite/test_core_events.c",
+      "root_cause": "NOT YET ROOT-CAUSED (logged, not investigated -- out of scope for the session that found it). Confirmed via git blame that the ENTIRE failing test function, including the failing sanity assertion at line 1364 itself, was newly added by commit eb06f16d (\"fix(scheduler): wire mem reservation reconciliation into live rusage event path\", .kiro/specs/scheduler-dispatch-commit/), not by worker-rejoin-resync or any earlier commit -- this is that commit's own new test failing against its own new code, not a regression it caused in pre-existing coverage. The failure is at the test's OWN pre-condition sanity check (line 1364, before the rusage/reconciliation event the test is actually about is even pushed): a JOBS_PUSH-dispatched job with a real 4.0 \"mem\" xce_alloc_t is asserted to have already charged 4.0 into w->resInUse (\"matching sched_charge_account()'s existing behavior\", per the test's own comment) -- but val[0] reads back 0. Two live hypotheses, neither yet confirmed: (a) core_events.c's XCE_CORE_EVENT_JOBS_PUSH handler (used here to simulate a dispatch by directly attaching job1->allocs) never actually calls sched_charge_account()/an equivalent resInUse-charging step for a pushed RUN job carrying allocs -- JOBS_PUSH may be resync-shaped (report already-existing state) rather than dispatch-shaped (perform a fresh charge) by design, making the test's own sanity-check premise wrong; or (b) charging happens but keyed off a different resource-index/worker-pointer path than this test's fixture reads back from. Needs an actual debugging pass (e.g. kiro-debug) to distinguish; not attempted here.",
+      "fix": "NOT FIXED. Out of scope for the worker-rejoin-resync feature that discovered it (confirmed via git blame this belongs entirely to eb06f16d/.kiro/specs/scheduler-dispatch-commit/, a separate, currently in-flight spec on this shared branch). Logged per this project's bug-tracking convention instead of silently ignoring a real, currently-red test on the shared branch. Needs a dedicated debugging/fix pass scoped to scheduler-dispatch-commit before that spec can be considered complete -- do not patch around it from an unrelated feature.",
+      "tags": [
+        "scheduler-dispatch-commit",
+        "core_events-c",
+        "test_core_events-c",
+        "sched_reconcile_job_resource_usage",
+        "mem-reservation",
+        "resInUse",
+        "unrelated-branch-breakage",
+        "discovered-by-worker-rejoin-resync-validation",
+        "not-fixed"
+      ],
+      "related_bugs": [],
+      "occurrences": 1,
+      "last_seen": "2026-09-02T03:23:00Z"
+    },
+    {
+      "id": "bug-1759",
+      "timestamp": "2026-09-02T03:47:00Z",
+      "error_message": "Live simulation run (xce_scheduler --sim, user-observed, not reproduced by me in this session) shows a JOBS_PUSH batch of 50 jobs where the first 22 (indices [0]-[21]) receive [sim] DISPATCH decisions, immediately followed by [sim] PEND decisions for ALL 50 jobs (indices [0]-[49]) -- including the 22 that had JUST been dispatched. Design expectation (confirmed correct, see root_cause): a job that has received a DISPATCH decision must be treated as RUN immediately, without waiting for any downstream job-start acknowledgment, and must never be reconsidered as a fresh scheduling candidate afterward -- in BOTH the live scheduler and simulation/replay mode.",
+      "file": "src/scheduler/src/sim_loop.c, src/scheduler/src/plugins/builtin_fcfs.c, src/scheduler/src/scheduler.c",
+      "root_cause": "NOT YET ROOT-CAUSED -- logged from a live-observed symptom, investigated only via static code reading this session (no live repro attempted), per explicit instruction to log and defer rather than context-switch into a full debug pass. Confirmed via source read: sched_commit_decision() (scheduler.c:1435-1455) DOES set job->state = JOB_STAT_RUN synchronously at the moment a DISPATCH decision is committed, with no wait for any job-start ack -- this is shared code (sched_allocator()/sched_commit_decision()) used by BOTH the normal-mode driver and the sim driver alike (scheduler/CLAUDE.md: \"Both drive the identical core session/decision-callback protocol\"), so the intended \"dispatch implies RUN, immediately\" design IS mechanically implemented at this layer already -- this is not a missing feature. builtin_fcfs.c's on_dispatch() carries its own documented, pre-existing concurrency-hazard comment (\"this runs in the allocator thread and can race the picker thread's next_job() -- jbuf's capacity-1 handoff unblocks the picker for job N+1 before this call for job N returns\"), guarded by fcfs_plugin_data_t.lock for candidate-list mutations, but reading the code alone does not confirm whether that race (or something else -- e.g. two back-to-back sim sessions/rounds failing to preserve job->state or the picker's candidate set between them) is what produces ALL 50 jobs reappearing in one PEND batch, including the 22 already-dispatched ones. Related prior findings: bug-1567 (fixed 2026-08-17) documented that simulation mode ONCE unconditionally PENDed every candidate with no real dispatch at all, deliberately deferred to a then-not-yet-designed scheduler-allocation spec -- the pasted live log now shows genuine DISPATCH decisions in sim mode, meaning that gap has since been closed (scheduler-sim-decision-parity, referenced by bug-1623, 2026-08-22, looks like the likely spec, not yet confirmed which task/commit). bug-1623 also documents sim_loop.c's sim_reconcile_unaccounted_jobs() logic (per-round decoded-decision matching for RUN-state unaccounted jobs) as a separate, related mechanism in the same file worth checking during a real debug pass.",
+      "fix": "NOT FIXED -- deliberately deferred per explicit instruction (log and continue other work, do not context-switch into a full debug pass now). Needs a dedicated investigation (e.g. /kiro-debug) scoped to sim_loop.c's session/round-driving loop to determine: (a) whether this is one scheduling session emitting a genuine duplicate/second decision for already-dispatched jobs (a real bug in the shared allocator/picker pipeline or in sim_loop.c's own session driving), or two distinct back-to-back sessions where job->state / the picker's candidate set was not correctly carried over between them; (b) whether the identical scenario is reproducible against the NORMAL-mode driver (scheduler.c/driver.c), which would point at the shared sched_allocator/builtin_fcfs code rather than anything sim-specific, or whether it only reproduces in --sim, which would point at sim_loop.c's own session/round handling (possibly interacting with bug-1623's sim_reconcile_unaccounted_jobs()). A live, controlled repro (fixed job count, fixed resource limits, single JOBS_PUSH batch, --sim-verify or a new targeted unit test) should be built first rather than relying on the user's own live-run log.",
+      "tags": [
+        "scheduler",
+        "sim-mode",
+        "sim_loop-c",
+        "builtin_fcfs",
+        "dispatch",
+        "pend",
+        "job-state",
+        "dispatch-implies-run",
+        "race-condition-suspected",
+        "needs-debug",
+        "discovered-live",
+        "user-reported",
+        "not-fixed"
+      ],
+      "related_bugs": [
+        "bug-1567",
+        "bug-1623"
+      ],
+      "occurrences": 1,
+      "last_seen": "2026-09-02T03:47:00Z"
+    },
+    {
+      "id": "bug-1760",
+      "timestamp": "2026-09-02T04:10:00Z",
+      "error_message": "Live-tested against a real 3-node HA cluster + xcecd_virtual worker + xce_scheduler (worker-rejoin-resync soak-harness development): after the CURRENT primary is killed and a NEW primary is elected+reconverged, a job submitted through the new primary stays PEND for 60+ seconds with ZERO scheduler.log activity of any kind (no WARN/INFO/ERROR, no reconnect attempt) -- not a slow recovery, a genuinely stuck one. Reproduced twice in a row with the same shape.",
+      "file": "src/scheduler/src/scheduler.c (sched_needs_rejoin:487, sched_on_reconnect:507, sched_publish_check_rejoin:1474), src/scheduler/src/main.c",
+      "root_cause": "CONFIRMED via source read (not yet live-instrumented/gdb-confirmed): the scheduler has NO periodic connection-liveness/reconnect check analogous to the worker daemon's checking_connection() (src/worker/src/controller_connector.c:505, called every xcecd_ctx.update_interval seconds from xcecd.c's main loop via xcecli_hello()). The scheduler's ONLY rejoin mechanism is sched_publish_check_rejoin() (scheduler.c:1474), which re-issues xcecli_join() ONLY as a side effect of its own NEXT SUCCESSFUL publish call (sched_publish_dispatch()/sched_publish_pend_reason() via sched_publish_retry_loop(), which calls sched_publish_check_rejoin() only after attempt_fn() already returned >=0). sched_needs_rejoin itself is set correctly by sched_on_reconnect() (the client library's own transparent-reconnect callback), so the SCHEDULER's own underlying connection to the cluster likely IS aware it reconnected somewhere -- but nothing forces a FRESH xcecli_join() against the actual NEW primary until the scheduler has something to publish. After a real controller failover, the scheduler has nothing queued to publish (no job ever reaches its picker/allocator, since the CONTROLLER-side registration of \"where the scheduler is\" is presumed to be live-connection-scoped state on whichever controller instance the scheduler last explicitly joined -- i.e. the dead primary -- and does not appear to carry over to the newly-elected primary), so the scheduler can never publish anything to trigger its own rejoin: a chicken-and-egg deadlock. NOT YET independently confirmed via gdb/live instrumentation which exact half of this loop is stuck (client-library-level reconnect vs. controller-side scheduler-registration loss) -- this is a plausible, evidence-consistent hypothesis from static reading + the observed symptom, not a fully root-caused finding.",
+      "fix": "NOT FIXED -- discovered as a side effect of unrelated worker-rejoin-resync soak-harness development; deliberately not investigated further or fixed in that session (explicit instruction: log and continue). This is a potentially serious standing HA-availability gap (every job submitted after ANY controller failover could stay PEND forever until the scheduler process is manually restarted) that deserves its own dedicated debugging investigation (e.g. /kiro-debug) scoped to component-failover-adoption's scheduler-rejoin design, ideally with live gdb/log instrumentation to confirm which half of the chicken-and-egg loop is actually broken. Pragmatic workaround adopted ONLY in the worker-rejoin-resync soak harness (.claude/skills/xce-test-ha/run-ha-soak.sh): restart the scheduler process after each round's controller failover, the same way the harness already restarts the downed CONTROLLER node -- a fresh scheduler process calls sched_init(), which tries all 3 candidates in order and joins whichever is actually reachable, sidestepping the stuck reactive-rejoin path entirely. This workaround is scoped to test infrastructure only and does NOT fix the underlying production behavior.",
+      "tags": [
+        "scheduler",
+        "ha",
+        "failover",
+        "rejoin",
+        "availability",
+        "component-failover-adoption",
+        "sched_publish_check_rejoin",
+        "chicken-and-egg",
+        "discovered-live",
+        "needs-debug",
+        "not-fixed",
+        "severity-high-suspected"
+      ],
+      "related_bugs": [],
+      "occurrences": 2,
+      "last_seen": "2026-09-02T04:10:00Z"
     }
   ]
 }