myxce/.wolf/bugs/bug-1548.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	legacy
bug-1548	fixed	2026-09-24		apiserver	unknown	2026-08-12T12:59:31Z	2026-09-19T09:00:00Z	3	src/frontend/apiserver/backend/reconnect.go heartbeat.go clib.go (component-failover-adoption task 4.3/4.4's checkAndPerformRejoin/needsRejoin mechanism)	apiserver rejoin push-notification failover component-failover-adoption task-5.1 heartbeat live-e2e-finding task-5.2 second-independent-repro		true

bug-1548: apiserver never re-registers (application-level xcecli_join with HostRole_apiserver) with the new…

Description

apiserver never re-registers (application-level xcecli_join with HostRole_apiserver) with the new primary after a live controller failover, even though its transport/write-call connectivity clearly recovers (job/host-control POSTs keep succeeding); observed for 10+ minutes and 20+ default-30s heartbeat ticks with zero occurrences of reconnect.go's own 'checkAndPerformRejoin: rejoin after transparent reconnect succeeded/failed' log lines in apiserver_out.log, across 3 separate write calls (2 job submits + 1 host control) issued after the primary was kill -9'd. TASK 5.2 UPDATE (2026-08-12, second independent live E2E reproduction, cleaner isolation): ran component-failover-adoption task 5.2's own scenario (apiserver-only, no worker/scheduler, 3-node controller cluster at localhost:19991-19993, apiserver -d /home/claude/tmp/xce-e2e-task52/apiserver) specifically to isolate the heartbeat-only idle-period path task 5.1's write-traffic-muddied run could not cleanly separate. Confirmed real election: ctrl-1 (primary, term=2) kill -9'd at 13:33:31Z; ctrl-2 promoted primary at term=3 within 9s (13:33:40Z, raft_cb_become_leader). Zero write requests issued for the full outage window: apiserver_out.log is BYTE-IDENTICAL (23 lines) from before the kill through 268s / ~9 heartbeat ticks after it (well past the task's 2-minute/3-4-tick minimum) -- zero occurrences of 'checkAndPerformRejoin'/'rejoin'/'needsRejoin'/'xcecli_join' anywhere. Diagnostic control issued only AFTER the observation window (13:39:52Z, ~6.5 min post-kill): POST /xce/v1/jobs/create succeeded with a real controller-assigned job ID (73930919093731328) and server-generated success message -- proving the transport had already reached the NEW primary (ctrl-2) -- yet this write call ALSO produced zero rejoin log lines, full apiserver_out.log grep for the same 4 terms still empty after it. This is strictly stronger evidence than task 5.1's: it directly demonstrates the write path's own XCECLI_FAILOVER_CALL/xcecli_invoke_rejoin also did not fire even when explicitly exercised, consistent with the transport having already been silently repaired by an earlier heartbeat tick's XCECLI_FF_CALL before the write call ever needed to retry -- exactly bug-1548's leading hypothesis, not merely 'consistent with' it as task 5.1 could only say. Corroborating bug-1549 downstream symptom also reproduced: GET /xce/v1/jobs immediately after the successful submit returned {"count":0,"rows":[]} -- the just-submitted job never reached apiserver's local store, consistent with apiserver never having actually re-issued xcecli_join(HostRole_apiserver) against ctrl-2. ctrl-2.log has no apiserver-join lines either direction (join.c logs only rejections, per task 5.1's prior finding -- reconfirmed, not independently informative here). Task 5.2's own observable-completion criterion ('the scenario passes without requiring any client-triggered write request during the outage window' -- i.e. apiserver rejoins and resumes push notifications via the heartbeat-triggered path alone) was NOT met: no rejoin evidence appeared at any point in the heartbeat-only window. Evidence preserved at /home/claude/tmp/xce-e2e-task52/ (apiserver/logs/apiserver_out.log, logs/ctrl-{1,2,3}.log, logon_response.json, diagnostic_job_submit_response.json, diagnostic_joblist_response.json, KILL_TIMESTAMP_UTC.txt, DIAGNOSTIC_CONTROL_START_UTC.txt, DIAGNOSTIC_CONTROL_SUBMIT_UTC.txt).

Environment

Not recorded (migrated from buglog.json).

Reproduction

Not recorded (migrated from buglog.json).

Root cause

NOT root-caused (out of task 5.1's E2E-validation boundary to fix). Leading hypothesis from live log analysis: XCECLI_FF_CALL (the fire-and-forget macro backing xcecli_heartbeat) can silently self-heal the TRANSPORT (rotate to a live candidate) entirely within a single heartbeat tick without that tick ever returning a failure to Go -- so heartbeat.go's prevFailed-edge detection (onFailureToSuccess) never observes a failure->success transition and never sets needsRejoin. Separately, none of the 3 post-kill write calls (which DO route through XCECLI_FAILOVER_CALL, which per design.md should invoke the rejoin callback mid-call on a genuine failover) show any evidence of having hit a failed connection either -- consistent with the connection having already been silently fixed by a prior heartbeat tick before any write call needed to retry, so XCECLI_FAILOVER_CALL's own rejoin-invoking path is also never exercised. Net effect: needsRejoin is never set by any path, so checkAndPerformRejoin() stays a permanent silent no-op and apiserver's push-registration (join) with the new primary is never renewed. Could not independently confirm via controller-side logs either way -- discovered during this same investigation that the controller's join.c only logs REJECTED/failed joins (WARN/ERROR), never a successful one at any verbosity, so controller logs are not usable evidence for this question in either direction. REVIEW UPDATE (task 5.1 review, 2026-08-12): confirmed via source read that XCECLI_FF_CALL (client_internal.h:290-302) never calls xcecli_invoke_rejoin under any circumstance -- this half of the hypothesis is not speculative, it's structurally true by construction. Elevated confidence this is a REPRODUCIBLE gap, not scenario-specific timing luck: since real elections in this codebase's own logs complete in single-digit seconds while the default heartbeat interval is 30s, a single heartbeat tick landing after the election has already completed will both discover the dead connection AND repair it within that same XCECLI_FF_CALL invocation -- the failure never surfaces to Go's heartbeat.go loop as a separate tick, so the cross-tick prevFailed-edge detection (onFailureToSuccess) structurally cannot observe it. This is likely to recur on MOST real single-election failovers, not just this run's specific timing -- materially risks Req 2 AC4 (component-failover-adoption requirements.md) in production. Recommend escalating past ordinary buglog cadence to a dedicated follow-up spec/task, given it affects the already-approved task 4.3/4.4 mechanism.

Fix

FIXED (component-failover-adoption tasks.md Phase 6, task 6.1). Root cause confirmed as hypothesized: XCECLI_FF_CALL (client_internal.h:290-302) can rotate+reconnect the transport entirely inside a single call, holding ac->failover.lock, and still return success -- so a failure->success transition on the caller's own return code is not a reliable 'a reconnect happened' signal whenever an election (single-digit seconds) completes faster than the 30s default heartbeat interval, which is the common case. Fix: added a monotonic _Atomic uint64_t reconnect_gen field to endpoint_t (src/include/netio.h:137), incremented unconditionally on the success path of xce_ctx_reconnect() (src/lib/xcectx.c:402) -- confirmed via repo-wide grep as the sole reconnect call-site reachable from xce_failover_rotate(), covering both the write-path retry loop and the heartbeat fire-and-forget path. apiserver's heartbeat.go runHeartbeat() now compares this generation on every tick (via the existing xcecli_get_my_endpoint() accessor, atomic-loaded through clib.go) instead of waiting for a failure->success transition, replacing the old detector entirely. Regression test TestRunHeartbeat_IdlePeriodRecovery_GenChangeAloneSetsNeedsRejoinEvenWhenAllTicksSucceed (heartbeat_rejoin_test.go) proves the fix's exact shape. Closed via live E2E re-run of task 5.2's scenario: rejoin fired 22s after primary kill (12s post-election) with zero write traffic for the preceding 5.5 minutes.

Verification

Not recorded (migrated from buglog.json).

References

Not recorded (migrated from buglog.json).