fix(component-failover-adoption): Phase 6 -- close apiserver's idle-period rejoin gap (bug-1548/bug-1549)

XCECLI_FF_CALL (backing xcecli_heartbeat) can silently rotate+reconnect
the transport entirely inside a single call and still return success, so
heartbeat.go's failure-to-success transition detector could never observe
a reconnect that completes within one heartbeat tick -- the common case,
since elections finish in single-digit seconds against a default 30s
interval. apiserver's application-level rejoin never fired on a real
idle-period failover, and job pushes silently stopped reaching it.

Adds a monotonic reconnect_gen counter to endpoint_t (netio.h), bumped by
xce_ctx_reconnect() (xcectx.c) on every successful reconnect regardless of
trigger path. heartbeat.go now compares this generation every tick instead
of tracking success/failure transitions. Re-validated live against task
5.2's exact scenario (apiserver alone, zero write traffic): rejoin fired
22s after a primary kill, 12s after election.

Reopens and closes out component-failover-adoption's own NO-GO -> GO
validation cycle from 2026-08-15.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-08-16 03:42:03 +00:00
parent 0cb5918517
commit d479b70764
Signed by: rshen
SSH key fingerprint: SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
11 changed files with 430 additions and 146 deletions

View file

@ -83,6 +83,7 @@ import (
"fmt"
"strings"
"sync"
"sync/atomic"
"time"
"unsafe"
@ -350,19 +351,35 @@ func Cleanup() {
C.idgen_ctx_fini(&gIDCtx)
}
// currentReconnectGen reads endpoint_t.reconnect_gen (netio.h) off the
// backend's live endpoint via the existing, previously-unused C accessor
// xcecli_get_my_endpoint() -- no new C getter needed (component-failover-
// adoption Phase 6, bug-1548). Uses an explicit atomic load (rather than a
// bare field dereference) so the Go-side read is formally atomic and pairs
// correctly with the C11 _Atomic write in xce_ctx_reconnect() (xcectx.c).
func currentReconnectGen() uint64 {
ep := C.xcecli_get_my_endpoint(gCtx.ctx)
if ep == nil {
return 0
}
return atomic.LoadUint64((*uint64)(unsafe.Pointer(&ep.reconnect_gen)))
}
// StartHeartbeat launches a goroutine that sends periodic xcecli_heartbeat
// keep-alive messages to the controller. Stops when ctx is cancelled.
//
// Wires runHeartbeat's failure-to-success and every-successful-tick hooks to
// Wires runHeartbeat's reconnect-detected and every-successful-tick hooks to
// reconnect.go's setNeedsRejoin/checkAndPerformRejoin (task 4.3) -- closing
// the idle-period reconnect gap (task 4.4, Req 2 AC4/AC5): XCECLI_FF_CALL
// (backing xcecli_heartbeat) never invokes the client library's rejoin
// callback, so nothing else would set the pending-rejoin flag if this
// heartbeat prober is the only traffic during a controller outage.
// the idle-period reconnect gap (task 4.4/Phase 6, Req 2 AC4/AC5):
// XCECLI_FF_CALL (backing xcecli_heartbeat) never invokes the client
// library's rejoin callback, so nothing else would set the pending-rejoin
// flag if this heartbeat prober is the only traffic during a controller
// outage. currentReconnectGen (above) is what actually detects the
// reconnect even when it completed silently inside a single XCECLI_FF_CALL.
func StartHeartbeat(ctx context.Context) {
go runHeartbeat(ctx, func(secs int) int {
return int(C.xcecli_heartbeat(gCtx.ctx, C.int(secs)))
}, setNeedsRejoin, checkAndPerformRejoin)
}, currentReconnectGen, setNeedsRejoin, checkAndPerformRejoin)
}
// go_push_handler is called by the C library on every incoming request.