fix(component-failover-adoption): Phase 6 -- close apiserver's idle-period rejoin gap (bug-1548/bug-1549)

XCECLI_FF_CALL (backing xcecli_heartbeat) can silently rotate+reconnect
the transport entirely inside a single call and still return success, so
heartbeat.go's failure-to-success transition detector could never observe
a reconnect that completes within one heartbeat tick -- the common case,
since elections finish in single-digit seconds against a default 30s
interval. apiserver's application-level rejoin never fired on a real
idle-period failover, and job pushes silently stopped reaching it.

Adds a monotonic reconnect_gen counter to endpoint_t (netio.h), bumped by
xce_ctx_reconnect() (xcectx.c) on every successful reconnect regardless of
trigger path. heartbeat.go now compares this generation every tick instead
of tracking success/failure transitions. Re-validated live against task
5.2's exact scenario (apiserver alone, zero write traffic): rejoin fired
22s after a primary kill, 12s after election.

Reopens and closes out component-failover-adoption's own NO-GO -> GO
validation cycle from 2026-08-15.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-08-16 03:42:03 +00:00
parent 0cb5918517
commit d479b70764
Signed by: rshen
SSH key fingerprint: SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
11 changed files with 430 additions and 146 deletions

View file

@ -120,6 +120,21 @@ typedef struct endpoint_t {
* their lifetime directly. */
_Atomic int refcnt;
int heap_alloc; /* 1 iff allocated by endpoint_get (unref may free) */
/* bug-1548: bumped once, unconditionally, by xce_ctx_reconnect() (xcectx.c)
* on every successful transport reconnect, regardless of trigger path
* (write-path XCECLI_FAILOVER_CALL retry or heartbeat XCECLI_FF_CALL
* best-effort self-heal). Neither endpoint_init() nor endpoint_fini()
* (netio.c) touch this field, so it survives reconnect cycles the same
* way refcnt/heap_alloc already do. Exists so a consumer (e.g. apiserver's
* heartbeat prober, client.h's xcecli_get_my_endpoint) can detect that a
* reconnect happened even when it completed silently inside a single
* fire-and-forget call and the call's own return code never surfaced a
* failure — XCECLI_FF_CALL can rotate+recover before evaluating its EXPR,
* so a failure->success transition on the caller's own return code is not
* a reliable signal (component-failover-adoption Phase 6). Does not
* change reconnect timing/ordering/semantics — a passive observation
* point at the existing success path, same shape as on_fini above. */
_Atomic uint64_t reconnect_gen;
} endpoint_t;
/* bug-1298 family: msg_handler runs SYNCHRONOUSLY on the io_worker thread (not