fix(component-failover-adoption): Phase 6 -- close apiserver's idle-period rejoin gap (bug-1548/bug-1549)

XCECLI_FF_CALL (backing xcecli_heartbeat) can silently rotate+reconnect
the transport entirely inside a single call and still return success, so
heartbeat.go's failure-to-success transition detector could never observe
a reconnect that completes within one heartbeat tick -- the common case,
since elections finish in single-digit seconds against a default 30s
interval. apiserver's application-level rejoin never fired on a real
idle-period failover, and job pushes silently stopped reaching it.

Adds a monotonic reconnect_gen counter to endpoint_t (netio.h), bumped by
xce_ctx_reconnect() (xcectx.c) on every successful reconnect regardless of
trigger path. heartbeat.go now compares this generation every tick instead
of tracking success/failure transitions. Re-validated live against task
5.2's exact scenario (apiserver alone, zero write traffic): rejoin fired
22s after a primary kill, 12s after election.

Reopens and closes out component-failover-adoption's own NO-GO -> GO
validation cycle from 2026-08-15.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

.kiro/specs/component-failover-adoption/spec.json

@@ -1,9 +1,9 @@
 {
   "feature_name": "component-failover-adoption",
   "created_at": "2026-07-20T04:23:58Z",
-  "updated_at": "2026-07-20T05:20:00Z",
+  "updated_at": "2026-08-15T14:20:00Z",
   "language": "en",
-  "phase": "tasks-generated",
+  "phase": "implementation-complete",
   "approvals": {
     "requirements": {
       "generated": true,
@@ -15,8 +15,24 @@
     },
     "tasks": {
       "generated": true,
-      "approved": false
+      "approved": true
     }
   },
-  "ready_for_implementation": false
+  "ready_for_implementation": true,
+  "validate_impl": {
+    "decision": "GO",
+    "validated_at": "2026-08-15T14:20:00Z",
+    "known_limitations": [],
+    "resolved_limitations": [
+      "bug-1548/bug-1549: apiserver never re-registered with a new primary after a fast controller failover. Fixed via tasks.md Phase 6 -- endpoint_t.reconnect_gen counter (netio.h/xcectx.c) bumped on every successful transport reconnect, heartbeat.go's runHeartbeat replaced its failure-to-success transition detector with a per-tick generation comparison. Confirmed via live E2E re-run of task 5.2's exact scenario: rejoin fired 22s after primary kill (12s after new election) with zero write traffic for the preceding 5.5 minutes; a post-recovery diagnostic job submit correctly appeared via GET /xce/v1/jobs.",
+      "bug-1559: xce-build.sh's scheduler case didn't build/run 4 newer scheduler test binaries. Fixed directly (tooling, not a spec task) -- now runs `meson test --suite scheduler`, exercising all 41 scheduler test binaries.",
+      "bug-1558: make test-backend broken standalone. Fixed directly (tooling, not a spec task) -- self-derives CGO_CFLAGS/CGO_LDFLAGS via pkg-config, confirmed working with zero manual env setup."
+    ],
+    "prior_decision": {
+      "decision": "NO-GO",
+      "validated_at": "2026-08-15T13:10:00Z",
+      "reopened_at": "2026-08-15T13:10:00Z",
+      "reopen_reason": "component-failover-adoption's own /kiro-validate-impl run found Req 2 AC4 unmet in the common case (fast controller election, apiserver idle) -- a defect inside this spec's own ApiserverBackendBridge/task 4.4 boundary, not legitimately out of scope. Remediated as tasks.md Phase 6 (6.1/6.2) plus two non-blocking build-tooling fixes (bug-1558, bug-1559) found during the same validation pass."
+    }
+  }
 }

.kiro/specs/component-failover-adoption/tasks.md

@@ -159,6 +159,26 @@
   - _Requirements: 1.5_
   - _Depends: 5.1, 5.2, 5.3_
 
+## 6. Post-validation remediation (bug-1548/bug-1549, NO-GO 2026-08-15)
+
+- [x] 6.1 Fix apiserver's idle-period rejoin detection gap
+  - Add a monotonic reconnect-generation counter to the client library's endpoint struct, incremented on every successful transport reconnect regardless of trigger path
+  - Replace the heartbeat prober's failure-to-success transition check with a per-tick comparison against this generation counter, triggering the same pending-rejoin flag on any change
+  - Observable completion: a new regression test proves the pending-rejoin flag is set on a generation change even when every heartbeat tick the Go loop observes reports success (the exact shape XCECLI_FF_CALL's silent inline reconnect produces) — its own doc comment states this proves detector logic, not reachability
+  - _Requirements: 2.4, 2.5_
+  - _Depends: 4.4_
+
+- [x] 6.2 E2E re-validation: re-run task 5.2's idle-period scenario against the fix
+  - Re-run the exact task 5.2 scenario (apiserver only, no worker/scheduler, 3-node controller cluster, default heartbeat interval, primary killed with zero write traffic during the observation window) against the fixed code
+  - Observable completion: identical to task 5.2's own bar — rejoin log lines appear within the observation window, and GET /xce/v1/jobs populates correctly after a post-recovery submit, with no client-triggered write request required
+  - _Requirements: 2.4, 2.5_
+  - _Depends: 6.1_
+
+## Implementation Notes (Phase 6)
+- Task 6.1: fix is a monotonic `endpoint_t.reconnect_gen` counter (`src/include/netio.h`), bumped once, unconditionally, on the success path of `xce_ctx_reconnect()` (`src/lib/xcectx.c`) — confirmed the sole reconnect call-site funnel for both the write-path retry loop and the heartbeat fire-and-forget path, so it observes every reconnect regardless of trigger. Read from Go via the existing, previously-unused `xcecli_get_my_endpoint()` accessor (`client.h`) — no new C getter needed, and none of the frozen mechanism files (`client.c`/`client_internal.h`/`client_failover.c`/`client_failover.h`) were touched. `heartbeat.go`'s `runHeartbeat()` fully replaced its `prevFailed` failure-to-success transition detector with a per-tick generation comparison (user's explicit direction: no compatibility shim needed, pick the objectively better approach) — the old detector is logically a strict subset of the new one under this codebase's own documented state machine (`ctx->state == XCE_CTX_ERR` is the single known-dead signal gating `XCECLI_FF_CALL`'s reconnect branch), and the new one additionally catches a reconnect triggered by a concurrent write-path call between heartbeat ticks. New regression tests (`heartbeat_test.go`, `heartbeat_rejoin_test.go`) prove the detector fires on a generation change even when every observed tick reports success — the literal shape of bug-1548 that the old detector's own green test suite could never catch.
+- Task 6.2: live E2E re-run of task 5.2's exact scenario (apiserver alone, no worker/scheduler, 3-node cluster, default 30s heartbeat interval, primary killed, zero write traffic during the observation window) — **PASS**, closing the gap. Primary killed 14:13:46 UTC; new primary elected 14:13:56 UTC (10s); `checkAndPerformRejoin: rejoin after transparent reconnect succeeded` logged at 14:14:08 UTC (22s after the kill, 12s after the election), with zero write/backend traffic in the preceding 5.5 minutes — the heartbeat path alone triggered it, exactly the case that previously showed zero rejoin evidence across a 268s+ window. bug-1549 downstream symptom also confirmed resolved: a diagnostic job submitted after recovery correctly appeared via `GET /xce/v1/jobs` (count:1, matching job data), proving apiserver's push/registration state genuinely recovered with the new primary, not just the transport.
+- Non-blocking tooling fixes made alongside (not spec tasks — `xce-build.sh`/the frontend `Makefile` are general repo tooling, not spec-owned files per design.md's File Structure Plan; tracked via buglog only): bug-1559 (`xce-build.sh`'s `scheduler` case now runs `meson test --suite scheduler`, exercising all 41 scheduler test binaries instead of just `sched_test` directly by path) and bug-1558 (`make test-backend` now self-derives `CGO_CFLAGS`/`CGO_LDFLAGS` via `pkg-config`, works standalone with zero manual env setup — one iteration needed: the first attempt's `-I../../include` was relative and silently resolved wrong under cgo's actual build cwd, fixed to an absolute `-I$(CURDIR)/../include`).
+
 ## Implementation Notes
 - Task 1.1: `src/lib/cluster.c:99-102`'s `xce_cluster_config_validate()` still has a third hardcoded `>3` WARN check (message says "<=3 only") of the same magic-number class Req 1.7 targets, and `src/include/xce/cluster.h:31`'s doc comment repeats the stale "3" figure. Task 1.1's text named only "the two" worker/scheduler checks, so this was left untouched there — task 1.2 already touches `cluster.c` to add the new conversion helper, so fold this fix in there rather than letting it silently persist as the one remaining un-reconciled magic number.
 - Task 1.1 follow-up resolved in 1.2: the `>3`→`>MAX_HA_CANDIDATES` rename initially proposed for `xce_cluster_config_validate()`'s WARN check turned out to be dead code — after task 1.1's constant reconciliation, that check tests the identical field against the identical constant as an earlier `-EINVAL` return in the same function, so it can never fire. Deleted rather than renamed (with the matching stale doc-comment line also removed), caught by task 1.2's review round 1.
@@ -175,11 +195,11 @@
 - Task 3.4: pure test-coverage task, zero production changes to `scheduler.c`/`driver.c` (correct outcome — task 3.2 already implemented the retry/shutdown/rejoin behavior correctly; 3.4 proves it holds at multi-decision QUEUE granularity through the real drain loop, which task 3.2's own tests never exercised, all 6 of which call `sched_publish_dispatch()`/`sched_publish_pend_reason()` directly on one decision). New `test_scheduler_decision_redelivery.c` links real `scheduler_srcs` with `-Wl,--wrap=xcecli_jobdispatch` (task 3.2's established technique) and drives the real `xce_sched_driver_on_decision()` (driver.c) over 3 queued, distinct-jobid decisions. Two load-bearing tests, both RED→GREEN-verified against the real code by two independent reviewer rounds (not just the implementer's claim): ordering (decision 1 stuck-then-succeeds fully completes before decision 2's call ever fires) and bounded shutdown latency (decisions queued behind one that triggers `core.stop` mid-retry get ZERO publish calls — the concrete, fast, deterministic proof that `sched_publish_retry_loop()`'s check-`stop`-before-each-attempt placement is correct, verified by reviewers moving the check to bottom-of-loop and confirming only this one test catches it). Test-fixture gotcha for any future test building a bare `xce_sched_core_t` via `xce_sched_core_init()` alone (not full `sched_init()`): do NOT call `xce_sched_core_fini()` on it — `fini()` calls `dataq_fini()` on `pend_updates[0/1]`, which only `sched_init()` (not `xce_sched_core_init()`) ever `dataq_init()`s, so `fini()` on a bare core SEGVs on a NULL dataq; omit the `fini()` call entirely, matching `test_core_events.c`'s established "fixtures aren't torn down" convention for this exact minimal-fixture pattern.
 - Task 4.5: subtle design question resolved and worth remembering for task 4.6/5.x — `callBackend()`'s (`clib.go`) degraded fast-path deliberately skips `checkAndPerformRejoin()` ENTIRELY (not just the underlying `C.xcecli_*` call), meaning write-path requests contribute NOTHING to rejoin-retry cadence while `IsDegraded()` is true; only task 4.4's heartbeat goroutine keeps retrying during that window. Reviewer independently confirmed this is correct, not a violation of "must never suppress the pending-rejoin flag's own retry-until-actually-joined behavior": that constraint is one-directional (the circuit breaker's own optimistic TTL-based state must never be READ BY the rejoin mechanism to gate it — confirmed `checkAndPerformRejoin()` never references `IsDegraded()` anywhere), not a requirement that every possible call site attempt a rejoin check regardless of cost. The alternative (calling `checkAndPerformRejoin()`, which can itself block ~100s inside `xcecli_join()`'s own retry budget, on every fast-failed request) would directly violate THIS task's own fail-fast acceptance criterion — the two constraints (fail-fast + never-suppress-retry) are jointly satisfiable only because `heartbeat.go`'s goroutine runs unconditionally for the process lifetime, independent of circuit-breaker state, and `backendDegradedTTL` (10s, chosen well below the ~100s `XCECLI_FAILOVER_CALL` budget) is shorter than the heartbeat interval (default 30s) so write traffic resumes contributing once the TTL clears anyway. `callBackend()` is a genuine refactor consolidating what tasks 4.3/4.4 left as duplicated inline `rc := C.xcecli_*(...); checkAndPerformRejoin()` pairs across 5 wrapper functions into one shared helper — `checkAndPerformRejoin()` now appears exactly once in `clib.go`, inside `callBackend`, matching design.md's explicit "not copy-pasted per wrapper" instruction. `E_XCE_NOT_READY`-triggered `MarkDegraded()` uses a plain-Go `notReadyRC` mirror var (`= -int(C.E_XCE_NOT_READY)`, computed live via cgo, not hardcoded) since `_test.go` files can't reference `C.*` symbols directly (task 4.1's established Go-toolchain restriction) — keep this pattern for any future test needing a C errno constant.
 - Task 4.6 (closes out apiserver group 4.1-4.6): audit task, same "zero new code is a valid outcome when genuinely true" precedent as scheduler's task 1.3 — 2 of 3 coverage areas (degraded-to-healthy transitions, candidate-list parity) were already fully satisfied by tasks 4.1/4.5's own tests, verified by reading the cited tests directly rather than trusting prior notes' claims. Found and closed exactly one genuine gap: `reconnect.go`'s `needsRejoin`/`checkAndPerformRejoin` (task 4.3) had zero concurrent-access test coverage despite being reached from 5 real concurrent HTTP-wrapper call sites plus the heartbeat goroutine — `health.go`'s own 50-goroutine stress test (task 4.5) only covered the degraded-until timestamp, not this separate atomic flag. One new test closes it. `gClusterCfg`/`gClientCfg` (task 4.2's CGO structs) correctly judged NOT a concurrency gap — written once at startup, single-threaded, before the HTTP server starts, never mutated again. Confirmed apiserver's Go-level `xce_parse_peers()` malformed-string coverage (`candidates_test.go`) is broader than any C-side component's own test — worker's/scheduler's C tests, and even `test_cluster.c`, only exercise `xce_cluster_config_validate()` against pre-built structs, never `xce_parse_peers()` against malformed raw strings at all. `go test -race` is confirmed, again, flatly unusable in this ASAN-linked CGO environment ("too many address space collisions") — this is now the third task (4.2, 4.5, 4.6) to independently hit and document this; don't waste time trying `-race` here, rely on atomic-correctness-by-construction plus deterministic concurrent-stress assertions instead. `xce-build.sh apiserver asan`'s binary-compile step can hit transient `goproxy.cn` network flakiness unrelated to any code change — when that happens, fall back to `cd src/frontend && make test` (CGO_ENABLED=0, whole apiserver module + xcecli) and `make test-backend` (CGO_ENABLED=1, `./backend/...`) directly, which is what the full pipeline's own `RUN_TESTS=1` branch invokes anyway once binaries are built. A pre-existing, unrelated `apiserver/controllers` build failure under a whole-module `CGO_ENABLED=1 go test ./...` (predates this spec entirely, `integration_test.go`'s `//go:build !cgo` tag vs. `output_test.go`'s unconditional references to its helpers) doesn't affect either authoritative Makefile target — don't chase it as a regression in future apiserver work.
-- Task 5.1 (first Group 5 task, live 4-process E2E — worker + scheduler + apiserver + 3-node controller cluster): `xcecd_virtual` (worker's own task 2.3 meson target, `-DVIRTUAL_CLUSTER -DDBUS_MANAGER`) is the correct non-root worker E2E vehicle in this sandbox — use it, NOT `unshare -r`, which hits the unresolved bug-010 ASAN double-join abort. Worker's reconnect proved cleanly (real `checking_connection: rejoin after transparent reconnect succeeded` log line, 16s after kill, PID unchanged). Scheduler's rejoin path was never naturally exercised (no job ever reached its picker/allocator) — clean initial join + PID stability shown instead, same partial-coverage shape task 3.3 already got approved with. **Found a likely-real, structural gap in the already-approved task 4.3/4.4 apiserver rejoin mechanism (bug-039, see buglog for full detail)**: `XCECLI_FF_CALL` (backing `xcecli_heartbeat`) can silently self-heal the transport within a SINGLE heartbeat tick (rotate+recover, then the send succeeds) without ever returning a failure to Go — so `heartbeat.go`'s cross-tick `onFailureToSuccess` transition detector structurally cannot observe it when election completes faster than one heartbeat interval (elections in this codebase complete in single-digit seconds; default heartbeat interval is 30s), and none of apiserver's post-kill write calls showed evidence of hitting a failed connection either (consistent with the transport having already healed via a prior heartbeat tick before any write needed to retry). Net: `needsRejoin` may never get set on a real single-election failover, so apiserver's application-level re-join with the new primary never fires even though transport-level requests keep succeeding — risks Req 2 AC4 in production. This is NOT a task-5.1 fix (out of its cross-component-validation boundary) — flagged for a dedicated follow-up spec/task, escalated above ordinary buglog cadence given it touches an already-approved mechanism. **Also found bug-040** (job pushes never reach apiserver's store, `GET /xce/v1/jobs` empty despite controller-confirmed commits) — very likely a downstream symptom of bug-039, not independent: `on_apiserver_join()` (`apiserver.c`) only bulk-resyncs jobs when a fresh join is actually processed, and `c->apiservers[]` (the push fan-out list) is per-process in-memory state with no replication — a new primary starts with an empty registration table until each apiserver instance re-joins against it. Fixing bug-039 should be verified against bug-040 clearing as confirmation they share a root cause. Controller-side `join.c` logs only rejected/failed joins, never successful ones at any verbosity — a real observability gap for anyone debugging this family of issues in the future, worth fixing alongside.
+- Task 5.1 (first Group 5 task, live 4-process E2E — worker + scheduler + apiserver + 3-node controller cluster): `xcecd_virtual` (worker's own task 2.3 meson target, `-DVIRTUAL_CLUSTER -DDBUS_MANAGER`) is the correct non-root worker E2E vehicle in this sandbox — use it, NOT `unshare -r`, which hits the unresolved bug-1519 ASAN double-join abort. Worker's reconnect proved cleanly (real `checking_connection: rejoin after transparent reconnect succeeded` log line, 16s after kill, PID unchanged). Scheduler's rejoin path was never naturally exercised (no job ever reached its picker/allocator) — clean initial join + PID stability shown instead, same partial-coverage shape task 3.3 already got approved with. **Found a likely-real, structural gap in the already-approved task 4.3/4.4 apiserver rejoin mechanism (bug-1548, see buglog for full detail)**: `XCECLI_FF_CALL` (backing `xcecli_heartbeat`) can silently self-heal the transport within a SINGLE heartbeat tick (rotate+recover, then the send succeeds) without ever returning a failure to Go — so `heartbeat.go`'s cross-tick `onFailureToSuccess` transition detector structurally cannot observe it when election completes faster than one heartbeat interval (elections in this codebase complete in single-digit seconds; default heartbeat interval is 30s), and none of apiserver's post-kill write calls showed evidence of hitting a failed connection either (consistent with the transport having already healed via a prior heartbeat tick before any write needed to retry). Net: `needsRejoin` may never get set on a real single-election failover, so apiserver's application-level re-join with the new primary never fires even though transport-level requests keep succeeding — risks Req 2 AC4 in production. This is NOT a task-5.1 fix (out of its cross-component-validation boundary) — flagged for a dedicated follow-up spec/task, escalated above ordinary buglog cadence given it touches an already-approved mechanism. **Also found bug-1549** (job pushes never reach apiserver's store, `GET /xce/v1/jobs` empty despite controller-confirmed commits) — very likely a downstream symptom of bug-1548, not independent: `on_apiserver_join()` (`apiserver.c`) only bulk-resyncs jobs when a fresh join is actually processed, and `c->apiservers[]` (the push fan-out list) is per-process in-memory state with no replication — a new primary starts with an empty registration table until each apiserver instance re-joins against it. Fixing bug-1548 should be verified against bug-1549 clearing as confirmation they share a root cause. Controller-side `join.c` logs only rejected/failed joins, never successful ones at any verbosity — a real observability gap for anyone debugging this family of issues in the future, worth fixing alongside.
-- Task 5.2: designed as a targeted, cleaner confirmation experiment for task 5.1's bug-039 finding (heartbeat-only, zero write traffic, isolated to apiserver alone — no worker/scheduler to confound the signal) — and it worked exactly as intended. Result: **bug-039 independently reproduced a second time, more cleanly** — a 268s+ observation window with zero rejoin log lines, AND (stronger than task 5.1) a post-window diagnostic write call that itself triggers zero rejoin activity despite succeeding against the confirmed-new primary, ruling out "task 5.1 was just unlucky timing." Reviewer independently re-derived the mechanism from source (not just citing task 5.1's analysis) and confirmed `checkAndPerformRejoin()` is called unconditionally after every backend call in `callBackend()`, so the diagnostic write's silence is a genuine negative control, not a skipped-check artifact. Controller-side `ctrl-2.log` corroborates independently — zero "apiserver" references anywhere, meaning no join attempt (success or rejection) ever reached the new primary. This task's own literal pass bar ("scenario passes without requiring any write request") was NOT met — approved anyway on the same precedent task 5.1 established: the validation itself succeeded rigorously and honestly surfaced a real gap in already-shipped code, which is the task's actual purpose. **bug-039 is now doubly-confirmed and should be treated as a real, live production gap, not a hypothesis** — a dedicated follow-up spec/task to fix `heartbeat.go`'s `onFailureToSuccess`/`onSuccessTick` wiring (make `XCECLI_FF_CALL`'s internal self-heal observable to Go, e.g. by having the heartbeat wrapper track transport state itself rather than relying on the call's return code) is warranted, separate from any remaining task in this spec's own scope. bug-040 also re-reproduced a third time in this run (same session, same isolated evidence) — buglog kept in sync on both bug-039 and bug-040's own objects (occurrences/last_seen) after review caught bug-040's object being out of sync with its own re-reproduction narrated inside bug-039's text.
+- Task 5.2: designed as a targeted, cleaner confirmation experiment for task 5.1's bug-1548 finding (heartbeat-only, zero write traffic, isolated to apiserver alone — no worker/scheduler to confound the signal) — and it worked exactly as intended. Result: **bug-1548 independently reproduced a second time, more cleanly** — a 268s+ observation window with zero rejoin log lines, AND (stronger than task 5.1) a post-window diagnostic write call that itself triggers zero rejoin activity despite succeeding against the confirmed-new primary, ruling out "task 5.1 was just unlucky timing." Reviewer independently re-derived the mechanism from source (not just citing task 5.1's analysis) and confirmed `checkAndPerformRejoin()` is called unconditionally after every backend call in `callBackend()`, so the diagnostic write's silence is a genuine negative control, not a skipped-check artifact. Controller-side `ctrl-2.log` corroborates independently — zero "apiserver" references anywhere, meaning no join attempt (success or rejection) ever reached the new primary. This task's own literal pass bar ("scenario passes without requiring any write request") was NOT met — approved anyway on the same precedent task 5.1 established: the validation itself succeeded rigorously and honestly surfaced a real gap in already-shipped code, which is the task's actual purpose. **bug-1548 is now doubly-confirmed and should be treated as a real, live production gap, not a hypothesis** — a dedicated follow-up spec/task to fix `heartbeat.go`'s `onFailureToSuccess`/`onSuccessTick` wiring (make `XCECLI_FF_CALL`'s internal self-heal observable to Go, e.g. by having the heartbeat wrapper track transport state itself rather than relying on the call's return code) is warranted, separate from any remaining task in this spec's own scope. bug-1549 also re-reproduced a third time in this run (same session, same isolated evidence) — buglog kept in sync on both bug-1548 and bug-1549's own objects (occurrences/last_seen) after review caught bug-1549's object being out of sync with its own re-reproduction narrated inside bug-1548's text.
 - Task 5.3: unlike 5.1/5.2 (killed only the primary, letting rotation succeed), this task needs a call to genuinely EXHAUST its retry budget — kill ALL 3 controller nodes, not just the primary. Corrected timing assumption worth remembering: `client_failover.c`'s `xce_failover_classify()` puts `ECONNREFUSED`/`ETIMEDOUT`/`ECONNRESET` (and the dead-ctx fast path) on the ZERO-backoff `XCE_FO_RETRY_NOW` track, not the ~70-100s exponential-backoff `XCE_FO_RETRY_BACKOFF` track (reserved for `E_XCE_NOMASTER`/`E_XCE_NOT_READY` responses from a reachable-but-electing controller) — so a fully-dead-cluster trigger call exhausts its 20-attempt budget in milliseconds, not minutes. Clean, strongly-corroborated PASS: exact literal string match on `ErrBackendUnavailable` (`clib.go`) proves the degraded fast-fail path genuinely skips the C call rather than the C call happening to also be fast; `E_XCE_NOT_READY`'s numeric value (10013 = `XCE_ERRNO_BASE` 10001 + 12 enum positions, `errnox.h`) independently confirmed the trigger call's `-10013` response was the real thing, not a fabricated failure; the raw C-library log (`client_failover.c`'s own WARN/error lines, a format-distinct second logging subsystem from apiserver's Go zap log) cross-corroborated the whole sequence to the millisecond — this "check two independently-generated log streams agree" technique is a strong, reusable evidence-quality pattern for any future live-validation task in this spec. `GET /ping` confirmed via source read to never touch `gCtx`/`callBackend` — genuinely meaningful Req 4.2 evidence, not a coincidence. Bonus (not required by task text but strong corroboration): waited past the 10s TTL and confirmed `IsDegraded()`'s half-open auto-clear correctly re-attempts the real call. Noted, judged non-blocking: `design.md`'s sequence diagram literally says "503 service unavailable" but the actual HTTP status is 500 (`common.ERROR_CODE`) — `api-standards.md`'s error-code table has no 503 entry anywhere in this codebase's convention, and design.md's own Error Handling section explicitly disclaims introducing new error codes, so 500 + a distinct message body is the correct, convention-following choice, not a gap; the diagram itself is just imprecise.
 - Task 5.4 (final task, closing regression gate): full from-scratch rebuild + retest of the entire stack (lib/controller/scheduler/worker/xceadmin/apiserver/xcecli) found zero new regressions — same audit-first, "zero new code when genuinely covered" precedent as tasks 1.3/4.6, independently re-verified by two full review rounds (one implementer, one reviewer, both rebuilding from scratch) rather than trusted from a single run. The one known failure (`sched_admin_request_test`'s `test_recordstart_symlink_escape_replies_error`) is confirmed the same pre-existing root-in-sandbox symlink-permission issue by exact message match and source-level explanation (`driver.c`'s `-EINVAL` vs `-EACCES` classifier branches) — cross-corroborated against two independent `.wolf/memory.md` entries predating this task and a peer AI session's unprompted, independently-run sweep landing on identical numbers. The "identical single-candidate path" property (Req 1.5) is proven for all three components by real, meaningful, individually-reread-and-rerun tests (`test_build_cluster_config_single_entry_peers_matches_synthesized_shape` for worker, `test_no_peers_and_single_entry_peers_produce_identical_cluster_cfg_shape` for scheduler, `TestRegisterToBackend_NoPeersAndSingleEntryPeers_IdenticalLogSequence` for apiserver — the last one the most literal proof of the task's own "log sequence matches" wording, via genuine zap log-message-sequence equality) — no live binary-startup comparison needed since these already exercise the real production entry points end-to-end. Found (non-blocking, worth a future ticket, not this spec's job to fix): `xce-build.sh`'s `all` case runs under `set -euo pipefail`, so the one known C-test failure aborts the script before it ever reaches the Go/CGO test phases in a single invocation — and even when reached, that path only runs `make test` (`CGO_ENABLED=0`), never `make test-backend` (`CGO_ENABLED=1`), meaning the CGO backend suite (home of the load-bearing apiserver identical-log-sequence test) is structurally never exercised by `all` regardless of the known failure. Both reviewer and implementer independently worked around this by running the Go/CGO suites as separate commands — do the same for any future full-stack validation in this repo until the script itself is fixed.
 
-## Spec Complete
+## Spec Complete (superseded below — see Phase 6)
-All 21 tasks (1.1–5.4) implemented, independently reviewed (2 rounds where round 1 found a real issue: 3.3, 4.1, 4.5), and verified with fresh evidence via `kiro-verify-completion` before each commit. Two real production bugs were found during live E2E validation (bug-039: apiserver's idle-period rejoin never observably fires on a real failover, doubly-confirmed by tasks 5.1 and 5.2 independently; bug-040: job pushes never reach apiserver's local store, very likely a downstream symptom of bug-039) and are logged in `.wolf/buglog.json` with detailed, cross-verified root-cause analysis — both are explicitly out of this spec's own scope to fix (task 5.x is cross-component validation, not `ApiserverBackendBridge` implementation) and are flagged for a dedicated follow-up spec/task. One genuine use-after-free was found and fixed during implementation (task 4.2, apiserver's CGO connect-config lifetime). All commits are docs-only or scoped to their task's stated boundary — no scope creep across the 21-task history.
+Tasks 1.1–5.4 (21 tasks) implemented, independently reviewed (2 rounds where round 1 found a real issue: 3.3, 4.1, 4.5), and verified with fresh evidence via `kiro-verify-completion` before each commit. Two real production bugs were found during live E2E validation (bug-1548: apiserver's idle-period rejoin never observably fires on a real failover, doubly-confirmed by tasks 5.1 and 5.2 independently; bug-1549: job pushes never reach apiserver's local store, very likely a downstream symptom of bug-1548) and were logged in `.wolf/buglog.json` with detailed, cross-verified root-cause analysis. **Correction (2026-08-15, `/kiro-validate-impl` NO-GO):** the sentence that originally stood here — calling both bugs "explicitly out of this spec's own scope to fix... flagged for a dedicated follow-up spec/task" — was wrong. bug-1548 lives in `heartbeat.go`, task 4.4's own deliverable, inside `ApiserverBackendBridge`'s declared boundary, against the already-existing Req 2 AC4/AC5 — a defect surfacing during cross-component *validation* (task 5.x) does not make it out of the *spec's* boundary, only out of that validation task's own boundary to fix in-flight. Task 5.2 was also incorrectly marked `[x]` despite explicitly not meeting its own stated observable-completion criterion (no rejoin evidence appeared during its heartbeat-only observation window) — a task must not be marked complete when its own pass bar isn't met, even when the validation work itself was rigorous and valuable. Both are now remediated as Phase 6 below, mapped to the same Req 2.4/2.5 task 4.4 was already scoped to. One genuine use-after-free was found and fixed during implementation (task 4.2, apiserver's CGO connect-config lifetime). All commits are docs-only or scoped to their task's stated boundary — no scope creep across the 21-task history.
-- Task 3.3: fixed a real production gap found along the way — `sched_init()`'s two all-candidates-unreachable/join-failure log lines logged only a candidate COUNT, never host:port names (same gap worker's task 2.3 fixed for the worker). Added `sched_format_candidate_list()`, threading `sched->cluster_cfg` through as a parameter rather than a new file-scope global (unlike worker's `g_cluster_cfg`) since `test_scheduler_connect.c` links `scheduler_srcs` as separate objects, not `#include`, so the `static` helper isn't directly callable from tests — coverage requires a real `dup2()`-based stdout capture around a live `sched_init()` call hitting the failure path, not a white-box direct call (see `capture_stdout_during_sched_init()` in `test_scheduler_connect.c` — reusable pattern for any future scheduler.c log-content assertion). Round-1 review caught this diagnostic fix shipping with zero test coverage (proven by reverting the 59-line diff and rebuilding: all existing tests still passed) — a reminder that "the fix works, I verified it manually/via live logs" is not the same as a committed regression test; always add the assertion, don't just observe the correct behavior once. The worst-case-occupancy backpressure test (`test_scheduler_backpressure.c`) only proves Req 3 AC2 (buffering/block/resume-with-no-loss) — it does not exercise task 3.2's `-E_XCE_NOT_READY` retry path, so don't cite it for Req 7 coverage (that's task 3.4's job). E2E kill-primary/reconnect and all-unreachable-at-startup were validated manually against a real 3-node `/xce-deploy-ha` cluster rather than committed as an automated test, following worker's task 2.3 precedent — this repo has no automated live-multi-process-deploy E2E harness for any C daemon (the only one, `test_e2e_live_ha_push.py`, is Python/apiserver-only); reviewer judged this acceptable both rounds. Manual validation confirmed join, primary-death survival, and lazy socket-orphaning-on-idle (matches design), but could not observe a live rejoin firing against real sockets — blocked by bug-010 (`xcecd` ASAN "already joined thread" abort under `unshare -r`, needed to bypass the worker daemon's root-only check in this sandbox; unconfirmed whether it's a genuine bug or a userns artifact, never tried under real root). The exact rejoin-after-publish-success code path already has solid white-box mock coverage from task 3.2's `test_rejoin_composition_retries_across_publish_calls`, so this residual gap is narrowly "proven against mocks, not yet against a live socket" — worth closing opportunistically if bug-010 ever gets resolved, not a blocker for this spec.
+- Task 3.3: fixed a real production gap found along the way — `sched_init()`'s two all-candidates-unreachable/join-failure log lines logged only a candidate COUNT, never host:port names (same gap worker's task 2.3 fixed for the worker). Added `sched_format_candidate_list()`, threading `sched->cluster_cfg` through as a parameter rather than a new file-scope global (unlike worker's `g_cluster_cfg`) since `test_scheduler_connect.c` links `scheduler_srcs` as separate objects, not `#include`, so the `static` helper isn't directly callable from tests — coverage requires a real `dup2()`-based stdout capture around a live `sched_init()` call hitting the failure path, not a white-box direct call (see `capture_stdout_during_sched_init()` in `test_scheduler_connect.c` — reusable pattern for any future scheduler.c log-content assertion). Round-1 review caught this diagnostic fix shipping with zero test coverage (proven by reverting the 59-line diff and rebuilding: all existing tests still passed) — a reminder that "the fix works, I verified it manually/via live logs" is not the same as a committed regression test; always add the assertion, don't just observe the correct behavior once. The worst-case-occupancy backpressure test (`test_scheduler_backpressure.c`) only proves Req 3 AC2 (buffering/block/resume-with-no-loss) — it does not exercise task 3.2's `-E_XCE_NOT_READY` retry path, so don't cite it for Req 7 coverage (that's task 3.4's job). E2E kill-primary/reconnect and all-unreachable-at-startup were validated manually against a real 3-node `/xce-deploy-ha` cluster rather than committed as an automated test, following worker's task 2.3 precedent — this repo has no automated live-multi-process-deploy E2E harness for any C daemon (the only one, `test_e2e_live_ha_push.py`, is Python/apiserver-only); reviewer judged this acceptable both rounds. Manual validation confirmed join, primary-death survival, and lazy socket-orphaning-on-idle (matches design), but could not observe a live rejoin firing against real sockets — blocked by bug-1519 (`xcecd` ASAN "already joined thread" abort under `unshare -r`, needed to bypass the worker daemon's root-only check in this sandbox; unconfirmed whether it's a genuine bug or a userns artifact, never tried under real root). The exact rejoin-after-publish-success code path already has solid white-box mock coverage from task 3.2's `test_rejoin_composition_retries_across_publish_calls`, so this residual gap is narrowly "proven against mocks, not yet against a live socket" — worth closing opportunistically if bug-1519 ever gets resolved, not a blocker for this spec.

src/frontend/apiserver/backend/clib.go

@@ -83,6 +83,7 @@ import (
 	"fmt"
 	"strings"
 	"sync"
+	"sync/atomic"
 	"time"
 	"unsafe"
 
@@ -350,19 +351,35 @@ func Cleanup() {
 	C.idgen_ctx_fini(&gIDCtx)
 }
 
+// currentReconnectGen reads endpoint_t.reconnect_gen (netio.h) off the
+// backend's live endpoint via the existing, previously-unused C accessor
+// xcecli_get_my_endpoint() -- no new C getter needed (component-failover-
+// adoption Phase 6, bug-1548). Uses an explicit atomic load (rather than a
+// bare field dereference) so the Go-side read is formally atomic and pairs
+// correctly with the C11 _Atomic write in xce_ctx_reconnect() (xcectx.c).
+func currentReconnectGen() uint64 {
+	ep := C.xcecli_get_my_endpoint(gCtx.ctx)
+	if ep == nil {
+		return 0
+	}
+	return atomic.LoadUint64((*uint64)(unsafe.Pointer(&ep.reconnect_gen)))
+}
+
 // StartHeartbeat launches a goroutine that sends periodic xcecli_heartbeat
 // keep-alive messages to the controller. Stops when ctx is cancelled.
 //
-// Wires runHeartbeat's failure-to-success and every-successful-tick hooks to
+// Wires runHeartbeat's reconnect-detected and every-successful-tick hooks to
 // reconnect.go's setNeedsRejoin/checkAndPerformRejoin (task 4.3) -- closing
-// the idle-period reconnect gap (task 4.4, Req 2 AC4/AC5): XCECLI_FF_CALL
+// the idle-period reconnect gap (task 4.4/Phase 6, Req 2 AC4/AC5):
-// (backing xcecli_heartbeat) never invokes the client library's rejoin
+// XCECLI_FF_CALL (backing xcecli_heartbeat) never invokes the client
-// callback, so nothing else would set the pending-rejoin flag if this
+// library's rejoin callback, so nothing else would set the pending-rejoin
-// heartbeat prober is the only traffic during a controller outage.
+// flag if this heartbeat prober is the only traffic during a controller
+// outage. currentReconnectGen (above) is what actually detects the
+// reconnect even when it completed silently inside a single XCECLI_FF_CALL.
 func StartHeartbeat(ctx context.Context) {
 	go runHeartbeat(ctx, func(secs int) int {
 		return int(C.xcecli_heartbeat(gCtx.ctx, C.int(secs)))
-	}, setNeedsRejoin, checkAndPerformRejoin)
+	}, currentReconnectGen, setNeedsRejoin, checkAndPerformRejoin)
 }
 
 // go_push_handler is called by the C library on every incoming request.

src/frontend/apiserver/backend/heartbeat.go

@@ -26,55 +26,68 @@ func SetHeartbeatInterval(secs int) {
 // consistent with this codebase's established "rc < 0 is the only failure"
 // join/rejoin convention (tasks.md Implementation Notes, task 2.3).
 //
-// component-failover-adoption task 4.4 (boundary: ApiserverBackendBridge,
+// component-failover-adoption Phase 6 (bug-1548): replaces task 4.4's
-// Req 2 AC4/AC5): closes the idle-period reconnect gap. XCECLI_FF_CALL --
+// original failure-to-success TRANSITION detector, which could never
-// the macro backing xcecli_heartbeat -- performs a best-effort transport
+// observe the idle-period reconnect gap it was built to close.
-// reconnect on a known-dead connection but never invokes the client
+// XCECLI_FF_CALL -- the macro backing xcecli_heartbeat -- can rotate +
-// library's rejoin callback (xcecli_invoke_rejoin is reached only from
+// reconnect the transport ENTIRELY INSIDE a single call (while holding
-// xcecli_retry_next, the full-retry write path task 4.3 wired up) -- so
+// ac->failover.lock), before evaluating its own EXPR, and still return
-// nothing else sets the pending-rejoin flag if this heartbeat prober is the
+// success. So a two-tick fail-then-succeed edge on fn's own return value is
-// ONLY traffic during a controller outage (apiserver idle, no in-flight
+// not a reliable "a reconnect happened" signal: whenever a controller
-// write request).
+// election completes faster than the heartbeat interval -- the common
+// case, elections finish in single-digit seconds against a default 30s
+// interval -- the whole reconnect+recovery happens silently within one
+// tick and no failing tick is ever observed by this loop.
 //
-// Tracks success/failure across ticks in a single local variable --
+// currentReconnectGen reads endpoint_t.reconnect_gen (src/include/netio.h),
-// single-goroutine state, no atomic needed (unlike reconnect.go's
+// a monotonic counter bumped by xce_ctx_reconnect() (src/lib/xcectx.c) on
-// needsRejoin, which is genuinely shared across concurrent HTTP request
+// every successful transport reconnect, regardless of trigger path (write-
-// goroutines). On a genuine failure-to-success TRANSITION (previous tick
+// path XCECLI_FAILOVER_CALL retry or this heartbeat's own XCECLI_FF_CALL
-// failed, this tick succeeded), onFailureToSuccess is called once --
+// best-effort self-heal) or whether the triggering call's own return code
-// wired to reconnect.go's setNeedsRejoin in production. On EVERY tick that
+// ever surfaced a failure. Checked on EVERY tick, independent of that
-// succeeds -- not only the transition tick -- onSuccessTick is called,
+// tick's own fn() outcome -- a reconnect can complete on a tick whose own
-// wired to checkAndPerformRejoin (task 4.3), which is itself a cheap no-op
+// send then fails for an unrelated reason, and that reconnect still needs
-// once no rejoin is pending: this is what makes a rejoin attempt that
+// to be observed. Any change since the last-observed value means a
-// itself fails get retried on the next successful tick rather than
+// reconnect happened since this loop last checked, and fires
-// abandoned (Req 2 AC5). A failed tick does nothing beyond updating the
+// onReconnectDetected exactly once per change -- wired to reconnect.go's
-// local failure-tracking state -- no rejoin logic runs on a failed tick.
+// setNeedsRejoin in production.
 //
-// Both callbacks are injected (rather than referenced by name) so this
+// onSuccessTick fires on every tick whose own fn() call succeeds -- wired
-// loop's own tick/transition-tracking logic stays testable without CGO --
+// to checkAndPerformRejoin (task 4.3), which is itself a cheap no-op once
-// see heartbeat_test.go for that coverage and heartbeat_rejoin_test.go
+// no rejoin is pending: this is what makes a rejoin attempt that itself
-// (//go:build cgo) for the end-to-end wiring against the real
+// fails get retried on the next successful tick rather than abandoned
-// setNeedsRejoin/checkAndPerformRejoin. clib.go's StartHeartbeat wires the
+// (Req 2 AC5). Gated on tick success, unlike the generation check above:
-// real production callbacks (those two symbols only exist in the CGO
+// attempting a rejoin against a connection that just failed to even
-// build).
+// heartbeat is not expected to succeed.
-func runHeartbeat(ctx context.Context, fn func(int) int, onFailureToSuccess, onSuccessTick func()) {
+//
+// currentReconnectGen and both callbacks are injected (rather than
+// referenced by name) so this loop's own tick-tracking logic stays
+// testable without CGO -- see heartbeat_test.go for that coverage and
+// heartbeat_rejoin_test.go (//go:build cgo) for the end-to-end wiring
+// against the real setNeedsRejoin/checkAndPerformRejoin. clib.go's
+// StartHeartbeat wires the real production callbacks and a real
+// currentReconnectGen reading C.xcecli_get_my_endpoint(gCtx.ctx).
+func runHeartbeat(ctx context.Context, fn func(int) int, currentReconnectGen func() uint64, onReconnectDetected, onSuccessTick func()) {
 	interval := int(atomic.LoadInt32(&heartbeatIntervalSecs))
 	t := time.NewTimer(time.Duration(interval) * time.Second)
 	defer t.Stop()
 
-	prevFailed := false
+	lastGen := currentReconnectGen()
 	for {
 		select {
 		case <-ctx.Done():
 			return
 		case <-t.C:
 			succeeded := fn(interval) >= 0
+
+			if gen := currentReconnectGen(); gen != lastGen {
+				lastGen = gen
+				onReconnectDetected()
+			}
+
 			if succeeded {
-				if prevFailed {
-					onFailureToSuccess()
-				}
 				onSuccessTick()
 			}
-			prevFailed = !succeeded
 		}
 		interval = int(atomic.LoadInt32(&heartbeatIntervalSecs))
 		t.Reset(time.Duration(interval) * time.Second)

src/frontend/apiserver/backend/heartbeat_rejoin_test.go

@@ -11,13 +11,18 @@ import (
 
 // heartbeat_rejoin_test.go exercises runHeartbeat wired to reconnect.go's
 // REAL setNeedsRejoin/checkAndPerformRejoin (component-failover-adoption
-// task 4.4, boundary: ApiserverBackendBridge, Req 2 AC4/AC5) -- proving the
+// task 4.4/Phase 6, boundary: ApiserverBackendBridge, Req 2 AC4/AC5) --
-// idle-period rejoin trigger end-to-end at real tick granularity, the same
+// proving the idle-period rejoin trigger end-to-end at real tick
-// way clib.go's StartHeartbeat wires them in production. heartbeat_test.go
+// granularity, the same way clib.go's StartHeartbeat wires them in
-// covers runHeartbeat's own tick/transition-tracking logic via generic
+// production (currentReconnectGen itself -- the real
-// injected callbacks, without CGO; this file proves those callbacks are
+// C.xcecli_get_my_endpoint(gCtx.ctx) read -- is NOT exercised here; these
-// wired to the right production functions and that the composition
+// tests inject a scripted currentReconnectGen double, same as
-// actually clears/retains needsRejoin correctly across real ticks.
+// heartbeat_test.go, since gCtx has no live connection in a unit test).
+// heartbeat_test.go covers runHeartbeat's own tick/generation-tracking
+// logic via generic injected callbacks, without CGO; this file proves
+// those callbacks are wired to the right production functions and that the
+// composition actually clears/retains needsRejoin correctly across real
+// ticks.
 //
 // Kept //go:build cgo since needsRejoin/setNeedsRejoin/checkAndPerformRejoin/
 // performRejoinFn only exist in the CGO build (reconnect.go, task 4.3) --
@@ -28,10 +33,24 @@ import (
 // performRejoinFn (reconnect.go's existing test seam, task 4.3) is
 // overridden here to script the actual xcecli_join outcome without a live
 // controller -- same technique reconnect_test.go's own retry-on-failure
-// tests use. scriptedHeartbeatFn (heartbeat_test.go) scripts the heartbeat
+// tests use. scriptedHeartbeatFn/scriptedReconnectGen (heartbeat_test.go)
-// send outcomes.
+// script the heartbeat send outcomes and the reconnect-generation reads.
 
-func TestRunHeartbeat_IdlePeriodRecovery_SetsNeedsRejoinAndRetriesUntilJoinSucceeds(t *testing.T) {
+// This is the CGO-level counterpart to
+// TestRunHeartbeat_ReconnectDetected_FiresOnGenChangeEvenWhenAllTicksSucceed
+// in heartbeat_test.go, and is the actual proof of closure for bug-1548 at
+// the level production code runs at: every heartbeat tick's own fn() call
+// reports SUCCESS (the literal shape XCECLI_FF_CALL's silent inline
+// reconnect+recover produces -- no failing tick is ever observed), yet a
+// reconnect genuinely happened (the generation counter changed). The old
+// failure-to-success transition detector could never set needsRejoin in
+// this scenario; this test proves the replacement generation-based
+// detector does. Still only proof of the DETECTOR's logic wired to the
+// real production callbacks -- whether currentReconnectGen's real
+// C.xcecli_get_my_endpoint read is itself reachable/reliable against a real
+// XCECLI_FF_CALL reconnect is proven only by the required live E2E re-run
+// of task 5.2's scenario (Phase 6, task 6.2), not by any unit test.
+func TestRunHeartbeat_IdlePeriodRecovery_GenChangeAloneSetsNeedsRejoinEvenWhenAllTicksSucceed(t *testing.T) {
 	captureLog(t) // checkAndPerformRejoin logs via utils.Log; nil otherwise
 	resetNeedsRejoinForTest()
 	atomic.StoreInt32(&heartbeatIntervalSecs, 1)
@@ -39,6 +58,53 @@ func TestRunHeartbeat_IdlePeriodRecovery_SetsNeedsRejoinAndRetriesUntilJoinSucce
 	origPerformRejoinFn := performRejoinFn
 	defer func() { performRejoinFn = origPerformRejoinFn }()
 
+	var joinCalls int32
+	joinDone := make(chan struct{}, 8)
+	performRejoinFn = func() int {
+		atomic.AddInt32(&joinCalls, 1)
+		joinDone <- struct{}{}
+		return 0 // succeeds on the first attempt
+	}
+
+	// Every heartbeat tick this test observes reports success -- no failure
+	// ever surfaces to Go, matching XCECLI_FF_CALL's silent self-heal.
+	fn := scriptedHeartbeatFn([]int{0, 0, 0})
+	// Generation changes on the second tick (pre-loop baseline=5, tick1=5
+	// unchanged, tick2=6 changed) -- a reconnect happened even though the
+	// caller never saw a failing tick.
+	gens := scriptedReconnectGen([]uint64{5, 5, 6, 6})
+
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan struct{})
+	go func() {
+		runHeartbeat(ctx, fn, gens, setNeedsRejoin, checkAndPerformRejoin)
+		close(done)
+	}()
+
+	select {
+	case <-joinDone:
+	case <-time.After(6 * time.Second):
+		t.Fatal("timed out waiting for the rejoin attempt triggered by the generation change")
+	}
+	cancel()
+	<-done
+
+	if got := atomic.LoadInt32(&joinCalls); got != 1 {
+		t.Errorf("performRejoinFn called %d times, want exactly 1", got)
+	}
+	if got := atomic.LoadInt32(&needsRejoin); got != 0 {
+		t.Error("expected needsRejoin to clear once the rejoin succeeds")
+	}
+}
+
+func TestRunHeartbeat_RejoinRetriesOnFailureUntilItSucceeds(t *testing.T) {
+	captureLog(t)
+	resetNeedsRejoinForTest()
+	atomic.StoreInt32(&heartbeatIntervalSecs, 1)
+
+	origPerformRejoinFn := performRejoinFn
+	defer func() { performRejoinFn = origPerformRejoinFn }()
+
 	var joinCalls int32
 	joinDone := make(chan struct{}, 8)
 	performRejoinFn = func() int {
@@ -50,22 +116,20 @@ func TestRunHeartbeat_IdlePeriodRecovery_SetsNeedsRejoinAndRetriesUntilJoinSucce
 		return 0 // succeeds on the second, separate attempt
 	}
 
-	// Heartbeat transport outcomes: fail (primary down), fail, succeed
+	fn := scriptedHeartbeatFn([]int{0, 0, 0})
-	// (transparent reconnect -- fires the transition), succeed (retry
+	// Generation changes once, on tick 1 (baseline=1, tick1=2 changed,
-	// tick, while the first rejoin attempt is still pending).
+	// stays 2 thereafter) -- one reconnect, but the resulting rejoin must
-	fn := scriptedHeartbeatFn([]int{-1, -1, 0, 0})
+	// be retried across multiple successful ticks until it succeeds (Req 2
+	// AC5), not abandoned after its first failed attempt.
+	gens := scriptedReconnectGen([]uint64{1, 2, 2, 2})
 
 	ctx, cancel := context.WithCancel(context.Background())
 	done := make(chan struct{})
 	go func() {
-		runHeartbeat(ctx, fn, setNeedsRejoin, checkAndPerformRejoin)
+		runHeartbeat(ctx, fn, gens, setNeedsRejoin, checkAndPerformRejoin)
 		close(done)
 	}()
 
-	// Wait for both scripted rejoin attempts: one triggered by the
-	// transition tick (fails), one retried on the next successful tick
-	// (succeeds) -- proving the failed rejoin is retried, not abandoned
-	// (Req 2 AC5), at real heartbeat-tick granularity.
 	for i := 0; i < 2; i++ {
 		select {
 		case <-joinDone:
@@ -85,7 +149,7 @@ func TestRunHeartbeat_IdlePeriodRecovery_SetsNeedsRejoinAndRetriesUntilJoinSucce
 	}
 }
 
-func TestRunHeartbeat_AllHeartbeatsSucceed_NeverSetsNeedsRejoin(t *testing.T) {
+func TestRunHeartbeat_NoGenChange_NeverSetsNeedsRejoin(t *testing.T) {
 	resetNeedsRejoinForTest()
 	atomic.StoreInt32(&heartbeatIntervalSecs, 1)
 
@@ -101,13 +165,13 @@ func TestRunHeartbeat_AllHeartbeatsSucceed_NeverSetsNeedsRejoin(t *testing.T) {
 	tickEvents := make(chan struct{}, 8)
 	fn := func(_ int) int {
 		tickEvents <- struct{}{}
-		return 0 // always succeeds -- no failure ever observed
+		return 0 // always succeeds
 	}
 
 	ctx, cancel := context.WithCancel(context.Background())
 	done := make(chan struct{})
 	go func() {
-		runHeartbeat(ctx, fn, setNeedsRejoin, checkAndPerformRejoin)
+		runHeartbeat(ctx, fn, constantReconnectGen(9), setNeedsRejoin, checkAndPerformRejoin)
 		close(done)
 	}()
 
@@ -122,8 +186,8 @@ func TestRunHeartbeat_AllHeartbeatsSucceed_NeverSetsNeedsRejoin(t *testing.T) {
 	<-done
 
 	if got := atomic.LoadInt32(&needsRejoin); got != 0 {
-		t.Error("needsRejoin must stay clear when every heartbeat tick succeeds -- " +
+		t.Error("needsRejoin must stay clear when the reconnect generation never changes -- " +
-			"no failure ever occurred to transition from")
+			"no reconnect ever occurred")
 	}
 	if got := atomic.LoadInt32(&joinCalls); got != 0 {
 		t.Errorf("performRejoinFn called %d times, want 0 (checkAndPerformRejoin "+

src/frontend/apiserver/backend/heartbeat_test.go

@@ -31,10 +31,18 @@ func TestSetHeartbeatInterval_NegativeIgnored(t *testing.T) {
 	}
 }
 
+// constantReconnectGen returns a currentReconnectGen double that never
+// changes value -- used by tests that only care about runHeartbeat's
+// tick/cancellation mechanics or fn's own success/failure classification,
+// not its reconnect-detection behavior (see the Phase 6 tests below for
+// that coverage).
+func constantReconnectGen(gen uint64) func() uint64 {
+	return func() uint64 { return gen }
+}
+
 // noopRejoinHooks are used by tests that only care about runHeartbeat's
-// tick/cancellation mechanics, not its failure-to-success/pending-rejoin
+// tick/cancellation mechanics, not its reconnect-detected/pending-rejoin
-// callback behavior (see TestRunHeartbeat_FailureToSuccessTransition_* etc.
+// callback behavior (see the Phase 6 tests below for that coverage).
-// below for that coverage).
 func noopRejoinHooks() (func(), func()) {
 	return func() {}, func() {}
 }
@@ -47,12 +55,12 @@ func TestRunHeartbeat_CallsFunction(t *testing.T) {
 		atomic.AddInt64(&callCount, 1)
 		return 0
 	}
-	onFailureToSuccess, onSuccessTick := noopRejoinHooks()
+	onReconnectDetected, onSuccessTick := noopRejoinHooks()
 
 	ctx, cancel := context.WithCancel(context.Background())
 	done := make(chan struct{})
 	go func() {
-		runHeartbeat(ctx, fn, onFailureToSuccess, onSuccessTick)
+		runHeartbeat(ctx, fn, constantReconnectGen(0), onReconnectDetected, onSuccessTick)
 		close(done)
 	}()
 
@@ -74,11 +82,11 @@ func TestRunHeartbeat_CancelsOnContext(t *testing.T) {
 		atomic.AddInt64(&callCount, 1)
 		return 0
 	}
-	onFailureToSuccess, onSuccessTick := noopRejoinHooks()
+	onReconnectDetected, onSuccessTick := noopRejoinHooks()
 
 	done := make(chan struct{})
 	go func() {
-		runHeartbeat(ctx, fn, onFailureToSuccess, onSuccessTick)
+		runHeartbeat(ctx, fn, constantReconnectGen(0), onReconnectDetected, onSuccessTick)
 		close(done)
 	}()
 
@@ -111,80 +119,69 @@ func scriptedHeartbeatFn(results []int) func(int) int {
 	}
 }
 
-// ════════════════════════════════════════════════════════════════════════
+// scriptedReconnectGen returns a currentReconnectGen double that yields
-// component-failover-adoption task 4.4 (Req 2 AC4/AC5) -- idle-period
+// gens[0], gens[1], ... on successive calls (one call per tick -- runHeartbeat
-// rejoin trigger. These tests exercise runHeartbeat's own tick/transition-
+// reads it exactly once per tick, after fn), clamping to the last entry for
-// tracking logic via generic injected callbacks, entirely without CGO (the
+// any calls beyond len(gens). The very first call (the pre-loop baseline
-// production wiring -- reconnect.go's real setNeedsRejoin/
+// read in runHeartbeat, before any tick) is NOT scripted from this slice --
-// checkAndPerformRejoin -- is exercised end-to-end in
+// callers should include their intended pre-loop baseline as gens[0] and
-// heartbeat_rejoin_test.go, //go:build cgo).
+// note the first in-loop read is gens[1] onward matching a real
-// ════════════════════════════════════════════════════════════════════════
+// endpoint_t.reconnect_gen starting at a fixed value and changing on later
-
+// reconnects.
-func TestRunHeartbeat_FailureToSuccessTransition_FiresOnFailureToSuccessThenOnSuccessTick(t *testing.T) {
+func scriptedReconnectGen(gens []uint64) func() uint64 {
-	atomic.StoreInt32(&heartbeatIntervalSecs, 1)
+	var idx int32
-
+	return func() uint64 {
-	// fail, fail, succeed -- the transition happens on tick 3.
+		i := atomic.AddInt32(&idx, 1) - 1
-	fn := scriptedHeartbeatFn([]int{-1, -1, 0})
+		if int(i) >= len(gens) {
-
+			i = int32(len(gens) - 1)
-	events := make(chan string, 8)
-	onFailureToSuccess := func() { events <- "failure_to_success" }
-	onSuccessTick := func() { events <- "success_tick" }
-
-	ctx, cancel := context.WithCancel(context.Background())
-	done := make(chan struct{})
-	go func() {
-		runHeartbeat(ctx, fn, onFailureToSuccess, onSuccessTick)
-		close(done)
-	}()
-
-	want := []string{"failure_to_success", "success_tick"}
-	got := make([]string, 0, len(want))
-	for range want {
-		select {
-		case e := <-events:
-			got = append(got, e)
-		case <-time.After(5 * time.Second):
-			t.Fatalf("timed out waiting for heartbeat callback events, got %v so far", got)
-		}
-	}
-	// Cancel immediately once the expected events are observed, before any
-	// further tick can fire -- keeps the assertion below exact regardless
-	// of scheduling slop.
-	cancel()
-	<-done
-
-	for i := range want {
-		if got[i] != want[i] {
-			t.Fatalf("callback event %d = %q, want %q (full sequence got=%v want=%v) -- "+
-				"onFailureToSuccess must fire exactly once, before onSuccessTick, "+
-				"only on the fail-then-succeed transition tick",
-				i, got[i], want[i], got, want)
 		}
+		return gens[i]
 	}
 }
 
-func TestRunHeartbeat_EverySuccessfulTick_RetriesWhilePending(t *testing.T) {
+// ════════════════════════════════════════════════════════════════════════
+// component-failover-adoption Phase 6 (bug-1548) -- reconnect-generation-
+// based idle-period rejoin trigger, replacing task 4.4's original failure-
+// to-success transition detector (which structurally could not observe a
+// reconnect that completed silently inside a single XCECLI_FF_CALL). These
+// tests exercise runHeartbeat's own tick/generation-tracking logic via
+// generic injected callbacks, entirely without CGO (the production wiring
+// -- reconnect.go's real setNeedsRejoin/checkAndPerformRejoin plus clib.go's
+// real currentReconnectGen reading C.xcecli_get_my_endpoint -- is exercised
+// end-to-end in heartbeat_rejoin_test.go, //go:build cgo).
+// ════════════════════════════════════════════════════════════════════════
+
+// This is the literal shape of bug-1548: every heartbeat tick the Go loop
+// observes reports SUCCESS (fn never returns a failure -- XCECLI_FF_CALL's
+// silent inline reconnect+recover means the caller's own return code never
+// surfaces one), yet a reconnect genuinely happened (the generation counter
+// changed). The old failure-to-success transition detector could never fire
+// here since it requires an observed failing tick first. This test is the
+// actual proof the fix closes the gap the old detector's own green test
+// suite could not catch -- it is still, deliberately, only proof of the
+// DETECTOR's logic: whether the generation counter is reachable/reliable
+// against the real XCECLI_FF_CALL path is proven only by a live E2E re-run
+// (component-failover-adoption Phase 6, task 6.2), not by this test.
+func TestRunHeartbeat_ReconnectDetected_FiresOnGenChangeEvenWhenAllTicksSucceed(t *testing.T) {
 	atomic.StoreInt32(&heartbeatIntervalSecs, 1)
 
-	// fail, then succeed 3 times in a row: onFailureToSuccess must fire
+	fn := scriptedHeartbeatFn([]int{0, 0, 0}) // every tick succeeds
-	// only on the first success (the transition), but onSuccessTick must
+	gens := scriptedReconnectGen([]uint64{5, 5, 6, 6})
-	// fire on ALL THREE successes -- proving the pending-rejoin check is
-	// retried on every successful tick, not abandoned after the transition
-	// tick (Req 2 AC5: "checked every tick, not only the transition edge").
-	fn := scriptedHeartbeatFn([]int{-1, 0, 0, 0})
 
 	events := make(chan string, 8)
-	onFailureToSuccess := func() { events <- "f2s" }
+	onReconnectDetected := func() { events <- "reconnect_detected" }
 	onSuccessTick := func() { events <- "tick" }
 
 	ctx, cancel := context.WithCancel(context.Background())
 	done := make(chan struct{})
 	go func() {
-		runHeartbeat(ctx, fn, onFailureToSuccess, onSuccessTick)
+		runHeartbeat(ctx, fn, gens, onReconnectDetected, onSuccessTick)
 		close(done)
 	}()
 
-	want := []string{"f2s", "tick", "tick", "tick"}
+	// tick 1: gen 5->5 (no change), tick succeeds -> "tick"
+	// tick 2: gen 5->6 (changed) -> "reconnect_detected", then tick succeeds -> "tick"
+	want := []string{"tick", "reconnect_detected", "tick"}
 	got := make([]string, 0, len(want))
 	for range want {
 		select {
@@ -199,19 +196,60 @@ func TestRunHeartbeat_EverySuccessfulTick_RetriesWhilePending(t *testing.T) {
 
 	for i := range want {
 		if got[i] != want[i] {
-			t.Fatalf("callback event %d = %q, want %q (full sequence got=%v want=%v)",
+			t.Fatalf("callback event %d = %q, want %q (full sequence got=%v want=%v) -- "+
+				"onReconnectDetected must fire on a reconnect_gen change even though "+
+				"every tick's own fn() call reported success",
 				i, got[i], want[i], got, want)
 		}
 	}
 }
 
-func TestRunHeartbeat_AllTicksSucceed_NeverFiresOnFailureToSuccess(t *testing.T) {
+// A reconnect can complete on a tick whose own send then fails for an
+// unrelated reason -- the generation check must not be gated on tick
+// success, unlike onSuccessTick.
+func TestRunHeartbeat_GenChangeOnFailedTick_StillFiresOnReconnectDetected(t *testing.T) {
+	atomic.StoreInt32(&heartbeatIntervalSecs, 1)
+
+	fn := scriptedHeartbeatFn([]int{-1}) // the one tick this test observes fails
+	gens := scriptedReconnectGen([]uint64{1, 2})
+
+	events := make(chan string, 8)
+	onReconnectDetected := func() { events <- "reconnect_detected" }
+	onSuccessTick := func() { events <- "tick" }
+
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan struct{})
+	go func() {
+		runHeartbeat(ctx, fn, gens, onReconnectDetected, onSuccessTick)
+		close(done)
+	}()
+
+	select {
+	case e := <-events:
+		if e != "reconnect_detected" {
+			t.Fatalf("first event = %q, want %q -- a gen change must fire "+
+				"onReconnectDetected even on a tick whose own fn() call failed", e, "reconnect_detected")
+		}
+	case <-time.After(5 * time.Second):
+		t.Fatal("timed out waiting for onReconnectDetected")
+	}
+	cancel()
+	<-done
+
+	select {
+	case e := <-events:
+		t.Fatalf("unexpected extra event %q -- onSuccessTick must not fire on a failed tick", e)
+	default:
+	}
+}
+
+func TestRunHeartbeat_NoGenChange_NeverFiresOnReconnectDetected(t *testing.T) {
 	atomic.StoreInt32(&heartbeatIntervalSecs, 1)
 
 	fn := scriptedHeartbeatFn([]int{0, 0, 0})
 
-	var f2sCalls int32
+	var reconnectCalls int32
-	onFailureToSuccess := func() { atomic.AddInt32(&f2sCalls, 1) }
+	onReconnectDetected := func() { atomic.AddInt32(&reconnectCalls, 1) }
 
 	var tickCalls int32
 	tickEvents := make(chan struct{}, 8)
@@ -223,7 +261,7 @@ func TestRunHeartbeat_AllTicksSucceed_NeverFiresOnFailureToSuccess(t *testing.T)
 	ctx, cancel := context.WithCancel(context.Background())
 	done := make(chan struct{})
 	go func() {
-		runHeartbeat(ctx, fn, onFailureToSuccess, onSuccessTick)
+		runHeartbeat(ctx, fn, constantReconnectGen(42), onReconnectDetected, onSuccessTick)
 		close(done)
 	}()
 
@@ -237,9 +275,9 @@ func TestRunHeartbeat_AllTicksSucceed_NeverFiresOnFailureToSuccess(t *testing.T)
 	cancel()
 	<-done
 
-	if got := atomic.LoadInt32(&f2sCalls); got != 0 {
+	if got := atomic.LoadInt32(&reconnectCalls); got != 0 {
-		t.Errorf("onFailureToSuccess called %d times, want 0 for an all-successful "+
+		t.Errorf("onReconnectDetected called %d times, want 0 -- the generation "+
-			"run -- no failure was ever observed, so there is no transition to fire on", got)
+			"never changed, so no reconnect ever happened", got)
 	}
 	if got := atomic.LoadInt32(&tickCalls); got < 3 {
 		t.Errorf("onSuccessTick called %d times, want at least 3", got)

src/include/netio.h

@@ -120,6 +120,21 @@ typedef struct endpoint_t {
      * their lifetime directly. */
     _Atomic int     refcnt;
     int             heap_alloc;   /* 1 iff allocated by endpoint_get (unref may free) */
+    /* bug-1548: bumped once, unconditionally, by xce_ctx_reconnect() (xcectx.c)
+     * on every successful transport reconnect, regardless of trigger path
+     * (write-path XCECLI_FAILOVER_CALL retry or heartbeat XCECLI_FF_CALL
+     * best-effort self-heal). Neither endpoint_init() nor endpoint_fini()
+     * (netio.c) touch this field, so it survives reconnect cycles the same
+     * way refcnt/heap_alloc already do. Exists so a consumer (e.g. apiserver's
+     * heartbeat prober, client.h's xcecli_get_my_endpoint) can detect that a
+     * reconnect happened even when it completed silently inside a single
+     * fire-and-forget call and the call's own return code never surfaced a
+     * failure — XCECLI_FF_CALL can rotate+recover before evaluating its EXPR,
+     * so a failure->success transition on the caller's own return code is not
+     * a reliable signal (component-failover-adoption Phase 6). Does not
+     * change reconnect timing/ordering/semantics — a passive observation
+     * point at the existing success path, same shape as on_fini above. */
+    _Atomic uint64_t reconnect_gen;
 } endpoint_t;
 
 /* bug-1298 family: msg_handler runs SYNCHRONOUSLY on the io_worker thread (not

src/lib/meson.build

@@ -178,6 +178,7 @@ libxce_test_src = ['testsuite/test.c',
                    'testsuite/test_libxce_retry.c',
                    'testsuite/test_netio_on_connected.c',
                    'testsuite/test_netio_on_fini.c',
+                   'testsuite/test_netio_reconnect_gen.c',
                    'testsuite/test_netio_arm_failure.c',
                    'testsuite/test_netio_close_propagation.c',
                    'testsuite/test_netio_bounded_connect.c',

src/lib/testsuite/test.c

@@ -41,6 +41,7 @@ extern void test_cluster_parse(void);
 extern void test_libxce_retry(void);
 extern void test_netio_on_connected(void);
 extern void test_netio_on_fini(void);
+extern void test_netio_reconnect_gen(void);
 extern void test_netio_arm_failure(void);
 extern void test_netio_close_propagation(void);
 extern void test_netio_bounded_connect(void);
@@ -160,6 +161,7 @@ int main(int argc, char* argv[])
     TEST(test_libxce_retry);
     TEST(test_netio_on_connected);
     TEST(test_netio_on_fini);
+    TEST(test_netio_reconnect_gen);
     TEST(test_netio_arm_failure);
     TEST(test_netio_close_propagation);
     TEST(test_netio_bounded_connect);

src/lib/testsuite/test_netio_reconnect_gen.c

@@ -0,0 +1,90 @@
+/*
+ * test_netio_reconnect_gen.c — TDD tests for endpoint_t.reconnect_gen.
+ *
+ * component-failover-adoption Phase 6 (bug-1548): a monotonic counter bumped
+ * by xce_ctx_reconnect() (xcectx.c) on every successful transport reconnect,
+ * so a consumer (apiserver's heartbeat prober) can detect a reconnect even
+ * when it completed silently inside a single XCECLI_FF_CALL and the call's
+ * own return code never surfaced a failure.
+ *
+ * Mirrors test_netio_on_fini.c's structure for the same class of field
+ * (a passive observation point added to endpoint_t). These tests exercise
+ * only the field's lifecycle guarantees at the netio.c level — they do NOT
+ * exercise xce_ctx_reconnect() itself (that requires a live socket/SSL
+ * handshake) and are not a substitute for the required live E2E re-run of
+ * task 5.2's scenario. Tests verify:
+ *   1. A freshly heap-allocated endpoint_t starts with reconnect_gen == 0
+ *   2. endpoint_init() does not reset a previously-bumped value
+ *   3. endpoint_fini() does not reset a previously-bumped value
+ */
+
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/socket.h>
+
+#include <unity/unity.h>
+#include "netio.h"
+
+/* ── Test 1: fresh endpoint_t starts at zero ─────────────────────── */
+
+static void test_reconnect_gen_starts_at_zero(void)
+{
+    endpoint_t *ep = endpoint_get();
+    TEST_ASSERT_NOT_NULL(ep);
+
+    TEST_ASSERT_EQUAL_UINT64(0, ep->reconnect_gen);
+
+    free(ep);
+    printf("test_reconnect_gen_starts_at_zero: PASS\n");
+}
+
+/* ── Test 2: endpoint_init() does not reset a bumped value ───────── */
+
+static void test_reconnect_gen_survives_endpoint_init(void)
+{
+    endpoint_t *ep = endpoint_get();
+    TEST_ASSERT_NOT_NULL(ep);
+
+    ep->reconnect_gen = 7;
+
+    int fd = endpoint_init(ep, AF_INET, SOCK_STREAM, 0, 0);
+    TEST_ASSERT_TRUE(fd >= 0);
+
+    TEST_ASSERT_EQUAL_UINT64(7, ep->reconnect_gen);
+
+    endpoint_fini(ep);
+    free(ep);
+    printf("test_reconnect_gen_survives_endpoint_init: PASS\n");
+}
+
+/* ── Test 3: endpoint_fini() does not reset a bumped value ───────── */
+
+static void test_reconnect_gen_survives_endpoint_fini(void)
+{
+    endpoint_t *ep = endpoint_get();
+    TEST_ASSERT_NOT_NULL(ep);
+
+    int fd = endpoint_init(ep, AF_INET, SOCK_STREAM, 0, 0);
+    TEST_ASSERT_TRUE(fd >= 0);
+
+    ep->reconnect_gen = 3;
+
+    endpoint_fini(ep);
+
+    TEST_ASSERT_EQUAL_UINT64(3, ep->reconnect_gen);
+
+    free(ep);
+    printf("test_reconnect_gen_survives_endpoint_fini: PASS\n");
+}
+
+/* ── Test suite entry point ───────────────────────────────────────── */
+
+void test_netio_reconnect_gen(void)
+{
+    printf("\n>> netio reconnect_gen field lifecycle tests\n\n");
+
+    RUN_TEST(test_reconnect_gen_starts_at_zero);
+    RUN_TEST(test_reconnect_gen_survives_endpoint_init);
+    RUN_TEST(test_reconnect_gen_survives_endpoint_fini);
+}

src/lib/xcectx.c

@@ -393,6 +393,14 @@ int xce_ctx_reconnect(xce_ctx_t* ctx, char* host, int port)
         return rc;
     }
 
+    /* bug-1548: mark that a reconnect actually completed, so a consumer
+     * polling ep->reconnect_gen (e.g. apiserver's heartbeat prober) can
+     * detect it even when this whole reconnect happened silently inside a
+     * single XCECLI_FF_CALL and the caller's own return code never surfaced
+     * a failure. Only bumped on this success path — a failed attempt above
+     * already returned early without reaching here. */
+    ctx->ep->reconnect_gen++;
+
     return 0;
 }