feat(failover): worker keeps retrying forever + apiserver re-joins on reconnect

component-failover-adoption Req 2 (partial): two of the three daemons now
self-heal after a controller restart without an operator restart.

- worker (controller_connector.c): checking_connection() previously gave up
  after 5 retries and called handle_disconnect() (which set running=false and
  exited the connector thread). It now retries indefinitely — the connector
  thread is dedicated to this loop so the unbounded retry/sleep cannot block
  the worker's other liveness duties. Verified live: killing the controller
  leaves all 3 workers alive and retrying; they re-join when it returns.

- apiserver (clib.go): xcecli_heartbeat is fire-and-forget — it best-effort
  reconnects the transport but never re-joins, so after a controller restart
  push notifications stop. StartHeartbeat now tracks heartbeat failure→success
  and, on recovery, calls xce_backend_rejoin (xcecli_join as HostRole_apiserver)
  to re-register and resume pushes. Verified live: "backend re-joined
  controller after reconnect" logged during a controller kill/restart.

Scheduler reconnect NOT included: the rejoin works but exposes a pre-existing
libxce race — xce_ctx_reconnect's endpoint_fini tears down the old endpoint
while a lib vtask/io thread is still processing it, a deterministic double-free
(bad-free in the loader, same freed address offset across runs). Root-cause in
libxce netio/endpoint lifecycle is a separate task (documented in
left_scheduler_work.md).

Co-Authored-By: Claude <noreply@anthropic.com>

src/frontend/apiserver/backend/clib.go

@@ -36,6 +36,15 @@ static int xce_backend_init(xce_client_config_t *config, backend_ctx_t *ctx) {
     return xcecli_join(ctx->ctx, HostRole_apiserver, NULL);
 }
 
+// xce_backend_rejoin re-registers the apiserver with the (re)connected
+// controller. The controller resets its peer table on restart, so after a
+// transport reconnect the apiserver must re-join to resume push notifications
+// (component-failover-adoption Req 2.4). Call this from Go AFTER a heartbeat
+// has re-established the transport — never from the lib's own reconnect hook.
+static int xce_backend_rejoin(backend_ctx_t *ctx) {
+    return xcecli_join(ctx->ctx, HostRole_apiserver, NULL);
+}
+
 // Helpers to extract typed row pointers from a dataset_t.
 static xce_job_t    **dataset_jobs(dataset_t *d)    { return (xce_job_t **)d->rows; }
 static xce_worker_t **dataset_workers(dataset_t *d) { return (xce_worker_t **)d->rows; }
@@ -60,6 +69,7 @@ import (
 	"context"
 	"fmt"
 	"sync"
+	"sync/atomic"
 	"unsafe"
 
 	"go.uber.org/zap"
@@ -185,11 +195,40 @@ func Cleanup() {
 	C.idgen_ctx_fini(&gIDCtx)
 }
 
+// rejoinBackend re-registers the apiserver with the controller after a
+// transport reconnect (component-failover-adoption Req 2.4). xcecli_heartbeat
+// is fire-and-forget: it best-effort reconnects the transport but never
+// re-joins, so without an explicit rejoin the controller's restarted peer
+// table never learns this apiserver again and push notifications stop.
+func rejoinBackend() {
+	if gCtx.ctx == nil {
+		return
+	}
+	rc := C.xce_backend_rejoin(&gCtx)
+	if rc < 0 {
+		logger.Warn("backend rejoin failed (will retry on next heartbeat)",
+			zap.Int("rc", int(rc)))
+		return
+	}
+	logger.Info("backend re-joined controller after reconnect")
+}
+
 // StartHeartbeat launches a goroutine that sends periodic xcecli_heartbeat
-// keep-alive messages to the controller. Stops when ctx is cancelled.
+// keep-alive messages to the controller. Stops when ctx is cancelled. When a
+// heartbeat transitions from failure to success (the transport reconnect
+// succeeded), it triggers an explicit backend rejoin so the apiserver resumes
+// receiving push notifications without a process restart.
 func StartHeartbeat(ctx context.Context) {
+	var prevFailed atomic.Bool
 	go runHeartbeat(ctx, func(secs int) {
-		C.xcecli_heartbeat(gCtx.ctx, C.int(secs))
+		rc := C.xcecli_heartbeat(gCtx.ctx, C.int(secs))
+		if rc < 0 {
+			prevFailed.Store(true)
+			return
+		}
+		if prevFailed.Swap(false) {
+			rejoinBackend()
+		}
 	})
 }