fix(apiserver): auto-recover after full controller death+restart (task #12)

The apiserver stayed circuit-breaker-degraded after a full controller
restart until manually restarted. Two root causes:

1. Breaker recovery was decoupled from the breaker itself. The breaker can
   be tripped by a backend write failure (e.g. E_XCE_STANDBY -10014 while
   the controller is electing) with no heartbeat ever failing, but
   StartHeartbeat only recovered on a fail→success heartbeat transition
   (prevFailed), so the breaker stayed degraded forever.
   → extract needsRejoinAfterHeartbeat(hbOK, prevFailed): recover when
   prevFailed OR the breaker is currently tripped. 5 new unit tests.

2. The 30s heartbeat's fire-and-forget reconnect only fires when the lib's
   internal ctx state is XCE_CTX_ERR. After a controller restart whose
   election outlasts the FAILOVER_CALL retry budget, the retry loop's
   xcecli_ctx_recover leaves the ctx RUN on a half-open connection, so the
   FF path never reconnects and the client stays disconnected forever.
   → lib: new xcecli_force_reconnect() forces a rotate+recover regardless
   of state; apiserver: 2s recoveryLoop force-reconnects + rejoins while
   degraded.

Verified on cluster: after controller kill+restart, the apiserver logs
"backend re-joined controller after reconnect" and job submissions resume
without a manual restart; a fresh job runs to DONE. Controller, scheduler
(rejoin ok, bug-1513) and workers all auto-recover.

Co-Authored-By: Claude <noreply@anthropic.com>

src/frontend/apiserver/backend/clib.go

@@ -45,6 +45,16 @@ static int xce_backend_rejoin(backend_ctx_t *ctx) {
     return xcecli_join(ctx->ctx, HostRole_apiserver, NULL);
 }
 
+// xce_backend_force_reconnect force-reconnects the transport regardless of the
+// lib's internal state (see xcecli_force_reconnect). Used by the Go recovery
+// loop when the circuit breaker is degraded: the 30s heartbeat's fire-and-forget
+// reconnect only fires when ctx->state == XCE_CTX_ERR, and after a controller
+// restart whose election outlasts the FAILOVER_CALL retry budget the ctx can
+// stay RUN on a half-open connection, never reconnecting on its own.
+static int xce_backend_force_reconnect(backend_ctx_t *ctx) {
+    return xcecli_force_reconnect(ctx->ctx);
+}
+
 // Helpers to extract typed row pointers from a dataset_t.
 static xce_job_t    **dataset_jobs(dataset_t *d)    { return (xce_job_t **)d->rows; }
 static xce_worker_t **dataset_workers(dataset_t *d) { return (xce_worker_t **)d->rows; }
@@ -70,6 +80,7 @@ import (
 	"fmt"
 	"sync"
 	"sync/atomic"
+	"time"
 	"unsafe"
 
 	"go.uber.org/zap"
@@ -222,16 +233,42 @@ func StartHeartbeat(ctx context.Context) {
 	var prevFailed atomic.Bool
 	go runHeartbeat(ctx, func(secs int) {
 		rc := C.xcecli_heartbeat(gCtx.ctx, C.int(secs))
-		if rc < 0 {
-			prevFailed.Store(true)
-			MarkDegraded()
-			return
-		}
-		if prevFailed.Swap(false) {
+		if needsRejoinAfterHeartbeat(rc >= 0, &prevFailed) {
 			rejoinBackend()
 			MarkHealthy()
 		}
 	})
+	go recoveryLoop(ctx)
+}
+
+// recoveryLoop retries a forced transport reconnect + rejoin on a short
+// interval while the circuit breaker is degraded (bug-1513 follow-up). The
+// 30s heartbeat's fire-and-forget reconnect only fires when the lib's internal
+// state is ERR; after a controller restart whose election outlasts the
+// FAILOVER_CALL retry budget the ctx can stay RUN on a half-open connection and
+// the heartbeat never reconnects — the backend stays degraded until a manual
+// restart. This loop force-reconnects deterministically and re-joins, so a
+// full controller death+restart recovers on its own.
+func recoveryLoop(ctx context.Context) {
+	t := time.NewTicker(2 * time.Second)
+	defer t.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+			if !IsDegraded() {
+				continue
+			}
+			rc := C.xce_backend_force_reconnect(&gCtx)
+			if rc < 0 {
+				// Still unreachable (controller electing/booting) — retry next tick.
+				continue
+			}
+			rejoinBackend()
+			MarkHealthy()
+		}
+	}
 }
 
 // go_push_handler is called by the C library on every incoming request.