fix(apiserver): auto-recover after full controller death+restart (task #12)

The apiserver stayed circuit-breaker-degraded after a full controller
restart until manually restarted. Two root causes:

1. Breaker recovery was decoupled from the breaker itself. The breaker can
   be tripped by a backend write failure (e.g. E_XCE_STANDBY -10014 while
   the controller is electing) with no heartbeat ever failing, but
   StartHeartbeat only recovered on a fail→success heartbeat transition
   (prevFailed), so the breaker stayed degraded forever.
   → extract needsRejoinAfterHeartbeat(hbOK, prevFailed): recover when
   prevFailed OR the breaker is currently tripped. 5 new unit tests.

2. The 30s heartbeat's fire-and-forget reconnect only fires when the lib's
   internal ctx state is XCE_CTX_ERR. After a controller restart whose
   election outlasts the FAILOVER_CALL retry budget, the retry loop's
   xcecli_ctx_recover leaves the ctx RUN on a half-open connection, so the
   FF path never reconnects and the client stays disconnected forever.
   → lib: new xcecli_force_reconnect() forces a rotate+recover regardless
   of state; apiserver: 2s recoveryLoop force-reconnects + rejoins while
   degraded.

Verified on cluster: after controller kill+restart, the apiserver logs
"backend re-joined controller after reconnect" and job submissions resume
without a manual restart; a fresh job runs to DONE. Controller, scheduler
(rejoin ok, bug-1513) and workers all auto-recover.

Co-Authored-By: Claude <noreply@anthropic.com>

docs/left_scheduler_work.md

@@ -93,7 +93,9 @@ spec 声称 36/36 任务完成，但 re-audit（research.md）发现缺陷。来
   - **worker 无限重试**（controller_connector.c `checking_connection`）：去掉 5 次上限，无限重试直到重连（连接器线程专用，不阻塞硬件/进程监控线程）。实测：杀 controller 后 3 worker 全部存活并持续重试，controller 回来后自动 rejoin。**永久故障**：持续重试不放弃（正确）；**HA**：`xcecd_join_cluster` 已按 --peers 候选列表迭代（best-effort）。
   - **apiserver rejoin**（clib.go `StartHeartbeat` + `xce_backend_rejoin`）：心跳从失败转成功时显式 rejoin（HostRole_apiserver），恢复 push 通知。实测：controller 重启后日志 `backend re-joined controller after reconnect`。
   - **apiserver fail-fast 电路熔断（Req 4，commit 2bbe894）**：health.go 原子熔断 + `callBackend` 包 4 个写调用。degraded 时立即返回 `backend controller unavailable`，不调 lib；传输错误或 XCE NOMASTER/REDIRECT/NOT_READY/STANDBY 触发熔断；语义错误不触发。实测：杀 controller 后第 2 次提交 ~11ms 返回 fail-fast。7 单测。
-  - **⚠️ 已知缺口**：**apiserver 心跳恢复在 controller 完整死亡+重启后不总是可靠**——lib transport reconnect 未稳定重建到新 controller（与 scheduler reconnect 同类，task #13）。当前需手动重启 apiserver 恢复（clean rejoin 正常）。
+  - **✅ apiserver 自动恢复已修复（2026-08-14）**——两个根因：
+    1. **熔断恢复与心跳解耦**：breaker 可被 backend 写失败（如选举期 E_XCE_STANDBY -10014）触发，但 `StartHeartbeat` 只按「心跳失败→成功」转移恢复，`prevFailed` 为 false 时永不恢复 → 一直 degraded 直到手动重启。修复：`needsRejoinAfterHeartbeat` 在 `prevFailed.Swap(false) || IsDegraded()` 时恢复（heartbeat.go + 5 个单测）。
+    2. **transport reconnect 靠 lib 状态启发式不可靠**：30s 心跳的 fire-and-forget 只在 `ctx->state==ERR` 时重连；controller 重启的选举期超出 FAILOVER_CALL 重试预算后，ctx 停在 RUN 的半开连接，永不重连。修复：lib 新增 `xcecli_force_reconnect`（无视状态强制 rotate+recover），apiserver 加 2s `recoveryLoop`（degraded 时强制重连+rejoin）。实测：controller 完整死亡+重启后 apiserver 自动 `backend re-joined controller after reconnect`，无需手动重启，提交恢复。
 - **✅ scheduler reconnect 已修复（2026-08-14 根因定位 + 集群验证）**：
   - **关键结论**：所谓 "libxce reconnect heap corruption" **不是 lib 竞态**——是 scheduler rejoin 状态重同步的 3 个真实 bug（均在 config/dataset 重推路径）。
   - **根因 1（group.c:234/247）**：`makeGroup` 刷新已有 group 的成员时 `FREEUP(m->u.name)` **raw-free 了共享 interned xstring**（`make_g_member` 用 `makeString` 建的是全局引用计数共享串，多个 group 共享同一指针）。正确释放是 `dereference()`（`group_killer` 就这么做）。ASAN 报 `bad-free`/`free(): invalid pointer` 即源于此。