e505066162
The apiserver stayed circuit-breaker-degraded after a full controller restart until manually restarted. Two root causes: 1. Breaker recovery was decoupled from the breaker itself. The breaker can be tripped by a backend write failure (e.g. E_XCE_STANDBY -10014 while the controller is electing) with no heartbeat ever failing, but StartHeartbeat only recovered on a fail→success heartbeat transition (prevFailed), so the breaker stayed degraded forever. → extract needsRejoinAfterHeartbeat(hbOK, prevFailed): recover when prevFailed OR the breaker is currently tripped. 5 new unit tests. 2. The 30s heartbeat's fire-and-forget reconnect only fires when the lib's internal ctx state is XCE_CTX_ERR. After a controller restart whose election outlasts the FAILOVER_CALL retry budget, the retry loop's xcecli_ctx_recover leaves the ctx RUN on a half-open connection, so the FF path never reconnects and the client stays disconnected forever. → lib: new xcecli_force_reconnect() forces a rotate+recover regardless of state; apiserver: 2s recoveryLoop force-reconnects + rejoins while degraded. Verified on cluster: after controller kill+restart, the apiserver logs "backend re-joined controller after reconnect" and job submissions resume without a manual restart; a fresh job runs to DONE. Controller, scheduler (rejoin ok, bug-1513) and workers all auto-recover. Co-Authored-By: Claude <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| superpowers/plans | ||
| aibot_poc_investiagting.md | ||
| Job-Finish-Robustness-设计文档.md | ||
| left_scheduler_work.md | ||
| pend-reason-investigation-design.md | ||
| Scheduler-Agent-设计文档.md | ||