lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(apiserver): auto-recover after full controller death+restart (task #12)

Browse source 
The apiserver stayed circuit-breaker-degraded after a full controller
restart until manually restarted. Two root causes:

1. Breaker recovery was decoupled from the breaker itself. The breaker can
   be tripped by a backend write failure (e.g. E_XCE_STANDBY -10014 while
   the controller is electing) with no heartbeat ever failing, but
   StartHeartbeat only recovered on a fail→success heartbeat transition
   (prevFailed), so the breaker stayed degraded forever.
   → extract needsRejoinAfterHeartbeat(hbOK, prevFailed): recover when
   prevFailed OR the breaker is currently tripped. 5 new unit tests.

2. The 30s heartbeat's fire-and-forget reconnect only fires when the lib's
   internal ctx state is XCE_CTX_ERR. After a controller restart whose
   election outlasts the FAILOVER_CALL retry budget, the retry loop's
   xcecli_ctx_recover leaves the ctx RUN on a half-open connection, so the
   FF path never reconnects and the client stays disconnected forever.
   → lib: new xcecli_force_reconnect() forces a rotate+recover regardless
   of state; apiserver: 2s recoveryLoop force-reconnects + rejoins while
   degraded.

Verified on cluster: after controller kill+restart, the apiserver logs
"backend re-joined controller after reconnect" and job submissions resume
without a manual restart; a fresh job runs to DONE. Controller, scheduler
(rejoin ok, bug-1513) and workers all auto-recover.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
lei.chen 2026-08-14 08:01:35 +00:00
parent 7f8c07ba85
commit e505066162
6 changed files with 171 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

1
src/include/client.h
View file

@ -48,6 +48,7 @@ extern int xcecli_wakeup(xcecli_ctx_t, int);
extern int xcecli_hello(xcecli_ctx_t, char*);
extern int xcecli_nop(xcecli_ctx_t);
extern int xcecli_heartbeat(xcecli_ctx_t, int);
extern int xcecli_force_reconnect(xcecli_ctx_t);
extern int xcecli_shutdown(xcecli_ctx_t);
extern int xcecli_reslist(xcecli_ctx_t, xce_resource_t***);
extern int xcecli_join(xcecli_ctx_t, int, hostattrs_t*);