2bbe894f99
component-failover-adoption Req 4: when the apiserver's controller connection is down, backend-requiring API requests must return a clear service-unavailable error rather than hanging. Adds: - health.go: an atomic circuit breaker (IsDegraded/MarkDegraded/MarkHealthy). - callBackend: the four write calls (jobsubmit/jobctrl/queuectrl/hostctrl) now route through it — degraded ⇒ immediate "backend controller unavailable" error without invoking the lib; a transport OR XCE not-ready/masterless/ standby failure (E_XCE_NOMASTER/REDIRECT/NOT_READY/STANDBY) trips the breaker so the NEXT call fails fast. Semantic rejections (-EINVAL/-ENOENT/E_XCE_BAD_*) never trip it. - StartHeartbeat: a failed tick marks degraded; recovery re-joins (rejoinBackend) and marks healthy. Verified live (t1 cluster): kill controller → first submit returns xcecli_jobsubmit failed: -10013 (bounded, no hang), second submit returns "backend controller unavailable (reconnecting)" in ~11ms. 7 unit tests. Recovery-via-heartbeat after a FULL controller death is NOT yet reliable — the lib transport reconnect does not always re-establish to the restarted controller (same class as the scheduler reconnect; tracked under task 13). Co-Authored-By: Claude <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| backend | ||
| cmd | ||
| common | ||
| config | ||
| controllers | ||
| docs | ||
| middleware | ||
| models | ||
| router | ||
| services | ||
| store | ||
| tools/encrypt | ||
| utils | ||
| go.mod | ||
| go.sum | ||
| meson.build | ||
| README.md | ||
(1) run server command:
cd cmd go run main.go
(2) run server with different log level: cd cmd go run main.go -level=debug
level options: info/debug/warn/error/fatal