2bbe894f99
component-failover-adoption Req 4: when the apiserver's controller connection is down, backend-requiring API requests must return a clear service-unavailable error rather than hanging. Adds: - health.go: an atomic circuit breaker (IsDegraded/MarkDegraded/MarkHealthy). - callBackend: the four write calls (jobsubmit/jobctrl/queuectrl/hostctrl) now route through it — degraded ⇒ immediate "backend controller unavailable" error without invoking the lib; a transport OR XCE not-ready/masterless/ standby failure (E_XCE_NOMASTER/REDIRECT/NOT_READY/STANDBY) trips the breaker so the NEXT call fails fast. Semantic rejections (-EINVAL/-ENOENT/E_XCE_BAD_*) never trip it. - StartHeartbeat: a failed tick marks degraded; recovery re-joins (rejoinBackend) and marks healthy. Verified live (t1 cluster): kill controller → first submit returns xcecli_jobsubmit failed: -10013 (bounded, no hang), second submit returns "backend controller unavailable (reconnecting)" in ~11ms. 7 unit tests. Recovery-via-heartbeat after a FULL controller death is NOT yet reliable — the lib transport reconnect does not always re-establish to the restarted controller (same class as the scheduler reconnect; tracked under task 13). Co-Authored-By: Claude <noreply@anthropic.com>
24 lines
1 KiB
Go
24 lines
1 KiB
Go
package backend
|
|
|
|
import "sync/atomic"
|
|
|
|
// backendDegraded is the fail-fast circuit breaker (component-failover-adoption
|
|
// Req 4): when the controller connection is down, backend-requiring API
|
|
// requests return a clear "backend unavailable" error immediately instead of
|
|
// hanging until the connection is restored.
|
|
//
|
|
// Lifecycle:
|
|
// - tripped by a write-path transport failure (callBackend) and by a failed
|
|
// heartbeat tick (StartHeartbeat);
|
|
// - cleared by a successful heartbeat recovery (StartHeartbeat → rejoinBackend
|
|
// → MarkHealthy), which also re-joins the apiserver session so pushes resume.
|
|
var backendDegraded atomic.Bool
|
|
|
|
// IsDegraded reports whether the backend controller is currently unreachable.
|
|
func IsDegraded() bool { return backendDegraded.Load() }
|
|
|
|
// MarkDegraded trips the circuit breaker — backend calls now fail fast.
|
|
func MarkDegraded() { backendDegraded.Store(true) }
|
|
|
|
// MarkHealthy clears the circuit breaker after a successful reconnect/rejoin.
|
|
func MarkHealthy() { backendDegraded.Store(false) }
|