test(component-failover-adoption): close apiserver concurrency coverage gap
Audit of apiserver's three task-4.6 coverage areas (degraded-to-healthy transitions, concurrent-request safety, candidate-list parity): degraded-to-healthy and candidate-list parity were already fully satisfied by tasks 4.1/4.5's own tests. Found one genuine gap: reconnect.go's needsRejoin/checkAndPerformRejoin (task 4.3) had zero concurrent-access test coverage despite being reached from 5 real concurrent HTTP-wrapper call sites plus the heartbeat goroutine -- health.go's existing 50-goroutine stress test only covered the separate degraded-until timestamp. One new test closes it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
This commit is contained in:
parent
c8325e7575
commit
79273a7f82
2 changed files with 71 additions and 1 deletions
|
|
@ -5,6 +5,7 @@ package backend
|
|||
import (
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
|
@ -303,3 +304,71 @@ func TestClibSource_BackendCallWrappersCallSharedRejoinHelper(t *testing.T) {
|
|||
"or callBackend itself no longer calls it", count)
|
||||
}
|
||||
}
|
||||
|
||||
// ════════════════════════════════════════════════════════════════════════
|
||||
// Concurrent-request safety (component-failover-adoption task 4.6,
|
||||
// boundary: ApiserverBackendBridge). health_test.go already stress-tests
|
||||
// health.go's degraded-until state under concurrent goroutines
|
||||
// (TestHealthState_ConcurrentReadersAndWriters_NoDataCorruption), but that
|
||||
// test never touches needsRejoin/checkAndPerformRejoin at all -- and unlike
|
||||
// a plain atomic load/store pair, checkAndPerformRejoin() is a genuine
|
||||
// check-then-act compound operation (load needsRejoin, conditionally call
|
||||
// performRejoinFn, conditionally store) reached from real concurrent
|
||||
// production call sites: all 5 of clib.go's callBackend-wrapped functions
|
||||
// (SubmitJob/ControlJob/ControlQueue/ControlHost/FetchFile) run on
|
||||
// per-HTTP-request goroutines, plus heartbeat.go's own goroutine calls it on
|
||||
// every successful tick (task 4.4) -- so multiple goroutines can genuinely
|
||||
// call checkAndPerformRejoin() at the same moment after a failover, exactly
|
||||
// as task 4.3's own Implementation Note already anticipated ("concurrent
|
||||
// goroutines can each observe needsRejoin==1 ... and each call the real
|
||||
// xcecli_join()"). That note explains why this is correctness-safe in
|
||||
// production (serialized by the C library's own per-ctx lock) but it was
|
||||
// never backed by a test proving the Go-level state machine itself (flag
|
||||
// clear/no-clear, no panic) holds up under real concurrent access -- this
|
||||
// closes that gap. Deliberately does NOT assert exactly-one call to
|
||||
// performRejoinFn: redundant concurrent calls are the documented, accepted
|
||||
// behavior (a CAS-based single-flight guard was explicitly judged a
|
||||
// reasonable-but-not-required fast-follow, not a bug), so asserting
|
||||
// call-count==1 would encode a guarantee the design never made.
|
||||
// ════════════════════════════════════════════════════════════════════════
|
||||
|
||||
func TestCheckAndPerformRejoin_ConcurrentCallers_NoDataCorruptionAndFlagClears(t *testing.T) {
|
||||
captureLog(t)
|
||||
resetNeedsRejoinForTest()
|
||||
defer resetNeedsRejoinForTest()
|
||||
|
||||
orig := performRejoinFn
|
||||
defer func() { performRejoinFn = orig }()
|
||||
|
||||
var joinCalls int32
|
||||
performRejoinFn = func() int {
|
||||
atomic.AddInt32(&joinCalls, 1)
|
||||
return 0 // always succeeds
|
||||
}
|
||||
|
||||
atomic.StoreInt32(&needsRejoin, 1)
|
||||
|
||||
const goroutines = 50
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(goroutines)
|
||||
for i := 0; i < goroutines; i++ {
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
checkAndPerformRejoin()
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
// Reaching here without a panic/deadlock, under concurrent goroutines
|
||||
// each racing through the load-call-store sequence, is itself part of
|
||||
// what this test proves.
|
||||
|
||||
if got := atomic.LoadInt32(&joinCalls); got < 1 {
|
||||
t.Errorf("expected performRejoinFn to be called at least once across "+
|
||||
"%d concurrent checkAndPerformRejoin() callers, got %d", goroutines, got)
|
||||
}
|
||||
if atomic.LoadInt32(&needsRejoin) != 0 {
|
||||
t.Error("expected needsRejoin to end up cleared: every concurrent " +
|
||||
"performRejoinFn call in this test succeeds (rc=0), so whichever " +
|
||||
"goroutine's store lands last still clears the flag to 0")
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue