test(component-failover-adoption): close apiserver concurrency coverage gap

Audit of apiserver's three task-4.6 coverage areas (degraded-to-healthy
transitions, concurrent-request safety, candidate-list parity):
degraded-to-healthy and candidate-list parity were already fully
satisfied by tasks 4.1/4.5's own tests. Found one genuine gap: reconnect.go's
needsRejoin/checkAndPerformRejoin (task 4.3) had zero concurrent-access
test coverage despite being reached from 5 real concurrent HTTP-wrapper
call sites plus the heartbeat goroutine -- health.go's existing
50-goroutine stress test only covered the separate degraded-until
timestamp. One new test closes it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
This commit is contained in:
Rongsong Shen 2026-08-12 12:21:54 +00:00
parent c8325e7575
commit 79273a7f82
Signed by: rshen
SSH key fingerprint: SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
2 changed files with 71 additions and 1 deletions

View file

@ -5,6 +5,7 @@ package backend
import (
"os"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
@ -303,3 +304,71 @@ func TestClibSource_BackendCallWrappersCallSharedRejoinHelper(t *testing.T) {
"or callBackend itself no longer calls it", count)
}
}
// ════════════════════════════════════════════════════════════════════════
// Concurrent-request safety (component-failover-adoption task 4.6,
// boundary: ApiserverBackendBridge). health_test.go already stress-tests
// health.go's degraded-until state under concurrent goroutines
// (TestHealthState_ConcurrentReadersAndWriters_NoDataCorruption), but that
// test never touches needsRejoin/checkAndPerformRejoin at all -- and unlike
// a plain atomic load/store pair, checkAndPerformRejoin() is a genuine
// check-then-act compound operation (load needsRejoin, conditionally call
// performRejoinFn, conditionally store) reached from real concurrent
// production call sites: all 5 of clib.go's callBackend-wrapped functions
// (SubmitJob/ControlJob/ControlQueue/ControlHost/FetchFile) run on
// per-HTTP-request goroutines, plus heartbeat.go's own goroutine calls it on
// every successful tick (task 4.4) -- so multiple goroutines can genuinely
// call checkAndPerformRejoin() at the same moment after a failover, exactly
// as task 4.3's own Implementation Note already anticipated ("concurrent
// goroutines can each observe needsRejoin==1 ... and each call the real
// xcecli_join()"). That note explains why this is correctness-safe in
// production (serialized by the C library's own per-ctx lock) but it was
// never backed by a test proving the Go-level state machine itself (flag
// clear/no-clear, no panic) holds up under real concurrent access -- this
// closes that gap. Deliberately does NOT assert exactly-one call to
// performRejoinFn: redundant concurrent calls are the documented, accepted
// behavior (a CAS-based single-flight guard was explicitly judged a
// reasonable-but-not-required fast-follow, not a bug), so asserting
// call-count==1 would encode a guarantee the design never made.
// ════════════════════════════════════════════════════════════════════════
func TestCheckAndPerformRejoin_ConcurrentCallers_NoDataCorruptionAndFlagClears(t *testing.T) {
captureLog(t)
resetNeedsRejoinForTest()
defer resetNeedsRejoinForTest()
orig := performRejoinFn
defer func() { performRejoinFn = orig }()
var joinCalls int32
performRejoinFn = func() int {
atomic.AddInt32(&joinCalls, 1)
return 0 // always succeeds
}
atomic.StoreInt32(&needsRejoin, 1)
const goroutines = 50
var wg sync.WaitGroup
wg.Add(goroutines)
for i := 0; i < goroutines; i++ {
go func() {
defer wg.Done()
checkAndPerformRejoin()
}()
}
wg.Wait()
// Reaching here without a panic/deadlock, under concurrent goroutines
// each racing through the load-call-store sequence, is itself part of
// what this test proves.
if got := atomic.LoadInt32(&joinCalls); got < 1 {
t.Errorf("expected performRejoinFn to be called at least once across "+
"%d concurrent checkAndPerformRejoin() callers, got %d", goroutines, got)
}
if atomic.LoadInt32(&needsRejoin) != 0 {
t.Error("expected needsRejoin to end up cleared: every concurrent " +
"performRejoinFn call in this test succeeds (rc=0), so whichever " +
"goroutine's store lands last still clears the flag to 0")
}
}