fix(component-failover-adoption): Phase 6 -- close apiserver's idle-period rejoin gap (bug-1548/bug-1549)

XCECLI_FF_CALL (backing xcecli_heartbeat) can silently rotate+reconnect
the transport entirely inside a single call and still return success, so
heartbeat.go's failure-to-success transition detector could never observe
a reconnect that completes within one heartbeat tick -- the common case,
since elections finish in single-digit seconds against a default 30s
interval. apiserver's application-level rejoin never fired on a real
idle-period failover, and job pushes silently stopped reaching it.

Adds a monotonic reconnect_gen counter to endpoint_t (netio.h), bumped by
xce_ctx_reconnect() (xcectx.c) on every successful reconnect regardless of
trigger path. heartbeat.go now compares this generation every tick instead
of tracking success/failure transitions. Re-validated live against task
5.2's exact scenario (apiserver alone, zero write traffic): rejoin fired
22s after a primary kill, 12s after election.

Reopens and closes out component-failover-adoption's own NO-GO -> GO
validation cycle from 2026-08-15.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

.kiro/specs/component-failover-adoption/spec.json

@@ -1,9 +1,9 @@
 {
   "feature_name": "component-failover-adoption",
   "created_at": "2026-07-20T04:23:58Z",
-  "updated_at": "2026-07-20T05:20:00Z",
+  "updated_at": "2026-08-15T14:20:00Z",
   "language": "en",
-  "phase": "tasks-generated",
+  "phase": "implementation-complete",
   "approvals": {
     "requirements": {
       "generated": true,
@@ -15,8 +15,24 @@
     },
     "tasks": {
       "generated": true,
-      "approved": false
+      "approved": true
     }
   },
-  "ready_for_implementation": false
+  "ready_for_implementation": true,
+  "validate_impl": {
+    "decision": "GO",
+    "validated_at": "2026-08-15T14:20:00Z",
+    "known_limitations": [],
+    "resolved_limitations": [
+      "bug-1548/bug-1549: apiserver never re-registered with a new primary after a fast controller failover. Fixed via tasks.md Phase 6 -- endpoint_t.reconnect_gen counter (netio.h/xcectx.c) bumped on every successful transport reconnect, heartbeat.go's runHeartbeat replaced its failure-to-success transition detector with a per-tick generation comparison. Confirmed via live E2E re-run of task 5.2's exact scenario: rejoin fired 22s after primary kill (12s after new election) with zero write traffic for the preceding 5.5 minutes; a post-recovery diagnostic job submit correctly appeared via GET /xce/v1/jobs.",
+      "bug-1559: xce-build.sh's scheduler case didn't build/run 4 newer scheduler test binaries. Fixed directly (tooling, not a spec task) -- now runs `meson test --suite scheduler`, exercising all 41 scheduler test binaries.",
+      "bug-1558: make test-backend broken standalone. Fixed directly (tooling, not a spec task) -- self-derives CGO_CFLAGS/CGO_LDFLAGS via pkg-config, confirmed working with zero manual env setup."
+    ],
+    "prior_decision": {
+      "decision": "NO-GO",
+      "validated_at": "2026-08-15T13:10:00Z",
+      "reopened_at": "2026-08-15T13:10:00Z",
+      "reopen_reason": "component-failover-adoption's own /kiro-validate-impl run found Req 2 AC4 unmet in the common case (fast controller election, apiserver idle) -- a defect inside this spec's own ApiserverBackendBridge/task 4.4 boundary, not legitimately out of scope. Remediated as tasks.md Phase 6 (6.1/6.2) plus two non-blocking build-tooling fixes (bug-1558, bug-1559) found during the same validation pass."
+    }
+  }
 }

.kiro/specs/component-failover-adoption/tasks.md

@@ -159,6 +159,26 @@
   - _Requirements: 1.5_
   - _Depends: 5.1, 5.2, 5.3_
 
+## 6. Post-validation remediation (bug-1548/bug-1549, NO-GO 2026-08-15)
+
+- [x] 6.1 Fix apiserver's idle-period rejoin detection gap
+  - Add a monotonic reconnect-generation counter to the client library's endpoint struct, incremented on every successful transport reconnect regardless of trigger path
+  - Replace the heartbeat prober's failure-to-success transition check with a per-tick comparison against this generation counter, triggering the same pending-rejoin flag on any change
+  - Observable completion: a new regression test proves the pending-rejoin flag is set on a generation change even when every heartbeat tick the Go loop observes reports success (the exact shape XCECLI_FF_CALL's silent inline reconnect produces) — its own doc comment states this proves detector logic, not reachability
+  - _Requirements: 2.4, 2.5_
+  - _Depends: 4.4_
+
+- [x] 6.2 E2E re-validation: re-run task 5.2's idle-period scenario against the fix
+  - Re-run the exact task 5.2 scenario (apiserver only, no worker/scheduler, 3-node controller cluster, default heartbeat interval, primary killed with zero write traffic during the observation window) against the fixed code
+  - Observable completion: identical to task 5.2's own bar — rejoin log lines appear within the observation window, and GET /xce/v1/jobs populates correctly after a post-recovery submit, with no client-triggered write request required
+  - _Requirements: 2.4, 2.5_
+  - _Depends: 6.1_
+
+## Implementation Notes (Phase 6)
+- Task 6.1: fix is a monotonic `endpoint_t.reconnect_gen` counter (`src/include/netio.h`), bumped once, unconditionally, on the success path of `xce_ctx_reconnect()` (`src/lib/xcectx.c`) — confirmed the sole reconnect call-site funnel for both the write-path retry loop and the heartbeat fire-and-forget path, so it observes every reconnect regardless of trigger. Read from Go via the existing, previously-unused `xcecli_get_my_endpoint()` accessor (`client.h`) — no new C getter needed, and none of the frozen mechanism files (`client.c`/`client_internal.h`/`client_failover.c`/`client_failover.h`) were touched. `heartbeat.go`'s `runHeartbeat()` fully replaced its `prevFailed` failure-to-success transition detector with a per-tick generation comparison (user's explicit direction: no compatibility shim needed, pick the objectively better approach) — the old detector is logically a strict subset of the new one under this codebase's own documented state machine (`ctx->state == XCE_CTX_ERR` is the single known-dead signal gating `XCECLI_FF_CALL`'s reconnect branch), and the new one additionally catches a reconnect triggered by a concurrent write-path call between heartbeat ticks. New regression tests (`heartbeat_test.go`, `heartbeat_rejoin_test.go`) prove the detector fires on a generation change even when every observed tick reports success — the literal shape of bug-1548 that the old detector's own green test suite could never catch.
+- Task 6.2: live E2E re-run of task 5.2's exact scenario (apiserver alone, no worker/scheduler, 3-node cluster, default 30s heartbeat interval, primary killed, zero write traffic during the observation window) — **PASS**, closing the gap. Primary killed 14:13:46 UTC; new primary elected 14:13:56 UTC (10s); `checkAndPerformRejoin: rejoin after transparent reconnect succeeded` logged at 14:14:08 UTC (22s after the kill, 12s after the election), with zero write/backend traffic in the preceding 5.5 minutes — the heartbeat path alone triggered it, exactly the case that previously showed zero rejoin evidence across a 268s+ window. bug-1549 downstream symptom also confirmed resolved: a diagnostic job submitted after recovery correctly appeared via `GET /xce/v1/jobs` (count:1, matching job data), proving apiserver's push/registration state genuinely recovered with the new primary, not just the transport.
+- Non-blocking tooling fixes made alongside (not spec tasks — `xce-build.sh`/the frontend `Makefile` are general repo tooling, not spec-owned files per design.md's File Structure Plan; tracked via buglog only): bug-1559 (`xce-build.sh`'s `scheduler` case now runs `meson test --suite scheduler`, exercising all 41 scheduler test binaries instead of just `sched_test` directly by path) and bug-1558 (`make test-backend` now self-derives `CGO_CFLAGS`/`CGO_LDFLAGS` via `pkg-config`, works standalone with zero manual env setup — one iteration needed: the first attempt's `-I../../include` was relative and silently resolved wrong under cgo's actual build cwd, fixed to an absolute `-I$(CURDIR)/../include`).
+
 ## Implementation Notes
 - Task 1.1: `src/lib/cluster.c:99-102`'s `xce_cluster_config_validate()` still has a third hardcoded `>3` WARN check (message says "<=3 only") of the same magic-number class Req 1.7 targets, and `src/include/xce/cluster.h:31`'s doc comment repeats the stale "3" figure. Task 1.1's text named only "the two" worker/scheduler checks, so this was left untouched there — task 1.2 already touches `cluster.c` to add the new conversion helper, so fold this fix in there rather than letting it silently persist as the one remaining un-reconciled magic number.
 - Task 1.1 follow-up resolved in 1.2: the `>3`→`>MAX_HA_CANDIDATES` rename initially proposed for `xce_cluster_config_validate()`'s WARN check turned out to be dead code — after task 1.1's constant reconciliation, that check tests the identical field against the identical constant as an earlier `-EINVAL` return in the same function, so it can never fire. Deleted rather than renamed (with the matching stale doc-comment line also removed), caught by task 1.2's review round 1.
@@ -175,11 +195,11 @@
 - Task 3.4: pure test-coverage task, zero production changes to `scheduler.c`/`driver.c` (correct outcome — task 3.2 already implemented the retry/shutdown/rejoin behavior correctly; 3.4 proves it holds at multi-decision QUEUE granularity through the real drain loop, which task 3.2's own tests never exercised, all 6 of which call `sched_publish_dispatch()`/`sched_publish_pend_reason()` directly on one decision). New `test_scheduler_decision_redelivery.c` links real `scheduler_srcs` with `-Wl,--wrap=xcecli_jobdispatch` (task 3.2's established technique) and drives the real `xce_sched_driver_on_decision()` (driver.c) over 3 queued, distinct-jobid decisions. Two load-bearing tests, both RED→GREEN-verified against the real code by two independent reviewer rounds (not just the implementer's claim): ordering (decision 1 stuck-then-succeeds fully completes before decision 2's call ever fires) and bounded shutdown latency (decisions queued behind one that triggers `core.stop` mid-retry get ZERO publish calls — the concrete, fast, deterministic proof that `sched_publish_retry_loop()`'s check-`stop`-before-each-attempt placement is correct, verified by reviewers moving the check to bottom-of-loop and confirming only this one test catches it). Test-fixture gotcha for any future test building a bare `xce_sched_core_t` via `xce_sched_core_init()` alone (not full `sched_init()`): do NOT call `xce_sched_core_fini()` on it — `fini()` calls `dataq_fini()` on `pend_updates[0/1]`, which only `sched_init()` (not `xce_sched_core_init()`) ever `dataq_init()`s, so `fini()` on a bare core SEGVs on a NULL dataq; omit the `fini()` call entirely, matching `test_core_events.c`'s established "fixtures aren't torn down" convention for this exact minimal-fixture pattern.
 - Task 4.5: subtle design question resolved and worth remembering for task 4.6/5.x — `callBackend()`'s (`clib.go`) degraded fast-path deliberately skips `checkAndPerformRejoin()` ENTIRELY (not just the underlying `C.xcecli_*` call), meaning write-path requests contribute NOTHING to rejoin-retry cadence while `IsDegraded()` is true; only task 4.4's heartbeat goroutine keeps retrying during that window. Reviewer independently confirmed this is correct, not a violation of "must never suppress the pending-rejoin flag's own retry-until-actually-joined behavior": that constraint is one-directional (the circuit breaker's own optimistic TTL-based state must never be READ BY the rejoin mechanism to gate it — confirmed `checkAndPerformRejoin()` never references `IsDegraded()` anywhere), not a requirement that every possible call site attempt a rejoin check regardless of cost. The alternative (calling `checkAndPerformRejoin()`, which can itself block ~100s inside `xcecli_join()`'s own retry budget, on every fast-failed request) would directly violate THIS task's own fail-fast acceptance criterion — the two constraints (fail-fast + never-suppress-retry) are jointly satisfiable only because `heartbeat.go`'s goroutine runs unconditionally for the process lifetime, independent of circuit-breaker state, and `backendDegradedTTL` (10s, chosen well below the ~100s `XCECLI_FAILOVER_CALL` budget) is shorter than the heartbeat interval (default 30s) so write traffic resumes contributing once the TTL clears anyway. `callBackend()` is a genuine refactor consolidating what tasks 4.3/4.4 left as duplicated inline `rc := C.xcecli_*(...); checkAndPerformRejoin()` pairs across 5 wrapper functions into one shared helper — `checkAndPerformRejoin()` now appears exactly once in `clib.go`, inside `callBackend`, matching design.md's explicit "not copy-pasted per wrapper" instruction. `E_XCE_NOT_READY`-triggered `MarkDegraded()` uses a plain-Go `notReadyRC` mirror var (`= -int(C.E_XCE_NOT_READY)`, computed live via cgo, not hardcoded) since `_test.go` files can't reference `C.*` symbols directly (task 4.1's established Go-toolchain restriction) — keep this pattern for any future test needing a C errno constant.
 - Task 4.6 (closes out apiserver group 4.1-4.6): audit task, same "zero new code is a valid outcome when genuinely true" precedent as scheduler's task 1.3 — 2 of 3 coverage areas (degraded-to-healthy transitions, candidate-list parity) were already fully satisfied by tasks 4.1/4.5's own tests, verified by reading the cited tests directly rather than trusting prior notes' claims. Found and closed exactly one genuine gap: `reconnect.go`'s `needsRejoin`/`checkAndPerformRejoin` (task 4.3) had zero concurrent-access test coverage despite being reached from 5 real concurrent HTTP-wrapper call sites plus the heartbeat goroutine — `health.go`'s own 50-goroutine stress test (task 4.5) only covered the degraded-until timestamp, not this separate atomic flag. One new test closes it. `gClusterCfg`/`gClientCfg` (task 4.2's CGO structs) correctly judged NOT a concurrency gap — written once at startup, single-threaded, before the HTTP server starts, never mutated again. Confirmed apiserver's Go-level `xce_parse_peers()` malformed-string coverage (`candidates_test.go`) is broader than any C-side component's own test — worker's/scheduler's C tests, and even `test_cluster.c`, only exercise `xce_cluster_config_validate()` against pre-built structs, never `xce_parse_peers()` against malformed raw strings at all. `go test -race` is confirmed, again, flatly unusable in this ASAN-linked CGO environment ("too many address space collisions") — this is now the third task (4.2, 4.5, 4.6) to independently hit and document this; don't waste time trying `-race` here, rely on atomic-correctness-by-construction plus deterministic concurrent-stress assertions instead. `xce-build.sh apiserver asan`'s binary-compile step can hit transient `goproxy.cn` network flakiness unrelated to any code change — when that happens, fall back to `cd src/frontend && make test` (CGO_ENABLED=0, whole apiserver module + xcecli) and `make test-backend` (CGO_ENABLED=1, `./backend/...`) directly, which is what the full pipeline's own `RUN_TESTS=1` branch invokes anyway once binaries are built. A pre-existing, unrelated `apiserver/controllers` build failure under a whole-module `CGO_ENABLED=1 go test ./...` (predates this spec entirely, `integration_test.go`'s `//go:build !cgo` tag vs. `output_test.go`'s unconditional references to its helpers) doesn't affect either authoritative Makefile target — don't chase it as a regression in future apiserver work.
-- Task 5.1 (first Group 5 task, live 4-process E2E — worker + scheduler + apiserver + 3-node controller cluster): `xcecd_virtual` (worker's own task 2.3 meson target, `-DVIRTUAL_CLUSTER -DDBUS_MANAGER`) is the correct non-root worker E2E vehicle in this sandbox — use it, NOT `unshare -r`, which hits the unresolved bug-010 ASAN double-join abort. Worker's reconnect proved cleanly (real `checking_connection: rejoin after transparent reconnect succeeded` log line, 16s after kill, PID unchanged). Scheduler's rejoin path was never naturally exercised (no job ever reached its picker/allocator) — clean initial join + PID stability shown instead, same partial-coverage shape task 3.3 already got approved with. **Found a likely-real, structural gap in the already-approved task 4.3/4.4 apiserver rejoin mechanism (bug-039, see buglog for full detail)**: `XCECLI_FF_CALL` (backing `xcecli_heartbeat`) can silently self-heal the transport within a SINGLE heartbeat tick (rotate+recover, then the send succeeds) without ever returning a failure to Go — so `heartbeat.go`'s cross-tick `onFailureToSuccess` transition detector structurally cannot observe it when election completes faster than one heartbeat interval (elections in this codebase complete in single-digit seconds; default heartbeat interval is 30s), and none of apiserver's post-kill write calls showed evidence of hitting a failed connection either (consistent with the transport having already healed via a prior heartbeat tick before any write needed to retry). Net: `needsRejoin` may never get set on a real single-election failover, so apiserver's application-level re-join with the new primary never fires even though transport-level requests keep succeeding — risks Req 2 AC4 in production. This is NOT a task-5.1 fix (out of its cross-component-validation boundary) — flagged for a dedicated follow-up spec/task, escalated above ordinary buglog cadence given it touches an already-approved mechanism. **Also found bug-040** (job pushes never reach apiserver's store, `GET /xce/v1/jobs` empty despite controller-confirmed commits) — very likely a downstream symptom of bug-039, not independent: `on_apiserver_join()` (`apiserver.c`) only bulk-resyncs jobs when a fresh join is actually processed, and `c->apiservers[]` (the push fan-out list) is per-process in-memory state with no replication — a new primary starts with an empty registration table until each apiserver instance re-joins against it. Fixing bug-039 should be verified against bug-040 clearing as confirmation they share a root cause. Controller-side `join.c` logs only rejected/failed joins, never successful ones at any verbosity — a real observability gap for anyone debugging this family of issues in the future, worth fixing alongside.
-- Task 5.2: designed as a targeted, cleaner confirmation experiment for task 5.1's bug-039 finding (heartbeat-only, zero write traffic, isolated to apiserver alone — no worker/scheduler to confound the signal) — and it worked exactly as intended. Result: **bug-039 independently reproduced a second time, more cleanly** — a 268s+ observation window with zero rejoin log lines, AND (stronger than task 5.1) a post-window diagnostic write call that itself triggers zero rejoin activity despite succeeding against the confirmed-new primary, ruling out "task 5.1 was just unlucky timing." Reviewer independently re-derived the mechanism from source (not just citing task 5.1's analysis) and confirmed `checkAndPerformRejoin()` is called unconditionally after every backend call in `callBackend()`, so the diagnostic write's silence is a genuine negative control, not a skipped-check artifact. Controller-side `ctrl-2.log` corroborates independently — zero "apiserver" references anywhere, meaning no join attempt (success or rejection) ever reached the new primary. This task's own literal pass bar ("scenario passes without requiring any write request") was NOT met — approved anyway on the same precedent task 5.1 established: the validation itself succeeded rigorously and honestly surfaced a real gap in already-shipped code, which is the task's actual purpose. **bug-039 is now doubly-confirmed and should be treated as a real, live production gap, not a hypothesis** — a dedicated follow-up spec/task to fix `heartbeat.go`'s `onFailureToSuccess`/`onSuccessTick` wiring (make `XCECLI_FF_CALL`'s internal self-heal observable to Go, e.g. by having the heartbeat wrapper track transport state itself rather than relying on the call's return code) is warranted, separate from any remaining task in this spec's own scope. bug-040 also re-reproduced a third time in this run (same session, same isolated evidence) — buglog kept in sync on both bug-039 and bug-040's own objects (occurrences/last_seen) after review caught bug-040's object being out of sync with its own re-reproduction narrated inside bug-039's text.
+- Task 5.1 (first Group 5 task, live 4-process E2E — worker + scheduler + apiserver + 3-node controller cluster): `xcecd_virtual` (worker's own task 2.3 meson target, `-DVIRTUAL_CLUSTER -DDBUS_MANAGER`) is the correct non-root worker E2E vehicle in this sandbox — use it, NOT `unshare -r`, which hits the unresolved bug-1519 ASAN double-join abort. Worker's reconnect proved cleanly (real `checking_connection: rejoin after transparent reconnect succeeded` log line, 16s after kill, PID unchanged). Scheduler's rejoin path was never naturally exercised (no job ever reached its picker/allocator) — clean initial join + PID stability shown instead, same partial-coverage shape task 3.3 already got approved with. **Found a likely-real, structural gap in the already-approved task 4.3/4.4 apiserver rejoin mechanism (bug-1548, see buglog for full detail)**: `XCECLI_FF_CALL` (backing `xcecli_heartbeat`) can silently self-heal the transport within a SINGLE heartbeat tick (rotate+recover, then the send succeeds) without ever returning a failure to Go — so `heartbeat.go`'s cross-tick `onFailureToSuccess` transition detector structurally cannot observe it when election completes faster than one heartbeat interval (elections in this codebase complete in single-digit seconds; default heartbeat interval is 30s), and none of apiserver's post-kill write calls showed evidence of hitting a failed connection either (consistent with the transport having already healed via a prior heartbeat tick before any write needed to retry). Net: `needsRejoin` may never get set on a real single-election failover, so apiserver's application-level re-join with the new primary never fires even though transport-level requests keep succeeding — risks Req 2 AC4 in production. This is NOT a task-5.1 fix (out of its cross-component-validation boundary) — flagged for a dedicated follow-up spec/task, escalated above ordinary buglog cadence given it touches an already-approved mechanism. **Also found bug-1549** (job pushes never reach apiserver's store, `GET /xce/v1/jobs` empty despite controller-confirmed commits) — very likely a downstream symptom of bug-1548, not independent: `on_apiserver_join()` (`apiserver.c`) only bulk-resyncs jobs when a fresh join is actually processed, and `c->apiservers[]` (the push fan-out list) is per-process in-memory state with no replication — a new primary starts with an empty registration table until each apiserver instance re-joins against it. Fixing bug-1548 should be verified against bug-1549 clearing as confirmation they share a root cause. Controller-side `join.c` logs only rejected/failed joins, never successful ones at any verbosity — a real observability gap for anyone debugging this family of issues in the future, worth fixing alongside.
+- Task 5.2: designed as a targeted, cleaner confirmation experiment for task 5.1's bug-1548 finding (heartbeat-only, zero write traffic, isolated to apiserver alone — no worker/scheduler to confound the signal) — and it worked exactly as intended. Result: **bug-1548 independently reproduced a second time, more cleanly** — a 268s+ observation window with zero rejoin log lines, AND (stronger than task 5.1) a post-window diagnostic write call that itself triggers zero rejoin activity despite succeeding against the confirmed-new primary, ruling out "task 5.1 was just unlucky timing." Reviewer independently re-derived the mechanism from source (not just citing task 5.1's analysis) and confirmed `checkAndPerformRejoin()` is called unconditionally after every backend call in `callBackend()`, so the diagnostic write's silence is a genuine negative control, not a skipped-check artifact. Controller-side `ctrl-2.log` corroborates independently — zero "apiserver" references anywhere, meaning no join attempt (success or rejection) ever reached the new primary. This task's own literal pass bar ("scenario passes without requiring any write request") was NOT met — approved anyway on the same precedent task 5.1 established: the validation itself succeeded rigorously and honestly surfaced a real gap in already-shipped code, which is the task's actual purpose. **bug-1548 is now doubly-confirmed and should be treated as a real, live production gap, not a hypothesis** — a dedicated follow-up spec/task to fix `heartbeat.go`'s `onFailureToSuccess`/`onSuccessTick` wiring (make `XCECLI_FF_CALL`'s internal self-heal observable to Go, e.g. by having the heartbeat wrapper track transport state itself rather than relying on the call's return code) is warranted, separate from any remaining task in this spec's own scope. bug-1549 also re-reproduced a third time in this run (same session, same isolated evidence) — buglog kept in sync on both bug-1548 and bug-1549's own objects (occurrences/last_seen) after review caught bug-1549's object being out of sync with its own re-reproduction narrated inside bug-1548's text.
 - Task 5.3: unlike 5.1/5.2 (killed only the primary, letting rotation succeed), this task needs a call to genuinely EXHAUST its retry budget — kill ALL 3 controller nodes, not just the primary. Corrected timing assumption worth remembering: `client_failover.c`'s `xce_failover_classify()` puts `ECONNREFUSED`/`ETIMEDOUT`/`ECONNRESET` (and the dead-ctx fast path) on the ZERO-backoff `XCE_FO_RETRY_NOW` track, not the ~70-100s exponential-backoff `XCE_FO_RETRY_BACKOFF` track (reserved for `E_XCE_NOMASTER`/`E_XCE_NOT_READY` responses from a reachable-but-electing controller) — so a fully-dead-cluster trigger call exhausts its 20-attempt budget in milliseconds, not minutes. Clean, strongly-corroborated PASS: exact literal string match on `ErrBackendUnavailable` (`clib.go`) proves the degraded fast-fail path genuinely skips the C call rather than the C call happening to also be fast; `E_XCE_NOT_READY`'s numeric value (10013 = `XCE_ERRNO_BASE` 10001 + 12 enum positions, `errnox.h`) independently confirmed the trigger call's `-10013` response was the real thing, not a fabricated failure; the raw C-library log (`client_failover.c`'s own WARN/error lines, a format-distinct second logging subsystem from apiserver's Go zap log) cross-corroborated the whole sequence to the millisecond — this "check two independently-generated log streams agree" technique is a strong, reusable evidence-quality pattern for any future live-validation task in this spec. `GET /ping` confirmed via source read to never touch `gCtx`/`callBackend` — genuinely meaningful Req 4.2 evidence, not a coincidence. Bonus (not required by task text but strong corroboration): waited past the 10s TTL and confirmed `IsDegraded()`'s half-open auto-clear correctly re-attempts the real call. Noted, judged non-blocking: `design.md`'s sequence diagram literally says "503 service unavailable" but the actual HTTP status is 500 (`common.ERROR_CODE`) — `api-standards.md`'s error-code table has no 503 entry anywhere in this codebase's convention, and design.md's own Error Handling section explicitly disclaims introducing new error codes, so 500 + a distinct message body is the correct, convention-following choice, not a gap; the diagram itself is just imprecise.
 - Task 5.4 (final task, closing regression gate): full from-scratch rebuild + retest of the entire stack (lib/controller/scheduler/worker/xceadmin/apiserver/xcecli) found zero new regressions — same audit-first, "zero new code when genuinely covered" precedent as tasks 1.3/4.6, independently re-verified by two full review rounds (one implementer, one reviewer, both rebuilding from scratch) rather than trusted from a single run. The one known failure (`sched_admin_request_test`'s `test_recordstart_symlink_escape_replies_error`) is confirmed the same pre-existing root-in-sandbox symlink-permission issue by exact message match and source-level explanation (`driver.c`'s `-EINVAL` vs `-EACCES` classifier branches) — cross-corroborated against two independent `.wolf/memory.md` entries predating this task and a peer AI session's unprompted, independently-run sweep landing on identical numbers. The "identical single-candidate path" property (Req 1.5) is proven for all three components by real, meaningful, individually-reread-and-rerun tests (`test_build_cluster_config_single_entry_peers_matches_synthesized_shape` for worker, `test_no_peers_and_single_entry_peers_produce_identical_cluster_cfg_shape` for scheduler, `TestRegisterToBackend_NoPeersAndSingleEntryPeers_IdenticalLogSequence` for apiserver — the last one the most literal proof of the task's own "log sequence matches" wording, via genuine zap log-message-sequence equality) — no live binary-startup comparison needed since these already exercise the real production entry points end-to-end. Found (non-blocking, worth a future ticket, not this spec's job to fix): `xce-build.sh`'s `all` case runs under `set -euo pipefail`, so the one known C-test failure aborts the script before it ever reaches the Go/CGO test phases in a single invocation — and even when reached, that path only runs `make test` (`CGO_ENABLED=0`), never `make test-backend` (`CGO_ENABLED=1`), meaning the CGO backend suite (home of the load-bearing apiserver identical-log-sequence test) is structurally never exercised by `all` regardless of the known failure. Both reviewer and implementer independently worked around this by running the Go/CGO suites as separate commands — do the same for any future full-stack validation in this repo until the script itself is fixed.
 
-## Spec Complete
-All 21 tasks (1.1–5.4) implemented, independently reviewed (2 rounds where round 1 found a real issue: 3.3, 4.1, 4.5), and verified with fresh evidence via `kiro-verify-completion` before each commit. Two real production bugs were found during live E2E validation (bug-039: apiserver's idle-period rejoin never observably fires on a real failover, doubly-confirmed by tasks 5.1 and 5.2 independently; bug-040: job pushes never reach apiserver's local store, very likely a downstream symptom of bug-039) and are logged in `.wolf/buglog.json` with detailed, cross-verified root-cause analysis — both are explicitly out of this spec's own scope to fix (task 5.x is cross-component validation, not `ApiserverBackendBridge` implementation) and are flagged for a dedicated follow-up spec/task. One genuine use-after-free was found and fixed during implementation (task 4.2, apiserver's CGO connect-config lifetime). All commits are docs-only or scoped to their task's stated boundary — no scope creep across the 21-task history.
-- Task 3.3: fixed a real production gap found along the way — `sched_init()`'s two all-candidates-unreachable/join-failure log lines logged only a candidate COUNT, never host:port names (same gap worker's task 2.3 fixed for the worker). Added `sched_format_candidate_list()`, threading `sched->cluster_cfg` through as a parameter rather than a new file-scope global (unlike worker's `g_cluster_cfg`) since `test_scheduler_connect.c` links `scheduler_srcs` as separate objects, not `#include`, so the `static` helper isn't directly callable from tests — coverage requires a real `dup2()`-based stdout capture around a live `sched_init()` call hitting the failure path, not a white-box direct call (see `capture_stdout_during_sched_init()` in `test_scheduler_connect.c` — reusable pattern for any future scheduler.c log-content assertion). Round-1 review caught this diagnostic fix shipping with zero test coverage (proven by reverting the 59-line diff and rebuilding: all existing tests still passed) — a reminder that "the fix works, I verified it manually/via live logs" is not the same as a committed regression test; always add the assertion, don't just observe the correct behavior once. The worst-case-occupancy backpressure test (`test_scheduler_backpressure.c`) only proves Req 3 AC2 (buffering/block/resume-with-no-loss) — it does not exercise task 3.2's `-E_XCE_NOT_READY` retry path, so don't cite it for Req 7 coverage (that's task 3.4's job). E2E kill-primary/reconnect and all-unreachable-at-startup were validated manually against a real 3-node `/xce-deploy-ha` cluster rather than committed as an automated test, following worker's task 2.3 precedent — this repo has no automated live-multi-process-deploy E2E harness for any C daemon (the only one, `test_e2e_live_ha_push.py`, is Python/apiserver-only); reviewer judged this acceptable both rounds. Manual validation confirmed join, primary-death survival, and lazy socket-orphaning-on-idle (matches design), but could not observe a live rejoin firing against real sockets — blocked by bug-010 (`xcecd` ASAN "already joined thread" abort under `unshare -r`, needed to bypass the worker daemon's root-only check in this sandbox; unconfirmed whether it's a genuine bug or a userns artifact, never tried under real root). The exact rejoin-after-publish-success code path already has solid white-box mock coverage from task 3.2's `test_rejoin_composition_retries_across_publish_calls`, so this residual gap is narrowly "proven against mocks, not yet against a live socket" — worth closing opportunistically if bug-010 ever gets resolved, not a blocker for this spec.
+## Spec Complete (superseded below — see Phase 6)
+Tasks 1.1–5.4 (21 tasks) implemented, independently reviewed (2 rounds where round 1 found a real issue: 3.3, 4.1, 4.5), and verified with fresh evidence via `kiro-verify-completion` before each commit. Two real production bugs were found during live E2E validation (bug-1548: apiserver's idle-period rejoin never observably fires on a real failover, doubly-confirmed by tasks 5.1 and 5.2 independently; bug-1549: job pushes never reach apiserver's local store, very likely a downstream symptom of bug-1548) and were logged in `.wolf/buglog.json` with detailed, cross-verified root-cause analysis. **Correction (2026-08-15, `/kiro-validate-impl` NO-GO):** the sentence that originally stood here — calling both bugs "explicitly out of this spec's own scope to fix... flagged for a dedicated follow-up spec/task" — was wrong. bug-1548 lives in `heartbeat.go`, task 4.4's own deliverable, inside `ApiserverBackendBridge`'s declared boundary, against the already-existing Req 2 AC4/AC5 — a defect surfacing during cross-component *validation* (task 5.x) does not make it out of the *spec's* boundary, only out of that validation task's own boundary to fix in-flight. Task 5.2 was also incorrectly marked `[x]` despite explicitly not meeting its own stated observable-completion criterion (no rejoin evidence appeared during its heartbeat-only observation window) — a task must not be marked complete when its own pass bar isn't met, even when the validation work itself was rigorous and valuable. Both are now remediated as Phase 6 below, mapped to the same Req 2.4/2.5 task 4.4 was already scoped to. One genuine use-after-free was found and fixed during implementation (task 4.2, apiserver's CGO connect-config lifetime). All commits are docs-only or scoped to their task's stated boundary — no scope creep across the 21-task history.
+- Task 3.3: fixed a real production gap found along the way — `sched_init()`'s two all-candidates-unreachable/join-failure log lines logged only a candidate COUNT, never host:port names (same gap worker's task 2.3 fixed for the worker). Added `sched_format_candidate_list()`, threading `sched->cluster_cfg` through as a parameter rather than a new file-scope global (unlike worker's `g_cluster_cfg`) since `test_scheduler_connect.c` links `scheduler_srcs` as separate objects, not `#include`, so the `static` helper isn't directly callable from tests — coverage requires a real `dup2()`-based stdout capture around a live `sched_init()` call hitting the failure path, not a white-box direct call (see `capture_stdout_during_sched_init()` in `test_scheduler_connect.c` — reusable pattern for any future scheduler.c log-content assertion). Round-1 review caught this diagnostic fix shipping with zero test coverage (proven by reverting the 59-line diff and rebuilding: all existing tests still passed) — a reminder that "the fix works, I verified it manually/via live logs" is not the same as a committed regression test; always add the assertion, don't just observe the correct behavior once. The worst-case-occupancy backpressure test (`test_scheduler_backpressure.c`) only proves Req 3 AC2 (buffering/block/resume-with-no-loss) — it does not exercise task 3.2's `-E_XCE_NOT_READY` retry path, so don't cite it for Req 7 coverage (that's task 3.4's job). E2E kill-primary/reconnect and all-unreachable-at-startup were validated manually against a real 3-node `/xce-deploy-ha` cluster rather than committed as an automated test, following worker's task 2.3 precedent — this repo has no automated live-multi-process-deploy E2E harness for any C daemon (the only one, `test_e2e_live_ha_push.py`, is Python/apiserver-only); reviewer judged this acceptable both rounds. Manual validation confirmed join, primary-death survival, and lazy socket-orphaning-on-idle (matches design), but could not observe a live rejoin firing against real sockets — blocked by bug-1519 (`xcecd` ASAN "already joined thread" abort under `unshare -r`, needed to bypass the worker daemon's root-only check in this sandbox; unconfirmed whether it's a genuine bug or a userns artifact, never tried under real root). The exact rejoin-after-publish-success code path already has solid white-box mock coverage from task 3.2's `test_rejoin_composition_retries_across_publish_calls`, so this residual gap is narrowly "proven against mocks, not yet against a live socket" — worth closing opportunistically if bug-1519 ever gets resolved, not a blocker for this spec.