docs(component-failover-adoption): mark task 5.4 complete, close spec

Final regression sweep (task 5.4) confirms zero new regressions across
the full stack (lib/controller/scheduler/worker/xceadmin/apiserver/xcecli)
and proves the single-candidate path is identical to the multi-candidate
path across all three components (Req 1.5), closing out all 21 tasks
(1.1-5.4) of component-failover-adoption.

Two production bugs found during live E2E validation (5.1/5.2) remain
open and out of this spec's scope: bug-039 (apiserver idle-period rejoin
structurally unobservable on a real single-election failover) and
bug-040 (job pushes not reaching apiserver's local store, likely
downstream of bug-039).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

.kiro/specs/component-failover-adoption/tasks.md

@@ -152,7 +152,7 @@
   - _Requirements: 4.1, 4.2_
   - _Depends: 4.6_
 
-- [ ] 5.4 Regression: unified single-candidate path is observably correct across all three components
+- [x] 5.4 Regression: unified single-candidate path is observably correct across all three components
   - Run the full worker, scheduler, and apiserver test suites, plus the startup path for each component with no candidate list configured
   - Confirm each component still starts and operates normally with no candidate list configured, and that this now takes the identical code path as a validated single-entry candidate list rather than a separately-maintained fallback
   - Observable completion: all existing tests pass unchanged, and each component's no-candidate-list startup log sequence matches its single-entry-candidate-list startup log sequence
@@ -178,4 +178,8 @@
 - Task 5.1 (first Group 5 task, live 4-process E2E — worker + scheduler + apiserver + 3-node controller cluster): `xcecd_virtual` (worker's own task 2.3 meson target, `-DVIRTUAL_CLUSTER -DDBUS_MANAGER`) is the correct non-root worker E2E vehicle in this sandbox — use it, NOT `unshare -r`, which hits the unresolved bug-010 ASAN double-join abort. Worker's reconnect proved cleanly (real `checking_connection: rejoin after transparent reconnect succeeded` log line, 16s after kill, PID unchanged). Scheduler's rejoin path was never naturally exercised (no job ever reached its picker/allocator) — clean initial join + PID stability shown instead, same partial-coverage shape task 3.3 already got approved with. **Found a likely-real, structural gap in the already-approved task 4.3/4.4 apiserver rejoin mechanism (bug-039, see buglog for full detail)**: `XCECLI_FF_CALL` (backing `xcecli_heartbeat`) can silently self-heal the transport within a SINGLE heartbeat tick (rotate+recover, then the send succeeds) without ever returning a failure to Go — so `heartbeat.go`'s cross-tick `onFailureToSuccess` transition detector structurally cannot observe it when election completes faster than one heartbeat interval (elections in this codebase complete in single-digit seconds; default heartbeat interval is 30s), and none of apiserver's post-kill write calls showed evidence of hitting a failed connection either (consistent with the transport having already healed via a prior heartbeat tick before any write needed to retry). Net: `needsRejoin` may never get set on a real single-election failover, so apiserver's application-level re-join with the new primary never fires even though transport-level requests keep succeeding — risks Req 2 AC4 in production. This is NOT a task-5.1 fix (out of its cross-component-validation boundary) — flagged for a dedicated follow-up spec/task, escalated above ordinary buglog cadence given it touches an already-approved mechanism. **Also found bug-040** (job pushes never reach apiserver's store, `GET /xce/v1/jobs` empty despite controller-confirmed commits) — very likely a downstream symptom of bug-039, not independent: `on_apiserver_join()` (`apiserver.c`) only bulk-resyncs jobs when a fresh join is actually processed, and `c->apiservers[]` (the push fan-out list) is per-process in-memory state with no replication — a new primary starts with an empty registration table until each apiserver instance re-joins against it. Fixing bug-039 should be verified against bug-040 clearing as confirmation they share a root cause. Controller-side `join.c` logs only rejected/failed joins, never successful ones at any verbosity — a real observability gap for anyone debugging this family of issues in the future, worth fixing alongside.
 - Task 5.2: designed as a targeted, cleaner confirmation experiment for task 5.1's bug-039 finding (heartbeat-only, zero write traffic, isolated to apiserver alone — no worker/scheduler to confound the signal) — and it worked exactly as intended. Result: **bug-039 independently reproduced a second time, more cleanly** — a 268s+ observation window with zero rejoin log lines, AND (stronger than task 5.1) a post-window diagnostic write call that itself triggers zero rejoin activity despite succeeding against the confirmed-new primary, ruling out "task 5.1 was just unlucky timing." Reviewer independently re-derived the mechanism from source (not just citing task 5.1's analysis) and confirmed `checkAndPerformRejoin()` is called unconditionally after every backend call in `callBackend()`, so the diagnostic write's silence is a genuine negative control, not a skipped-check artifact. Controller-side `ctrl-2.log` corroborates independently — zero "apiserver" references anywhere, meaning no join attempt (success or rejection) ever reached the new primary. This task's own literal pass bar ("scenario passes without requiring any write request") was NOT met — approved anyway on the same precedent task 5.1 established: the validation itself succeeded rigorously and honestly surfaced a real gap in already-shipped code, which is the task's actual purpose. **bug-039 is now doubly-confirmed and should be treated as a real, live production gap, not a hypothesis** — a dedicated follow-up spec/task to fix `heartbeat.go`'s `onFailureToSuccess`/`onSuccessTick` wiring (make `XCECLI_FF_CALL`'s internal self-heal observable to Go, e.g. by having the heartbeat wrapper track transport state itself rather than relying on the call's return code) is warranted, separate from any remaining task in this spec's own scope. bug-040 also re-reproduced a third time in this run (same session, same isolated evidence) — buglog kept in sync on both bug-039 and bug-040's own objects (occurrences/last_seen) after review caught bug-040's object being out of sync with its own re-reproduction narrated inside bug-039's text.
 - Task 5.3: unlike 5.1/5.2 (killed only the primary, letting rotation succeed), this task needs a call to genuinely EXHAUST its retry budget — kill ALL 3 controller nodes, not just the primary. Corrected timing assumption worth remembering: `client_failover.c`'s `xce_failover_classify()` puts `ECONNREFUSED`/`ETIMEDOUT`/`ECONNRESET` (and the dead-ctx fast path) on the ZERO-backoff `XCE_FO_RETRY_NOW` track, not the ~70-100s exponential-backoff `XCE_FO_RETRY_BACKOFF` track (reserved for `E_XCE_NOMASTER`/`E_XCE_NOT_READY` responses from a reachable-but-electing controller) — so a fully-dead-cluster trigger call exhausts its 20-attempt budget in milliseconds, not minutes. Clean, strongly-corroborated PASS: exact literal string match on `ErrBackendUnavailable` (`clib.go`) proves the degraded fast-fail path genuinely skips the C call rather than the C call happening to also be fast; `E_XCE_NOT_READY`'s numeric value (10013 = `XCE_ERRNO_BASE` 10001 + 12 enum positions, `errnox.h`) independently confirmed the trigger call's `-10013` response was the real thing, not a fabricated failure; the raw C-library log (`client_failover.c`'s own WARN/error lines, a format-distinct second logging subsystem from apiserver's Go zap log) cross-corroborated the whole sequence to the millisecond — this "check two independently-generated log streams agree" technique is a strong, reusable evidence-quality pattern for any future live-validation task in this spec. `GET /ping` confirmed via source read to never touch `gCtx`/`callBackend` — genuinely meaningful Req 4.2 evidence, not a coincidence. Bonus (not required by task text but strong corroboration): waited past the 10s TTL and confirmed `IsDegraded()`'s half-open auto-clear correctly re-attempts the real call. Noted, judged non-blocking: `design.md`'s sequence diagram literally says "503 service unavailable" but the actual HTTP status is 500 (`common.ERROR_CODE`) — `api-standards.md`'s error-code table has no 503 entry anywhere in this codebase's convention, and design.md's own Error Handling section explicitly disclaims introducing new error codes, so 500 + a distinct message body is the correct, convention-following choice, not a gap; the diagram itself is just imprecise.
+- Task 5.4 (final task, closing regression gate): full from-scratch rebuild + retest of the entire stack (lib/controller/scheduler/worker/xceadmin/apiserver/xcecli) found zero new regressions — same audit-first, "zero new code when genuinely covered" precedent as tasks 1.3/4.6, independently re-verified by two full review rounds (one implementer, one reviewer, both rebuilding from scratch) rather than trusted from a single run. The one known failure (`sched_admin_request_test`'s `test_recordstart_symlink_escape_replies_error`) is confirmed the same pre-existing root-in-sandbox symlink-permission issue by exact message match and source-level explanation (`driver.c`'s `-EINVAL` vs `-EACCES` classifier branches) — cross-corroborated against two independent `.wolf/memory.md` entries predating this task and a peer AI session's unprompted, independently-run sweep landing on identical numbers. The "identical single-candidate path" property (Req 1.5) is proven for all three components by real, meaningful, individually-reread-and-rerun tests (`test_build_cluster_config_single_entry_peers_matches_synthesized_shape` for worker, `test_no_peers_and_single_entry_peers_produce_identical_cluster_cfg_shape` for scheduler, `TestRegisterToBackend_NoPeersAndSingleEntryPeers_IdenticalLogSequence` for apiserver — the last one the most literal proof of the task's own "log sequence matches" wording, via genuine zap log-message-sequence equality) — no live binary-startup comparison needed since these already exercise the real production entry points end-to-end. Found (non-blocking, worth a future ticket, not this spec's job to fix): `xce-build.sh`'s `all` case runs under `set -euo pipefail`, so the one known C-test failure aborts the script before it ever reaches the Go/CGO test phases in a single invocation — and even when reached, that path only runs `make test` (`CGO_ENABLED=0`), never `make test-backend` (`CGO_ENABLED=1`), meaning the CGO backend suite (home of the load-bearing apiserver identical-log-sequence test) is structurally never exercised by `all` regardless of the known failure. Both reviewer and implementer independently worked around this by running the Go/CGO suites as separate commands — do the same for any future full-stack validation in this repo until the script itself is fixed.
+
+## Spec Complete
+All 21 tasks (1.1–5.4) implemented, independently reviewed (2 rounds where round 1 found a real issue: 3.3, 4.1, 4.5), and verified with fresh evidence via `kiro-verify-completion` before each commit. Two real production bugs were found during live E2E validation (bug-039: apiserver's idle-period rejoin never observably fires on a real failover, doubly-confirmed by tasks 5.1 and 5.2 independently; bug-040: job pushes never reach apiserver's local store, very likely a downstream symptom of bug-039) and are logged in `.wolf/buglog.json` with detailed, cross-verified root-cause analysis — both are explicitly out of this spec's own scope to fix (task 5.x is cross-component validation, not `ApiserverBackendBridge` implementation) and are flagged for a dedicated follow-up spec/task. One genuine use-after-free was found and fixed during implementation (task 4.2, apiserver's CGO connect-config lifetime). All commits are docs-only or scoped to their task's stated boundary — no scope creep across the 21-task history.
 - Task 3.3: fixed a real production gap found along the way — `sched_init()`'s two all-candidates-unreachable/join-failure log lines logged only a candidate COUNT, never host:port names (same gap worker's task 2.3 fixed for the worker). Added `sched_format_candidate_list()`, threading `sched->cluster_cfg` through as a parameter rather than a new file-scope global (unlike worker's `g_cluster_cfg`) since `test_scheduler_connect.c` links `scheduler_srcs` as separate objects, not `#include`, so the `static` helper isn't directly callable from tests — coverage requires a real `dup2()`-based stdout capture around a live `sched_init()` call hitting the failure path, not a white-box direct call (see `capture_stdout_during_sched_init()` in `test_scheduler_connect.c` — reusable pattern for any future scheduler.c log-content assertion). Round-1 review caught this diagnostic fix shipping with zero test coverage (proven by reverting the 59-line diff and rebuilding: all existing tests still passed) — a reminder that "the fix works, I verified it manually/via live logs" is not the same as a committed regression test; always add the assertion, don't just observe the correct behavior once. The worst-case-occupancy backpressure test (`test_scheduler_backpressure.c`) only proves Req 3 AC2 (buffering/block/resume-with-no-loss) — it does not exercise task 3.2's `-E_XCE_NOT_READY` retry path, so don't cite it for Req 7 coverage (that's task 3.4's job). E2E kill-primary/reconnect and all-unreachable-at-startup were validated manually against a real 3-node `/xce-deploy-ha` cluster rather than committed as an automated test, following worker's task 2.3 precedent — this repo has no automated live-multi-process-deploy E2E harness for any C daemon (the only one, `test_e2e_live_ha_push.py`, is Python/apiserver-only); reviewer judged this acceptable both rounds. Manual validation confirmed join, primary-death survival, and lazy socket-orphaning-on-idle (matches design), but could not observe a live rejoin firing against real sockets — blocked by bug-010 (`xcecd` ASAN "already joined thread" abort under `unshare -r`, needed to bypass the worker daemon's root-only check in this sandbox; unconfirmed whether it's a genuine bug or a userns artifact, never tried under real root). The exact rejoin-after-publish-success code path already has solid white-box mock coverage from task 3.2's `test_rejoin_composition_retries_across_publish_calls`, so this residual gap is narrowly "proven against mocks, not yet against a live socket" — worth closing opportunistically if bug-010 ever gets resolved, not a blocker for this spec.