myxce/.kiro/specs/component-failover-adoption/tasks.md

Implementation Plan

• 1. Foundation: candidate-count constant reconciliation and shared conversion helper

• 1.1 Reconcile the candidate-count limit constant across libxce

  • Derive client.h's N_MAX_MASTERS from config.h's MAX_HA_CANDIDATES instead of an independent literal, so the two constants can never silently diverge again
  • Fix the two hardcoded > 3 candidate-count checks (worker's and scheduler's startup argument validation) to reference the constant symbolically instead of a magic number, so they track the raised value automatically
  • Update the client-retry test fixture that currently hardcodes exactly 3 literal host/port entries so it doesn't silently exercise an uninitialized 4th slot once the constant is raised
  • Observable completion: the project builds clean with the candidate-master array now sized to 4; the two updated startup checks and the updated test fixture all reference the same symbolic constant, not a literal
  • Requirements: 1.7

• 1.2 Implement the candidate-list-to-client-config conversion helper in libxce

  • Add a pure, socket-free helper that converts a validated candidate list into a fully-populated client config, copying candidates in order and leaving identity/auth fields for the caller to set
  • A one-candidate input is handled through the identical code path as any other count — no special-cased branch for "single controller"
  • The truncate-with-warning safety net for an over-count input stays in place as a defensive guard, even though it can no longer be reached via a properly validated input now that the two limits match
  • Observable completion: given a validated candidate list with 1, 3, and 4 candidates, the helper returns a correctly populated client config in each case, exercising the defensive truncation path only when called with an artificially invalid over-count input
  • Requirements: 1.1, 1.2, 1.3, 1.5, 1.6, 1.7
  • Depends: 1.1

• 1.3 Unit test the conversion helper and the constant reconciliation

  • Cover: candidate order preserved; one-candidate input produces a one-master output with no branch taken; truncation-with-warning exercised directly against the defensive path; NULL/empty-candidate-list rejection
  • Add a compile-time or unit-level assertion that the two candidate-count constants stay equal, guarding against reintroducing the divergence
  • Observable completion: new test cases pass under the existing libxce unit test run, including the constant-equality assertion
  • Requirements: 1.5, 1.7
  • Depends: 1.2

• 2. Worker: unified candidate-list connect and transparent reconnect

• 2.1 (P) Replace worker's dual connect mechanisms with the single unified pipeline

  • Delete the standalone client-config construction used for worker's default connection and the separate single-host branch used when no candidate list is configured — both bypass the shared candidate-list validation and conversion helper entirely today
  • Build exactly one validated candidate list before connecting, whether it came from an operator-supplied peer list or was synthesized as a single candidate from the existing default when none was given, then run it through the shared helper and connect once
  • Register a rejoin callback that only sets a flag and returns immediately — it must not call back into the client library itself, since the library invokes it while still holding its own lock
  • Reconcile the periodic liveness probe's own bounded give-up behavior (today it tears the worker's connection down for good after a handful of failed probes) with the library's own indefinite retry, so an outage longer than that old bound no longer causes the worker to give up permanently — the library now owns connection recovery
  • Observable completion: starting the worker with no candidate list configured and starting it with a single-entry candidate list produce the identical connect log sequence; a three-candidate config where the first is a live standby joins correctly, showing the standby redirect and the successful candidate; a simulated outage longer than the old give-up bound no longer tears the connection down for good; invoking the registered rejoin callback while the client's lock is held returns immediately without blocking and performs no I/O
  • Requirements: 1.1, 1.2, 1.3, 1.5, 2.1, 2.2, 2.3, 5.1, 5.2
  • Boundary: WorkerControllerConnector
  • Depends: 1.2

• 2.2 Fix worker's candidate-list startup ordering and add rejoin retry-on-failure

  • Move the candidate-list assignment to before the connector starts, so the very first connection attempt already sees the full configured list instead of only a synthesized single candidate
  • After each periodic liveness probe returns, check whether a rejoin is pending; if so, issue the rejoin call as a separate step and clear the pending indicator only if that call succeeds — on its own failure, leave the indicator set so the next liveness probe retries the rejoin rather than leaving the worker connected but never actually re-registered
  • Observable completion: starting the worker with a peer list where the first listed candidate is unreachable succeeds on the very first connect attempt, not only on a later reconnect; a rejoin call that fails once before succeeding is retried on the next liveness-probe cycle rather than abandoned
  • Requirements: 1.4, 2.5
  • Depends: 2.1

• 2.3 Validate worker thread isolation and add worker candidate/reconnect test coverage

  • Confirm the connector thread is isolated from the worker's other recurring liveness duties, and that a blocked connector thread during retry/backoff does not stall them
  • Add end-to-end coverage: worker starts and joins across a multi-candidate config; killing the joined primary results in reconnect and re-join to a surviving candidate without a worker restart
  • Add coverage for the case where every configured candidate is unreachable or rejects the join at startup: assert the worker reports a clear failure identifying every candidate it attempted, not a generic error
  • Confirm the worker's periodic liveness probe to a connected controller keeps its regular cadence and keeps succeeding even while a separate reconnect is in progress elsewhere, so a stalled reconnect attempt cannot itself look like a missed heartbeat and trigger an unwanted failover
  • Observable completion: the new test passes, and the worker's other liveness duties continue observably updating while a connect/reconnect retry is in progress on the connector thread; the all-candidates-unreachable case produces a failure message naming each attempted candidate; the liveness probe's cadence to the connected controller is unaffected by a concurrent reconnect elsewhere
  • Requirements: 1.6, 3.1, 3.3
  • Depends: 2.1, 2.2

• 3. Scheduler: unified candidate-list connect, decision-publish redelivery, and runtime reconnect

• 3.1 (P) Replace scheduler's dual connect mechanisms with the single unified pipeline

  • Delete the scheduler's inline per-candidate loop and its separate single-host branch used when no candidate list is configured — both bypass the shared candidate-list validation and conversion helper entirely today
  • Build exactly one validated candidate list before connecting, whether from an operator-supplied peer list or synthesized as a single candidate from the existing default when none was given, then run it through the shared helper and connect once
  • Register a rejoin callback that only sets a flag and returns immediately, for the same self-deadlock reason as worker's
  • Observable completion: starting the scheduler with no candidate list and with a single-entry candidate list produce the identical connect log sequence; a three-candidate config where the first is a live standby joins correctly; invoking the registered rejoin callback while the client's lock is held returns immediately without blocking and performs no I/O
  • Requirements: 1.1, 1.2, 1.3, 1.5, 1.7, 2.1, 2.2, 2.3, 5.1, 5.2
  • Boundary: SchedulerControllerClient
  • Depends: 1.2

• 3.2 Add decision-publish retry-until-success-or-shutdown, composed with rejoin retry-on-failure

  • Wrap the scheduler's per-decision publish call so that a retry-budget-exhausted outcome is retried rather than treated as final and the decision discarded — checking whether the scheduler is shutting down before each attempt, not only after, so decisions still queued behind the one actively retrying do not each burn a full retry cycle just to discover shutdown was requested
  • After a publish finally succeeds, check whether a rejoin is pending; if so, issue it as a separate step and clear the pending indicator only on that call's success
  • Log an extended-retry success distinctly from an ordinary first-attempt success
  • Observable completion: a publish call that fails repeatedly before succeeding results in the same decision retried — never dropped, never skipped to the next queued one; setting the shutdown flag mid-retry breaks the loop within one retry cycle
  • Requirements: 7.1, 7.2, 7.3, 7.4, 2.5
  • Depends: 3.1

• 3.3 Validate scheduler allocation/decision backpressure behavior and add connect/reconnect test coverage

  • Confirm the bounded, occupancy-dependent buffering behavior: the scheduler's allocation/decision processing continues briefly past the point a retry begins, then blocks until connectivity is restored and resumes automatically — assert the worst-case (least-slack) occupancy scenario explicitly, not only the most-generous one, since that is the guarantee actually being made
  • Add end-to-end coverage: scheduler starts and joins across a multi-candidate config; killing the joined primary results in reconnect and re-join without a scheduler restart
  • Add coverage for the case where every configured candidate is unreachable or rejects the join at startup: assert the scheduler reports a clear failure identifying every candidate it attempted
  • Confirm by code inspection that the scheduler makes no periodic liveness/heartbeat call of its own to a connected controller — the reconnect-retries-don't-delay-heartbeat-responses guarantee is satisfied by construction here, not by a dedicated runtime assertion, since there is no such response to delay
  • Observable completion: the new test passes, showing allocation/decision processing continuing briefly during a simulated reconnect delay, then stalling for the remainder of the outage, then resuming automatically once the connection recovers; the all-candidates-unreachable case produces a failure message naming each attempted candidate
  • Requirements: 1.6, 3.2, 3.3
  • Depends: 3.1

• 3.4 Add scheduler decision-redelivery test coverage

  • Assert decisions already queued behind a stuck one publish afterward in their original order, not out of order and not dropped
  • With several decisions queued when scheduler shutdown fires mid-retry on the first, assert total shutdown latency stays bounded to roughly one retry cycle rather than accumulating one cycle per queued decision
  • Confirm this fix is scoped to the scheduler's own decision-publishing direction only — it does not touch worker job-state resync and adds no persistence for queued-but-unpublished decisions, so a scheduler process restart mid-outage still loses whatever was queued, same as before this work
  • Observable completion: all decision-redelivery acceptance criteria are exercised by passing tests, including the bounded-shutdown-latency case and a restart-loses-in-flight-state confirmation
  • Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 2.5
  • Depends: 3.2

• 4. apiserver: unified candidate-list connect via CGO, config parity, rejoin, and fail-fast

• 4.1 (P) Route apiserver's candidate-list parsing through the real C parser

  • Replace the hand-written Go candidate-list parser with the shared C parser used by worker and scheduler, retiring the Go parser and its call sites entirely
  • Map validation errors to clear Go error messages naming the malformed entry
  • Observable completion: the same malformed-candidate-string cases worker/scheduler already reject are rejected identically when passed through apiserver's config path
  • Requirements: 6.1, 6.2
  • Boundary: ApiserverBackendBridge
  • Depends: 1.2

• 4.2 Replace apiserver's dual connect mechanisms with the single unified pipeline

  • Delete apiserver's per-peer Go connect loop and its separate direct-construction path used when no peer list is configured — both bypass the shared candidate-list validation and conversion helper entirely today
  • Build exactly one validated candidate list before connecting, whether from the configured peer list (via Task 4.1's parser) or synthesized as a single candidate from the legacy address/port fields when no peer list was given, then run it through the shared helper (via CGO) and connect once
  • Observable completion: starting apiserver with no peer list and with a single-entry peer list produce the identical connect log sequence; a three-peer config where the first is a live standby registers against the actual primary in one call
  • Requirements: 1.1, 1.2, 1.3, 1.5, 1.7, 5.1, 5.2
  • Depends: 4.1

• 4.3 Add apiserver's rejoin logic with a correctly-placed cgo registration and retry-on-failure

  • Add a flag-setter registered as the library's rejoin callback that does no I/O, only sets a pending-rejoin flag and returns immediately, and a separate function that performs the actual rejoin call
  • Register the flag-setter's C shim in the same file as its extern declaration (cgo symbol resolution is per-file)
  • Add one shared helper wrapping every backend call: after the call returns, check the pending-rejoin flag; if set, issue the rejoin as a separate step, clearing the flag only if that call succeeds — on its own failure, leave it set so the next backend call's return retries the rejoin
  • Observable completion: after killing and restoring connectivity while a write request is in flight, apiserver resumes receiving backend push notifications without a process restart; a rejoin call that fails once before succeeding is retried on the next backend call rather than abandoned; invoking the registered flag-setter while the client's lock is held returns immediately without blocking and performs no I/O
  • Requirements: 2.1, 2.2, 2.3, 2.4, 2.5
  • Depends: 4.2

• 4.4 Trigger apiserver's rejoin from the idle-period heartbeat path

  • On a transport failure-to-success transition detected by the heartbeat prober, set the same pending-rejoin flag the write path uses (the heartbeat's own probe mechanism never sets it itself)
  • On every subsequent heartbeat tick where the probe succeeds and a rejoin is still pending, retry the rejoin — checked every tick, not only on the transition edge, so a rejoin attempt that itself fails still gets retried on a later tick
  • Do not gate this retry on the separate backend-degraded circuit-breaker state, since that state clears itself on its own timer independent of whether a rejoin actually succeeded
  • Observable completion: killing the primary while apiserver has no in-flight write request still results in apiserver rejoining and resuming push notifications; a rejoin that fails on the first post-recovery tick before succeeding on a later tick confirms the retry, not just the single-attempt case
  • Requirements: 2.4, 2.5
  • Depends: 4.3

• 4.5 Add apiserver's backend-health circuit breaker

  • Add a degraded-until state that lets request handlers fail fast instead of blocking on the full retry budget when the backend is known-degraded, clearing on either a successful rejoin or its own timeout
  • Wire it into the shared backend-call helper: check before every call, mark degraded when a call's outcome indicates retry-budget exhaustion, mark healthy on a successful rejoin from either the write path or the idle-period path
  • Keep this state's own timeout independent from the pending-rejoin flag — the circuit breaker may reset optimistically on its timer, but that must never suppress the pending-rejoin flag's own retry-until-actually-joined behavior
  • Observable completion: with the state forced degraded, a wrapped backend call returns immediately with a clear unavailable error instead of invoking the backend call
  • Requirements: 4.1, 4.2
  • Depends: 4.4

• 4.6 apiserver unit test coverage

  • Cover: degraded-to-healthy transitions via both an explicit successful rejoin and timeout expiry; concurrent-request safety; candidate-list parity cases from Task 4.1
  • Observable completion: the full apiserver test suite passes with the new cases included
  • Requirements: 1.5, 4.1, 4.2, 6.1, 6.2
  • Depends: 4.5

• 5. Integration and cross-component validation

• 5.1 E2E: full-cluster reconnect after primary failover

  • Extend the project's existing HA soak pattern to a scenario with worker, scheduler, and apiserver all configured with the full candidate list; kill the primary mid-session with a write request in flight
  • Assert: worker reconnects and resumes job-state reporting; scheduler reconnects and resumes dispatch; apiserver reconnects, resumes serving API requests, and resumes receiving push notifications — none of the three restarted
  • Observable completion: the soak scenario passes with all three components showing a successful re-join to the new primary in their logs and no manual intervention
  • Requirements: 2.1, 2.2, 2.3, 2.4
  • Depends: 2.3, 3.3, 3.4, 4.6

• 5.2 E2E: apiserver reconnects during an idle period

  • Kill the primary while apiserver has only its heartbeat prober active, no pending write requests; assert apiserver still rejoins and resumes push notifications via the heartbeat-triggered path
  • Observable completion: the scenario passes without requiring any client-triggered write request during the outage window
  • Requirements: 2.4, 2.5
  • Depends: 4.6

• 5.3 E2E: apiserver fails fast while backend is degraded, and keeps serving requests that don't need it

  • With apiserver's backend connection killed and not yet reconnected, issue an API request that requires the backend and assert it returns a clear service-unavailable error within a bounded, sub-second time — not after the full retry budget
  • In the same degraded window, issue a request that does not require the backend connection and assert it still succeeds normally, confirming degradation is scoped to backend-dependent requests only
  • Observable completion: both timed assertions pass against the same live apiserver instance while it remains degraded
  • Requirements: 4.1, 4.2
  • Depends: 4.6

• 5.4 Regression: unified single-candidate path is observably correct across all three components

  • Run the full worker, scheduler, and apiserver test suites, plus the startup path for each component with no candidate list configured
  • Confirm each component still starts and operates normally with no candidate list configured, and that this now takes the identical code path as a validated single-entry candidate list rather than a separately-maintained fallback
  • Observable completion: all existing tests pass unchanged, and each component's no-candidate-list startup log sequence matches its single-entry-candidate-list startup log sequence
  • Requirements: 1.5
  • Depends: 5.1, 5.2, 5.3

6. Post-validation remediation (bug-1548/bug-1549, NO-GO 2026-08-15)

• 6.1 Fix apiserver's idle-period rejoin detection gap

  • Add a monotonic reconnect-generation counter to the client library's endpoint struct, incremented on every successful transport reconnect regardless of trigger path
  • Replace the heartbeat prober's failure-to-success transition check with a per-tick comparison against this generation counter, triggering the same pending-rejoin flag on any change
  • Observable completion: a new regression test proves the pending-rejoin flag is set on a generation change even when every heartbeat tick the Go loop observes reports success (the exact shape XCECLI_FF_CALL's silent inline reconnect produces) — its own doc comment states this proves detector logic, not reachability
  • Requirements: 2.4, 2.5
  • Depends: 4.4

• 6.2 E2E re-validation: re-run task 5.2's idle-period scenario against the fix

  • Re-run the exact task 5.2 scenario (apiserver only, no worker/scheduler, 3-node controller cluster, default heartbeat interval, primary killed with zero write traffic during the observation window) against the fixed code
  • Observable completion: identical to task 5.2's own bar — rejoin log lines appear within the observation window, and GET /xce/v1/jobs populates correctly after a post-recovery submit, with no client-triggered write request required
  • Requirements: 2.4, 2.5
  • Depends: 6.1

Implementation Notes (Phase 6)

• Task 6.1: fix is a monotonic endpoint_t.reconnect_gen counter (src/include/netio.h), bumped once, unconditionally, on the success path of xce_ctx_reconnect() (src/lib/xcectx.c) — confirmed the sole reconnect call-site funnel for both the write-path retry loop and the heartbeat fire-and-forget path, so it observes every reconnect regardless of trigger. Read from Go via the existing, previously-unused xcecli_get_my_endpoint() accessor (client.h) — no new C getter needed, and none of the frozen mechanism files (client.c/client_internal.h/client_failover.c/client_failover.h) were touched. heartbeat.go's runHeartbeat() fully replaced its prevFailed failure-to-success transition detector with a per-tick generation comparison (user's explicit direction: no compatibility shim needed, pick the objectively better approach) — the old detector is logically a strict subset of the new one under this codebase's own documented state machine (ctx->state == XCE_CTX_ERR is the single known-dead signal gating XCECLI_FF_CALL's reconnect branch), and the new one additionally catches a reconnect triggered by a concurrent write-path call between heartbeat ticks. New regression tests (heartbeat_test.go, heartbeat_rejoin_test.go) prove the detector fires on a generation change even when every observed tick reports success — the literal shape of bug-1548 that the old detector's own green test suite could never catch.
• Task 6.2: live E2E re-run of task 5.2's exact scenario (apiserver alone, no worker/scheduler, 3-node cluster, default 30s heartbeat interval, primary killed, zero write traffic during the observation window) — PASS, closing the gap. Primary killed 14:13:46 UTC; new primary elected 14:13:56 UTC (10s); checkAndPerformRejoin: rejoin after transparent reconnect succeeded logged at 14:14:08 UTC (22s after the kill, 12s after the election), with zero write/backend traffic in the preceding 5.5 minutes — the heartbeat path alone triggered it, exactly the case that previously showed zero rejoin evidence across a 268s+ window. bug-1549 downstream symptom also confirmed resolved: a diagnostic job submitted after recovery correctly appeared via GET /xce/v1/jobs (count:1, matching job data), proving apiserver's push/registration state genuinely recovered with the new primary, not just the transport.
• Non-blocking tooling fixes made alongside (not spec tasks — xce-build.sh/the frontend Makefile are general repo tooling, not spec-owned files per design.md's File Structure Plan; tracked via buglog only): bug-1559 (xce-build.sh's scheduler case now runs meson test --suite scheduler, exercising all 41 scheduler test binaries instead of just sched_test directly by path) and bug-1558 (make test-backend now self-derives CGO_CFLAGS/CGO_LDFLAGS via pkg-config, works standalone with zero manual env setup — one iteration needed: the first attempt's -I../../include was relative and silently resolved wrong under cgo's actual build cwd, fixed to an absolute -I$(CURDIR)/../include).

Implementation Notes

• Task 1.1: src/lib/cluster.c:99-102's xce_cluster_config_validate() still has a third hardcoded >3 WARN check (message says "<=3 only") of the same magic-number class Req 1.7 targets, and src/include/xce/cluster.h:31's doc comment repeats the stale "3" figure. Task 1.1's text named only "the two" worker/scheduler checks, so this was left untouched there — task 1.2 already touches cluster.c to add the new conversion helper, so fold this fix in there rather than letting it silently persist as the one remaining un-reconciled magic number.
• Task 1.1 follow-up resolved in 1.2: the >3→>MAX_HA_CANDIDATES rename initially proposed for xce_cluster_config_validate()'s WARN check turned out to be dead code — after task 1.1's constant reconciliation, that check tests the identical field against the identical constant as an earlier -EINVAL return in the same function, so it can never fire. Deleted rather than renamed (with the matching stale doc-comment line also removed), caught by task 1.2's review round 1.
• Task 1.2: its 8 new tests in test_cluster.c substantively cover task 1.3's stated checklist already (candidate order, one-candidate-no-branch, truncation-with-warning, NULL/empty rejection) — task 1.3 should scope down to just the constant-equality assertion (N_MAX_MASTERS == MAX_HA_CANDIDATES) rather than duplicating this coverage.
• Task 1.3: closed with zero new code. Verified fresh (source read + a clean xce-build.sh lib asan run) that every item in its checklist was already satisfied by prior tasks: the _Static_assert(N_MAX_MASTERS == MAX_HA_CANDIDATES, ...) landed in task 1.1 (test_libxce_retry.c:44), and the four behavioral cases (order/no-branch/truncation/NULL-empty) landed in task 1.2's 8 tests (test_cluster.c). No implementer/reviewer cycle was dispatched since there was no diff to review — dispatching one would have either duplicated existing test names (build conflict) or been a no-op review of an empty diff.
• Task 2.1: xce_cluster_config_to_client_config() (task 1.2) aliases out->masters[i] directly to cfg->candidates[i].hostname — no strdup. Any caller building the source xce_cluster_config_t on the stack and passing it through the helper then using the resulting client config beyond that stack frame's lifetime has a use-after-scope bug. Worker's fix keeps the source config in a file-scope static for this reason; tasks 3.1 (scheduler) and 4.2 (apiserver) hit the identical hazard and must do the same (or otherwise guarantee the source config outlives the client config's use).
• Task 2.2: fixed the startup-ordering bug by threading cluster_cfg by value through xcecd_config_t → xcecd_context_t → cc_ctx (set in main.c before xcecd_init()/xcecd_start() run, instead of after). Safe as a plain struct copy because xce_candidate_t.hostname is a fixed char[] array, not a pointer — no aliasing hazard here (unlike the xce_cluster_config_to_client_config() output struct noted above). Scheduler (3.1) likely has the same ordering gap (sched_init loops candidates correctly already per design.md, so may not need this fix — verify against current source, don't assume). Rejoin-retry pattern established: checking_connection() checks needs_rejoin after every probe return, calls xcecli_join() directly (not full re-entry into the join-cluster function), clears the flag only on success — same pattern scheduler/apiserver rejoin tasks (3.1, 4.3) should follow.
• Task 2.3 (real E2E caught what unit tests missed — 2 production bugs slipped through tasks 2.1/2.2's approved reviews): (1) xcecli_start() (src/lib/client.c) unconditionally resets ctx->failover.rejoin_cb = NULL as part of its own init — any xcecli_set_reconnect_handler() call before xcecli_start() is silently wiped. Register the reconnect handler AFTER xcecli_start() succeeds, not before. Scheduler/apiserver's own reconnect-handler registration (3.1, 4.3) must follow this same ordering or they'll hit the identical silent-wipe bug. (2) A successful xcecli_join()/worker join returns the assigned slot index as reply.status (any rc >= 0, not always exactly 0) — checking rc == 0 for success is wrong; use rc < 0 as the only failure test (matches xcecd_join_cluster()'s own established convention). Any new code checking a join/rejoin call's success must use this same rc < 0 convention, not rc == 0. (3) Security lesson: an env-var runtime bypass of a security check (XCECD_ALLOW_NONROOT) added for test/E2E convenience was rejected on review — the correct pattern for skipping a production-only check (root privilege, etc.) in a test/virtual build is a compile-time guard against the project's existing VIRTUAL_CLUSTER or XCE_TESTING flags, never a runtime environment variable reachable in the shipped production binary.
• Task 3.2: implemented as a shared sched_publish_retry_loop()/sched_publish_check_rejoin() pair called from both sched_publish_dispatch() and sched_publish_pend_reason(), not literally inlined into sched_publish_dispatch() alone as design.md's prose names it — reviewed and judged compliant (same thread, fires once after the whole retry loop's successful return, not per-attempt, not in sched_allocator). Testing lesson for any future scheduler.c unit test needing to intercept xcecli_* calls: xcecli_jobdispatch/xcecli_set_pend_reason/xcecli_join/xcecli_start/xcecli_set_reconnect_handler all live in the single client.c translation unit inside libxce.a alongside symbols scheduler.c genuinely needs — the "test double linked ahead of the archive" trick (used by test_driver_decision_cb.c one layer up) causes duplicate-definition errors here. Use -Wl,--wrap=<symbol> linker interposition instead (see test_scheduler_decision_publish_retry.c/its meson.build target) when a test needs scheduler.c itself linked live alongside intercepted libxce calls. Also confirmed: sched_admin_request_test's test_recordstart_symlink_escape_replies_error fails independent of this spec's changes (its white-box #include list doesn't pull in scheduler.c at all) — a pre-existing, unrelated, environment-specific (root-in-sandbox symlink-permission) failure; do not re-investigate it as a regression in later tasks.
• Task 4.1: first apiserver-group (Go+CGO) task. xce_parse_peers() alone (src/lib/cluster.c) does syntax parsing only — no semantic checks (accepts port 0, non-numeric port via silent atoi()-to-0, empty host). Worker/scheduler always pair it with a second xce_cluster_config_validate() call in their main.cs; round-1 review caught parseCandidates() (candidates.go) skipping that second call, silently accepting malformed candidates worker/scheduler reject — a reminder that "the C function I called is correct" isn't the same as "I called every C function the precedent calls." Fix: call xce_cluster_config_validate(&cCfg) after xce_parse_peers(), stamping a local placeholder into cCfg.identity first since that function also checks a non-empty top-level identity (unrelated to per-candidate validation, never propagated to the real connection identity elsewhere in clib.go). xce_cluster_config_validate() checks hostname/port/top-level identity only, never per-candidate name — so the name= prefix stays legitimately optional. Established convention confirmed pre-existing (not invented this task): a //go:build cgo test file with no import "C" of its own, pairing with a plain-Go-signature function in a separate cgo-preamble file (_test.go files can never import "C" themselves — a Go-toolchain restriction, not just a project style choice) — see wire_dispatch_test.go/wire_dispatch_support.go for the pre-existing precedent this task's candidates_test.go/candidates.go split follows. Task 4.1 is parsing-only by design — RegisterToBackend()'s connect loop and tryConnect() stay untouched; task 4.2 is the one that collapses them into a single connect call, so don't expect this diff to touch that territory. Behavior change (deliberate, disclosed, not a regression): a malformed entry now rejects the WHOLE candidate list (matches worker/scheduler's all-or-nothing precedent, xce_parse_peers() has no partial-success mode by construction) instead of the old Go parser's skip-with-warning leniency — surfaces via RegisterToBackend's error → cmd/main.go's logger.Error+os.Exit(1), loud not silent. Tooling gap for future apiserver tasks (4.2-4.6): make test-backend's documented CGO env-var setup doesn't actually export CGO_CFLAGS/CGO_LDFLAGS to a general shell (they only exist as meson custom_target env vars) — reconstruct via pkg-config against the deps apiserver/meson.build lists, or run the full xce-build.sh apiserver asan (rebuilds libxce.a fresh too) rather than a bare go test.
• Task 4.2: found and fixed a genuine, independently-verified use-after-free while collapsing apiserver's two connect mechanisms into one — the deleted tryConnect() did C.CString(host) then defer C.free(cAddr) right after its single xce_backend_init() call returned, but xcecli_ctx_init() stores ctx->conf = cfg as a raw pointer (src/lib/client.c:38, never copied), and client_failover.c's rotation path (xcecli_retry_next) re-dereferences ac->conf->masters[cur] on every failover attempt, not just at init — so any reconnect after the first would read freed heap memory. (from/key were NOT part of this bug — xce_ctx_init() strdup()s those synchronously before the defer ever fires.) Fix mirrors worker's/scheduler's own file-scope-static pattern for the Go/CGO side: two package-level vars (gClusterCfg C.xce_cluster_config_t, gClientCfg C.xce_client_config_t, alongside the pre-existing gCtx/gIDCtx) give the CGO structs a stable, process-lifetime address, and the connect path now writes host/from/key into fixed-size C arrays (writeCCharArray()) instead of C.CString/C.free-ing heap pointers the C library retains forever. General lesson for any future CGO code in this codebase that hands a struct to a C API storing a bare pointer to it (check the C function's own source, not just its header, for whether it copies or aliases): a Go function-local var is NOT safe once the call returns unless the C side genuinely copies everything reachable from it — verify via source read, the same standard task 2.1's original aliasing-hazard note set for the C-side daemons. parseCandidates() (task 4.1's deliverable) is now production-dead code — buildClusterConfig() calls xce_parse_peers() directly into the package-level gClusterCfg instead, since routing through parseCandidates()'s Go-slice return would mean re-parsing/re-copying into a fresh C struct that doesn't have the required stable address; parseCandidates() is exercised only by its own tests now, worth a follow-up cleanup task if it stays unused.
• Task 4.3: rejoin flag-setter (go_backend_rejoin_handler), its C shim, extern declaration, and xcecli_set_reconnect_handler() registration call all live together in clib.go's xce_backend_init() — confirms design.md's "cgo symbol resolution is per-file" constraint (see wire_dispatch_support.go's own documented rationale): a C.<exported-Go-fn> reference only resolves against an extern declared in THAT file's own cgo preamble, even though the Go //export mechanism makes the function callable package-wide — splitting the shim into a separate reconnect.go preamble while the extern lived in clib.go (or vice versa) would silently fail to link. Registration placed strictly AFTER xcecli_start()'s success check (task 2.3's ordering lesson ported to Go: xcecli_start() unconditionally wipes rejoin_cb to NULL as part of its own init). needsRejoin is a proper sync/atomic flag, not a plain bool — apiserver serves concurrent HTTP requests (unlike worker's/scheduler's single-threaded daemons), so this is a real correctness requirement here, not just style. checkAndPerformRejoin() wired into all 5 gCtx.ctx-based backend-call wrappers (SubmitJob/ControlJob/ControlQueue/ControlHost/FetchFile) — note xcecli_fetchfile() itself isn't XCECLI_FAILOVER_CALL-wrapped in libxce (never triggers the C-side rejoin invoke), so wrapping it here is safe but less load-bearing than the other four; that's a libxce gap, out of this task's ApiserverBackendBridge boundary to fix. Reviewer found (non-blocking, correctness-safe): concurrent goroutines can each observe needsRejoin==1 after a failover and each call the real xcecli_join(), serialized by the C library's own per-ctx lock (no corruption, no incorrect flag state) but each can block up to the ~100s retry budget on a redundant join — a latency/traffic concern, not a Requirement-2 violation; a CAS-based single-flight guard is a reasonable fast-follow, not required. Scope reminder for task 4.4/4.5: ApiserverBackendHealth/IsDegraded()/MarkHealthy() deliberately do not exist yet — checkAndPerformRejoin() is the POST-call half only; task 4.5 adds a pre-call check to the same helper, not a new one.
• Task 4.4: thin adapter, not a second rejoin mechanism — runHeartbeat()'s new onFailureToSuccess/onSuccessTick callback params are wired directly to task 4.3's real setNeedsRejoin/checkAndPerformRejoin (reconnect.go) from StartHeartbeat(), zero duplicated flag-check logic added in heartbeat.go itself. Necessary because xcecli_heartbeat() is XCECLI_FF_CALL-wrapped (client_internal.h), a deliberately non-retrying fire-and-forget path that does its own best-effort one-shot reconnect but NEVER calls xcecli_invoke_rejoin() (that only fires from xcecli_retry_next, the write-call retry path) — so nothing sets needsRejoin on a pure idle-period recovery unless the heartbeat loop does it itself. rc >= 0 success classification for xcecli_heartbeat confirmed against xceapi.c/request.c: it only ever returns 0 or a negative errno, never a positive value, so this is equivalent to (not just consistent with) the rc<0-only-failure convention used elsewhere. onSuccessTick fires unconditionally on every successful tick (no pending-check duplicated in heartbeat.go) — safe because checkAndPerformRejoin() is itself a cheap single-atomic-load no-op when nothing is pending. prevFailed (transition-tracking state) is correctly a plain non-atomic local, not sync/atomic — verified runHeartbeat() has exactly one call site (StartHeartbeat(), called once from cmd/main.go), one goroutine for the process lifetime; contrast with needsRejoin itself, which stays atomic since concurrent HTTP-handler goroutines reach it via the write-path wrappers. Known, accepted, already-documented (task 4.3) tradeoff carried forward: a rejoin attempt from the heartbeat goroutine shares ac->failover.lock with concurrent HTTP-handler backend calls, so one can block the other for up to the ~1-2 minute retry budget — not a new issue this task introduces, same mechanism task 4.3 already reviewed and accepted, task 4.4 just adds one more caller of it.
• Task 3.4: pure test-coverage task, zero production changes to scheduler.c/driver.c (correct outcome — task 3.2 already implemented the retry/shutdown/rejoin behavior correctly; 3.4 proves it holds at multi-decision QUEUE granularity through the real drain loop, which task 3.2's own tests never exercised, all 6 of which call sched_publish_dispatch()/sched_publish_pend_reason() directly on one decision). New test_scheduler_decision_redelivery.c links real scheduler_srcs with -Wl,--wrap=xcecli_jobdispatch (task 3.2's established technique) and drives the real xce_sched_driver_on_decision() (driver.c) over 3 queued, distinct-jobid decisions. Two load-bearing tests, both RED→GREEN-verified against the real code by two independent reviewer rounds (not just the implementer's claim): ordering (decision 1 stuck-then-succeeds fully completes before decision 2's call ever fires) and bounded shutdown latency (decisions queued behind one that triggers core.stop mid-retry get ZERO publish calls — the concrete, fast, deterministic proof that sched_publish_retry_loop()'s check-stop-before-each-attempt placement is correct, verified by reviewers moving the check to bottom-of-loop and confirming only this one test catches it). Test-fixture gotcha for any future test building a bare xce_sched_core_t via xce_sched_core_init() alone (not full sched_init()): do NOT call xce_sched_core_fini() on it — fini() calls dataq_fini() on pend_updates[0/1], which only sched_init() (not xce_sched_core_init()) ever dataq_init()s, so fini() on a bare core SEGVs on a NULL dataq; omit the fini() call entirely, matching test_core_events.c's established "fixtures aren't torn down" convention for this exact minimal-fixture pattern.
• Task 4.5: subtle design question resolved and worth remembering for task 4.6/5.x — callBackend()'s (clib.go) degraded fast-path deliberately skips checkAndPerformRejoin() ENTIRELY (not just the underlying C.xcecli_* call), meaning write-path requests contribute NOTHING to rejoin-retry cadence while IsDegraded() is true; only task 4.4's heartbeat goroutine keeps retrying during that window. Reviewer independently confirmed this is correct, not a violation of "must never suppress the pending-rejoin flag's own retry-until-actually-joined behavior": that constraint is one-directional (the circuit breaker's own optimistic TTL-based state must never be READ BY the rejoin mechanism to gate it — confirmed checkAndPerformRejoin() never references IsDegraded() anywhere), not a requirement that every possible call site attempt a rejoin check regardless of cost. The alternative (calling checkAndPerformRejoin(), which can itself block ~100s inside xcecli_join()'s own retry budget, on every fast-failed request) would directly violate THIS task's own fail-fast acceptance criterion — the two constraints (fail-fast + never-suppress-retry) are jointly satisfiable only because heartbeat.go's goroutine runs unconditionally for the process lifetime, independent of circuit-breaker state, and backendDegradedTTL (10s, chosen well below the ~100s XCECLI_FAILOVER_CALL budget) is shorter than the heartbeat interval (default 30s) so write traffic resumes contributing once the TTL clears anyway. callBackend() is a genuine refactor consolidating what tasks 4.3/4.4 left as duplicated inline rc := C.xcecli_*(...); checkAndPerformRejoin() pairs across 5 wrapper functions into one shared helper — checkAndPerformRejoin() now appears exactly once in clib.go, inside callBackend, matching design.md's explicit "not copy-pasted per wrapper" instruction. E_XCE_NOT_READY-triggered MarkDegraded() uses a plain-Go notReadyRC mirror var (= -int(C.E_XCE_NOT_READY), computed live via cgo, not hardcoded) since _test.go files can't reference C.* symbols directly (task 4.1's established Go-toolchain restriction) — keep this pattern for any future test needing a C errno constant.
• Task 4.6 (closes out apiserver group 4.1-4.6): audit task, same "zero new code is a valid outcome when genuinely true" precedent as scheduler's task 1.3 — 2 of 3 coverage areas (degraded-to-healthy transitions, candidate-list parity) were already fully satisfied by tasks 4.1/4.5's own tests, verified by reading the cited tests directly rather than trusting prior notes' claims. Found and closed exactly one genuine gap: reconnect.go's needsRejoin/checkAndPerformRejoin (task 4.3) had zero concurrent-access test coverage despite being reached from 5 real concurrent HTTP-wrapper call sites plus the heartbeat goroutine — health.go's own 50-goroutine stress test (task 4.5) only covered the degraded-until timestamp, not this separate atomic flag. One new test closes it. gClusterCfg/gClientCfg (task 4.2's CGO structs) correctly judged NOT a concurrency gap — written once at startup, single-threaded, before the HTTP server starts, never mutated again. Confirmed apiserver's Go-level xce_parse_peers() malformed-string coverage (candidates_test.go) is broader than any C-side component's own test — worker's/scheduler's C tests, and even test_cluster.c, only exercise xce_cluster_config_validate() against pre-built structs, never xce_parse_peers() against malformed raw strings at all. go test -race is confirmed, again, flatly unusable in this ASAN-linked CGO environment ("too many address space collisions") — this is now the third task (4.2, 4.5, 4.6) to independently hit and document this; don't waste time trying -race here, rely on atomic-correctness-by-construction plus deterministic concurrent-stress assertions instead. xce-build.sh apiserver asan's binary-compile step can hit transient goproxy.cn network flakiness unrelated to any code change — when that happens, fall back to cd src/frontend && make test (CGO_ENABLED=0, whole apiserver module + xcecli) and make test-backend (CGO_ENABLED=1, ./backend/...) directly, which is what the full pipeline's own RUN_TESTS=1 branch invokes anyway once binaries are built. A pre-existing, unrelated apiserver/controllers build failure under a whole-module CGO_ENABLED=1 go test ./... (predates this spec entirely, integration_test.go's //go:build !cgo tag vs. output_test.go's unconditional references to its helpers) doesn't affect either authoritative Makefile target — don't chase it as a regression in future apiserver work.
• Task 5.1 (first Group 5 task, live 4-process E2E — worker + scheduler + apiserver + 3-node controller cluster): xcecd_virtual (worker's own task 2.3 meson target, -DVIRTUAL_CLUSTER -DDBUS_MANAGER) is the correct non-root worker E2E vehicle in this sandbox — use it, NOT unshare -r, which hits the unresolved bug-1519 ASAN double-join abort. Worker's reconnect proved cleanly (real checking_connection: rejoin after transparent reconnect succeeded log line, 16s after kill, PID unchanged). Scheduler's rejoin path was never naturally exercised (no job ever reached its picker/allocator) — clean initial join + PID stability shown instead, same partial-coverage shape task 3.3 already got approved with. Found a likely-real, structural gap in the already-approved task 4.3/4.4 apiserver rejoin mechanism (bug-1548, see buglog for full detail): XCECLI_FF_CALL (backing xcecli_heartbeat) can silently self-heal the transport within a SINGLE heartbeat tick (rotate+recover, then the send succeeds) without ever returning a failure to Go — so heartbeat.go's cross-tick onFailureToSuccess transition detector structurally cannot observe it when election completes faster than one heartbeat interval (elections in this codebase complete in single-digit seconds; default heartbeat interval is 30s), and none of apiserver's post-kill write calls showed evidence of hitting a failed connection either (consistent with the transport having already healed via a prior heartbeat tick before any write needed to retry). Net: needsRejoin may never get set on a real single-election failover, so apiserver's application-level re-join with the new primary never fires even though transport-level requests keep succeeding — risks Req 2 AC4 in production. This is NOT a task-5.1 fix (out of its cross-component-validation boundary) — flagged for a dedicated follow-up spec/task, escalated above ordinary buglog cadence given it touches an already-approved mechanism. Also found bug-1549 (job pushes never reach apiserver's store, GET /xce/v1/jobs empty despite controller-confirmed commits) — very likely a downstream symptom of bug-1548, not independent: on_apiserver_join() (apiserver.c) only bulk-resyncs jobs when a fresh join is actually processed, and c->apiservers[] (the push fan-out list) is per-process in-memory state with no replication — a new primary starts with an empty registration table until each apiserver instance re-joins against it. Fixing bug-1548 should be verified against bug-1549 clearing as confirmation they share a root cause. Controller-side join.c logs only rejected/failed joins, never successful ones at any verbosity — a real observability gap for anyone debugging this family of issues in the future, worth fixing alongside.
• Task 5.2: designed as a targeted, cleaner confirmation experiment for task 5.1's bug-1548 finding (heartbeat-only, zero write traffic, isolated to apiserver alone — no worker/scheduler to confound the signal) — and it worked exactly as intended. Result: bug-1548 independently reproduced a second time, more cleanly — a 268s+ observation window with zero rejoin log lines, AND (stronger than task 5.1) a post-window diagnostic write call that itself triggers zero rejoin activity despite succeeding against the confirmed-new primary, ruling out "task 5.1 was just unlucky timing." Reviewer independently re-derived the mechanism from source (not just citing task 5.1's analysis) and confirmed checkAndPerformRejoin() is called unconditionally after every backend call in callBackend(), so the diagnostic write's silence is a genuine negative control, not a skipped-check artifact. Controller-side ctrl-2.log corroborates independently — zero "apiserver" references anywhere, meaning no join attempt (success or rejection) ever reached the new primary. This task's own literal pass bar ("scenario passes without requiring any write request") was NOT met — approved anyway on the same precedent task 5.1 established: the validation itself succeeded rigorously and honestly surfaced a real gap in already-shipped code, which is the task's actual purpose. bug-1548 is now doubly-confirmed and should be treated as a real, live production gap, not a hypothesis — a dedicated follow-up spec/task to fix heartbeat.go's onFailureToSuccess/onSuccessTick wiring (make XCECLI_FF_CALL's internal self-heal observable to Go, e.g. by having the heartbeat wrapper track transport state itself rather than relying on the call's return code) is warranted, separate from any remaining task in this spec's own scope. bug-1549 also re-reproduced a third time in this run (same session, same isolated evidence) — buglog kept in sync on both bug-1548 and bug-1549's own objects (occurrences/last_seen) after review caught bug-1549's object being out of sync with its own re-reproduction narrated inside bug-1548's text.
• Task 5.3: unlike 5.1/5.2 (killed only the primary, letting rotation succeed), this task needs a call to genuinely EXHAUST its retry budget — kill ALL 3 controller nodes, not just the primary. Corrected timing assumption worth remembering: client_failover.c's xce_failover_classify() puts ECONNREFUSED/ETIMEDOUT/ECONNRESET (and the dead-ctx fast path) on the ZERO-backoff XCE_FO_RETRY_NOW track, not the ~70-100s exponential-backoff XCE_FO_RETRY_BACKOFF track (reserved for E_XCE_NOMASTER/E_XCE_NOT_READY responses from a reachable-but-electing controller) — so a fully-dead-cluster trigger call exhausts its 20-attempt budget in milliseconds, not minutes. Clean, strongly-corroborated PASS: exact literal string match on ErrBackendUnavailable (clib.go) proves the degraded fast-fail path genuinely skips the C call rather than the C call happening to also be fast; E_XCE_NOT_READY's numeric value (10013 = XCE_ERRNO_BASE 10001 + 12 enum positions, errnox.h) independently confirmed the trigger call's -10013 response was the real thing, not a fabricated failure; the raw C-library log (client_failover.c's own WARN/error lines, a format-distinct second logging subsystem from apiserver's Go zap log) cross-corroborated the whole sequence to the millisecond — this "check two independently-generated log streams agree" technique is a strong, reusable evidence-quality pattern for any future live-validation task in this spec. GET /ping confirmed via source read to never touch gCtx/callBackend — genuinely meaningful Req 4.2 evidence, not a coincidence. Bonus (not required by task text but strong corroboration): waited past the 10s TTL and confirmed IsDegraded()'s half-open auto-clear correctly re-attempts the real call. Noted, judged non-blocking: design.md's sequence diagram literally says "503 service unavailable" but the actual HTTP status is 500 (common.ERROR_CODE) — api-standards.md's error-code table has no 503 entry anywhere in this codebase's convention, and design.md's own Error Handling section explicitly disclaims introducing new error codes, so 500 + a distinct message body is the correct, convention-following choice, not a gap; the diagram itself is just imprecise.
• Task 5.4 (final task, closing regression gate): full from-scratch rebuild + retest of the entire stack (lib/controller/scheduler/worker/xceadmin/apiserver/xcecli) found zero new regressions — same audit-first, "zero new code when genuinely covered" precedent as tasks 1.3/4.6, independently re-verified by two full review rounds (one implementer, one reviewer, both rebuilding from scratch) rather than trusted from a single run. The one known failure (sched_admin_request_test's test_recordstart_symlink_escape_replies_error) is confirmed the same pre-existing root-in-sandbox symlink-permission issue by exact message match and source-level explanation (driver.c's -EINVAL vs -EACCES classifier branches) — cross-corroborated against two independent .wolf/memory.md entries predating this task and a peer AI session's unprompted, independently-run sweep landing on identical numbers. The "identical single-candidate path" property (Req 1.5) is proven for all three components by real, meaningful, individually-reread-and-rerun tests (test_build_cluster_config_single_entry_peers_matches_synthesized_shape for worker, test_no_peers_and_single_entry_peers_produce_identical_cluster_cfg_shape for scheduler, TestRegisterToBackend_NoPeersAndSingleEntryPeers_IdenticalLogSequence for apiserver — the last one the most literal proof of the task's own "log sequence matches" wording, via genuine zap log-message-sequence equality) — no live binary-startup comparison needed since these already exercise the real production entry points end-to-end. Found (non-blocking, worth a future ticket, not this spec's job to fix): xce-build.sh's all case runs under set -euo pipefail, so the one known C-test failure aborts the script before it ever reaches the Go/CGO test phases in a single invocation — and even when reached, that path only runs make test (CGO_ENABLED=0), never make test-backend (CGO_ENABLED=1), meaning the CGO backend suite (home of the load-bearing apiserver identical-log-sequence test) is structurally never exercised by all regardless of the known failure. Both reviewer and implementer independently worked around this by running the Go/CGO suites as separate commands — do the same for any future full-stack validation in this repo until the script itself is fixed.

Spec Complete (superseded below — see Phase 6)

Tasks 1.1–5.4 (21 tasks) implemented, independently reviewed (2 rounds where round 1 found a real issue: 3.3, 4.1, 4.5), and verified with fresh evidence via kiro-verify-completion before each commit. Two real production bugs were found during live E2E validation (bug-1548: apiserver's idle-period rejoin never observably fires on a real failover, doubly-confirmed by tasks 5.1 and 5.2 independently; bug-1549: job pushes never reach apiserver's local store, very likely a downstream symptom of bug-1548) and were logged in .wolf/buglog.json with detailed, cross-verified root-cause analysis. Correction (2026-08-15, /kiro-validate-impl NO-GO): the sentence that originally stood here — calling both bugs "explicitly out of this spec's own scope to fix... flagged for a dedicated follow-up spec/task" — was wrong. bug-1548 lives in heartbeat.go, task 4.4's own deliverable, inside ApiserverBackendBridge's declared boundary, against the already-existing Req 2 AC4/AC5 — a defect surfacing during cross-component validation (task 5.x) does not make it out of the spec's boundary, only out of that validation task's own boundary to fix in-flight. Task 5.2 was also incorrectly marked [x] despite explicitly not meeting its own stated observable-completion criterion (no rejoin evidence appeared during its heartbeat-only observation window) — a task must not be marked complete when its own pass bar isn't met, even when the validation work itself was rigorous and valuable. Both are now remediated as Phase 6 below, mapped to the same Req 2.4/2.5 task 4.4 was already scoped to. One genuine use-after-free was found and fixed during implementation (task 4.2, apiserver's CGO connect-config lifetime). All commits are docs-only or scoped to their task's stated boundary — no scope creep across the 21-task history.

• Task 3.3: fixed a real production gap found along the way — sched_init()'s two all-candidates-unreachable/join-failure log lines logged only a candidate COUNT, never host:port names (same gap worker's task 2.3 fixed for the worker). Added sched_format_candidate_list(), threading sched->cluster_cfg through as a parameter rather than a new file-scope global (unlike worker's g_cluster_cfg) since test_scheduler_connect.c links scheduler_srcs as separate objects, not #include, so the static helper isn't directly callable from tests — coverage requires a real dup2()-based stdout capture around a live sched_init() call hitting the failure path, not a white-box direct call (see capture_stdout_during_sched_init() in test_scheduler_connect.c — reusable pattern for any future scheduler.c log-content assertion). Round-1 review caught this diagnostic fix shipping with zero test coverage (proven by reverting the 59-line diff and rebuilding: all existing tests still passed) — a reminder that "the fix works, I verified it manually/via live logs" is not the same as a committed regression test; always add the assertion, don't just observe the correct behavior once. The worst-case-occupancy backpressure test (test_scheduler_backpressure.c) only proves Req 3 AC2 (buffering/block/resume-with-no-loss) — it does not exercise task 3.2's -E_XCE_NOT_READY retry path, so don't cite it for Req 7 coverage (that's task 3.4's job). E2E kill-primary/reconnect and all-unreachable-at-startup were validated manually against a real 3-node /xce-deploy-ha cluster rather than committed as an automated test, following worker's task 2.3 precedent — this repo has no automated live-multi-process-deploy E2E harness for any C daemon (the only one, test_e2e_live_ha_push.py, is Python/apiserver-only); reviewer judged this acceptable both rounds. Manual validation confirmed join, primary-death survival, and lazy socket-orphaning-on-idle (matches design), but could not observe a live rejoin firing against real sockets — blocked by bug-1519 (xcecd ASAN "already joined thread" abort under unshare -r, needed to bypass the worker daemon's root-only check in this sandbox; unconfirmed whether it's a genuine bug or a userns artifact, never tried under real root). The exact rejoin-after-publish-success code path already has solid white-box mock coverage from task 3.2's test_rejoin_composition_retries_across_publish_calls, so this residual gap is narrowly "proven against mocks, not yet against a live socket" — worth closing opportunistically if bug-1519 ever gets resolved, not a blocker for this spec.