myxce/.kiro/specs/component-failover-adoption/research.md

Research & Design Decisions

Summary

• Feature: component-failover-adoption
• Discovery Scope: Extension (light discovery — libxce, worker, scheduler, apiserver already exist; this wires three consumers onto an existing library mechanism)
• Key Findings:
  • libxce-failover (shipped, GO 2026-07-18) already built a complete, transparent, per-call multi-master retry/rotate mechanism (XCECLI_FAILOVER_CALL / XCECLI_FAILOVER_CALL_COUNT / XCECLI_FF_CALL in client_internal.h, wired into every public xcecli_* call in client.c, including xcecli_join and the initial xcecli_start connect). It requires only xce_client_config_t.nmasters > 1 to activate. No production code sets nmasters above 1.
  • A rejoin hook (xcecli_set_reconnect_handler) exists specifically so a consumer can re-send its join after a transparent reconnect. It has zero production callers — only its own setter and a unit test reference it.
  • A second, older, half-built and unused primitive (xce_connect_cluster() / xce_reconnect_init/step/fini() in src/lib/cluster.c) also exists. Its production connect path (do_connect()) is a stub that always returns NULL outside unit tests, and it operates on raw endpoint_t*, not xcecli_ctx_t. No production code calls it. It predates/overlaps libxce-failover and is superseded by it.
  • This is a genuine "wire it up" feature, not a "build failover" feature — confirmed by the roadmap: component-failover-adoption (roadmap.md:64) was already recorded 2026-06-16 with this exact framing, HARD-blocked on libxce-failover, now unblocked.

Research Log

Why do worker/scheduler/apiserver each hand-roll a candidate loop instead of using the library mechanism?

• Context: Initial verification (this conversation) found worker, scheduler, and apiserver each independently loop over configured candidates, calling xcecli_ctx_init/xcecli_start/xcecli_join fresh for each candidate with nmasters hardcoded to 1 per attempt.
• Sources Consulted: src/lib/client.c, src/lib/client_internal.h, src/worker/src/controller_connector.c, src/scheduler/src/scheduler.c, src/frontend/apiserver/backend/clib.go, roadmap.md.
• Findings: xcecli_start (client.c:125-188) already rotates through ALL configured masters on the very first connect (bug-854 fix, comment at line 136). xcecli_join (client.c:291-300) is wrapped in XCECLI_FAILOVER_CALL, so it already retries and rotates on standby/redirect/transport-error responses. The three daemons' hand-rolled loops predate this (or were never updated after it landed) and now duplicate — imperfectly — what the library does correctly.
• Implications: The fix is subtractive as much as additive: delete the per-daemon candidate loops, populate nmasters/masters[]/master_ports[] fully once, and let the existing library mechanism do the rest. This directly satisfies Requirements 1 and most of Requirement 2 without new retry/rotation logic.

Does the library mechanism, once enabled, also cover runtime reconnect (Requirement 2) automatically?

• Findings: Yes for any component whose normal operation already calls xcecli_* functions periodically or on demand — every one of those calls is wrapped in XCECLI_FAILOVER_CALL/XCECLI_FAILOVER_CALL_COUNT, so a transport failure on any call transparently triggers rotate+reconnect+retry within that same call. Fire-and-forget calls (xcecli_nop, xcecli_heartbeat) use XCECLI_FF_CALL instead: a bounded single best-effort reconnect attempt, never the full retry budget — this is the library's own liveness-safety design for exactly the concern in Requirement 3.
• Implications: Scheduler's xcecli_jobdispatch calls (scheduler.c:1259, called from sched_publish_dispatch on the main driver-loop thread — see "Thread-affinity" correction below, not sched_allocator) and apiserver's per-request backend calls (gCtx.ctx in clib.go) get transparent reconnect for free once nmasters is set correctly — no new polling/reconnect loop needs to be written for either component.

Does anything need a NEW re-join after a transparent reconnect, beyond what the library does automatically?

• Findings: The library's rejoin hook exists precisely because reconnecting the transport does not re-establish application-level cluster membership — a new primary does not automatically know a worker/scheduler/apiserver rejoined unless it re-sends its join (role) message. xcecli_reconnect_cb "runs on the calling thread" and returns 0 to resume the interrupted op or <0 to treat the rejoin as failed (loop rotates/backs off again).
• Implications: All three components must register a rejoin callback that re-issues xcecli_join(ctx, <role>, ...) (with worker also re-sending hostattrs_t). Without it, a component would reconnect at the transport level but the new primary would not recognize it as joined — likely surfacing as silently-dropped pushes/notifications rather than an obvious error.

Thread-affinity of blocking retry calls (Requirement 3)

• Findings: Worker already isolates all controller-connection work (controller_connector_start, xcecd_join_cluster, and the periodic checking_connection() health-check) onto one dedicated thread (start_controller_connector_thread / connector_to_controller), separate from hardware_monitor/system_monitor/process_manager update calls in xcecd_start(). This isolation already satisfies Requirement 3 structurally for the worker — it is not something this feature needs to newly build, only preserve while removing the redundant candidate loop.
• Corrected 2026-08-10: xcecli_jobdispatch (scheduler.c:1259) does NOT execute from sched_allocator. It is called from sched_publish_dispatch, reached only via the decision-drain loop inside xce_sched_driver_run_normal_mode_round (driver.c:765-786), which runs on the single main-loop thread (main.c's while(1), main.c:348-349). sched_allocator (scheduler.c:295-392) only does cbuf_get(&core.jbuf, ...) then cbuf_put(&core.decisions, ...) — it never calls xcecli_jobdispatch itself. The scheduler's real pipeline is three stages joined by two cbufs: sched_job_picker → jbuf (capacity 1, core.c:161, the scheduler-core determinism invariant) → sched_allocator → decisions (capacity ndecisions=5, core.c:84) → the main-loop thread → xcecli_jobdispatch. When that call blocks in retry/backoff, the main thread stops draining decisions; sched_allocator and sched_job_picker can still produce a few more jobs before their own cbuf_put calls block. So the two threads are buffered for a bounded, occupancy-dependent grace window, not indefinitely isolated — the earlier "benign, does not stall the job picker" claim was wrong; both threads do eventually stall for the remainder of an outage. Corrected 2026-08-10 (second pass): the grace window's size depends on how full decisions was at the moment of the stall (guaranteed floor as low as 1-2 jobs, ceiling ndecisions+1=6) — an earlier version of this note claimed "~6-job" as if it were the guaranteed size; that was the best-case ceiling, not a floor. See design.md's SchedulerControllerClient section for the full derivation, the Req 3 AC1→AC2 scope-narrowing, and Requirement 7 (the actual load-bearing guarantee: no job lost regardless of how many the buffer absorbs).
• apiserver: unlike the C daemons, ALL request-handling goroutines that touch the backend share the same gCtx.ctx, serialized by the library's per-context failover.lock. Under a real outage this could queue every concurrent API request behind the full retry budget, which is a client-experience problem (Requirement 4), not a cluster-liveness problem. Needs a lightweight fail-fast guard in the Go layer — see Design Decisions.
• Implications: No new dedicated thread is required for worker (existing isolation already suffices). Scheduler needs no new thread either, but Req 3 AC1 needs its wording narrowed to match the bounded (not indefinite) buffering the existing jbuf/decisions cbufs actually provide — see design.md Critical Issue note. apiserver needs a new, small piece of state (not a thread) to short-circuit blocking calls when the backend is known-down.

Configuration surface mismatch: N_MAX_MASTERS vs MAX_HA_CANDIDATES

• Findings: src/include/client.h:28 defines N_MAX_MASTERS (3) — the hard cap on xce_client_config_t.masters[]. src/include/config.h:78 defines MAX_HA_CANDIDATES 4 — the cap on xce_cluster_config_t.candidates[] used by xce_parse_peers. These are different constants. xce_test (test.c:360-367) already handles the mismatch correctly: parse up to 4 candidates, then clamp to N_MAX_MASTERS (3) with a WARN if truncated.
• Implications: Any new shared conversion helper must replicate this exact clamp-and-warn behavior (not silently drop entries, not error out) so all components behave identically when 4 candidates are configured. Scope expanded 2026-08-10 (explicit user direction): reconciling the two constants is now IN scope — N_MAX_MASTERS is redefined to derive from MAX_HA_CANDIDATES (config.h becomes the single source of truth, Req 1 AC7), raising it 3→4. Source audit confirmed this is safe: no wire-protocol array assumes 3 (masters[] is never serialized), no header-include cycle (config.h depends only on <stdint.h>), and xce_test's clamp already references N_MAX_MASTERS symbolically so it absorbs the raise automatically. Two latent bugs found by this audit that the raise would otherwise silently trip: src/worker/src/main.c:414 and src/scheduler/src/main.c:314 both hardcode a literal > 3 check instead of referencing the macro — need fixing to > N_MAX_MASTERS alongside the raise. src/lib/testsuite/test_libxce_retry.c's fo_make_client_masters() fixture also hardcodes exactly 3 literal host/port entries then copies N_MAX_MASTERS slots — needs a 4th literal added or it silently copies an uninitialized slot post-raise.

TLS/CA-cert wiring gap (pre-existing, out of boundary)

• Findings: xce_client_config_t (the struct actually consumed by xcecli_ctx_init/xcecli_start, i.e. the real connection path) has no CA-cert field at all — only nmasters/masters/master_ports/from/key. The ca_cert_path that worker/scheduler's --ca-cert flag populates on xce_cluster_config_t is validated by xce_cluster_config_validate() but is only ever read by the orphaned xce_connect_cluster() stub, never by the real connect path.
• Implications: This is a pre-existing gap unrelated to candidate-list wiring — the real connection already establishes TLS today (single-master) via whatever global SSL context setup exists outside this struct. Not this feature's concern; noted here so it isn't mistaken for something this design should fix.

Architecture Pattern Evaluation

Option	Description	Strengths	Risks / Limitations	Notes
Adopt existing xce_client_config_t / XCECLI_FAILOVER_CALL mechanism (chosen)	Populate the full candidate list once per component, register a rejoin callback, delete hand-rolled loops	Already built, tested (libxce-failover), and exactly what xce_test uses today; minimal new code	None material — this is the reference-verified path	Matches roadmap's own framing of this backlog item
Complete and adopt xce_connect_cluster()/xce_reconnect_*	Finish the stubbed do_connect(), wire it under each daemon	N/A	Duplicate mechanism to the one already shipped; raw-endpoint_t API doesn't carry the join/request/reply protocol the daemons need — would need its own substantial new integration layer on top	Rejected — would build a second, redundant failover mechanism
Build a new per-component reconnect loop	Fix each daemon's existing hand-rolled loop in place, independently	Minimal library changes	Perpetuates 3 divergent implementations (the root problem this feature exists to fix); does not gain the library's already-tested retry/backoff/classification logic	Rejected — directly contradicts the roadmap item's stated goal

Design Decisions

Decision: Adopt the existing xcecli_ctx_t / XCECLI_FAILOVER_CALL mechanism; do not build new failover logic

• Context: Requirements 1-3 need multi-candidate connect + runtime reconnect + liveness-safe retry.
• Alternatives Considered: see Architecture Pattern Evaluation above.
• Selected Approach: Each component populates a fully-formed xce_client_config_t (all configured candidates, not just the first) once, calls xcecli_ctx_init/xcecli_start/xcecli_join once, and registers a rejoin callback via xcecli_set_reconnect_handler. All subsequent xcecli_* calls already retry/rotate/reconnect transparently.
• Rationale: The mechanism is shipped, tested, and is the exact one xce_test's masters setting already uses correctly. Building anything new would duplicate it.
• Trade-offs: None material for worker/scheduler. For apiserver, moving off the current Go-level per-peer loop means the CGO layer becomes responsible for candidate rotation instead of Go — acceptable since it is strictly more capable (transparent runtime reconnect, not just startup).
• Follow-up: Confirm during implementation that worker's hardware/process-monitor update calls truly run off the connector thread (Validation Hook, not re-verified byte-for-byte in this research pass).

Decision: One new shared C helper converts xce_cluster_config_t → xce_client_config_t

• Context: All three components need to turn a parsed/validated candidate list into the nmasters/masters[]/master_ports[] triple, replicating xce_test's clamp-to-N_MAX_MASTERS-with-WARN behavior (see mismatch finding above). Today this logic exists ad hoc in test.c only.
• Alternatives Considered: (1) Duplicate the clamp/copy logic in worker, scheduler, and apiserver's CGO layer separately. (2) One shared helper in src/lib/cluster.c (where xce_parse_peers/xce_cluster_config_validate already live), declared in xce/cluster.h.
• Selected Approach: (2) — a single xce_cluster_config_to_client_config() helper, reused by worker (C), scheduler (C), and apiserver (via CGO).
• Rationale: DRY; the clamp-and-warn behavior must be identical across components (Requirement 6's config-syntax-parity spirit extends naturally to the derived struct, not just the input string); one function is one thing to test.
• Trade-offs: None — this is pure generalization of logic that would otherwise be copy-pasted three times.

Decision: apiserver calls the existing C xce_parse_peers/xce_cluster_config_validate via CGO instead of maintaining a second Go parser

• Context: Requirement 6 requires apiserver's candidate-list syntax to match scheduler/worker exactly. apiserver currently has its own hand-written Go parsePeer function.
• Alternatives Considered: (1) Keep/harden the Go parser to match C syntax by inspection. (2) Call the real C parser via CGO (apiserver already links libxce via CGO for everything else).
• Selected Approach: (2).
• Rationale: Guarantees byte-identical parsing forever (no drift risk), and apiserver already pays the CGO cost for every other backend interaction.
• Trade-offs: One more CGO surface, consistent with the existing backend/ package pattern.

Decision: apiserver gets a small Go-side "backend health" circuit breaker for Requirement 4

• Context: Requirement 4 needs apiserver to fail fast toward its own HTTP clients when the backend is down, rather than queue every request behind the library's full retry budget.
• Alternatives Considered: (1) Let every request block on the existing per-ctx lock and rely on Gin/client-side timeouts. (2) Track backend health from call outcomes + the rejoin callback, and check it before issuing a blocking backend call.
• Selected Approach: (2) — new backend/health.go: an atomic "degraded since" state, set when a call's classified outcome is retry-budget-exhaustion, cleared when the rejoin callback fires (successful reconnect+rejoin). Backend-call wrappers in clib.go check it first and return a clear error immediately while degraded, instead of entering the blocking call.
• Rationale: Small, localized, reuses signals the library already produces (call rc, rejoin callback) rather than adding new polling.
• Trade-offs: A short window exists where the flag is stale (a request arrives just as the backend goes down, before any call has failed) — acceptable, matches typical circuit-breaker behavior; the request in flight simply experiences the one blocking retry cycle before the flag is set for subsequent requests.

Decision: rejoin callbacks must be lock-free flag-setters, never direct xcecli_join calls

• Context: A second /kiro-validate-design pass, cross-checking the library-invoked rejoin path against client_failover.c directly rather than only its header comments, found that XCECLI_FAILOVER_CALL holds ac->failover.lock — a plain, non-recursive pthread_mutex_t (mylock_init → pthread_mutex_init(l, NULL)) — for its entire retry loop, and xcecli_invoke_rejoin() (which calls the registered rejoin callback) runs synchronously inside that loop, lock held (client_failover.c:397). Every component's original design had its rejoin callback call xcecli_join(...) directly — itself XCECLI_FAILOVER_CALL-wrapped, so it would try to re-lock the same mutex on the same thread: a guaranteed self-deadlock on the very first real failover. This was never caught because xcecli_set_reconnect_handler has zero production callers and the existing unit test's callback never calls back into the client (it's a no-op probe).
• Alternatives Considered: (1) Have the callback call xcecli_join directly (original design — deadlocks). (2) Expose a new, lock-free/raw join primitive from libxce for rejoin callbacks to use (would require modifying libxce-failover, violating this spec's Allowed Dependencies boundary of consuming it read-only). (3) Callback only sets a lightweight flag and returns immediately (no I/O, no lock); the daemon's own calling code checks and clears the flag after the triggering call returns (lock free by then) and issues the real xcecli_join as a separate call.
• Selected Approach: (3), applied uniformly to worker (checking_connection), scheduler (sched_publish_dispatch, the main-loop thread's actual xcecli_jobdispatch call site — not the allocator, corrected 2026-08-10), and apiserver (each backend-call wrapper, after its C.xcecli_* call). apiserver's heartbeat-triggered path (Task 4.4) was already structured this way by construction (it acts after xcecli_heartbeat returns) and needed no change.
• Rationale: Fixes the deadlock without touching libxce at all, staying inside this spec's stated boundary; the fix is a small, uniform pattern applied identically across all three components rather than a special case.
• Trade-offs: One extra check-and-clear step after each relevant call site, in exchange for correctness; negligible cost.
• Follow-up: Worth flagging upstream to libxce-failover's owners that the rejoin-hook contract (client.h's doc comment for xcecli_reconnect_cb) doesn't document this lock-held constraint — a future consumer without this spec's context could reintroduce the deadlock. Not fixed here (out of boundary), but the regression-guard unit test added in design.md's Testing Strategy exists specifically because of this gap in the upstream contract's documentation.

Decision: apiserver's rejoin registration uses a C shim in clib.go, not a standalone reconnect.go export

• Context: /kiro-validate-design review flagged that design.md originally described reconnect.go as "a //exported Go function registered via xcecli_set_reconnect_handler," implying direct registration. This codebase already documents (wire_dispatch_support.go:154-159, explaining why go_push_handler needs the xce_backend_handler C shim) that cgo symbol resolution is per-file — a C.<symbol> reference only resolves in the file holding its own matching extern declaration.
• Alternatives Considered: (1) Register the exported Go function directly from reconnect.go. (2) Add a static C shim + extern declaration in clib.go's own cgo preamble (mirroring the proven xce_backend_handler pattern) and register that.
• Selected Approach: (2).
• Rationale: Matches the codebase's own documented, working pattern instead of risking the same resolution failure xce_backend_handler was built to avoid.
• Trade-offs: None — same amount of code, just correctly placed.
• Follow-up: none; resolved by construction in design.md.

Decision: apiserver's heartbeat prober explicitly triggers rejoin on recovery

• Context: /kiro-validate-design review found, by direct inspection of client_internal.h/client_failover.c, that XCECLI_FF_CALL (backing xcecli_heartbeat, apiserver's only idle-period liveness probe) never calls xcecli_invoke_rejoin — only XCECLI_FAILOVER_CALL (the full-retry path used by write calls) does. Left unaddressed, apiserver would repair its TCP connection via the heartbeat but never re-join, so push notifications (Req 2.4) would stay dead during any failover where no write request happens to arrive first.
• Alternatives Considered: (1) Switch the periodic prober from xcecli_heartbeat (fire-and-forget) to xcecli_hello (full retry, rejoin-bearing) — the same mechanism worker's checking_connection() already uses. (2) Keep xcecli_heartbeat (preserves its existing wire semantics/side effects on the controller) and have the Go-side tick callback explicitly detect a failure-to-success transition and manually invoke the same rejoin logic used elsewhere.
• Selected Approach: (2).
• Rationale: Preserves xcecli_heartbeat's existing wire behavior (unknown/unverified whether the controller treats the heartbeat opcode differently from hello for apiserver accounting) while closing the gap with a small, self-contained Go-side check — no library changes, no behavior change to any other component.
• Trade-offs: A few extra lines in heartbeat.go; the rejoin logic now has two call sites (the cgo shim and the heartbeat callback), both invoking the same reconnect.go function, so no duplicated join-payload logic.
• Follow-up: none; resolved by construction in design.md.

Decision: Do not touch xce_connect_cluster()/xce_reconnect_*

• Context: This second, unused, stubbed primitive exists in the same area of the codebase.
• Selected Approach: Leave as-is; do not adopt, complete, or remove it in this spec.
• Rationale: Out of this feature's boundary — removing dead code is a separate hardening decision, not required to satisfy any requirement here, and touching it risks scope creep into an unrelated cleanup.
• Follow-up: Worth a future hardening backlog note (superseded/dead code) — not filed as part of this spec.

---

Gap Re-Validation (2026-08-10)

Re-verified all code citations against the current scheduler-framework branch (post scheduler-core refactor, post worker CLI cleanup commits 9ffb1d5–0df1cf5).

Worker (xcecd)

CLI changes orthogonal to spec. The 6 worker commits removed -c/-p/-m flags, config-file loading, xcecd_conf.c/h, and worker.yaml. Added -v verbose logging, fixed logdir default and -h ordering. None touched controller_connector.c/controller_connector.h — the failover-relevant code is unchanged.

Bug-1510 confirmed still present. cc_ctx->cluster_cfg = cluster_cfg at current main.c:537 still runs AFTER xcecd_start() at main.c:531. controller_connector_start() at xcecd.c:515 performs the first join via init_client_config() (controller_connector.c:129-160, nmasters=1 at line 144) before cluster_cfg is populated. Spec tasks 2.1–2.3 correctly target this.

Line drift (minor):

spec cite	current
main.c:610	main.c:537
controller_connector.c:127-158	controller_connector.c:129-160
controller_connector.c:237-311	controller_connector.c:239-312

Task 2.3 validation hook stale. Task says to verify hardware_monitor_update/system_monitor_update/process_manager_update don't run on connector thread. These three are one-shot init calls at xcecd.c:476-480 — never called again. The actual recurring liveness duties are worker_process_update() (main loop, line 577) and resource_sync_thread (dedicated thread, resource_sync.c:293-330). The architectural property (connector thread isolated) IS correct, but the named evidence is wrong. Recommend rewording to reference main loop/resource_sync_thread.

Scheduler

HA candidate loop untouched. scheduler.c:564-624 (was ~462-495 at spec-write). Loop logic unchanged; only line numbers shifted by scheduler-core refactor (doc comments, core.c extraction, --sim CLI additions in main.c).

No-reconnect claim confirmed. No xcecli_set_reconnect_handler call in any scheduler file. No reconnect path after sched_init().

Line drift (minor):

spec cite	current
scheduler.c:462-495	scheduler.c:564-624
scheduler.c:1289 (xcecli_jobdispatch)	scheduler.c:1259
scheduler.c:511,516 (pthread_create)	scheduler.c:626,632
research.md L22: scheduler.c:1289	scheduler.c:1259
research.md L30: scheduler.c:511,516	scheduler.c:626,632

New structural note for design: main.c now early-returns for --sim mode (line 296), inserted between peers-parsing block and sched_init(). Doesn't affect normal-mode path — peers block still runs before sched_init() for non---sim — but design should note this guard exists.

Apiserver

Not re-verified in depth (no flagged changes in this area). Light sanity check confirms RegisterToBackend still loops config.Peers with per-peer tryConnect.

Requirements.md pre-existing issues

Duplicate Background section. Lines 10-22 and 24-34 are byte-identical — copy-paste artifact at spec-creation time. Should deduplicate.

Summary

All architectural findings and target bugs in the spec remain substantively accurate. Only citation line numbers and task 2.3 wording need refreshing. No design decisions invalidated. Spec is ready for implementation.

Risks & Mitigations

• Deleting worker's/scheduler's hand-rolled candidate loops could regress subtle existing behavior (e.g., worker's specific WARN logging on standby redirect) — mitigate by preserving equivalent log statements around the new single-call path, verified in code review.
• apiserver's circuit breaker could mask a fixed backend as still-degraded if the clearing signal (rejoin callback) doesn't fire in some edge case — mitigate with a bounded TTL on the degraded flag (re-probe after a short timeout) rather than relying solely on the rejoin callback.
• N_MAX_MASTERS (3) vs MAX_HA_CANDIDATES (4) mismatch could silently drop a 4th configured candidate on any component — mitigate by making the shared helper's WARN-and-clamp behavior loud and identical everywhere (already the plan).

References

• src/lib/client.c, src/lib/client_internal.h, src/include/xce/client_failover.h — the shipped libxce-failover mechanism this feature adopts.
• src/controller/testsuite/test.c:345-382 — the reference (xce_test) usage pattern.
• .kiro/steering/roadmap.md:64 — component-failover-adoption backlog item this spec formalizes.
• .kiro/steering/ha-async-io.md — thread-role contract informing Requirement 3.