lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.kiro/specs/component-failover-adoption/tasks.md
Rongsong Shen 4ad31d774d save status
2026-07-21 08:24:44 +00:00

15 KiB
Raw Blame History

Implementation Plan

  • 1. Foundation: shared candidate-list conversion helper

  • 1.1 Implement the candidate-list-to-client-config conversion helper in libxce

    • Add xce_cluster_config_to_client_config() to src/lib/cluster.c, declared in src/include/xce/cluster.h, generalizing the clamp-to-N_MAX_MASTERS-with-WARN logic xce_test already applies inline
    • Copies candidates in order; leaves from/key untouched for the caller to set
    • Observable completion: given a validated xce_cluster_config_t with 1, 3, and 4 candidates, the function returns a correctly populated xce_client_config_t in each case, truncating and logging a WARN only for the 4-candidate case
    • Requirements: 1.5

  • 1.2 Unit test the conversion helper

    • Cover: candidate order preserved, truncation at N_MAX_MASTERS with WARN when given more, NULL/empty-candidate-list rejection with -EINVAL
    • Add cases to src/lib/testsuite/test_cluster.c
    • Observable completion: new test cases pass under the existing libxce unit test run
    • Requirements: 1.5

  • 2. Worker: adopt full-candidate-list connect and transparent reconnect

  • 2.1 (P) Replace worker's per-candidate join loop with a single library-driven connect/join

    • In src/worker/src/controller_connector.c: delete xcecd_join_cluster's manual per-candidate iteration and init_client_config's hardcoded nmasters=1; build the client config via the Task 1 helper from cc_ctx->cluster_cfg (or the existing single-host fallback when no --peers given) and call xcecli_ctx_init/xcecli_start/xcecli_join once
    • Register a rejoin callback via xcecli_set_reconnect_handler (declared in src/worker/include/controller_connector.h) that ONLY sets a needs_rejoin flag and returns 0 — it must NOT call xcecli_join or any other xcecli_* function itself, because the library invokes it while ac->failover.lock is still held (a non-recursive mutex) and re-entering that lock via a wrapped call self-deadlocks
    • In checking_connection(), immediately after each xcecli_hello call returns (lock now released), check and clear needs_rejoin; if set, call xcecli_join(ctx, HostRole_worker, &host_attrs) as a separate, subsequent call
    • Preserve existing WARN/INFO log statements for standby-redirect and per-candidate outcomes around the new single-call path
    • Reconcile checking_connection()'s bounded MAX_RETRY_COUNT/RETRY_INTERVAL-then-handle_disconnect() loop with the library's own indefinite retry: the library now owns connection recovery (Requirement 2.3 — keep retrying, never permanently give up), so checking_connection()'s outer bound must no longer tear the worker down after 5 failed hello probes; it becomes a liveness-probe cadence only
    • Observable completion: starting the worker against a 3-candidate config where candidate[0] is a live standby results in the worker joined as expected, with log lines showing the standby redirect and the successful candidate; a simulated outage longer than the old 5-retry window no longer triggers handle_disconnect(); invoking the registered rejoin callback while the ctx's failover lock is held returns immediately without blocking
    • Requirements: 1.1, 1.2, 1.3, 1.5, 1.6, 2.1, 2.2, 2.3, 5.1, 5.2
    • Boundary: WorkerControllerConnector
    • Depends: 1.1

  • 2.2 Fix worker's candidate-list startup ordering

    • In src/worker/src/main.c: move the cc_ctx->cluster_cfg = cluster_cfg assignment to before xcecd_start() is called, so the very first connection attempt (inside xcecd_start/controller_connector_start) already sees the full --peers candidate list instead of only the single-host fallback
    • Observable completion: starting the worker with --peers where the FIRST listed candidate is unreachable succeeds on the first connect attempt (no restart needed), verified by a log trace showing candidate rotation during the initial join, not only during a later reconnect
    • Requirements: 1.4

  • 2.3 Validate worker thread isolation and add worker candidate/reconnect test coverage

    • Confirm (via code read) that hardware_monitor_update/system_monitor_update/process_manager_update do not execute on the connector thread that now carries the library's blocking retry/backoff — fix isolation if this is not already true
    • Add src/worker/testsuite/test_e2e_multimaster_join.c: worker starts and joins across a multi-candidate config; killing the joined primary results in reconnect + successful re-join to a surviving candidate without a worker restart
    • Observable completion: the new test passes, and worker's hardware/process monitoring continues updating (observable via existing monitor state) while a connect/reconnect retry is in progress
    • Requirements: 1.6, 3.1, 3.2
    • Depends: 2.1, 2.2

  • 3. Scheduler: adopt full-candidate-list connect and add runtime reconnect

  • 3.1 (P) Replace scheduler's inline per-candidate loop with a single library-driven connect/join

    • In src/scheduler/src/scheduler.c: delete sched_init's inline per-candidate loop; build the client config via the Task 1 helper from sched->cluster_cfg (or the existing single-host fallback) and call xcecli_ctx_init/xcecli_start/xcecli_join once
    • Register a rejoin callback via xcecli_set_reconnect_handler that ONLY sets a needs_rejoin flag and returns 0 — same constraint as worker's Task 2.1: the library invokes it while ac->failover.lock is held, so it must not call xcecli_join itself or it self-deadlocks
    • After each xcecli_jobdispatch call returns (lock now released), check and clear needs_rejoin; if set, call xcecli_join(ctx, HostRole_scheduler, NULL) as a separate, subsequent call before continuing
    • Preserve/add per-candidate outcome logging equivalent to what the deleted loop produced
    • Observable completion: starting the scheduler against a 3-candidate config where candidate[0] is a live standby results in the scheduler joined as expected, with log lines showing the standby redirect and the successful candidate — and killing the joined primary afterward results in automatic reconnect and re-join to a surviving candidate, which the scheduler has never done before this task; invoking the registered rejoin callback while the ctx's failover lock is held returns immediately without blocking
    • Requirements: 1.1, 1.2, 1.3, 1.5, 1.6, 2.1, 2.2, 2.3, 5.1, 5.2
    • Boundary: SchedulerControllerClient
    • Depends: 1.1

  • 3.2 Add scheduler candidate/reconnect test coverage

    • Add src/scheduler/testsuite/test_e2e_multimaster_join.c: scheduler starts and joins across a multi-candidate config; killing the joined primary results in reconnect and re-join without a scheduler restart
    • Confirm sched_job_picker continues running on its own thread, unaffected, while sched_allocator's xcecli_jobdispatch call is blocked in reconnect/backoff (existing thread split, not new code)
    • Observable completion: the new test passes, and the job-picker thread's activity (e.g. its own state/counters) continues advancing during a simulated reconnect delay on the allocator thread
    • Requirements: 3.1, 3.2
    • Depends: 3.1

  • 4. apiserver: adopt full-candidate-list connect via CGO, config parity, rejoin, and fail-fast

  • 4.1 (P) Route apiserver's candidate-list parsing through the real C parser

    • In src/frontend/apiserver/backend/clib.go: call xce_parse_peers/xce_cluster_config_validate via CGO to parse config.Peers into a xce_cluster_config_t, replacing the hand-written Go parsePeer function and its call sites
    • Map C validation errors (-EINVAL cases) to clear Go error messages naming the malformed entry
    • Observable completion: the same malformed-candidate-string test cases that scheduler/worker already reject (from Task 1.2's neighboring validation coverage) are rejected identically when passed through apiserver's config path
    • Requirements: 6.1, 6.2
    • Boundary: ApiserverBackendBridge
    • Depends: 1.1

  • 4.2 Replace apiserver's per-peer connect loop with a single library-driven connect

    • In src/frontend/apiserver/backend/clib.go: replace RegisterToBackend's for _, peer := range config.Peers { tryConnect(...) } loop with one call that builds the client config via the Task 1 helper (through CGO) from the full parsed candidate list (or the single Address/Port fallback) and calls xce_backend_init once
    • Preserve/extend existing connect-outcome logging (zap) to report each candidate attempted
    • Observable completion: starting apiserver with 3 configured peers where the first is a live standby results in apiserver registered against the actual primary in a single RegisterToBackend call, with log lines showing the standby redirect
    • Requirements: 1.1, 1.2, 1.3, 1.5, 1.6, 5.1, 5.2
    • Depends: 4.1

  • 4.3 Add apiserver's rejoin logic with a correctly-placed cgo registration

    • New src/frontend/apiserver/backend/reconnect.go: two plain Go functions — (1) a //export-annotated flag-setter that ONLY sets a needs_rejoin atomic flag and returns 0 (no I/O — the library invokes it while ac->failover.lock is held, so calling xcecli_join directly here self-deadlocks); (2) a plain function that performs the actual rejoin — calls xcecli_join(gCtx.ctx, HostRole_apiserver, NULL) and clears backend-health degraded state on success
    • In src/frontend/apiserver/backend/clib.go's cgo preamble: add a static C shim (mirroring the existing xce_backend_handler/go_push_handler pattern) with its own extern declaration for the flag-setter export, and pass the shim to xcecli_set_reconnect_handler — the registration must live in clib.go, not in reconnect.go, because cgo symbol resolution is per-file (same reason go_push_handler needs xce_backend_handler, documented in wire_dispatch_support.go)
    • Add one shared internal helper in clib.go that wraps a C.xcecli_* call: after the call returns (lock now released), check and clear needs_rejoin; if set, call reconnect.go's actual-rejoin function as a separate step. Route all four backend-call wrappers (job submit, job control, queue control, host control) through this ONE helper instead of duplicating the check in each (Task 4.5 extends this same helper with the pre-call health check, rather than adding a second one)
    • Observable completion: after killing and restoring connectivity to apiserver's joined controller while a write request is in flight (job submit retried through the full failover path), apiserver resumes receiving backend push notifications (existing go_push_handler path) without an apiserver process restart; invoking the registered flag-setter while the ctx's failover lock is held returns immediately without blocking
    • Requirements: 2.1, 2.2, 2.3, 2.4
    • Depends: 4.2

  • 4.4 Trigger apiserver's rejoin from the idle-period heartbeat path

    • In src/frontend/apiserver/backend/heartbeat.go's tick callback: track the previous tick's xcecli_heartbeat outcome; when a tick transitions from failure to success (the fire-and-forget call's own best-effort transport reconnect just succeeded), explicitly call Task 4.3's actual-rejoin function (function (2), not the flag-setter) directly — safe here since XCECLI_FF_CALL has already released the lock by the time the heartbeat call returns. XCECLI_FF_CALL (backing xcecli_heartbeat) never invokes the library's own rejoin hook, unlike the full-retry path write calls use, so without this the idle-period case never re-joins
    • Observable completion: killing the primary while apiserver has no in-flight write request (only the heartbeat ticking) still results in apiserver rejoining and resuming push notifications, verified by the same push-notification check as Task 4.3 but with no write traffic present during the outage
    • Requirements: 2.4
    • Depends: 4.3

  • 4.5 Add apiserver's backend-health circuit breaker

    • New src/frontend/apiserver/backend/health.go: atomic degraded-until state with IsDegraded(), MarkDegraded(ttl), MarkHealthy()
    • Extend Task 4.3's shared internal helper (not a second, separate hook) to also check IsDegraded() before the C.xcecli_* call and return a clear "backend unavailable" error immediately instead of invoking it when degraded; wire MarkDegraded into the same helper's post-call path when a call's result indicates retry-budget exhaustion; wire MarkHealthy into both Task 4.3's rejoin function (write-path recovery) and Task 4.4's heartbeat-triggered rejoin (idle-path recovery)
    • Observable completion: with the flag forced degraded in a unit test, a wrapped backend call returns immediately with the unavailable error instead of invoking the C call
    • Requirements: 4.1, 4.2
    • Depends: 4.4

  • 4.6 apiserver unit test coverage

    • src/frontend/apiserver/backend/health_test.go: degraded→healthy via MarkHealthy() and via TTL expiry; concurrent-goroutine safety
    • Extend src/frontend/apiserver/backend/clib_test.go with candidate-list parity cases from Task 4.1
    • Observable completion: make test in src/frontend/ passes with the new cases included
    • Requirements: 4.1, 4.2, 6.1, 6.2
    • Depends: 4.5

  • 5. Integration and cross-component validation

  • 5.1 E2E: full-cluster reconnect after primary failover

    • Extend the project's existing /xce-deploy-ha + /xce-test-ha pattern to a scenario with worker, scheduler, and apiserver all configured with the full 3-candidate list; kill the primary mid-session with a write request in flight
    • Assert: worker reconnects and resumes job-state reporting; scheduler reconnects and resumes dispatch; apiserver reconnects, resumes serving API requests, and resumes receiving push notifications — none of the three restarted
    • Observable completion: the soak scenario passes with all three components showing a successful re-join to the new primary in their logs and no manual intervention
    • Requirements: 2.1, 2.2, 2.3, 2.4
    • Depends: 2.3, 3.2, 4.6

  • 5.2 E2E: apiserver reconnects during an idle period (no in-flight write traffic)

    • Kill the primary while apiserver has only its heartbeat prober active (no pending write requests); assert apiserver still rejoins and resumes push notifications via the Task 4.4 heartbeat-triggered path
    • Observable completion: the scenario passes without requiring any client-triggered write request during the outage window
    • Requirements: 2.4
    • Depends: 4.6

  • 5.3 E2E: apiserver fails fast while backend is degraded

    • With apiserver's backend connection killed and not yet reconnected, issue an API request that requires the backend and assert it returns a clear 503-style error within a bounded, sub-second time — not after the full retry budget
    • Observable completion: the timed assertion passes against a live apiserver instance
    • Requirements: 4.1, 4.2
    • Depends: 4.6

  • 5.4 Regression: confirm existing single-controller mode is unaffected

    • Run the full worker, scheduler, and apiserver test suites (ASAN where applicable) plus the existing non-HA (no --peers/no backend.peers) startup path for each component
    • Observable completion: all existing tests pass unchanged, and each component still starts and operates normally when no candidate list is configured (Requirement 1.5's single/no-candidate fallback)
    • Requirements: 1.5
    • Depends: 5.1, 5.2, 5.3