126 lines
15 KiB
Markdown
126 lines
15 KiB
Markdown
# Implementation Plan
|
|
|
|
- [ ] 1. Foundation: shared candidate-list conversion helper
|
|
- [ ] 1.1 Implement the candidate-list-to-client-config conversion helper in libxce
|
|
- Add `xce_cluster_config_to_client_config()` to `src/lib/cluster.c`, declared in `src/include/xce/cluster.h`, generalizing the clamp-to-`N_MAX_MASTERS`-with-WARN logic `xce_test` already applies inline
|
|
- Copies candidates in order; leaves `from`/`key` untouched for the caller to set
|
|
- Observable completion: given a validated `xce_cluster_config_t` with 1, 3, and 4 candidates, the function returns a correctly populated `xce_client_config_t` in each case, truncating and logging a WARN only for the 4-candidate case
|
|
- _Requirements: 1.5_
|
|
|
|
- [ ] 1.2 Unit test the conversion helper
|
|
- Cover: candidate order preserved, truncation at `N_MAX_MASTERS` with WARN when given more, `NULL`/empty-candidate-list rejection with `-EINVAL`
|
|
- Add cases to `src/lib/testsuite/test_cluster.c`
|
|
- Observable completion: new test cases pass under the existing libxce unit test run
|
|
- _Requirements: 1.5_
|
|
|
|
- [ ] 2. Worker: adopt full-candidate-list connect and transparent reconnect
|
|
- [ ] 2.1 (P) Replace worker's per-candidate join loop with a single library-driven connect/join
|
|
- In `src/worker/src/controller_connector.c`: delete `xcecd_join_cluster`'s manual per-candidate iteration and `init_client_config`'s hardcoded `nmasters=1`; build the client config via the Task 1 helper from `cc_ctx->cluster_cfg` (or the existing single-host fallback when no `--peers` given) and call `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` once
|
|
- Register a rejoin callback via `xcecli_set_reconnect_handler` (declared in `src/worker/include/controller_connector.h`) that ONLY sets a `needs_rejoin` flag and returns `0` — it must NOT call `xcecli_join` or any other `xcecli_*` function itself, because the library invokes it while `ac->failover.lock` is still held (a non-recursive mutex) and re-entering that lock via a wrapped call self-deadlocks
|
|
- In `checking_connection()`, immediately after each `xcecli_hello` call returns (lock now released), check and clear `needs_rejoin`; if set, call `xcecli_join(ctx, HostRole_worker, &host_attrs)` as a separate, subsequent call
|
|
- Preserve existing WARN/INFO log statements for standby-redirect and per-candidate outcomes around the new single-call path
|
|
- Reconcile `checking_connection()`'s bounded `MAX_RETRY_COUNT`/`RETRY_INTERVAL`-then-`handle_disconnect()` loop with the library's own indefinite retry: the library now owns connection recovery (Requirement 2.3 — keep retrying, never permanently give up), so `checking_connection()`'s outer bound must no longer tear the worker down after 5 failed hello probes; it becomes a liveness-probe cadence only
|
|
- Observable completion: starting the worker against a 3-candidate config where candidate[0] is a live standby results in the worker joined as expected, with log lines showing the standby redirect and the successful candidate; a simulated outage longer than the old 5-retry window no longer triggers `handle_disconnect()`; invoking the registered rejoin callback while the ctx's failover lock is held returns immediately without blocking
|
|
- _Requirements: 1.1, 1.2, 1.3, 1.5, 1.6, 2.1, 2.2, 2.3, 5.1, 5.2_
|
|
- _Boundary: WorkerControllerConnector_
|
|
- _Depends: 1.1_
|
|
|
|
- [ ] 2.2 Fix worker's candidate-list startup ordering
|
|
- In `src/worker/src/main.c`: move the `cc_ctx->cluster_cfg = cluster_cfg` assignment to before `xcecd_start()` is called, so the very first connection attempt (inside `xcecd_start`/`controller_connector_start`) already sees the full `--peers` candidate list instead of only the single-host fallback
|
|
- Observable completion: starting the worker with `--peers` where the FIRST listed candidate is unreachable succeeds on the first connect attempt (no restart needed), verified by a log trace showing candidate rotation during the initial join, not only during a later reconnect
|
|
- _Requirements: 1.4_
|
|
|
|
- [ ] 2.3 Validate worker thread isolation and add worker candidate/reconnect test coverage
|
|
- Confirm (via code read) that `hardware_monitor_update`/`system_monitor_update`/`process_manager_update` do not execute on the connector thread that now carries the library's blocking retry/backoff — fix isolation if this is not already true
|
|
- Add `src/worker/testsuite/test_e2e_multimaster_join.c`: worker starts and joins across a multi-candidate config; killing the joined primary results in reconnect + successful re-join to a surviving candidate without a worker restart
|
|
- Observable completion: the new test passes, and worker's hardware/process monitoring continues updating (observable via existing monitor state) while a connect/reconnect retry is in progress
|
|
- _Requirements: 1.6, 3.1, 3.2_
|
|
- _Depends: 2.1, 2.2_
|
|
|
|
- [ ] 3. Scheduler: adopt full-candidate-list connect and add runtime reconnect
|
|
- [ ] 3.1 (P) Replace scheduler's inline per-candidate loop with a single library-driven connect/join
|
|
- In `src/scheduler/src/scheduler.c`: delete `sched_init`'s inline per-candidate loop; build the client config via the Task 1 helper from `sched->cluster_cfg` (or the existing single-host fallback) and call `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` once
|
|
- Register a rejoin callback via `xcecli_set_reconnect_handler` that ONLY sets a `needs_rejoin` flag and returns `0` — same constraint as worker's Task 2.1: the library invokes it while `ac->failover.lock` is held, so it must not call `xcecli_join` itself or it self-deadlocks
|
|
- After each `xcecli_jobdispatch` call returns (lock now released), check and clear `needs_rejoin`; if set, call `xcecli_join(ctx, HostRole_scheduler, NULL)` as a separate, subsequent call before continuing
|
|
- Preserve/add per-candidate outcome logging equivalent to what the deleted loop produced
|
|
- Observable completion: starting the scheduler against a 3-candidate config where candidate[0] is a live standby results in the scheduler joined as expected, with log lines showing the standby redirect and the successful candidate — and killing the joined primary afterward results in automatic reconnect and re-join to a surviving candidate, which the scheduler has never done before this task; invoking the registered rejoin callback while the ctx's failover lock is held returns immediately without blocking
|
|
- _Requirements: 1.1, 1.2, 1.3, 1.5, 1.6, 2.1, 2.2, 2.3, 5.1, 5.2_
|
|
- _Boundary: SchedulerControllerClient_
|
|
- _Depends: 1.1_
|
|
|
|
- [ ] 3.2 Add scheduler candidate/reconnect test coverage
|
|
- Add `src/scheduler/testsuite/test_e2e_multimaster_join.c`: scheduler starts and joins across a multi-candidate config; killing the joined primary results in reconnect and re-join without a scheduler restart
|
|
- Confirm `sched_job_picker` continues running on its own thread, unaffected, while `sched_allocator`'s `xcecli_jobdispatch` call is blocked in reconnect/backoff (existing thread split, not new code)
|
|
- Observable completion: the new test passes, and the job-picker thread's activity (e.g. its own state/counters) continues advancing during a simulated reconnect delay on the allocator thread
|
|
- _Requirements: 3.1, 3.2_
|
|
- _Depends: 3.1_
|
|
|
|
- [ ] 4. apiserver: adopt full-candidate-list connect via CGO, config parity, rejoin, and fail-fast
|
|
- [ ] 4.1 (P) Route apiserver's candidate-list parsing through the real C parser
|
|
- In `src/frontend/apiserver/backend/clib.go`: call `xce_parse_peers`/`xce_cluster_config_validate` via CGO to parse `config.Peers` into a `xce_cluster_config_t`, replacing the hand-written Go `parsePeer` function and its call sites
|
|
- Map C validation errors (`-EINVAL` cases) to clear Go error messages naming the malformed entry
|
|
- Observable completion: the same malformed-candidate-string test cases that scheduler/worker already reject (from Task 1.2's neighboring validation coverage) are rejected identically when passed through apiserver's config path
|
|
- _Requirements: 6.1, 6.2_
|
|
- _Boundary: ApiserverBackendBridge_
|
|
- _Depends: 1.1_
|
|
|
|
- [ ] 4.2 Replace apiserver's per-peer connect loop with a single library-driven connect
|
|
- In `src/frontend/apiserver/backend/clib.go`: replace `RegisterToBackend`'s `for _, peer := range config.Peers { tryConnect(...) }` loop with one call that builds the client config via the Task 1 helper (through CGO) from the full parsed candidate list (or the single `Address`/`Port` fallback) and calls `xce_backend_init` once
|
|
- Preserve/extend existing connect-outcome logging (`zap`) to report each candidate attempted
|
|
- Observable completion: starting apiserver with 3 configured peers where the first is a live standby results in apiserver registered against the actual primary in a single `RegisterToBackend` call, with log lines showing the standby redirect
|
|
- _Requirements: 1.1, 1.2, 1.3, 1.5, 1.6, 5.1, 5.2_
|
|
- _Depends: 4.1_
|
|
|
|
- [ ] 4.3 Add apiserver's rejoin logic with a correctly-placed cgo registration
|
|
- New `src/frontend/apiserver/backend/reconnect.go`: two plain Go functions — (1) a `//export`-annotated flag-setter that ONLY sets a `needs_rejoin` atomic flag and returns `0` (no I/O — the library invokes it while `ac->failover.lock` is held, so calling `xcecli_join` directly here self-deadlocks); (2) a plain function that performs the actual rejoin — calls `xcecli_join(gCtx.ctx, HostRole_apiserver, NULL)` and clears backend-health degraded state on success
|
|
- In `src/frontend/apiserver/backend/clib.go`'s cgo preamble: add a static C shim (mirroring the existing `xce_backend_handler`/`go_push_handler` pattern) with its own `extern` declaration for the flag-setter export, and pass the shim to `xcecli_set_reconnect_handler` — the registration must live in `clib.go`, not in `reconnect.go`, because cgo symbol resolution is per-file (same reason `go_push_handler` needs `xce_backend_handler`, documented in `wire_dispatch_support.go`)
|
|
- Add one shared internal helper in `clib.go` that wraps a `C.xcecli_*` call: after the call returns (lock now released), check and clear `needs_rejoin`; if set, call `reconnect.go`'s actual-rejoin function as a separate step. Route all four backend-call wrappers (job submit, job control, queue control, host control) through this ONE helper instead of duplicating the check in each (Task 4.5 extends this same helper with the pre-call health check, rather than adding a second one)
|
|
- Observable completion: after killing and restoring connectivity to apiserver's joined controller while a write request is in flight (job submit retried through the full failover path), apiserver resumes receiving backend push notifications (existing `go_push_handler` path) without an apiserver process restart; invoking the registered flag-setter while the ctx's failover lock is held returns immediately without blocking
|
|
- _Requirements: 2.1, 2.2, 2.3, 2.4_
|
|
- _Depends: 4.2_
|
|
|
|
- [ ] 4.4 Trigger apiserver's rejoin from the idle-period heartbeat path
|
|
- In `src/frontend/apiserver/backend/heartbeat.go`'s tick callback: track the previous tick's `xcecli_heartbeat` outcome; when a tick transitions from failure to success (the fire-and-forget call's own best-effort transport reconnect just succeeded), explicitly call Task 4.3's actual-rejoin function (function (2), not the flag-setter) directly — safe here since `XCECLI_FF_CALL` has already released the lock by the time the heartbeat call returns. `XCECLI_FF_CALL` (backing `xcecli_heartbeat`) never invokes the library's own rejoin hook, unlike the full-retry path write calls use, so without this the idle-period case never re-joins
|
|
- Observable completion: killing the primary while apiserver has no in-flight write request (only the heartbeat ticking) still results in apiserver rejoining and resuming push notifications, verified by the same push-notification check as Task 4.3 but with no write traffic present during the outage
|
|
- _Requirements: 2.4_
|
|
- _Depends: 4.3_
|
|
|
|
- [ ] 4.5 Add apiserver's backend-health circuit breaker
|
|
- New `src/frontend/apiserver/backend/health.go`: atomic degraded-until state with `IsDegraded()`, `MarkDegraded(ttl)`, `MarkHealthy()`
|
|
- Extend Task 4.3's shared internal helper (not a second, separate hook) to also check `IsDegraded()` before the `C.xcecli_*` call and return a clear "backend unavailable" error immediately instead of invoking it when degraded; wire `MarkDegraded` into the same helper's post-call path when a call's result indicates retry-budget exhaustion; wire `MarkHealthy` into both Task 4.3's rejoin function (write-path recovery) and Task 4.4's heartbeat-triggered rejoin (idle-path recovery)
|
|
- Observable completion: with the flag forced degraded in a unit test, a wrapped backend call returns immediately with the unavailable error instead of invoking the C call
|
|
- _Requirements: 4.1, 4.2_
|
|
- _Depends: 4.4_
|
|
|
|
- [ ] 4.6 apiserver unit test coverage
|
|
- `src/frontend/apiserver/backend/health_test.go`: degraded→healthy via `MarkHealthy()` and via TTL expiry; concurrent-goroutine safety
|
|
- Extend `src/frontend/apiserver/backend/clib_test.go` with candidate-list parity cases from Task 4.1
|
|
- Observable completion: `make test` in `src/frontend/` passes with the new cases included
|
|
- _Requirements: 4.1, 4.2, 6.1, 6.2_
|
|
- _Depends: 4.5_
|
|
|
|
- [ ] 5. Integration and cross-component validation
|
|
- [ ] 5.1 E2E: full-cluster reconnect after primary failover
|
|
- Extend the project's existing `/xce-deploy-ha` + `/xce-test-ha` pattern to a scenario with worker, scheduler, and apiserver all configured with the full 3-candidate list; kill the primary mid-session with a write request in flight
|
|
- Assert: worker reconnects and resumes job-state reporting; scheduler reconnects and resumes dispatch; apiserver reconnects, resumes serving API requests, and resumes receiving push notifications — none of the three restarted
|
|
- Observable completion: the soak scenario passes with all three components showing a successful re-join to the new primary in their logs and no manual intervention
|
|
- _Requirements: 2.1, 2.2, 2.3, 2.4_
|
|
- _Depends: 2.3, 3.2, 4.6_
|
|
|
|
- [ ] 5.2 E2E: apiserver reconnects during an idle period (no in-flight write traffic)
|
|
- Kill the primary while apiserver has only its heartbeat prober active (no pending write requests); assert apiserver still rejoins and resumes push notifications via the Task 4.4 heartbeat-triggered path
|
|
- Observable completion: the scenario passes without requiring any client-triggered write request during the outage window
|
|
- _Requirements: 2.4_
|
|
- _Depends: 4.6_
|
|
|
|
- [ ] 5.3 E2E: apiserver fails fast while backend is degraded
|
|
- With apiserver's backend connection killed and not yet reconnected, issue an API request that requires the backend and assert it returns a clear 503-style error within a bounded, sub-second time — not after the full retry budget
|
|
- Observable completion: the timed assertion passes against a live apiserver instance
|
|
- _Requirements: 4.1, 4.2_
|
|
- _Depends: 4.6_
|
|
|
|
- [ ] 5.4 Regression: confirm existing single-controller mode is unaffected
|
|
- Run the full worker, scheduler, and apiserver test suites (ASAN where applicable) plus the existing non-HA (no `--peers`/no `backend.peers`) startup path for each component
|
|
- Observable completion: all existing tests pass unchanged, and each component still starts and operates normally when no candidate list is configured (Requirement 1.5's single/no-candidate fallback)
|
|
- _Requirements: 1.5_
|
|
- _Depends: 5.1, 5.2, 5.3_
|