feat(component-failover-adoption): fix worker startup ordering, add rejoin retry-on-failure

Thread cluster_cfg by value through xcecd_config_t -> xcecd_context_t ->
cc_ctx so it's populated before controller_connector_start() fires the
first join, instead of after xcecd_start() returns. checking_connection()
now retries a failed post-reconnect xcecli_join() on the next liveness
probe instead of leaving needs_rejoin cleared on mere attempt.
This commit is contained in:
Rongsong Shen 2026-08-11 06:21:36 +00:00
parent ce9f734a38
commit 331fa02423
Signed by: rshen
SSH key fingerprint: SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
8 changed files with 663 additions and 12 deletions

View file

@ -34,7 +34,7 @@
- _Boundary: WorkerControllerConnector_
- _Depends: 1.2_
- [ ] 2.2 Fix worker's candidate-list startup ordering and add rejoin retry-on-failure
- [x] 2.2 Fix worker's candidate-list startup ordering and add rejoin retry-on-failure
- Move the candidate-list assignment to before the connector starts, so the very first connection attempt already sees the full configured list instead of only a synthesized single candidate
- After each periodic liveness probe returns, check whether a rejoin is pending; if so, issue the rejoin call as a separate step and clear the pending indicator only if that call succeeds — on its own failure, leave the indicator set so the next liveness probe retries the rejoin rather than leaving the worker connected but never actually re-registered
- Observable completion: starting the worker with a peer list where the first listed candidate is unreachable succeeds on the very first connect attempt, not only on a later reconnect; a rejoin call that fails once before succeeding is retried on the next liveness-probe cycle rather than abandoned
@ -165,3 +165,4 @@
- Task 1.2: its 8 new tests in `test_cluster.c` substantively cover task 1.3's stated checklist already (candidate order, one-candidate-no-branch, truncation-with-warning, NULL/empty rejection) — task 1.3 should scope down to just the constant-equality assertion (`N_MAX_MASTERS == MAX_HA_CANDIDATES`) rather than duplicating this coverage.
- Task 1.3: closed with zero new code. Verified fresh (source read + a clean `xce-build.sh lib asan` run) that every item in its checklist was already satisfied by prior tasks: the `_Static_assert(N_MAX_MASTERS == MAX_HA_CANDIDATES, ...)` landed in task 1.1 (`test_libxce_retry.c:44`), and the four behavioral cases (order/no-branch/truncation/NULL-empty) landed in task 1.2's 8 tests (`test_cluster.c`). No implementer/reviewer cycle was dispatched since there was no diff to review — dispatching one would have either duplicated existing test names (build conflict) or been a no-op review of an empty diff.
- Task 2.1: `xce_cluster_config_to_client_config()` (task 1.2) aliases `out->masters[i]` directly to `cfg->candidates[i].hostname` — no `strdup`. Any caller building the source `xce_cluster_config_t` on the stack and passing it through the helper then using the resulting client config beyond that stack frame's lifetime has a use-after-scope bug. Worker's fix keeps the source config in a file-scope `static` for this reason; tasks 3.1 (scheduler) and 4.2 (apiserver) hit the identical hazard and must do the same (or otherwise guarantee the source config outlives the client config's use).
- Task 2.2: fixed the startup-ordering bug by threading `cluster_cfg` by value through `xcecd_config_t` → `xcecd_context_t` → `cc_ctx` (set in `main.c` before `xcecd_init()`/`xcecd_start()` run, instead of after). Safe as a plain struct copy because `xce_candidate_t.hostname` is a fixed `char[]` array, not a pointer — no aliasing hazard here (unlike the `xce_cluster_config_to_client_config()` output struct noted above). Scheduler (3.1) likely has the same ordering gap (`sched_init` loops candidates correctly already per design.md, so may not need this fix — verify against current source, don't assume). Rejoin-retry pattern established: `checking_connection()` checks `needs_rejoin` after every probe return, calls `xcecli_join()` directly (not full re-entry into the join-cluster function), clears the flag only on success — same pattern scheduler/apiserver rejoin tasks (3.1, 4.3) should follow.