2165b1ab12
component-failover-adoption Req 2 (partial): two of the three daemons now self-heal after a controller restart without an operator restart. - worker (controller_connector.c): checking_connection() previously gave up after 5 retries and called handle_disconnect() (which set running=false and exited the connector thread). It now retries indefinitely — the connector thread is dedicated to this loop so the unbounded retry/sleep cannot block the worker's other liveness duties. Verified live: killing the controller leaves all 3 workers alive and retrying; they re-join when it returns. - apiserver (clib.go): xcecli_heartbeat is fire-and-forget — it best-effort reconnects the transport but never re-joins, so after a controller restart push notifications stop. StartHeartbeat now tracks heartbeat failure→success and, on recovery, calls xce_backend_rejoin (xcecli_join as HostRole_apiserver) to re-register and resume pushes. Verified live: "backend re-joined controller after reconnect" logged during a controller kill/restart. Scheduler reconnect NOT included: the rejoin works but exposes a pre-existing libxce race — xce_ctx_reconnect's endpoint_fini tears down the old endpoint while a lib vtask/io thread is still processing it, a deterministic double-free (bad-free in the loader, same freed address offset across runs). Root-cause in libxce netio/endpoint lifecycle is a separate task (documented in left_scheduler_work.md). Co-Authored-By: Claude <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| autoscaler | ||
| configs | ||
| controller | ||
| docs | ||
| frontend | ||
| include | ||
| lib | ||
| nixSupport | ||
| scheduler | ||
| schema | ||
| tools | ||
| worker | ||
| xceadmin | ||
| flake.lock | ||
| flake.nix | ||
| meson.build | ||
| meson.options | ||