--- id: "bug-1876" status: "fixed" status_updated: "2026-09-28" status_note: "" component: "controller" severity: "noise" timestamp: "2026-09-28T03:52:37Z" last_seen: "2026-09-28T03:58:49Z" occurrences: 1 files: ["src/controller/cluster.c", "src/controller/myapp.c", "src/controller/ha.c"] tags: ["controller", "ha", "cold-boot", "logging"] related: ["bug-1477"] fix_commits: ["44173f74"] gitea_issue: 314 gitea_synced_status: "fixed" gitea_synced_hash: "7c372ba63dad" --- # bug-1876: standby logs cluster_init_phase2 failed (rc=-22) WARN every 3 s while awaiting the primary's config ## Description **Observed:** ``` [2026-09-28 11:52:38,939] [51034] [ WARN] [ myapp_run] myapp_run: cluster_init_phase2 on standby rejoin failed (rc=-22) — WAL not recovered; member stays alive and will retry (bug-1477: ...) [2026-09-28 11:52:41,941] [51034] [ WARN] [ ha_phase2_retry_fn] HA: cluster_init_phase2 retry rc=-22 — member stays alive and will retry again (bug-1477, never self-terminate) (ctrl-2 and ctrl-3, every 3 s until 11:52:47) ``` **Expected:** the standby waits for the primary's config push quietly (INFO), as cluster_init_phase2() itself describes that state. **Impact:** WARN lines on every cold start of every standby that read like a failed WAL recovery; noise, but alarming. **Found by:** user report from a pc13v HA deploy, 2026-09-28 ## Environment - OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 - libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) - Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 - Build: asan (xce.build) - Code: scheduler-framework @ 8cadeeb2 (last commit without the fix) - Deployment: first seen on pc13v: 3-node HA (ctrl-1..3) + scheduler + apiserver + real root xcecd under systemd, deploy dir /tmp/xce-ha-deploy, 2026-09-28 11:52 +0800 (pc13v's OS, libc and exact commit were not recorded; the values above are from the reproduction on this dev host, same nix toolchain) ## Reproduction 1. Deploy a fresh 3-node HA cluster at 8cadeeb2 (`/xce-deploy-ha`) 2. `grep 'cluster_init_phase2' ` during the first seconds - Frequency: deterministic on a cold start (until the primary's config push arrives) - Evidence: the xcecd/scheduler/controller/apiserver logs pasted in the user's report, 2026-09-28 (quoted above) ## Root cause cluster_init_phase2() (controller/cluster.c) returns early when c->myconf->cluster_config is still NULL -- a standby that has not received the primary's config push yet -- and logs that at INFO as normal, but returned -EINVAL. Both callers, myapp_run()'s standby-rejoin path (myapp.c) and ha_phase2_retry_fn() (ha.c), treat any nonzero result as a failure and log WARN with rc=-22. ## Fix cluster_init_phase2() returns -EAGAIN for that wait; ha_phase2_rc_is_waiting() (ha.h) names it, and myapp_run() and ha_phase2_retry_fn() log it at INFO while other failures still WARN. The promotion-refused path in ha.c keeps its WARN. ## Verification - Test: test_phase2_without_config_is_a_wait_not_an_error in src/controller/testsuite/test_phase2_cold_boot.c: cluster_init_phase2() with no cluster_config returns -EAGAIN (it returned -EINVAL before), and ha_phase2_rc_is_waiting() accepts only -EAGAIN - Suites: full ASAN build (xce-build all): 113/113 suites after the updated scheduler picker-registration test; controller 1250/0, harness 141/0, worker diskstats 4/0, scheduler picker wiring 11/0 + picker registration 4/0, apiserver TestRootWarning PASS - Live: pc13v retest at 4c134127 (debug build, 2026-09-28 14:47 +0800): `ag -i error logs/` finds nothing; `ag -i warn logs/` shows only the WAL-unencrypted advisory; no cluster_init_phase2 warnings on ctrl-2/ctrl-3 ## References - Related: bug-1477