| id |
status |
status_updated |
status_note |
component |
severity |
timestamp |
last_seen |
occurrences |
files |
tags |
related |
fix_commits |
gitea_issue |
gitea_synced_status |
gitea_synced_hash |
| bug-1876 |
fixed |
2026-09-28 |
|
controller |
noise |
2026-09-28T03:52:37Z |
2026-09-28T03:58:49Z |
1 |
| src/controller/cluster.c |
| src/controller/myapp.c |
| src/controller/ha.c |
|
| controller |
| ha |
| cold-boot |
| logging |
|
|
|
314 |
fixed |
7c372ba63dad |
bug-1876: standby logs cluster_init_phase2 failed (rc=-22) WARN every 3 s while awaiting the primary's config
Description
Observed:
[2026-09-28 11:52:38,939] [51034] [ WARN] [ myapp_run] myapp_run: cluster_init_phase2 on standby rejoin failed (rc=-22) — WAL not recovered; member stays alive and will retry (bug-1477: ...)
[2026-09-28 11:52:41,941] [51034] [ WARN] [ ha_phase2_retry_fn] HA: cluster_init_phase2 retry rc=-22 — member stays alive and will retry again (bug-1477, never self-terminate)
(ctrl-2 and ctrl-3, every 3 s until 11:52:47)
Expected: the standby waits for the primary's config push quietly (INFO), as cluster_init_phase2() itself describes that state.
Impact: WARN lines on every cold start of every standby that read like a failed WAL recovery; noise, but alarming.
Found by: user report from a pc13v HA deploy, 2026-09-28
Environment
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: asan (xce.build)
- Code: scheduler-framework @
8cadeeb2 (last commit without the fix)
- Deployment: first seen on pc13v: 3-node HA (ctrl-1..3) + scheduler + apiserver + real root xcecd under systemd, deploy dir /tmp/xce-ha-deploy, 2026-09-28 11:52 +0800 (pc13v's OS, libc and exact commit were not recorded; the values above are from the reproduction on this dev host, same nix toolchain)
Reproduction
- Deploy a fresh 3-node HA cluster at
8cadeeb2 (/xce-deploy-ha)
grep 'cluster_init_phase2' <standby logs> during the first seconds
- Frequency: deterministic on a cold start (until the primary's config push arrives)
- Evidence: the xcecd/scheduler/controller/apiserver logs pasted in the user's report, 2026-09-28 (quoted above)
Root cause
cluster_init_phase2() (controller/cluster.c) returns early when c->myconf->cluster_config is still NULL -- a standby that has not received the primary's config push yet -- and logs that at INFO as normal, but returned -EINVAL. Both callers, myapp_run()'s standby-rejoin path (myapp.c) and ha_phase2_retry_fn() (ha.c), treat any nonzero result as a failure and log WARN with rc=-22.
Fix
cluster_init_phase2() returns -EAGAIN for that wait; ha_phase2_rc_is_waiting() (ha.h) names it, and myapp_run() and ha_phase2_retry_fn() log it at INFO while other failures still WARN. The promotion-refused path in ha.c keeps its WARN.
Verification
- Test: test_phase2_without_config_is_a_wait_not_an_error in src/controller/testsuite/test_phase2_cold_boot.c: cluster_init_phase2() with no cluster_config returns -EAGAIN (it returned -EINVAL before), and ha_phase2_rc_is_waiting() accepts only -EAGAIN
- Suites: full ASAN build (xce-build all): 113/113 suites after the updated scheduler picker-registration test; controller 1250/0, harness 141/0, worker diskstats 4/0, scheduler picker wiring 11/0 + picker registration 4/0, apiserver TestRootWarning PASS
- Live: pc13v retest at
4c134127 (debug build, 2026-09-28 14:47 +0800): ag -i error logs/ finds nothing; ag -i warn logs/ shows only the WAL-unencrypted advisory; no cluster_init_phase2 warnings on ctrl-2/ctrl-3
References
