59 lines
3.7 KiB
Markdown
59 lines
3.7 KiB
Markdown
---
|
|
id: "bug-1876"
|
|
status: "fixed"
|
|
status_updated: "2026-09-28"
|
|
status_note: ""
|
|
component: "controller"
|
|
severity: "noise"
|
|
timestamp: "2026-09-28T03:52:37Z"
|
|
last_seen: "2026-09-28T03:58:49Z"
|
|
occurrences: 1
|
|
files: ["src/controller/cluster.c", "src/controller/myapp.c", "src/controller/ha.c"]
|
|
tags: ["controller", "ha", "cold-boot", "logging"]
|
|
related: ["bug-1477"]
|
|
fix_commits: ["44173f74"]
|
|
gitea_issue: 314
|
|
gitea_synced_status: "fixed"
|
|
gitea_synced_hash: "7c372ba63dad"
|
|
---
|
|
|
|
# bug-1876: standby logs cluster_init_phase2 failed (rc=-22) WARN every 3 s while awaiting the primary's config
|
|
|
|
## Description
|
|
**Observed:**
|
|
```
|
|
[2026-09-28 11:52:38,939] [51034] [ WARN] [ myapp_run] myapp_run: cluster_init_phase2 on standby rejoin failed (rc=-22) — WAL not recovered; member stays alive and will retry (bug-1477: ...)
|
|
[2026-09-28 11:52:41,941] [51034] [ WARN] [ ha_phase2_retry_fn] HA: cluster_init_phase2 retry rc=-22 — member stays alive and will retry again (bug-1477, never self-terminate)
|
|
(ctrl-2 and ctrl-3, every 3 s until 11:52:47)
|
|
```
|
|
**Expected:** the standby waits for the primary's config push quietly (INFO), as cluster_init_phase2() itself describes that state.
|
|
**Impact:** WARN lines on every cold start of every standby that read like a failed WAL recovery; noise, but alarming.
|
|
**Found by:** user report from a pc13v HA deploy, 2026-09-28
|
|
|
|
## Environment
|
|
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
|
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: asan (xce.build)
|
|
- Code: scheduler-framework @ 8cadeeb2 (last commit without the fix)
|
|
- Deployment: first seen on pc13v: 3-node HA (ctrl-1..3) + scheduler + apiserver + real root xcecd under systemd, deploy dir /tmp/xce-ha-deploy, 2026-09-28 11:52 +0800 (pc13v's OS, libc and exact commit were not recorded; the values above are from the reproduction on this dev host, same nix toolchain)
|
|
|
|
## Reproduction
|
|
1. Deploy a fresh 3-node HA cluster at 8cadeeb2 (`/xce-deploy-ha`)
|
|
2. `grep 'cluster_init_phase2' <standby logs>` during the first seconds
|
|
- Frequency: deterministic on a cold start (until the primary's config push arrives)
|
|
- Evidence: the xcecd/scheduler/controller/apiserver logs pasted in the user's report, 2026-09-28 (quoted above)
|
|
|
|
## Root cause
|
|
cluster_init_phase2() (controller/cluster.c) returns early when c->myconf->cluster_config is still NULL -- a standby that has not received the primary's config push yet -- and logs that at INFO as normal, but returned -EINVAL. Both callers, myapp_run()'s standby-rejoin path (myapp.c) and ha_phase2_retry_fn() (ha.c), treat any nonzero result as a failure and log WARN with rc=-22.
|
|
|
|
## Fix
|
|
cluster_init_phase2() returns -EAGAIN for that wait; ha_phase2_rc_is_waiting() (ha.h) names it, and myapp_run() and ha_phase2_retry_fn() log it at INFO while other failures still WARN. The promotion-refused path in ha.c keeps its WARN.
|
|
|
|
## Verification
|
|
- Test: test_phase2_without_config_is_a_wait_not_an_error in src/controller/testsuite/test_phase2_cold_boot.c: cluster_init_phase2() with no cluster_config returns -EAGAIN (it returned -EINVAL before), and ha_phase2_rc_is_waiting() accepts only -EAGAIN
|
|
- Suites: full ASAN build (xce-build all): 113/113 suites after the updated scheduler picker-registration test; controller 1250/0, harness 141/0, worker diskstats 4/0, scheduler picker wiring 11/0 + picker registration 4/0, apiserver TestRootWarning PASS
|
|
- Live: pc13v retest at 4c134127 (debug build, 2026-09-28 14:47 +0800): `ag -i error logs/` finds nothing; `ag -i warn logs/` shows only the WAL-unencrypted advisory; no cluster_init_phase2 warnings on ctrl-2/ctrl-3
|
|
|
|
## References
|
|
- Related: bug-1477
|