61 lines
3.1 KiB
Markdown
61 lines
3.1 KiB
Markdown
---
|
|
id: "bug-1878"
|
|
status: "open"
|
|
status_updated: "2026-09-28"
|
|
status_note: "logged 2026-09-28; reproduced locally; fix not started"
|
|
component: "controller"
|
|
severity: "wrong-result"
|
|
timestamp: "2026-09-28T07:11:38Z"
|
|
last_seen: "2026-09-28T07:11:38Z"
|
|
occurrences: 1
|
|
files: ["src/controller/ctrlstatus.c"]
|
|
tags: ["controller", "ctrlstatus", "ha", "display"]
|
|
related: ["bug-1819"]
|
|
---
|
|
|
|
# bug-1878: ctrlstatus reports appliedLsn 0 on the primary (standby-only counter)
|
|
|
|
## Description
|
|
**Observed:**
|
|
```
|
|
xce/client>ctrlstatus
|
|
Primary: ctrl-1
|
|
...
|
|
answered by ctrl-1 (primary)
|
|
term: 2
|
|
committedLsn: 4900
|
|
appliedLsn: 0
|
|
```
|
|
(pc13v, 2026-09-28 14:56; locally: committedLsn 2, appliedLsn 0)
|
|
**Expected:** the primary's real apply position, next to its committed position.
|
|
**Impact:** every primary's ctrlstatus says it has applied nothing, which reads like a stalled apply pump; operators and scripts comparing applied vs committed get a false alarm. The data path itself is fine.
|
|
**Found by:** user running xce_test ctrlstatus on pc13v after the bug-1872..1877 retest
|
|
|
|
## Environment
|
|
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
|
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: asan (xce.build)
|
|
- Code: scheduler-framework @ 44173f74 (controller unchanged since; pc13v ran 4c134127)
|
|
- Deployment: local 3-node HA (localhost:9991-9993, src/tools/deploy-ha.sh, nodes started 1 s apart by start-all.sh, xcecd_virtual) on this dev host; first seen on pc13v: NixOS 26.05, kernel 6.18.33, glibc 2.42-47, debug build, 3-node HA (ctrl-1..3, localhost:9901-9903) + scheduler + apiserver + real root xcecd, /tmp/xce-ha-deploy, 2026-09-28 14:47 +0800
|
|
|
|
## Reproduction
|
|
1. Deploy and start a 3-node HA cluster (`/xce-deploy-ha`, then `scripts/start-all.sh`)
|
|
2. `xce_test --norc`, then `set masters ctrl1=localhost:9991,ctrl2=localhost:9992,ctrl3=localhost:9993`, `client`, `ctrlstatus`
|
|
- Frequency: deterministic (any primary)
|
|
- Evidence: the ctrlstatus output quoted above (pc13v, pasted in the user's report) and the local run, 2026-09-28
|
|
|
|
## Root cause
|
|
ctrlstatus_fill() (src/controller/ctrlstatus.c:178) sets appliedLsn from ha->applied_lsn for every role. That counter (ha.h:161, "Standby: last applied WAL LSN") is only advanced when a standby applies replicated entries; ha_init() zeroes it and nothing advances it on a primary. The primary's apply cursor is persist->last_applied_lsn, which ctrlstatus never reads. The two use different conventions: ha->applied_lsn is the NEXT lsn to apply (entry->lsn+1, ha.h:969), persist->last_applied_lsn the last applied entry -- a fix has to report one convention for both roles.
|
|
|
|
## Fix
|
|
TODO: required for fixed/partial
|
|
|
|
## Verification
|
|
- Test: TODO: name + file; RED before the fix, GREEN after
|
|
- Suites: TODO: suite results and build profile
|
|
- Live: TODO: soak/deploy run, or n/a and why
|
|
|
|
## References
|
|
- Related: bug-1819 (ctrlstatus_fill stale-slot bug, same function)
|
|
- Docs: .kiro/specs/controller-status/requirements.md (Req 6.2 commit/apply positions; lines 102-107)
|