myxce/.wolf/bugs/bug-1878.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	gitea_issue	gitea_synced_status	gitea_synced_hash
bug-1878	fixed	2026-09-28		controller	wrong-result	2026-09-28T07:11:38Z	2026-09-28T07:11:38Z	1	src/controller/ctrlstatus.c	controller ctrlstatus ha display	bug-1819	aa7fc65a	329	fixed	b6e3f549c655

bug-1878: ctrlstatus reports appliedLsn 0 on the primary (standby-only counter)

Description

Observed:

xce/client>ctrlstatus
Primary: ctrl-1
...
answered by ctrl-1 (primary)
term: 2
committedLsn: 4900
appliedLsn: 0

(pc13v, 2026-09-28 14:56; locally: committedLsn 2, appliedLsn 0) Expected: the primary's real apply position, next to its committed position. Impact: every primary's ctrlstatus says it has applied nothing, which reads like a stalled apply pump; operators and scripts comparing applied vs committed get a false alarm. The data path itself is fine. Found by: user running xce_test ctrlstatus on pc13v after the bug-1872..1877 retest

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (xce.build)
• Code: scheduler-framework @ 44173f74 (controller unchanged since; pc13v ran 4c134127)
• Deployment: local 3-node HA (localhost:9991-9993, src/tools/deploy-ha.sh, nodes started 1 s apart by start-all.sh, xcecd_virtual) on this dev host; first seen on pc13v: NixOS 26.05, kernel 6.18.33, glibc 2.42-47, debug build, 3-node HA (ctrl-1..3, localhost:9901-9903) + scheduler + apiserver + real root xcecd, /tmp/xce-ha-deploy, 2026-09-28 14:47 +0800

Reproduction

1. Deploy and start a 3-node HA cluster (/xce-deploy-ha, then scripts/start-all.sh)
2. xce_test --norc, then set masters ctrl1=localhost:9991,ctrl2=localhost:9992,ctrl3=localhost:9993, client, ctrlstatus

• Frequency: deterministic (any primary)
• Evidence: the ctrlstatus output quoted above (pc13v, pasted in the user's report) and the local run, 2026-09-28

Root cause

ctrlstatus_fill() (src/controller/ctrlstatus.c:178) sets appliedLsn from ha->applied_lsn for every role. That counter (ha.h:161, "Standby: last applied WAL LSN") is only advanced when a standby applies replicated entries; ha_init() zeroes it and nothing advances it on a primary. The primary's apply cursor is persist->last_applied_lsn, which ctrlstatus never reads. The two use different conventions: ha->applied_lsn is the NEXT lsn to apply (entry->lsn+1, ha.h:969), persist->last_applied_lsn the last applied entry -- a fix has to report one convention for both roles.

Fix

ctrlstatus_fill() (src/controller/ctrlstatus.c) reports a primary's appliedLsn from persist->last_applied_lsn (the last entry applied, never above committedLsn); a standby still reports ha->applied_lsn, as the spec defines. The existing primary fixture set ha.applied_lsn=42 on a primary, a state that never occurs live -- it now uses persist.last_applied_lsn. The controller-status design doc's "already correct in every role" note is corrected.

Verification

• Test: test_ctrlstatus_primary_reports_its_apply_cursor in src/controller/testsuite/test_ctrlstatus.c (pc13v's values): RED before the fix (Expected 4900 Was 0), GREEN after
• Suites: controller suite 1256/0 (asan); test_ha_failover_prevote 11/0, test_ha_liveness 38/0, test_ha_failover 8/0, test_ctrlstatus 23/0
• Live: HA soaks on aa7fc65a's tree: sequential 6 rounds PASS (21 jobs, zero job loss, all 3 nodes led); chaotic 6 rounds PASS (zero committed divergence, zero committed job loss); pc13v retest pending; ctrlstatus on the soak clusters not captured separately

References

• Related: bug-1819 (ctrlstatus_fill stale-slot bug, same function)
• Docs: .kiro/specs/controller-status/requirements.md (Req 6.2 commit/apply positions; lines 102-107)