myxce/.wolf/bugs/bug-1872.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	gitea_issue	gitea_synced_status	gitea_synced_hash
bug-1872	fixed	2026-09-28		worker	wrong-result	2026-09-28T03:52:37Z	2026-09-28T03:58:49Z	1	src/worker/src/node_information.c src/worker/include/diskstats.h	worker io diskstats nvme	bug-1586	44173f74	311	fixed	759d8ecd7a44

bug-1872: xcecd logs 'Failed to find any disk statistics' every 15 s on NVMe-only hosts; io never reported

Description

Observed:

[2026-09-28 11:53:03,567][ERROR][node_information.c:951 ] Failed to find any disk statistics in /proc/diskstats
(repeated every 15 s; pc13v's /proc/diskstats lists nvme0n1, nvme0n1p1, nvme0n1p2, loop0..loop7)

Expected: the io metric computed from nvme0n1's sector counters, no error. Impact: the io resource is never reported on a host whose disks are all NVMe or mmc (also any sd* name containing 'p', e.g. sdp), and xcecd logs an ERROR every 15 s. Likely the real cause of the io=0 readings in bug-1586. Found by: user report from a pc13v HA deploy, 2026-09-28

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (xce.build)
• Code: scheduler-framework @ 8cadeeb2 (last commit without the fix)
• Deployment: first seen on pc13v: 3-node HA (ctrl-1..3) + scheduler + apiserver + real root xcecd under systemd, deploy dir /tmp/xce-ha-deploy, 2026-09-28 11:52 +0800 (pc13v's OS, libc and exact commit were not recorded; the values above are from the reproduction on this dev host, same nix toolchain)

Reproduction

1. /xce-build worker
2. At 8cadeeb2 put the old filter in src/worker/include/diskstats.h (the name test from node_information.c) and run ASAN_OPTIONS=detect_leaks=0 src/xce.build/worker/testsuite/diskstats_test, which feeds pc13v's /proc/diskstats

• Frequency: deterministic
• Evidence: the xcecd/scheduler/controller/apiserver logs pasted in the user's report, 2026-09-28 (quoted above)

Root cause

read_disk_io_rate() (src/worker/src/node_information.c) decided whole disks by name: it skipped names starting loop/ram/dm-, names containing 'p', and names longer than 3 characters ending in a digit. Every NVMe disk (nvme0n1) ends in a digit and contains 'p' in its partitions only, so the disk itself was skipped for the digit; mmcblk0 likewise; sdp for its 'p'. With no disk counted it returned -1 and logged ERROR each sample. Pre-existing, origin unknown.

Fix

src/worker/include/diskstats.h: diskstats_sum_physical() counts a device only when /sys/block//device exists -- the kernel's own mark of a real disk; partitions are not in /sys/block and loop, ram, zram, dm- and md devices have no device (counting dm/md would double-count I/O already seen on their member disks). node_information.c uses it and warns once, not every 15 s, when a host really has no physical disk.

Verification

• Test: test_nvme_disk_is_counted_and_its_partitions_are_not (pc13v's /proc/diskstats) and test_sd_mmc_and_names_with_p_are_disks_virtual_ones_are_not in src/worker/testsuite/test_diskstats.c: both RED against the old name filter (0 disks; sdp/mmcblk0 missed), GREEN after; plus two more cases
• Suites: full ASAN build (xce-build all): 113/113 suites after the updated scheduler picker-registration test; controller 1250/0, harness 141/0, worker diskstats 4/0, scheduler picker wiring 11/0 + picker registration 4/0, apiserver TestRootWarning PASS
• Live: pc13v retest at 4c134127 (debug build, 2026-09-28 14:47 +0800): ag -i error logs/ finds nothing; ag -i warn logs/ shows only the WAL-unencrypted advisory; the diskstats ERROR is gone

References

• Related: bug-1586