--- id: "bug-1872" status: "fixed" status_updated: "2026-09-28" status_note: "" component: "worker" severity: "wrong-result" timestamp: "2026-09-28T03:52:37Z" last_seen: "2026-09-28T03:58:49Z" occurrences: 1 files: ["src/worker/src/node_information.c", "src/worker/include/diskstats.h"] tags: ["worker", "io", "diskstats", "nvme"] related: ["bug-1586"] fix_commits: ["44173f74"] gitea_issue: 311 gitea_synced_status: "fixed" gitea_synced_hash: "759d8ecd7a44" --- # bug-1872: xcecd logs 'Failed to find any disk statistics' every 15 s on NVMe-only hosts; io never reported ## Description **Observed:** ``` [2026-09-28 11:53:03,567][ERROR][node_information.c:951 ] Failed to find any disk statistics in /proc/diskstats (repeated every 15 s; pc13v's /proc/diskstats lists nvme0n1, nvme0n1p1, nvme0n1p2, loop0..loop7) ``` **Expected:** the io metric computed from nvme0n1's sector counters, no error. **Impact:** the `io` resource is never reported on a host whose disks are all NVMe or mmc (also any `sd*` name containing 'p', e.g. sdp), and xcecd logs an ERROR every 15 s. Likely the real cause of the io=0 readings in bug-1586. **Found by:** user report from a pc13v HA deploy, 2026-09-28 ## Environment - OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 - libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) - Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 - Build: asan (xce.build) - Code: scheduler-framework @ 8cadeeb2 (last commit without the fix) - Deployment: first seen on pc13v: 3-node HA (ctrl-1..3) + scheduler + apiserver + real root xcecd under systemd, deploy dir /tmp/xce-ha-deploy, 2026-09-28 11:52 +0800 (pc13v's OS, libc and exact commit were not recorded; the values above are from the reproduction on this dev host, same nix toolchain) ## Reproduction 1. `/xce-build worker` 2. At 8cadeeb2 put the old filter in `src/worker/include/diskstats.h` (the name test from node_information.c) and run `ASAN_OPTIONS=detect_leaks=0 src/xce.build/worker/testsuite/diskstats_test`, which feeds pc13v's /proc/diskstats - Frequency: deterministic - Evidence: the xcecd/scheduler/controller/apiserver logs pasted in the user's report, 2026-09-28 (quoted above) ## Root cause read_disk_io_rate() (src/worker/src/node_information.c) decided whole disks by name: it skipped names starting loop/ram/dm-, names containing 'p', and names longer than 3 characters ending in a digit. Every NVMe disk (nvme0n1) ends in a digit and contains 'p' in its partitions only, so the disk itself was skipped for the digit; mmcblk0 likewise; sdp for its 'p'. With no disk counted it returned -1 and logged ERROR each sample. Pre-existing, origin unknown. ## Fix src/worker/include/diskstats.h: diskstats_sum_physical() counts a device only when /sys/block//device exists -- the kernel's own mark of a real disk; partitions are not in /sys/block and loop, ram, zram, dm- and md devices have no `device` (counting dm/md would double-count I/O already seen on their member disks). node_information.c uses it and warns once, not every 15 s, when a host really has no physical disk. ## Verification - Test: test_nvme_disk_is_counted_and_its_partitions_are_not (pc13v's /proc/diskstats) and test_sd_mmc_and_names_with_p_are_disks_virtual_ones_are_not in src/worker/testsuite/test_diskstats.c: both RED against the old name filter (0 disks; sdp/mmcblk0 missed), GREEN after; plus two more cases - Suites: full ASAN build (xce-build all): 113/113 suites after the updated scheduler picker-registration test; controller 1250/0, harness 141/0, worker diskstats 4/0, scheduler picker wiring 11/0 + picker registration 4/0, apiserver TestRootWarning PASS - Live: pc13v retest at 4c134127 (debug build, 2026-09-28 14:47 +0800): `ag -i error logs/` finds nothing; `ag -i warn logs/` shows only the WAL-unencrypted advisory; the diskstats ERROR is gone ## References - Related: bug-1586