58 lines
3.9 KiB
Markdown
58 lines
3.9 KiB
Markdown
---
|
|
id: "bug-1872"
|
|
status: "fixed"
|
|
status_updated: "2026-09-28"
|
|
status_note: ""
|
|
component: "worker"
|
|
severity: "wrong-result"
|
|
timestamp: "2026-09-28T03:52:37Z"
|
|
last_seen: "2026-09-28T03:58:49Z"
|
|
occurrences: 1
|
|
files: ["src/worker/src/node_information.c", "src/worker/include/diskstats.h"]
|
|
tags: ["worker", "io", "diskstats", "nvme"]
|
|
related: ["bug-1586"]
|
|
fix_commits: ["44173f74"]
|
|
gitea_issue: 311
|
|
gitea_synced_status: "fixed"
|
|
gitea_synced_hash: "759d8ecd7a44"
|
|
---
|
|
|
|
# bug-1872: xcecd logs 'Failed to find any disk statistics' every 15 s on NVMe-only hosts; io never reported
|
|
|
|
## Description
|
|
**Observed:**
|
|
```
|
|
[2026-09-28 11:53:03,567][ERROR][node_information.c:951 ] Failed to find any disk statistics in /proc/diskstats
|
|
(repeated every 15 s; pc13v's /proc/diskstats lists nvme0n1, nvme0n1p1, nvme0n1p2, loop0..loop7)
|
|
```
|
|
**Expected:** the io metric computed from nvme0n1's sector counters, no error.
|
|
**Impact:** the `io` resource is never reported on a host whose disks are all NVMe or mmc (also any `sd*` name containing 'p', e.g. sdp), and xcecd logs an ERROR every 15 s. Likely the real cause of the io=0 readings in bug-1586.
|
|
**Found by:** user report from a pc13v HA deploy, 2026-09-28
|
|
|
|
## Environment
|
|
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
|
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: asan (xce.build)
|
|
- Code: scheduler-framework @ 8cadeeb2 (last commit without the fix)
|
|
- Deployment: first seen on pc13v: 3-node HA (ctrl-1..3) + scheduler + apiserver + real root xcecd under systemd, deploy dir /tmp/xce-ha-deploy, 2026-09-28 11:52 +0800 (pc13v's OS, libc and exact commit were not recorded; the values above are from the reproduction on this dev host, same nix toolchain)
|
|
|
|
## Reproduction
|
|
1. `/xce-build worker`
|
|
2. At 8cadeeb2 put the old filter in `src/worker/include/diskstats.h` (the name test from node_information.c) and run `ASAN_OPTIONS=detect_leaks=0 src/xce.build/worker/testsuite/diskstats_test`, which feeds pc13v's /proc/diskstats
|
|
- Frequency: deterministic
|
|
- Evidence: the xcecd/scheduler/controller/apiserver logs pasted in the user's report, 2026-09-28 (quoted above)
|
|
|
|
## Root cause
|
|
read_disk_io_rate() (src/worker/src/node_information.c) decided whole disks by name: it skipped names starting loop/ram/dm-, names containing 'p', and names longer than 3 characters ending in a digit. Every NVMe disk (nvme0n1) ends in a digit and contains 'p' in its partitions only, so the disk itself was skipped for the digit; mmcblk0 likewise; sdp for its 'p'. With no disk counted it returned -1 and logged ERROR each sample. Pre-existing, origin unknown.
|
|
|
|
## Fix
|
|
src/worker/include/diskstats.h: diskstats_sum_physical() counts a device only when /sys/block/<name>/device exists -- the kernel's own mark of a real disk; partitions are not in /sys/block and loop, ram, zram, dm- and md devices have no `device` (counting dm/md would double-count I/O already seen on their member disks). node_information.c uses it and warns once, not every 15 s, when a host really has no physical disk.
|
|
|
|
## Verification
|
|
- Test: test_nvme_disk_is_counted_and_its_partitions_are_not (pc13v's /proc/diskstats) and test_sd_mmc_and_names_with_p_are_disks_virtual_ones_are_not in src/worker/testsuite/test_diskstats.c: both RED against the old name filter (0 disks; sdp/mmcblk0 missed), GREEN after; plus two more cases
|
|
- Suites: full ASAN build (xce-build all): 113/113 suites after the updated scheduler picker-registration test; controller 1250/0, harness 141/0, worker diskstats 4/0, scheduler picker wiring 11/0 + picker registration 4/0, apiserver TestRootWarning PASS
|
|
- Live: pc13v retest at 4c134127 (debug build, 2026-09-28 14:47 +0800): `ag -i error logs/` finds nothing; `ag -i warn logs/` shows only the WAL-unencrypted advisory; the diskstats ERROR is gone
|
|
|
|
## References
|
|
- Related: bug-1586
|