myxce/.wolf/bugs/bug-1890.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	gitea_issue	gitea_synced_status	gitea_synced_hash
bug-1890	fixed	2026-09-29		lib	noise	2026-09-29T14:01:08Z	2026-09-29T14:01:08Z	1	src/lib/netio.c (socket_create, socket_connect, endpoint_accept)	netio tcp latency ha snapshot	bug-1871	8b561547	336	fixed	6ce3dfb46b4d

bug-1890: netio sockets leave Nagle on: last TLS records of a message burst arrive ~40 ms late

Description

Observed: a 9-chunk HA snapshot transfer is fully written by the primary within 1 ms, but the standby reads the last chunks ~40 ms later. Primary trace (xce-1871-fs1, ctrl-1) shows the final send drained at 13:20:33.967 (bug1871 upd ... 0x5->0x1 outq_empty=1 fsm_out=5); the standby logs:

[2026-09-29 13:20:33,969] ... ha_handle_snapshot_chunk: chunk crc32 checkpoint_lsn=114 chunk_index=6 ...
[2026-09-29 13:20:34,010] ... ha_handle_snapshot_chunk: chunk crc32 checkpoint_lsn=114 chunk_index=7 ...

Expected: on loopback the whole burst arrives within a few ms (most runs: max inter-chunk gap 1-4 ms). Impact: latency only, ~40 ms on the tail of a burst (snapshot/WAL batches, request-reply pairs after bulk data); no loss. grep -rn TCP_NODELAY src/lib finds nothing, so every netio TCP socket runs with Nagle, and a small trailing TLS record waits for the receiver's delayed ACK. Found by: soak -- bug-1871's instrumented force-snapshot runs, 2026-09-29. Suspected (not yet proven with TCP_NODELAY on): Nagle + delayed ACK.

UPDATE (2026-09-29): confirmed -- with TCP_NODELAY set the ~40 ms tail gaps disappear (see Verification), so the Nagle + delayed-ACK suspicion was right.

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (src/xce.build.bug1871, with temporary netio TRACE instrumentation)
• Code: scheduler-framework @ 3444cf16
• Deployment: local 3-node HA controller cluster (xce-test-ha --force-snapshot), localhost:9991-9993, 32 CPUs

Reproduction

1. nix develop ./src --command bash .claude/skills/xce-test-ha/run-ha-soak.sh --force-snapshot <dir>
2. On the rejoined standby's log, compute the gap between consecutive ha_handle_snapshot_chunk: chunk crc32 lines of one transfer; a ~40 ms gap (while the primary finished sending 1 ms earlier) is the symptom.

• Frequency: 5/40 transfers had a 39-41 ms gap, 2026-09-29 (xce-1871-fs1..fs16)
• Evidence: /home/claude/tmp/xce-1871-fs1, /home/claude/tmp/xce-1871-fs5, /home/claude/tmp/xce-1871-fs16 (ctrl-*.log)

Root cause

netio never set TCP_NODELAY (socket_create() and socket_accept() in src/lib/netio.c set only SO_REUSEADDR/FD_CLOEXEC), so every controller/worker/scheduler/apiserver TCP connection ran with Nagle's algorithm. netio writes a message as separate TLS records (an 8-byte size header, then the body split into <=16 KB records); when a small record follows data the peer has not ACKed yet, Nagle holds it until the ACK arrives, and the receiver delays that ACK (Linux delayed ACK, ~40 ms). The tail of a burst -- the last short chunk of a snapshot, a header, a reply -- therefore arrived ~40 ms after the sender had finished writing.

Fix

src/lib/netio.c: new socket_set_nodelay() sets TCP_NODELAY; socket_create() calls it for AF_INET stream sockets (outbound endpoints from endpoint_init, and listeners) and socket_accept() for every accepted connection. Small records are sent immediately instead of waiting for the peer's delayed ACK. Best effort: a failure leaves the socket working as before.

Verification

• Test: test_outbound_socket_has_nodelay and test_accepted_socket_has_nodelay in src/lib/testsuite/test_netio_nodelay.c check TCP_NODELAY on an endpoint_init socket and an endpoint_accept socket: RED before the fix (bug-1890: outbound netio socket runs with Nagle on, same for accepted), GREEN after
• Suites: xcelib_test 699/699 and controller 1256/1256, asan, no sanitizer errors
• Live: xce-test-ha --force-snapshot, 16 runs (2026-09-29, xce-1890-nd1..16): 40/40 snapshot transfers complete, max inter-chunk gap 4 ms, 0 gaps >= 20 ms -- versus 5/40 transfers with a 39-41 ms gap before the fix. 15/16 PASS; the one FAIL (nd5) is bug-1891 (snapshot sent to the wrong standby), unrelated to this change and seen before it

References

• Fix commit: 8b561547
• Related: bug-1871 (found while instrumenting it), bug-1891
• Gitea: #336