docs(buglog): bug-1683, bug-1485, bug-1725 fixed with corrected root causes; log bug-1913

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-03 14:51:27 +00:00
parent 362098b12a
commit 4cd5fb7dd8
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
5 changed files with 133 additions and 37 deletions

View file

@ -1,8 +1,8 @@
---
id: "bug-1683"
status: "open"
status: "fixed"
status_updated: "2026-10-03"
status_note: "2026-10-03, controller-peers-env-fallback task 4.1: xce-build.sh xceadmin --incremental run 1 failed test_wire.py::test_send_frame_writes_exact_known_good_byte_sequence (header captured, xce! payload missing); run 2 immediately after passed (314 passed). Diff under test touched only discovery.py/errors.py/test_discovery.py, so unrelated."
status_note: ""
component: "xceadmin"
severity: "unknown"
timestamp: "2026-08-25T00:05:00Z"
@ -11,6 +11,7 @@ occurrences: 3
files: ["src/xceadmin/src/xcecommon/wire.py"]
tags: ["xceadmin", "xcecommon", "wire-protocol", "python", "framing", "test-failure", "unreproducible"]
related: ["bug-1604"]
fix_commits: ["362098b1"]
legacy: true
---
@ -19,6 +20,8 @@ legacy: true
## Description
xceadmin pytest failure: test_send_frame_writes_exact_known_good_byte_sequence (src/xceadmin/tests/test_wire.py:200). Test sends a 4-byte payload b'xce!' via wire.send_frame() over a real TLS socket to a raw capture handler that reads exactly 12 bytes (8-byte length header + 4-byte payload) directly off the wire, bypassing this module's own recv_frame() decoder. Expected the captured bytes to be struct.pack('=Q', 4) + b'xce!' (header correctly encodes length=4, followed by the literal payload bytes). Actual captured bytes: the 8-byte header is correct (b'\x04\x00\x00\x00\x00\x00\x00\x00', i.e. length=4 little-endian native uint64_t), but the trailing 4 bytes are all zero (b'\x00\x00\x00\x00') instead of b'xce!' -- the real payload bytes never reached the wire. 1 failed, 266 passed, 3 skipped -- isolated to this one test.
UPDATE (2026-10-03): Correction: the trailing bytes were never zeroed -- pytest's truncated diff showed the 8-byte header alone. See Root cause (test-harness TLS RST race).
## Environment
_Not recorded (migrated from buglog.json)._
@ -26,15 +29,15 @@ _Not recorded (migrated from buglog.json)._
_Not recorded (migrated from buglog.json)._
## Root cause
UNRESOLVED -- not reproducible. send_frame() (wire.py:102-113) is two sequential sock.sendall() calls (header, then payload) -- nothing in it, or in the test's _TlsEchoServer/_capture_handler harness (test_wire.py), or conftest.py's session-scoped tls_certs fixture, could zero out payload bytes; all reviewed and found correct. Confirmed only bare pytest 8.4.2 is installed (no pytest-xdist, no pytest-randomly, no addopts) -- test order is deterministic, ruling out parallel-worker/reordering interference.
45+ reproduction attempts (2026-08-25), zero failures: isolated run x5, full test_wire.py x3, full 270-test suite x3 (ad-hoc PYTHONPATH), full meson-driven build+test pipeline (xce-build.sh xceadmin, the project's real test-running convention) x1 clean (267 passed, 3 skipped), isolated via the meson-built venv x15, full suite via meson-built venv x5, full suite under artificial CPU stress (20x `yes` spinners, load 11+) x5, 8-way concurrent parallel execution of the exact test x8 -- all clean. TCP/TLS cannot spontaneously zero application bytes mid-stream, so a genuine code defect producing this exact symptom (correct 8-byte header, real zeroed 4-byte payload) should be deterministically reproducible, and it isn't.
Corrected 2026-10-03, superseding the earlier "not reproducible / build-dir race (bug-1604)" analysis (kept in git history). Test-harness race in `src/xceadmin/tests/test_wire.py` `test_send_frame_writes_exact_known_good_byte_sequence`: the client called `send_frame()` and then `sock.close()` immediately. Under TLS 1.3 the server sends NewSessionTicket records after the handshake that the client never reads; closing a TCP socket with unread inbound data makes Linux send RST instead of FIN. Each `sendall()` is its own TLS record, so when the RST lands between the server's read of the 8-byte header record and the 4-byte payload record, the unread payload is discarded and `recv()` raises ConnectionResetError/BrokenPipeError, which `_TlsEchoServer._run` swallowed with a bare `pass` -- leaving `captured` = header only. The "4 zero bytes" in the original description was a misreading of pytest's truncated diff of the 8-byte header. `wire.send_frame()` is correct; production paths read the reply before closing and are not exposed.
## Fix
Not a code fix -- treating as a suspected ONE-OFF ENVIRONMENTAL/INFRASTRUCTURE artifact, not a defect in current source. This project is documented (.wolf/cerebrum.md, bug-1604) to run many concurrent automated sessions that race on shared build directories -- e.g. a concurrent xce-build.sh/meson-build.sh invocation's `rm -rf "$WORK_DIR"` + `cp -r` mirror step, if raced by another concurrent invocation touching the same build dir, is a plausible analogous corruption mechanism (already confirmed for other components under bug-1604's pattern) -- but no direct evidence this happened for bug-1683 specifically, since it has not recurred. No code changes made. If this recurs, capture `ps aux` (or similar) for a concurrent xce-build.sh/meson-build.sh run racing the same build directory at failure time, per the bug-1604 precedent -- that is the most useful missing evidence right now.
`test_wire.py` only: the client waits on a `capture_done` Event (set in the handler's `finally`) before `sock.close()`; `_TlsEchoServer` records the handler's OSError in `handler_error` (asserted None) instead of discarding it; the test is parametrized with `settle_s` [0.0, 0.05] (`immediate` / `tickets-pending`), the 0.05 case being a deterministic regression for the RST path. Fix commit 362098b1.
## Verification
_Not recorded (migrated from buglog.json)._
- Test: `test_send_frame_writes_exact_known_good_byte_sequence[tickets-pending]` in `src/xceadmin/tests/test_wire.py` — RED before the fix (`assert b'\x04\x00...\x00' == b'\x04\x00...xce!'`; reviewer's scratch copy without the wait: 5/5 ConnectionResetError 104), GREEN after; 200 looped runs 0 failures; scratch harness 8/2000 natural failures before vs 0/2000 after.
- Suites: `bash src/tools/xce-build.sh xceadmin --incremental` 373 passed, 2026-10-03.
- Live: n/a
## References
- Related: bug-1604