---
id: "bug-650"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "lib"
severity: "hang"
timestamp: "2026-06-17T18:30:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/lib/request.c"]
tags: ["libxce", "request", "xce_request", "failover", "silent-failure", "hang", "cbuf_get", "endpoint_enq", "send-propagation"]
related: ["bug-646", "bug-648"]
legacy: true
---

# bug-650: xce_test jobsub hangs in cbuf_get (xceapi.c:304) after failover even with bug-646/648 deployed

## Description
xce_test jobsub hangs in cbuf_get (xceapi.c:304) after failover even with bug-646/648 deployed. Log shows 'endpoint marked dead' (bug-646 firing) then a permanent hang; target primary (ctrl-2) is fully idle (no vtask in handle_jobsubmit) and the client fd is an unconnected TCP socket — the request was never transmitted yet the client waits for a reply forever.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
xce_request() and the 5 other request.c senders (xce_reply, xce_replconfigpush_req_raw, xce_wal_batch_req_raw, xce_snapshot_chunk_req_raw, xce_replconfig_reply_raw) called endpoint_enq(ep,&pkt->link) and IGNORED its return, always `return 0`. So when the transport was dead, endpoint_enq() correctly returned <0 and ran handle_error() (bug-646 fix) — but xce_request() swallowed that <0 and reported success. xce_jobsubmit() (xceapi.c:299-304) then saw rc>=0 and fell through to the BLOCKING cbuf_get(); the request was never flushed (server idle) and when err_handler was NULL after a failed reconnect, handle_disconnect never injected -ECONNRESET, so nothing woke the waiter. The bug-646 netio fix was defeated one layer up: the request layer reported success on a failed send. This was the missing link making every prior netio fix insufficient for the live hang.

## Fix
All six request.c senders propagate the enq result: `int enq_rc = endpoint_enq(ep,&pkt->link); return (enq_rc < 0) ? enq_rc : 0;`. A failed send now returns <0 through xce_jobsubmit_req -> xce_jobsubmit, which returns before cbuf_get; the failover loop classifies the transport error and rotates instead of blocking on a reply that never arrives. TDD: test_request_send_propagation.c builds an OPCode_nop request (encodes OK) on a dead endpoint (EPOLL_CTL_MOD ENOENT) and asserts xce_request returns <0 (verified red with old `return 0`, green with fix). libxce 169/0; controller_unit_test links clean.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-646, bug-648