---
id: "bug-652"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "lib"
severity: "hang"
timestamp: "2026-06-17T19:10:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/lib/netio.c"]
tags: ["libxce", "netio", "endpoint_enq", "deadlock", "self-deadlock", "ha-lock", "reentrancy", "regression", "controller", "vote_ep_err_handler"]
related: ["bug-646", "bug-650"]
legacy: true
---

# bug-652: Controller (primary) self-deadlocks on ha->lock after bug-646/650 deploy: jobsub now works, but…

## Description
Controller (primary) self-deadlocks on ha->lock after bug-646/650 deploy: jobsub now works, but the controller freezes. gdb shows Thread 3 in primary_heartbeat_fn holding ha->lock (owner=self) and re-acquiring it via xce_nop_req -> xce_request -> endpoint_enq -> handle_error -> vote_ep_err_handler (ha.c:3478); Thread 9 (io_worker) also blocked on the same ha->lock. Whole controller frozen.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
REGRESSION from bug-646: endpoint_enq() invoked handle_error() synchronously on the send path when the poll arm failed. handle_error() calls the endpoint's err_handler, which on a controller vote endpoint is vote_ep_err_handler() — it locks ha->lock. But endpoint_enq runs on the CALLER's thread, and the controller's primary_heartbeat_fn already holds ha->lock when it sends heartbeat NOPs. Non-recursive mutex -> self-deadlock. The err_handler is contractually an io_worker-thread callback (off an epoll error event, holding no caller locks); calling it from the synchronous send path violated that. With bug-650 now propagating the enq error up the return chain, the handle_error call in endpoint_enq was both redundant and harmful.

## Fix
endpoint_enq() no longer calls handle_error(); on arm failure it just returns a RETRIABLE -ECONNRESET. The caller propagates it (request.c senders return it -> client failover classifies ECONNRESET as RETRY_NOW and rotates; controller heartbeat path simply gets the error, no lock re-entry). The real teardown stays on the io_worker thread (handle_input/handle_output -> io_handler -> handle_error), which holds no caller locks. Updated test_netio_arm_failure.c: endpoint_enq must return <0 AND must NOT invoke the err_handler (g_err_count==0). libxce 169/0; controller_unit_test links clean.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-646, bug-650