---
id: "bug-1975"
status: "fixed"
status_updated: "2026-10-27"
status_note: ""
component: "controller"
severity: "wrong-result"
timestamp: "2026-10-27T03:40:05Z"
last_seen: "2026-10-27T03:40:05Z"
occurrences: 1
files: ["src/esd/src/esd.c", "src/lib/client.c", "src/lib/client_internal.h"]
tags: ["esd", "external-license", "reconnect", "E2"]
related: ["bug-1760", "bug-1973", "bug-1974"]
fix_commits: ["2fcacac2"]
summary: "xceesd idle on dead connection after controller restart (stable value: no re-push)"
title: "xceesd idle on dead connection after controller restart (stable value: no re-push)"
---

# bug-1975: xceesd idle on dead connection after controller restart (stable value: no re-push)

## Description
**Observed:**
Old xceesd (pid 162609) sat idle on a dead connection for 50+ minutes after
the controller was restarted:

```
(no socket() attempts; strace showed only collector forks + epoll_wait;
 /proc/net socket table had no trace of the controller connection)
```

Last log line was the boot-time `esd full re-push: 0/1 source(s)` — no
further activity.

**Expected:** within one poll cycle of the controller's restart the daemon
must re-establish the connection and re-push its value (the new primary has
no record of the daemon or its value).

**Impact:** after any controller restart/failover with a stable license
value, the controller restarts with extlic read 0 (seed skipped for EXTERNAL
by design) and no daemon re-push — every rusage[extlic] job PENDs forever
with NO_RESOURCE until the license value happens to change.

**Found by:** E2 Phase 1 live E2E on t1 (controller restart step).

## Environment
- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
- libc: glibc 2.42 (de-nix deployment closure)
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: debug (build.dbg)
- Code: feature/license-resources @ ba42fdd8-dirty
- Deployment: t1 single-node; xceesd extres daemon

## Reproduction
1. xceesd running, pushing a value the collector keeps returning unchanged
2. Restart (or fail over) the controller
3. Observe the daemon: no reconnect attempts, pushes never resume
- Frequency: deterministic whenever the value is stable across the restart
- Evidence: strace captures /tmp/esd-strace*.out (t1, 2026-10-26); log silence quoted above

## Root cause
Three mechanisms conspired:
1. `esd_poll_round()` debounces unchanged values (`last_value == v` → skip
   push) — with a stable value it pushes NOTHING.
2. A reconnect inside the client library only fires while INSIDE a push's
   failover call — no pushes, no reconnect.
3. `esd_on_reconnect` (the rejoin hook) only runs on a successful rotate;
   with no push attempt it never runs.

## Fix
`esd_run()`'s main loop now checks `xcecli_is_connection_broken()`
(bug-1760's pure predicate — the transport's RDHUP handler sets
XCE_CTX_ERR asynchronously) and forces one `esd_push_all()` when it latches.
The write attempt drives the rotate/reconnect; `esd_on_reconnect` keeps the
full re-push contract for the failover path.

## Verification
- Test: covered by the live contract (client-side predicate already unit
  tested by bug-1760's suite); esd-side loop is integration-shaped (the
  trigger is a transport callback), so live E2E is the proof.
- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).
- Live: t1 — controller killed and restarted; esd logged
  `connection broken -- forcing a full re-push to reconnect`, rotated to the
  new primary, re-pushed; `xcecli resources distribution list` shows extlic=4
  and a fresh rusage[extlic=3] job RUNS.

## References
- Related: bug-1760 (the pure broken-connection predicate this reuses),
  E2 spec design.md §3.5 E2E checklist (daemon kill/restart + controller
  failover items)