---
id: "bug-1820"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "build"
severity: "unknown"
timestamp: "2026-09-17T02:45:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/tools/deploy-ha.sh (generated scripts/stop-all.sh and scripts/cleanup.sh kill loops; generated scripts/start-worker.sh's sudo branch)"]
tags: ["deploy-ha", "stop-all.sh", "cleanup.sh", "start-worker.sh", "xcecd", "sudo", "orphaned-process", "real-worker", "sigkill", "live-verified"]
related: ["bug-1720"]
legacy: true
---

# bug-1820: User report: after deploying a real (root, sudo-launched) xcecd worker via /xce-deploy-ha and…

## Description
User report: after deploying a real (root, sudo-launched) xcecd worker via /xce-deploy-ha and stopping the cluster with scripts/stop-all.sh, xcecd keeps running -- ps shows a leftover xcecd process (a different PID than the one stop-all.sh reported stopping) still bound to the deploy dir's logs after stop-all.sh reported "Stopping worker-1 (PID <pid>)" with no warning.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
start-worker.sh's real-worker branch writes the pidfile via `echo $$ > "$PIDFILE"` and then does `exec sudo "$WORKER_BIN" ...` -- `exec` replaces the pidfile-writing shell's own process image with sudo IN PLACE (same PID), so the PID recorded in the pidfile is sudo's own PID, not the actual worker daemon's. sudo then forks a SEPARATE, root-owned child process that execs and runs the real xcecd daemon; sudo itself stays alive as a thin parent/monitor. stop-all.sh's (and cleanup.sh's, identical logic) kill loop did `kill -9 "$PID"` against the recorded (sudo) PID -- SIGKILL terminates sudo instantly but is never forwarded to its child (SIGKILL cannot be caught/relayed by the dying parent), so the real xcecd process is silently orphaned (reparented to init) and keeps running. The scripts' own pre-existing "still alive after kill -9" warning branch also never fired for this exact case, because it re-checks the SAME (sudo) PID, which genuinely is dead by then -- so the orphaned child produced zero warning output, making the bug fully silent.

## Fix
FIXED (2026-09-17). In both generated scripts/stop-all.sh and scripts/cleanup.sh (src/tools/deploy-ha.sh), before killing a pidfile'd PID, check whether that PID's own /proc/$PID/cmdline starts with "sudo " (the telltale sign left by `exec sudo ...`); if so, find its child process(es) via `pgrep -P "$PID"` and kill them first via `sudo kill -9` (they run as root), before killing the sudo PID itself. The post-kill "still alive" check was also extended to additionally test for the child PID's continued existence (via `/proc/$CHILD` presence, which needs no kill permission, since `kill -0` on a root-owned PID from a non-root caller returns EPERM regardless of whether the process is alive). Live-verified end-to-end: deployed a fresh 3-node cluster with XCE_REAL_WORKER=1 (forcing the sudo xcecd path), confirmed the exact sudo-parent/root-child PID pair via `ps axwww`, ran the fixed stop-all.sh and confirmed both processes died with the new "is a sudo wrapper -- also stopping its child" line printed and zero processes remaining afterward; repeated the same reproduction against cleanup.sh with the same clean result. No unit test exists for this bash tooling (deploy-ha.sh has no meson-registered test suite, consistent with other shell tooling in this project e.g. xce-build.sh); verification is this live end-to-end reproduction.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1720