fix(deploy-ha): stop-all.sh/cleanup.sh must also kill the real worker's sudo child (bug-1820)

start-worker.sh's real-worker branch writes the pidfile via `echo $$`
then `exec sudo "$WORKER_BIN" ...` -- exec replaces the pidfile-writing
shell with sudo in place (same PID), so the pidfile records sudo's own
PID, not the actual xcecd daemon's. sudo forks a separate, root-owned
child to run the real worker; kill -9 on sudo's PID kills only sudo
(SIGKILL can't be forwarded to a child), silently orphaning xcecd. The
existing "still alive after kill -9" warning never caught this either,
since it re-checked the same (already-dead) sudo PID.

Both generated stop-all.sh and cleanup.sh now detect a sudo-wrapped
pidfile entry (its /proc/$PID/cmdline starts with "sudo "), find its
child via pgrep -P, and kill that child with sudo first. The
still-alive check now also tests for the child's continued existence.

Live-verified end-to-end: deployed a 3-node cluster with a real
(XCE_REAL_WORKER=1) worker, confirmed the sudo-parent/root-child PID
pair, ran the fixed stop-all.sh and cleanup.sh and confirmed zero
processes remain after each, with the new "is a sudo wrapper" line
printed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC

.wolf/buglog.json

@@ -22291,6 +22291,32 @@
       ],
       "occurrences": 1,
       "last_seen": "2026-09-16T17:15:00Z"
+    },
+    {
+      "id": "bug-1820",
+      "timestamp": "2026-09-17T02:45:00Z",
+      "error_message": "User report: after deploying a real (root, sudo-launched) xcecd worker via /xce-deploy-ha and stopping the cluster with scripts/stop-all.sh, xcecd keeps running -- ps shows a leftover xcecd process (a different PID than the one stop-all.sh reported stopping) still bound to the deploy dir's logs after stop-all.sh reported \"Stopping worker-1 (PID <pid>)\" with no warning.",
+      "file": "src/tools/deploy-ha.sh (generated scripts/stop-all.sh and scripts/cleanup.sh kill loops; generated scripts/start-worker.sh's sudo branch)",
+      "root_cause": "start-worker.sh's real-worker branch writes the pidfile via `echo $$ > \"$PIDFILE\"` and then does `exec sudo \"$WORKER_BIN\" ...` -- `exec` replaces the pidfile-writing shell's own process image with sudo IN PLACE (same PID), so the PID recorded in the pidfile is sudo's own PID, not the actual worker daemon's. sudo then forks a SEPARATE, root-owned child process that execs and runs the real xcecd daemon; sudo itself stays alive as a thin parent/monitor. stop-all.sh's (and cleanup.sh's, identical logic) kill loop did `kill -9 \"$PID\"` against the recorded (sudo) PID -- SIGKILL terminates sudo instantly but is never forwarded to its child (SIGKILL cannot be caught/relayed by the dying parent), so the real xcecd process is silently orphaned (reparented to init) and keeps running. The scripts' own pre-existing \"still alive after kill -9\" warning branch also never fired for this exact case, because it re-checks the SAME (sudo) PID, which genuinely is dead by then -- so the orphaned child produced zero warning output, making the bug fully silent.",
+      "fix": "FIXED (2026-09-17). In both generated scripts/stop-all.sh and scripts/cleanup.sh (src/tools/deploy-ha.sh), before killing a pidfile'd PID, check whether that PID's own /proc/$PID/cmdline starts with \"sudo \" (the telltale sign left by `exec sudo ...`); if so, find its child process(es) via `pgrep -P \"$PID\"` and kill them first via `sudo kill -9` (they run as root), before killing the sudo PID itself. The post-kill \"still alive\" check was also extended to additionally test for the child PID's continued existence (via `/proc/$CHILD` presence, which needs no kill permission, since `kill -0` on a root-owned PID from a non-root caller returns EPERM regardless of whether the process is alive). Live-verified end-to-end: deployed a fresh 3-node cluster with XCE_REAL_WORKER=1 (forcing the sudo xcecd path), confirmed the exact sudo-parent/root-child PID pair via `ps axwww`, ran the fixed stop-all.sh and confirmed both processes died with the new \"is a sudo wrapper -- also stopping its child\" line printed and zero processes remaining afterward; repeated the same reproduction against cleanup.sh with the same clean result. No unit test exists for this bash tooling (deploy-ha.sh has no meson-registered test suite, consistent with other shell tooling in this project e.g. xce-build.sh); verification is this live end-to-end reproduction.",
+      "tags": [
+        "deploy-ha",
+        "stop-all.sh",
+        "cleanup.sh",
+        "start-worker.sh",
+        "xcecd",
+        "sudo",
+        "orphaned-process",
+        "real-worker",
+        "sigkill",
+        "fixed",
+        "live-verified"
+      ],
+      "related_bugs": [
+        "bug-1720"
+      ],
+      "occurrences": 1,
+      "last_seen": "2026-09-17T02:45:00Z"
     }
   ]
 }

src/tools/deploy-ha.sh

@@ -325,10 +325,32 @@ for PF in $TOP/run/*.pid; do
     NAME=\$(basename "\$PF" .pid)
     if [ -n "\$PID" ] && kill -0 "\$PID" 2>/dev/null; then
         echo "cleanup.sh: stopping \$NAME (PID \$PID)"
+        # bug-1820: the real (sudo-launched) worker's pidfile records sudo's
+        # OWN pid -- start-worker.sh's "exec sudo \$WORKER_BIN ..." replaces
+        # the pidfile-writing shell with sudo in place (same pid), but sudo
+        # then forks a SEPARATE, root-owned child that actually runs the
+        # worker. kill -9 on sudo's pid kills only sudo -- SIGKILL cannot be
+        # forwarded to a child -- silently orphaning the real worker, and the
+        # "still alive" check below never catches it either (sudo's own pid,
+        # owned by the invoking user, dies cleanly). Detect a sudo wrapper by
+        # its cmdline and kill its child(ren) first, via sudo (root-owned).
+        CHILDREN=""
+        if tr '\0' ' ' < /proc/\$PID/cmdline 2>/dev/null | grep -q '^sudo '; then
+            CHILDREN=\$(pgrep -P "\$PID" 2>/dev/null || true)
+            for CHILD in \$CHILDREN; do
+                echo "  \$NAME (PID \$PID) is a sudo wrapper -- also stopping its child (PID \$CHILD)"
+                sudo kill -9 "\$CHILD" 2>/dev/null || true
+            done
+        fi
         kill -9 "\$PID" 2>/dev/null || true
         sleep 0.2
-        if kill -0 "\$PID" 2>/dev/null; then
-            echo "Warning: \$NAME (PID \$PID) still alive after kill -9 -- likely the real worker running as root; try: sudo kill -9 \$PID"
+        STILL_ALIVE=0
+        kill -0 "\$PID" 2>/dev/null && STILL_ALIVE=1
+        for CHILD in \$CHILDREN; do
+            [ -d "/proc/\$CHILD" ] && STILL_ALIVE=1
+        done
+        if [ "\$STILL_ALIVE" -eq 1 ]; then
+            echo "Warning: \$NAME (PID \$PID \$CHILDREN) still alive after kill -9 -- likely the real worker running as root; try: sudo kill -9 \$PID \$CHILDREN"
         fi
     fi
 done
@@ -358,10 +380,33 @@ for PF in $TOP/run/*.pid; do
     NAME=\$(basename "\$PF" .pid)
     if [ -n "\$PID" ] && kill -0 "\$PID" 2>/dev/null; then
         echo "Stopping \$NAME (PID \$PID)"
+        # bug-1820: see cleanup.sh's identical comment above -- the real
+        # (sudo-launched) worker's pidfile records sudo's OWN pid (its
+        # start-worker.sh does "exec sudo \$WORKER_BIN ...", so the
+        # pidfile-writing shell becomes sudo in place, same pid), but sudo
+        # forks a SEPARATE, root-owned child that actually runs the worker.
+        # kill -9 on sudo's pid only kills sudo -- SIGKILL cannot be
+        # forwarded to a child -- silently orphaning the real worker, and
+        # the "still alive" check below never caught it either (sudo's own
+        # pid, owned by the invoking user, dies cleanly). Detect a sudo
+        # wrapper by its cmdline and kill its child(ren) first, via sudo.
+        CHILDREN=""
+        if tr '\0' ' ' < /proc/\$PID/cmdline 2>/dev/null | grep -q '^sudo '; then
+            CHILDREN=\$(pgrep -P "\$PID" 2>/dev/null || true)
+            for CHILD in \$CHILDREN; do
+                echo "  \$NAME (PID \$PID) is a sudo wrapper -- also stopping its child (PID \$CHILD)"
+                sudo kill -9 "\$CHILD" 2>/dev/null || true
+            done
+        fi
         kill -9 "\$PID" 2>/dev/null || true
         sleep 0.2
-        if kill -0 "\$PID" 2>/dev/null; then
-            echo "Warning: \$NAME (PID \$PID) still alive after kill -9 -- likely the real worker running as root; try: sudo kill -9 \$PID"
+        STILL_ALIVE=0
+        kill -0 "\$PID" 2>/dev/null && STILL_ALIVE=1
+        for CHILD in \$CHILDREN; do
+            [ -d "/proc/\$CHILD" ] && STILL_ALIVE=1
+        done
+        if [ "\$STILL_ALIVE" -eq 1 ]; then
+            echo "Warning: \$NAME (PID \$PID \$CHILDREN) still alive after kill -9 -- likely the real worker running as root; try: sudo kill -9 \$PID \$CHILDREN"
         else
             STOPPED=1
         fi