From baa07d78db8e5d4c1e524255fdc0d656b9e9c8e2 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Thu, 17 Sep 2026 02:43:18 +0000 Subject: [PATCH] fix(deploy-ha): stop-all.sh/cleanup.sh must also kill the real worker's sudo child (bug-1820) start-worker.sh's real-worker branch writes the pidfile via `echo $$` then `exec sudo "$WORKER_BIN" ...` -- exec replaces the pidfile-writing shell with sudo in place (same PID), so the pidfile records sudo's own PID, not the actual xcecd daemon's. sudo forks a separate, root-owned child to run the real worker; kill -9 on sudo's PID kills only sudo (SIGKILL can't be forwarded to a child), silently orphaning xcecd. The existing "still alive after kill -9" warning never caught this either, since it re-checked the same (already-dead) sudo PID. Both generated stop-all.sh and cleanup.sh now detect a sudo-wrapped pidfile entry (its /proc/$PID/cmdline starts with "sudo "), find its child via pgrep -P, and kill that child with sudo first. The still-alive check now also tests for the child's continued existence. Live-verified end-to-end: deployed a 3-node cluster with a real (XCE_REAL_WORKER=1) worker, confirmed the sudo-parent/root-child PID pair, ran the fixed stop-all.sh and cleanup.sh and confirmed zero processes remain after each, with the new "is a sudo wrapper" line printed. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC --- .wolf/buglog.json | 26 +++++++++++++++++++++ src/tools/deploy-ha.sh | 53 ++++++++++++++++++++++++++++++++++++++---- 2 files changed, 75 insertions(+), 4 deletions(-) diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 6ffceef3..060b651e 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -22291,6 +22291,32 @@ ], "occurrences": 1, "last_seen": "2026-09-16T17:15:00Z" + }, + { + "id": "bug-1820", + "timestamp": "2026-09-17T02:45:00Z", + "error_message": "User report: after deploying a real (root, sudo-launched) xcecd worker via /xce-deploy-ha and stopping the cluster with scripts/stop-all.sh, xcecd keeps running -- ps shows a leftover xcecd process (a different PID than the one stop-all.sh reported stopping) still bound to the deploy dir's logs after stop-all.sh reported \"Stopping worker-1 (PID )\" with no warning.", + "file": "src/tools/deploy-ha.sh (generated scripts/stop-all.sh and scripts/cleanup.sh kill loops; generated scripts/start-worker.sh's sudo branch)", + "root_cause": "start-worker.sh's real-worker branch writes the pidfile via `echo $$ > \"$PIDFILE\"` and then does `exec sudo \"$WORKER_BIN\" ...` -- `exec` replaces the pidfile-writing shell's own process image with sudo IN PLACE (same PID), so the PID recorded in the pidfile is sudo's own PID, not the actual worker daemon's. sudo then forks a SEPARATE, root-owned child process that execs and runs the real xcecd daemon; sudo itself stays alive as a thin parent/monitor. stop-all.sh's (and cleanup.sh's, identical logic) kill loop did `kill -9 \"$PID\"` against the recorded (sudo) PID -- SIGKILL terminates sudo instantly but is never forwarded to its child (SIGKILL cannot be caught/relayed by the dying parent), so the real xcecd process is silently orphaned (reparented to init) and keeps running. The scripts' own pre-existing \"still alive after kill -9\" warning branch also never fired for this exact case, because it re-checks the SAME (sudo) PID, which genuinely is dead by then -- so the orphaned child produced zero warning output, making the bug fully silent.", + "fix": "FIXED (2026-09-17). In both generated scripts/stop-all.sh and scripts/cleanup.sh (src/tools/deploy-ha.sh), before killing a pidfile'd PID, check whether that PID's own /proc/$PID/cmdline starts with \"sudo \" (the telltale sign left by `exec sudo ...`); if so, find its child process(es) via `pgrep -P \"$PID\"` and kill them first via `sudo kill -9` (they run as root), before killing the sudo PID itself. The post-kill \"still alive\" check was also extended to additionally test for the child PID's continued existence (via `/proc/$CHILD` presence, which needs no kill permission, since `kill -0` on a root-owned PID from a non-root caller returns EPERM regardless of whether the process is alive). Live-verified end-to-end: deployed a fresh 3-node cluster with XCE_REAL_WORKER=1 (forcing the sudo xcecd path), confirmed the exact sudo-parent/root-child PID pair via `ps axwww`, ran the fixed stop-all.sh and confirmed both processes died with the new \"is a sudo wrapper -- also stopping its child\" line printed and zero processes remaining afterward; repeated the same reproduction against cleanup.sh with the same clean result. No unit test exists for this bash tooling (deploy-ha.sh has no meson-registered test suite, consistent with other shell tooling in this project e.g. xce-build.sh); verification is this live end-to-end reproduction.", + "tags": [ + "deploy-ha", + "stop-all.sh", + "cleanup.sh", + "start-worker.sh", + "xcecd", + "sudo", + "orphaned-process", + "real-worker", + "sigkill", + "fixed", + "live-verified" + ], + "related_bugs": [ + "bug-1720" + ], + "occurrences": 1, + "last_seen": "2026-09-17T02:45:00Z" } ] } diff --git a/src/tools/deploy-ha.sh b/src/tools/deploy-ha.sh index 24038047..37f7d500 100755 --- a/src/tools/deploy-ha.sh +++ b/src/tools/deploy-ha.sh @@ -325,10 +325,32 @@ for PF in $TOP/run/*.pid; do NAME=\$(basename "\$PF" .pid) if [ -n "\$PID" ] && kill -0 "\$PID" 2>/dev/null; then echo "cleanup.sh: stopping \$NAME (PID \$PID)" + # bug-1820: the real (sudo-launched) worker's pidfile records sudo's + # OWN pid -- start-worker.sh's "exec sudo \$WORKER_BIN ..." replaces + # the pidfile-writing shell with sudo in place (same pid), but sudo + # then forks a SEPARATE, root-owned child that actually runs the + # worker. kill -9 on sudo's pid kills only sudo -- SIGKILL cannot be + # forwarded to a child -- silently orphaning the real worker, and the + # "still alive" check below never catches it either (sudo's own pid, + # owned by the invoking user, dies cleanly). Detect a sudo wrapper by + # its cmdline and kill its child(ren) first, via sudo (root-owned). + CHILDREN="" + if tr '\0' ' ' < /proc/\$PID/cmdline 2>/dev/null | grep -q '^sudo '; then + CHILDREN=\$(pgrep -P "\$PID" 2>/dev/null || true) + for CHILD in \$CHILDREN; do + echo " \$NAME (PID \$PID) is a sudo wrapper -- also stopping its child (PID \$CHILD)" + sudo kill -9 "\$CHILD" 2>/dev/null || true + done + fi kill -9 "\$PID" 2>/dev/null || true sleep 0.2 - if kill -0 "\$PID" 2>/dev/null; then - echo "Warning: \$NAME (PID \$PID) still alive after kill -9 -- likely the real worker running as root; try: sudo kill -9 \$PID" + STILL_ALIVE=0 + kill -0 "\$PID" 2>/dev/null && STILL_ALIVE=1 + for CHILD in \$CHILDREN; do + [ -d "/proc/\$CHILD" ] && STILL_ALIVE=1 + done + if [ "\$STILL_ALIVE" -eq 1 ]; then + echo "Warning: \$NAME (PID \$PID \$CHILDREN) still alive after kill -9 -- likely the real worker running as root; try: sudo kill -9 \$PID \$CHILDREN" fi fi done @@ -358,10 +380,33 @@ for PF in $TOP/run/*.pid; do NAME=\$(basename "\$PF" .pid) if [ -n "\$PID" ] && kill -0 "\$PID" 2>/dev/null; then echo "Stopping \$NAME (PID \$PID)" + # bug-1820: see cleanup.sh's identical comment above -- the real + # (sudo-launched) worker's pidfile records sudo's OWN pid (its + # start-worker.sh does "exec sudo \$WORKER_BIN ...", so the + # pidfile-writing shell becomes sudo in place, same pid), but sudo + # forks a SEPARATE, root-owned child that actually runs the worker. + # kill -9 on sudo's pid only kills sudo -- SIGKILL cannot be + # forwarded to a child -- silently orphaning the real worker, and + # the "still alive" check below never caught it either (sudo's own + # pid, owned by the invoking user, dies cleanly). Detect a sudo + # wrapper by its cmdline and kill its child(ren) first, via sudo. + CHILDREN="" + if tr '\0' ' ' < /proc/\$PID/cmdline 2>/dev/null | grep -q '^sudo '; then + CHILDREN=\$(pgrep -P "\$PID" 2>/dev/null || true) + for CHILD in \$CHILDREN; do + echo " \$NAME (PID \$PID) is a sudo wrapper -- also stopping its child (PID \$CHILD)" + sudo kill -9 "\$CHILD" 2>/dev/null || true + done + fi kill -9 "\$PID" 2>/dev/null || true sleep 0.2 - if kill -0 "\$PID" 2>/dev/null; then - echo "Warning: \$NAME (PID \$PID) still alive after kill -9 -- likely the real worker running as root; try: sudo kill -9 \$PID" + STILL_ALIVE=0 + kill -0 "\$PID" 2>/dev/null && STILL_ALIVE=1 + for CHILD in \$CHILDREN; do + [ -d "/proc/\$CHILD" ] && STILL_ALIVE=1 + done + if [ "\$STILL_ALIVE" -eq 1 ]; then + echo "Warning: \$NAME (PID \$PID \$CHILDREN) still alive after kill -9 -- likely the real worker running as root; try: sudo kill -9 \$PID \$CHILDREN" else STOPPED=1 fi