lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(deploy-ha): stop-all.sh/cleanup.sh must also kill the real worker's sudo child (bug-1820)

Browse source 
start-worker.sh's real-worker branch writes the pidfile via `echo $$`
then `exec sudo "$WORKER_BIN" ...` -- exec replaces the pidfile-writing
shell with sudo in place (same PID), so the pidfile records sudo's own
PID, not the actual xcecd daemon's. sudo forks a separate, root-owned
child to run the real worker; kill -9 on sudo's PID kills only sudo
(SIGKILL can't be forwarded to a child), silently orphaning xcecd. The
existing "still alive after kill -9" warning never caught this either,
since it re-checked the same (already-dead) sudo PID.

Both generated stop-all.sh and cleanup.sh now detect a sudo-wrapped
pidfile entry (its /proc/$PID/cmdline starts with "sudo "), find its
child via pgrep -P, and kill that child with sudo first. The
still-alive check now also tests for the child's continued existence.

Live-verified end-to-end: deployed a 3-node cluster with a real
(XCE_REAL_WORKER=1) worker, confirmed the sudo-parent/root-child PID
pair, ran the fixed stop-all.sh and cleanup.sh and confirmed zero
processes remain after each, with the new "is a sudo wrapper" line
printed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
Rongsong Shen 2026-09-17 02:43:18 +00:00
parent d19e9f2763
commit baa07d78db
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
2 changed files with 75 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

53
src/tools/deploy-ha.sh
View file

@ -325,10 +325,32 @@ for PF in $TOP/run/*.pid; do
NAME=\$(basename "\$PF" .pid)
if [ -n "\$PID" ] && kill -0 "\$PID" 2>/dev/null; then
echo "cleanup.sh: stopping \$NAME (PID \$PID)"
# bug-1820: the real (sudo-launched) worker's pidfile records sudo's
# OWN pid -- start-worker.sh's "exec sudo \$WORKER_BIN ..." replaces
# the pidfile-writing shell with sudo in place (same pid), but sudo
# then forks a SEPARATE, root-owned child that actually runs the
# worker. kill -9 on sudo's pid kills only sudo -- SIGKILL cannot be
# forwarded to a child -- silently orphaning the real worker, and the
# "still alive" check below never catches it either (sudo's own pid,
# owned by the invoking user, dies cleanly). Detect a sudo wrapper by
# its cmdline and kill its child(ren) first, via sudo (root-owned).
CHILDREN=""
if tr '\0' ' ' < /proc/\$PID/cmdline 2>/dev/null | grep -q '^sudo '; then
CHILDREN=\$(pgrep -P "\$PID" 2>/dev/null || true)
for CHILD in \$CHILDREN; do
echo " \$NAME (PID \$PID) is a sudo wrapper -- also stopping its child (PID \$CHILD)"
sudo kill -9 "\$CHILD" 2>/dev/null || true
done
fi
kill -9 "\$PID" 2>/dev/null || true
sleep 0.2
if kill -0 "\$PID" 2>/dev/null; then
echo "Warning: \$NAME (PID \$PID) still alive after kill -9 -- likely the real worker running as root; try: sudo kill -9 \$PID"
STILL_ALIVE=0
kill -0 "\$PID" 2>/dev/null && STILL_ALIVE=1
for CHILD in \$CHILDREN; do
[ -d "/proc/\$CHILD" ] && STILL_ALIVE=1
done
if [ "\$STILL_ALIVE" -eq 1 ]; then
echo "Warning: \$NAME (PID \$PID \$CHILDREN) still alive after kill -9 -- likely the real worker running as root; try: sudo kill -9 \$PID \$CHILDREN"
fi
fi
done
@ -358,10 +380,33 @@ for PF in $TOP/run/*.pid; do
NAME=\$(basename "\$PF" .pid)
if [ -n "\$PID" ] && kill -0 "\$PID" 2>/dev/null; then
echo "Stopping \$NAME (PID \$PID)"
# bug-1820: see cleanup.sh's identical comment above -- the real
# (sudo-launched) worker's pidfile records sudo's OWN pid (its
# start-worker.sh does "exec sudo \$WORKER_BIN ...", so the
# pidfile-writing shell becomes sudo in place, same pid), but sudo
# forks a SEPARATE, root-owned child that actually runs the worker.
# kill -9 on sudo's pid only kills sudo -- SIGKILL cannot be
# forwarded to a child -- silently orphaning the real worker, and
# the "still alive" check below never caught it either (sudo's own
# pid, owned by the invoking user, dies cleanly). Detect a sudo
# wrapper by its cmdline and kill its child(ren) first, via sudo.
CHILDREN=""
if tr '\0' ' ' < /proc/\$PID/cmdline 2>/dev/null | grep -q '^sudo '; then
CHILDREN=\$(pgrep -P "\$PID" 2>/dev/null || true)
for CHILD in \$CHILDREN; do
echo " \$NAME (PID \$PID) is a sudo wrapper -- also stopping its child (PID \$CHILD)"
sudo kill -9 "\$CHILD" 2>/dev/null || true
done
fi
kill -9 "\$PID" 2>/dev/null || true
sleep 0.2
if kill -0 "\$PID" 2>/dev/null; then
echo "Warning: \$NAME (PID \$PID) still alive after kill -9 -- likely the real worker running as root; try: sudo kill -9 \$PID"
STILL_ALIVE=0
kill -0 "\$PID" 2>/dev/null && STILL_ALIVE=1
for CHILD in \$CHILDREN; do
[ -d "/proc/\$CHILD" ] && STILL_ALIVE=1
done
if [ "\$STILL_ALIVE" -eq 1 ]; then
echo "Warning: \$NAME (PID \$PID \$CHILDREN) still alive after kill -9 -- likely the real worker running as root; try: sudo kill -9 \$PID \$CHILDREN"
else
STOPPED=1
fi