lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(deploy-ha): stop-all.sh/cleanup.sh must find the real worker regardless of sudo process-tree depth (bug-1823)

Browse source 
bug-1820's own fix assumed sudo always forks exactly one direct child
that IS the real target, and looked for it via `pgrep -P $PID` (direct
children only). On a real host, sudo commonly interposes an additional
"monitor" process between the invoked sudo and the actual target
(sudo -> monitor -> target, three processes, not two -- standard
sudo 1.8.7+ architecture for signal/exit-status relay) -- a shape this
sandbox's own simpler sudo build doesn't reproduce, which is why
bug-1820 verified clean here but still left the real worker orphaned
on the reporting user's box. `pgrep -P` found and killed the monitor
(a direct child) while the actual worker (the monitor's own child, not
a direct child of the pidfile'd PID) survived untouched.

Replaces the parent/child tree-walking approach entirely with a
depth-independent identity-based re-scan: both generated scripts now
search system-wide (pgrep -f) for any still-running process whose own
cmdline matches the worker's distinguishing identity (-n $NAME plus
this deploy's -l $TOP/logs -- the same two args start-worker.sh's own
"already running" guard checks) and kill every match directly, however
many hops deep it landed. No longer needs to know or guess sudo's
internal process-tree depth.

Live-verified: regression-checked against this sandbox's own real
(2-level) sudo deploy (still clean), and built + killed a synthetic
3-level process tree modeled on the user's exact live report,
confirming the deepest process is found directly and the intermediate
hop then exits on its own once its child is gone.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
Rongsong Shen 2026-09-17 04:20:03 +00:00
parent bfb0db4af9
commit 7b274ec0df
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
2 changed files with 92 additions and 55 deletions
Download patch file Download diff file Expand all files Collapse all files

27
.wolf/buglog.json
View file

@ -22374,6 +22374,33 @@
],
"occurrences": 1,
"last_seen": "2026-09-17T04:10:00Z"
},
{
"id": "bug-1823",
"timestamp": "2026-09-17T04:20:00Z",
"error_message": "User-reported (regression on top of bug-1820's own fix, live on the user's real host, 'pc13v'): after ./scripts/stop-all.sh reported 'Stopping worker-1 (PID N)' with no warning, the real xcecd worker was STILL running afterward as a different PID -- and a second `sudo ./scripts/stop-all.sh` run immediately reported 'Nothing was running for this deploy' (the pidfile had already been deleted by the first run despite the kill not actually succeeding).",
"file": "src/tools/deploy-ha.sh (generated scripts/stop-all.sh and scripts/cleanup.sh)",
"root_cause": "bug-1820's own fix assumed sudo always has exactly ONE level of indirection (sudo forks exactly one direct child that IS the real target), and looked for that child via `pgrep -P $PID` (direct children of the pidfile'd PID only). On the user's real host, sudo interposes an ADDITIONAL 'monitor' process between the invoked sudo and the actual target (sudo -> monitor -> target, THREE processes, not two -- standard sudo 1.8.7+ architecture for proper signal/exit-status relay, not present in this sandbox's own simpler sudo build, which is why bug-1820's fix appeared to work here but not on the user's real box). `pgrep -P $PID` found and killed the MONITOR (a direct child of the pidfile'd PID) while the actual worker -- the monitor's OWN child, not a direct child of the pidfile'd PID -- survived completely untouched, silently orphaned exactly as in the original bug-1820 symptom. Confirmed and reproduced deterministically via a synthetic 3-level process tree (fake-outer -> fake-monitor -> fake-worker, no real sudo needed) built specifically to isolate the tree-walking logic from any particular sudo version's actual behavior.",
"fix": "FIXED (2026-09-17). Replaced the parent/child tree-walking approach entirely with a depth-independent identity-based re-scan: after attempting to kill the pidfile'd PID, both generated scripts/stop-all.sh and scripts/cleanup.sh now search system-wide (`pgrep -f`) for ANY still-running process whose own cmdline matches the worker's distinguishing identity -- its `-n $NAME` flag plus this specific deploy's `-l $TOP/logs` flag, the same two args start-worker.sh's own 'already running' guard already checks -- and kills every match directly (both a plain `kill -9` and a `sudo kill -9`, since ownership can't be known in advance), regardless of how many hops deep in the process tree it landed. No longer needs to know or guess sudo's internal process-tree depth. A no-op for ctrl-N/scheduler/apiserver pidfiles, which use neither flag. Live-verified: (1) regression-checked against this sandbox's own real (2-level) sudo worker deploy -- still cleans up correctly; (2) built and killed a synthetic 3-level tree (fake-outer/fake-monitor/fake-worker, exact process-tree shape modeled on the user's live report) via the exact identity-pgrep mechanism the fix uses, confirming the deepest (target) process is found and killed directly, and the intermediate 'monitor' hop then exits on its own once its child is gone (it was just wait()-ing/relaying) -- proving the fix generalizes to arbitrary sudo process-tree depth rather than being tuned to one specific sudo version's behavior.",
"tags": [
"deploy-ha",
"stop-all.sh",
"cleanup.sh",
"xcecd",
"sudo",
"orphaned-process",
"real-worker",
"sudo-monitor",
"process-tree-depth",
"fixed",
"live-verified",
"synthetic-repro"
],
"related_bugs": [
"bug-1820"
],
"occurrences": 1,
"last_seen": "2026-09-17T04:20:00Z"
}
]
}