chore(wolf): checkpoint buglog.json (957 bugs)

.wolf/buglog.json

@@ -17109,6 +17109,51 @@
       ],
       "occurrences": 1,
       "last_seen": "2026-08-17T18:55:00Z"
+    },
+    {
+      "id": "bug-1575",
+      "timestamp": "2026-08-18T03:15:00Z",
+      "error_message": "run-resource-verify.sh's trap-based cleanup() (EXIT/INT/TERM) non-deterministically left the scheduler and/or apiserver process running after the script was interrupted mid-deployment via an external SIGTERM -- confirmed processes still alive (e.g. `xce_scheduler --peers ... --logfile .../scheduler.log`, `/tmp/apiserver -d .../apiserver`) seconds after the orchestrator's own `wait` returned with exit code 143. The SAME cleanup() code, reached via a normal fail()/exit-1 path (no external signal involved) inside the same single foreground invocation, worked perfectly every time (verified zero leftover processes).",
+      "file": ".claude/skills/xce-test-resources/run-resource-verify.sh",
+      "root_cause": "cleanup() issued four SEPARATE sequential `pkill -9 -f -- \"<pattern>\"` invocations (one per component: controller/worker/scheduler/apiserver), each forking+execing its own `pkill` subprocess. When cleanup() ran as a normal function call/return within the script's own top-to-bottom control flow (e.g. from fail()), all four completed reliably. When cleanup() instead ran from on_exit as the EXIT trap fired by an asynchronously-delivered SIGTERM (via a separate `trap 'exit 143' TERM` translating the signal into an explicit exit), the sequence of pkill forks was intermittently and non-deterministically cut short partway through (confirmed via added debug instrumentation: one run stopped after the 1st pkill, another after the 3rd, another after the 2nd -- no fixed failure point, consistent with a timing race rather than a deterministic pattern-matching bug). Root cause not fully isolated at the syscall level (ruled out: pattern-matching correctness -- each individual pkill pattern was independently verified correct via direct pgrep; ruled out: cgroup pids/resource exhaustion -- process count stayed low, ~20-25 total throughout). Most likely a bash/signal-trap interaction where starting new child processes (fork/exec for each pkill) from within a trap handler invoked asynchronously (as opposed to synchronous top-level exit) has some window of unreliability in this environment.",
+      "fix": "Rewrote cleanup() to (1) combine all four per-component patterns into ONE `pkill -9 -f` call using extended-regex alternation (`\"pat1|pat2|pat3|pat4\"`) instead of four separate invocations -- fewer fork/exec round trips through the trap-handler code path -- and (2) wrap it in a retry-until-verified loop (`pgrep -f` check, up to 10 attempts at 0.2s intervals) so a slow/racy single kill attempt can no longer silently leave a process behind. Verified reliable across 5+ repeated live runs deliberately interrupted via external SIGTERM at different points mid-deployment (right after controller-only confirmation, and after controller+scheduler confirmation) -- zero leftover processes every time after the fix, versus non-deterministic leftovers before it.",
+      "tags": [
+        "trap",
+        "cleanup",
+        "pkill",
+        "SIGTERM",
+        "signal-handler",
+        "race-condition",
+        "run-resource-verify.sh",
+        "resource-value-verification",
+        "bash"
+      ],
+      "related_bugs": [
+        "bug-1436"
+      ],
+      "occurrences": 1,
+      "last_seen": "2026-08-18T03:15:00Z"
+    },
+    {
+      "id": "bug-1576",
+      "timestamp": "2026-08-18T03:15:00Z",
+      "error_message": "Multiple interactive diagnostic Bash calls (`pgrep -af \"<pattern>\"`, `pkill -9 -f -- \"<pattern>\"`, `ps -eo pid,args | grep \"<pattern>\"`) unexpectedly matched -- and in one case SIGKILLed -- the CURRENTLY-RUNNING Bash tool call's own wrapper shell process, not just the intended target process(es). One instance: `pkill -9 -f -- \"rrv_dummy_marker_TEST1|...\"` killed the invoking Bash call itself mid-script, truncating all output after the pkill line with no further commands executing (looked like a mysterious 'exit code 1, script just stops' failure).",
+      "file": "N/A (diagnostic/interactive Bash tool usage, not a repo source file)",
+      "root_cause": "This harness wraps every Bash tool invocation as roughly `bash -c \"... && eval '<the submitted script text>' < /dev/null && ...\"`. The wrapper's own process argv (visible via /proc/<pid>/cmdline and matched by `pgrep -f`/`pkill -f`/`ps ... | grep`) therefore contains the FULL LITERAL TEXT of the submitted script -- including any pattern string typed directly into that same script (e.g. inside the pgrep/pkill command's own quoted argument, or in an unrelated echo/comment). Any `pgrep -f`/`pkill -f` search run from within a Bash tool call will self-match (and, for pkill, potentially self-kill) that call's own wrapper process whenever the search pattern also appears as literal text anywhere else in the SAME submitted command block -- which is nearly unavoidable, since typing the pattern to search for IS typing it into the submitted script.",
+      "fix": "Not a code fix (methodology only). When diagnosing process state from an interactive Bash tool call: (1) prefer reading /proc/<pid>/cmdline directly via a `for p in /proc/[0-9]*; do ...; done` scan and filtering in a way that doesn't require typing the exact search substring inline where it could also match the wrapper's own transcript (e.g. write the pattern to a variable/file first, or accept that the wrapper itself will show up as one extra line and visually distinguish it by its distinctive `bash -c \"source .../shell-snapshots/... && eval '...'\"` shape); (2) never assume a `pgrep -f`/`pkill -f` result of exactly 1 match confirms a target is unique -- it may be exactly 1 SELF-match with zero real matches; (3) for any pkill run interactively for diagnostic purposes (not from inside a dedicated script/process whose own argv doesn't contain the pattern), sanity-check the matched PID's `ps -o cmd` BEFORE trusting the pkill's effect, since a self-kill silently truncates the rest of that same Bash call with no distinguishing error message beyond a generic nonzero/truncated-output exit.",
+      "tags": [
+        "pgrep",
+        "pkill",
+        "self-match",
+        "diagnostic-methodology",
+        "bash-tool",
+        "sandbox-artifact"
+      ],
+      "related_bugs": [
+        "bug-1575"
+      ],
+      "occurrences": 1,
+      "last_seen": "2026-08-18T03:15:00Z"
     }
   ]
 }