fix(harness): reset getopt_long via optind=0 to stop cross-command corruption

Every cmd_* REPL handler in test.c/join_worker_args.c reset flag parsing
with `optind = 1;` only. glibc's getopt_long only fully reinitializes its
internal scan state (nextchar/first_nonopt/last_nonopt) when optind==0, or
on the process's very first getopt_long() call ever -- optind=1 alone
leaves that state stale for every call after the first. Since xce_test's
REPL dispatches many independent cmd_*() calls per process, a command
whose parse ends on a short no-argument flag (`qctrl -a`, `hostctrl -o`)
left state that corrupted the NEXT unrelated command's parse badly enough
to misread its own name as an unrecognized flag. This was the actual
cause of harness_unit_test's 4 deterministic (not flaky -- 20/20 identical
repeat runs) test_test_cli.c failures, each burning its full 5s
confirmation-prompt timeout. Fixed by using optind=0 at all 21 reset
sites; confirmed via 3 isolated pty repros plus 10 consecutive clean
harness_unit_test runs (132/132, was 4 failures/run every time).

Also fixes a resource leak found while investigating: test_test_cli.c's
spawned xce_test sessions were only torn down by inline
clitest_kill_and_reap()/clitest_close() calls at the bottom of each test
function, which Unity's longjmp-based TEST_ASSERT abort skips entirely on
failure, leaking the child process and its pty/pipe fds. Added a static
session mirror (safe across the longjmp, unlike a stack pointer) reaped
from harness_test.c's tearDown(), which Unity guarantees runs after every
test regardless of outcome.

Logged as bug-1807/bug-1808 in .wolf/buglog.json, with the underlying
getopt/longjmp gotchas recorded in .wolf/cerebrum.md. Full ASAN
/xce-build harness also passes 5/5 with no sanitizer findings.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
Rongsong Shen 2026-09-14 23:16:11 +00:00
parent cbf822fc53
commit 868c6427c8
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
9 changed files with 672 additions and 74 deletions

View file

@ -21859,6 +21859,90 @@
],
"occurrences": 1,
"last_seen": "2026-09-11T09:55:00Z"
},
{
"id": "bug-1805",
"timestamp": "2026-09-14T05:07:17.752283Z",
"error_message": "xce-harness-component task 1.1: controller_unit_test had 29 test failures (all in test_test_cli.c/test_xcetest_process_entry.c) after xce_test moved from src/controller/testsuite/ to src/harness/ -- Expected 0 Was 127 and missing xce> prompt",
"file": "src/controller/meson.build",
"root_cause": "A bridging custom_target (xce_test_controller_link) created a symlink via command: ['ln', '-sf', '@INPUT0@', '@OUTPUT0@']. Meson expands @INPUT0@ to a build-root-relative path (harness/xce_test), but a relative symlink TARGET is resolved relative to the SYMLINK OWN DIRECTORY at dereference time, not the ln invocation cwd or the string reference frame it was written in. The resulting symlink at <build_dir>/controller/xce_test therefore looked for <build_dir>/controller/harness/xce_test, which does not exist, causing every subprocess-spawning test (resolve_xce_test_path()) to exec a missing binary (exit 127).",
"fix": "FIXED (2026-09-14, xce-harness-component task 1.1). Changed the custom_target command to use the literal relative path ['ln', '-sf', '../harness/xce_test', '@OUTPUT0@'] instead of @INPUT0@, so the symlink text correctly resolves relative to controller/ up to harness/xce_test. Verified: controller_unit_test dropped from 29 failures to 0 (1228 Tests 0 Failures 0 Ignored), confirmed via the Unity binary's own printed summary line, not just meson's aggregate Ok/Fail count.",
"tags": [
"meson",
"symlink",
"custom_target",
"relative-path",
"build-system",
"xce-harness-component"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-09-14T05:07:17.752317Z"
},
{
"id": "bug-1806",
"timestamp": "2026-09-14T09:20:27.103747+00:00",
"error_message": "xce-harness-component task 4.3: /xce-build controller alone still rebuilds harness/xce_test on disk under the default asan profile, apparently violating requirement 2.2 (controller-only build shall not produce xce_test as a side effect)",
"file": "src/harness/meson.build, src/tools/xce-build.sh",
"root_cause": "Deterministic, not a flake -- confirmed via build-log line analysis. meson test (run by /xce-build controller's default RUN_TESTS=1 asan profile via meson test --suite controller) rebuilds every registered test()'s build-order dependencies PROJECT-WIDE before filtering which tests actually RUN by --suite. harness_unit_test's test() registration (src/harness/meson.build) has depends: [xce_test] (added task 2.1, verified load-bearing in task 4.2), so any meson test invocation -- even one scoped to --suite controller -- transitively relinks xce_test. The explicit step is clean (never references xce_test); only the SUBSEQUENT meson test step touches it. Task 3.1's own validation missed this because it checked for the OLD path (controller/xce_test, correctly absent) rather than the NEW path (harness/xce_test, which does get rebuilt).",
"fix": "NOT FIXED -- accepted as a documented Meson platform limitation (user decision 2026-09-14). Requirement 2.2 narrowed to scope the build system's guarantee to what it directly controls: the explicit compile-target list never names xce_test (still true and verified). Rejected alternatives: (a) add --no-rebuild to xce-build.sh's meson test calls -- non-trivial, since controller's OWN test suite already has 4 small standalone test binaries (sched_push_unit_test, sched_worker_change_unit_test, apiserver_push_unit_test, host_runusage_unit_test) not in its explicit compile target list, relying on this same auto-rebuild; fixing this properly means auditing and completing every component's explicit target list, with real regression risk; (b) restructure harness_unit_test's pty-spawning tests to drop the depends: edge and locate xce_test at runtime instead -- directly reverses the build-order guarantee task 4.2 just verified is real and load-bearing, reintroducing the stale/missing-binary risk it was added to prevent. requirements.md/design.md/tasks.md updated to reflect the narrowed, accurate guarantee.",
"tags": [
"meson",
"test-runner",
"build-dependency",
"xce-harness-component",
"accepted-limitation"
],
"related_bugs": [
"bug-1805"
],
"occurrences": 1,
"last_seen": "2026-09-14T09:20:27.103759+00:00"
},
{
"id": "bug-1807",
"timestamp": "2026-09-14T15:50:00Z",
"error_message": "harness_unit_test's test_test_cli.c: 4 of 132 tests deterministically FAIL every run -- test_migrated_commands_sample_reject_unrecognized_flag (jobkill unrecognized-flag rejection), test_qctrl_hostctrl_confirm_gate_destructiveness_determination (qctrl -c or hostctrl -c prompt missing -- WHICH one shifts between builds), test_destructive_commands_prompt_by_default_and_yes_bypasses (upgradeconfig prompt missing), test_confirm_gate_session_toggle_skips_and_restores ('set +y' restore prompt missing). Each failing assertion burns its full 5000ms clitest_wait_for_text() timeout (total suite runtime ~28-29s vs ~9s once fixed) -- not a race, a genuine miss every time.",
"file": "src/harness/test.c (all ~20 cmd_* handlers), src/harness/join_worker_args.c",
"root_cause": "CONFIRMED via isolated pty repro (bypassing Unity/gdb entirely): every cmd_*() REPL command handler resets getopt_long() state with `optind = 1; opterr = 0;` before its own parse loop. glibc's getopt_long only re-initializes its internal static parser state (nextchar, first_nonopt/last_nonopt) when optind==0 OR on the process's very first-ever getopt_long() call (guarded by the internal __getopt_initialized flag) -- optind=1 alone is NOT sufficient once any earlier call in the same process has already run once. Since xce_test's REPL dispatches many independent cmd_*() calls (each with its OWN, differently-sized argv array) across the lifetime of one process, a command whose parse ends by consuming a short no-argument flag as its LAST token (e.g. `qctrl -q normal -a`, `hostctrl -h node01 -o`) leaves stale internal state that corrupts the NEXT unrelated command's getopt_long() call -- observed corrupting the NEXT command so badly that argv[0] (the command's own name, e.g. 'upgradeconfig' or 'jobkill') is misread as an unrecognized flag via flagspec_reject(). 100% reproducible in isolation: `qctrl -q normal -a` immediately followed by ANY other command (upgradeconfig, jobkill) corrupts that next command every time; the same session's commands run individually/first are unaffected. Explains why the specific victim test shifted between builds (bug-1807 was first seen failing on hostctrl -c, then qctrl -c after an unrelated same-file edit) -- runtime sequence-dependent, not random.",
"fix": "FIXED. Changed all 20 occurrences of `optind = 1;` in src/harness/test.c and the 1 occurrence in src/harness/join_worker_args.c to `optind = 0;` (the documented glibc idiom that forces getopt_long to fully reinitialize, including nextchar/first_nonopt/last_nonopt, on every independent parse). Verified: (1) the 3 isolated minimal pty repros that previously reproduced the corruption 100% of the time now pass cleanly; (2) harness_unit_test run 10x consecutively under the debug profile: 132 Tests 0 Failures every run (was 4 Failures every run pre-fix), runtime dropped from ~28-29s to ~8-9s per run, confirming the failures were full timeout burns, not near-miss races; (3) full `/xce-build harness asan` (meson test, ASAN-instrumented) also reports 5/5 OK including harness_unit_test, no sanitizer findings.",
"tags": [
"confirmed",
"fixed",
"getopt",
"getopt_long",
"cli-parsing",
"xce_test",
"harness_unit_test",
"test_test_cli.c",
"flagspec"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-09-14T15:50:00Z"
},
{
"id": "bug-1808",
"timestamp": "2026-09-14T15:50:00Z",
"error_message": "src/harness/testsuite/test_test_cli.c: a spawned xce_test child process (clitest_spawn/clitest_spawn_ex) plus its pty master fd and stderr pipe fd leak whenever a TEST_ASSERT_* assertion fails partway through the owning test function, because every test relies on its OWN clitest_kill_and_reap()+clitest_close() calls placed inline at the bottom of the function body -- code that Unity's longjmp-based TEST_PROTECT()/TEST_ABORT() abort mechanism skips entirely on assertion failure. Directly observed: 4 live, idle `xce_test --norc` child processes still running (blocked forever on their own prompt) under one in-progress harness_unit_test invocation, discovered while investigating bug-1807's failures.",
"file": "src/harness/testsuite/test_test_cli.c, src/harness/testsuite/harness_test.c",
"root_cause": "Design gap, not a logic error in any specific test: clitest_kill_and_reap()/clitest_close() were documented as a mandatory safety net ('every caller is required to always end a session with clitest_kill_and_reap() regardless of which path was taken') but placed only at each test function's own normal-exit code path. Unity's RUN_TEST() macro DOES guarantee tearDown() runs after every test (pass, fail, or aborted-via-longjmp), but harness_test.c's shared tearDown() (used by all 13 Unity test groups linked into this binary, most unrelated to clitest) was a no-op beyond a cosmetic separator print -- it never reaped a still-open clitest session.",
"fix": "FIXED. Added a static clitest_session_t mirror (g_last_session) in test_test_cli.c updated by clitest_spawn_ex() (set on successful spawn), clitest_kill_and_reap() (reaped flag synced), and clitest_close() (cleared) -- a STATIC-storage mirror was required rather than tracking a pointer into the aborted test's own local `sess`, since that stack frame is unwound by the longjmp and would leave a dangling pointer. Exposed a new clitest_reap_leaked_session() function, called from harness_test.c's tearDown() (now declared extern there), which force-kills+reaps+closes any session left open when a test aborted before reaching its own inline cleanup. Only one clitest session is ever open at a time in this file (verified), so a single mirror slot is sufficient. Verified via ASAN build (xce.build): no LeakSanitizer or other sanitizer findings from the new code across a clean 5/5 harness suite run.",
"tags": [
"fixed",
"process-leak",
"fd-leak",
"unity",
"longjmp",
"test-infrastructure",
"test_test_cli.c",
"clitest"
],
"related_bugs": [
"bug-1807"
],
"occurrences": 1,
"last_seen": "2026-09-14T15:50:00Z"
}
]
}